When do they need replacing? When they are sufficiently worn.
Fret wire material, play frequency/duration and play technique have a bearing on how fast you wear down your frets.
The more common nickel silver fret wire wears more rapidly than stainless steel.
Frets on instruments that get only a few hours of play per week might last for several years.
If you concentrate your playing in a particular area of the fingerboard, those frets will wear more quickly.
If you are aggressive in your string to fret wire contact during play, the fret will wear more quickly.
If you can see a divot in the crown of a fret, ideally, it should be replaced, if possible. Why? Primarily for playability reasons, as mentioned in the above posts.
There is a relationship between the height of the frets and the path of the string. Properly set up frets form a level plane beneath the string (neck relief, if desired, is added after the frets are leveled to one another). Fretting a string involves pressing it down to make contact with the fret wire. This shortens the length of the vibrating string and increases the pitch as you move up the fingerboard. Needing to press farther in order to make contact with a worn fret can cause the string to make unwanted contact with additional frets, resulting in string buzz. But it will also alter the intonation, which may or may not bother you. The amount of fret wear you can tolerate can be relative to string action (height). A player comfortable with a very high action may ignore or endure more fret wear. In contrast, it is a bit self-defeating to have painstakingly set up a guitar to have the lowest possible action, and then allow a fret or frets to wear severely.
I always chuckle as I recall the late John Denver introducing his 12 string guitar to audiences that many have never seen one before. He would explain that the greatest challenge of performing with the 12 string was in learning how to tune it. In jest, he suggested that the correct strategy was to first be able to identify that one string that was out of pitch, a skill that could take years to master. Then, of course, tune all other 11 strings to it.
As with that silly approach to tuning, I am not a fan of filing down 20 + frets because 6 + frets show some wear, often referred to as a fret dressing or leveling, as Al mentioned, above. I would rather replace the worn frets and adjust them to match the height of the existing frets. Why? If you just lowered the overall height of the frets, in order to maintain the continuity of your set up you should also lower the string height at the nut to match. And then you may need to adjust the saddle height. Why all this extra effort (and risk), when you could have simply replaced the worn frets and returned to playing? While not as dramatic, I liken the typical full fret leveling operation to shaving the saddle in order to stave off a neck reset. Each approach comes with it's own price.
It is fully acceptable that a full fret job (all frets replaced) would never occur in the life of most guitars. Indeed, such a dramatic operation would typically only happen when changing fret material (switching to stainless or Gold EVO) or fret size (changing width or height). Of course, if you have worn them all down, then you need to replace them all. However, a partial fret job, merely replacing (and dressing) the worn frets, is not only economically advantageous, but advisable to maintain your guitar's set up.