I'm still trying to figure out how the hi-fi speaker knows that the waveform that it's reproducing is distortion.
That one was puzzling me until I thought of the square wave. Normally, a speaker gets a waveform that rises, peaks, and tapers off. The peak amplitude is of short duration, so the average amplitude isn't all that great. But a square wave rises, stays at peak for a longer duration, the tapers off. The average amplitude is a lot greater even though the peak amplitude is the same.
That's why clipping is bad, it results in squaring of the waveform, heating up the voice coils with no increase in actual volume being produced. So playing a square wave through an amp that's not clipping would accomplish the same thing.
But we are talking about distorted music, not listening to square waves. I still tend to agree that distorted music, played within the limits of the system, shouldn't damage the system.
One other thing about clipping- When an amp clips, it produces nasty harmonics. These harmonics pass through the crossover and end up getting passed right through to the tweeter. That's why tweeters seem to suffer the brunt of the damage done by clipping amps. Heavy bass won't pass through the crossover and fry your tweeter, but if it drives your amp into clipping, the harmonics caused can fry your tweeter.
Unless you run active crossovers. Then any amp that clips will only affect the driver attached to it.
At least that's my take on it.