A bit of historical perspective here, or at least my view of history.
When audio was in its infancy, say pre-1940's, hi-fi was a minority hobby pursued by enthusiastic DIYers who would make their own amps and speakers. When rock'n'roll and pop record sales started to become huge most people were listening on fairly crappy gear and "hi-fi" was still the pursuit of the enthusiastic hobbyist. As the years went on, electronic production techniques evolved - printed circuit boards etc. - and the cost of manufactured audio equipment fell. By the 1970's the Japanese were producing amplifiers etc. at prices previously impossible and audio equipment became a desirable purchase for the general consumer. Through the 1970's, 1980's and into the 1990's, "hi-fi" was widely popular, at least compared to what it had been before the 1970's. These days, there are many more things for consumers to spend their money on and hi-fi is no longer popular with the general public. It hasn't gone away, it's just returned to how it started. Hi-fi was a consumer-buying trend that came and went as far as the general public is concerned, and now they consume their music differently.
The biggest change, as I see it, isn't so much the equipment used by the general public as the fact that they consider music should be free.
The other thing that's significant is that listening to music via speakers can be a social thing - listening on headphones is an isolating thing.