Most audio equipment still sounds pretty good despite using polarized caps in the signal path which could've been bi-polar to begin with (in terms of the distortion we notice from the equipment). It becomes more of an issue at the working voltage of a capacitor. For instance, at low-level listening it is going to be lower and more unnoticeable. Things still sound "good" and hold together nicely, so the assumption is made everything's fine. At louder volumes however, it exponentially increases and things fall apart sooner on the volume dial and the music turns into mush and undifferentiated noise that hurts your ears. If you think the effect of distortion is minimal and can still be ignored, then the idea of prioritizing other capacitor parameters falls apart somewhat. People mainly choose caps based on specs, and absent distortion figures is not one to ignore -or at least be made more aware of and its effect on signal coupling where it becomes more noticeable at higher volumes when a film cap can't be used. While human ears aren't as sensitive as test equipment, lower distortion results in cleaner audio at higher volumes. Even though it is not as noticeable at lower volumes, polarized caps are still introducing more distortion into lower-level signals as well -whether it registers in our brain, or not.
At the same time, larger bp electrolytics often can't be used to replace smaller, polarized caps due to physical size limitations. They may also introduce parasitic inductance and higher ESL which should be taken into consideration as well. Their larger size also picks up more heat from other nearby components which can reduce longevity, and they also have higher ESR. As with anything, it's a give and take, but should be part of the equation people make when deciding which caps to use.
One of the main takeaways is that electrolytics are a necessary evil. It is more than just their polarization and is due to their material construction. Aside from their advantage of providing bulk capacitance, they're essentially one of the poorest performing capacitors overall -especially at HF. In my opinion, they can (and should be) avoided wherever possible, especially at low values where film or ceramic may be used instead.