Evidently we're definitely not all born equal or raised evenly ear wise... For example this should explain the too often sterile and volatile discussions between those whose preferences for "higher end" cables claim to ear those differences where others can only scream "snake oil". So we not only have different earbuds, but capacities and sensitivities as well. Then there is (wo)men's worse handicap for any tolerant approach in understanding of the yet unknown ; pride & assertiveness in his/her phono gear and intelligence.
Not that this is your case evidently, but for you to say that it's not really bothering on headphones then should be even less noticeable on speakers is something I can't agree with. These "noises" even through speakers can 1) bring distortions in the higher frequencies 2) act like a "veil" not allowing a good to better "focus" on the music 3) and finally "hide" the depth in sound that should allow to create a 2-D or even 3-D imagery in sound. My vintage MONO records do already suffer from wear to inflict them improper groove retrieval.
On my old lap top fitted with powered speakers and a sub (both given to me), so nothing fancy, I instantly heard the differences between both sound clips presented here. Admit-ably I am a MONO guy that knows what he wants. MONO grove retrieval is very important to me since I do do play and listen to my beloved vintage MONO 45rpm records. I do understand the sound technicians who needs to hear with headphones for a better "visibility" of the details, as it is their job. Still I truly prefer the music that fills the room when the acoustic allows to get "that". My playback sound is of the neutral, clear and dynamic kind so I do get a lot of details out of my speakers. And there those noises are a nuisance.
It took me ages to gather my Ri-Fi phono gear as it is today to get that right-fidelity playback. If some can live with a MONO switch in their chain, good for them. As far as I can remember, that is long ago, such switch on the amps I heard took off all dynamic from the signal as well. Maybe such switch in some preamps are better at that... So I embarked on a quest in the technical understanding of what is required for the right phono gear to do justice to my beloved MONO 45rpm's. Since I have learn to work-out, build and tweak my phono chain on and around my MONO cartridge. This not to impair as much as possible the cartridge micro volt signal and on the contrary in how to "improve" it's rendition in musicality.