It's been my understanding that even though you're wearing hearing protection, the vibrations in your ear drum can still have a negative effect. Tinnitus in particular.
I'm a heavy equipment operator and wear ear protection most of the time depending on the machine. Besides ear wax build-up...