Obviously, you need at least a third category:
BOTH
But your post reveals the difficulties and balancing acts involved with this hobby of ours.
Whenever you alter the organism that is the system, you alter its synergy. Over the decades, I’ve added new pieces of gear that in many ways sounded much better (were noticeably higher in the hi-end chain) than what I already had; but, sometimes, despite the virtues of some new pieces of better gear, the system then sounded too refined, too clinical, too distracting, less live, less alive, less emotionally involving. Or the new and better components just didn’t work with what I already had. (Less synergy.) Sometimes, my gear, improved and more revealing, has highlighted bad recordings so that I’ve had to jettison music from the lineup. And sometimes, new pieces of gear have forced me to be too invested in and focused on what newfound nuances they revealed about the music, which brought me too far away from the pure enjoyment of the music itself. In those instances, I’ve, perhaps, pulled back on some gains to maintain that delicate balance and to retain what I love about listening to my stereo: that it makes me love and appreciate the music/musicians, without actually inspiring me, instead, merely to love and to worship at the altar of my stereo.