More stuff in the path. Also circuitry to change load.
Usually it's nominal, if at all noticeable. Becomes more an issue with higher-up gear.
Ever activate the "bypass" switch (bypasses tone controls and filters) on your pre-amp? I have, on my PSE, B&K, Holman, Philips and Sansui CA-2000, and the sound becomes nominally, but noticeably improved.
Less stuff in the signal path.