The answer is it really depends upon the output impedance of the source driving the VU meter, how much loading the VU meter does, and how it does the measuring.
This is more of an engineering question than an audibility issue, as the amount in question is incredibly tiny. But it does distort, so that likely answers the question.
Here's a short discussion:
http://sound.whsites.net/project128.htm
While some users may simply connect a VU meter directly across the output signal lines from the mixer, this approach will cause extra loading on the signal, which is non-linear. While the distortion introduced might not be high (it depends a lot on the output impedance of the mixer), it is definitely measurable. The extra load may also reduce headroom, because the output stages may clip earlier than expected due to the additional load. The meters shown above are quite sensitive (100uA DC full scale), but still cause over 0.3% distortion when connected to a 600 ohm source. Distortion is worst at around mid-scale - right where it will sit for much of the time with typical programme material. The nominal impedance for a "true" VU meter is 3,900 ohms - the resistor can be seen in Figure 2.