Sean Olive, who certainly would know, has extensively discussed the problems of sighted, i.e. non-blind, testing. Here's one of his comments.
And, to forestall the inevitable "
who the hell is Sean Olive and why should I care what he thinks":
Sean Olive is Acoustic Research Fellow for Harman International, a major manufacturer of audio products for consumer, professional and automotive spaces.
He directs the Corporate R&D group, and
oversees the subjective evaluation of new audio products including Harman's OEM automotive audio systems. Prior to 1993, he was a
research scientist at the National Research Council of Canada where his research focused on the perception and measurement of loudspeakers, listening rooms, and microphones. Sean received a Bachelors degree in Music from the University of Toronto, and his Masters and
Ph.D. degrees in Sound Recording from McGill University in Montreal. His Ph.D. research was on room acoustic adaptation and the acoustical interaction between loudspeakers and rooms.
Dr. Olive has written over 30 research papers on the perception and measurement of audio for which he was awarded the Audio Engineering Society (AES) Fellowship Award in 1996, and two Publication Awards (1990 and 1995). Sean is the current President of the Audio Engineering Society. For more info see
www.linkedin.com/in/seanolive
One of his many writings on the subject:
http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html
The Dishonesty of Sighted Listening Tests
by Sean Olive
Thursday, April 9, 2009
An ongoing controversy within the high-end audio community is the efficacy of blind versus sighted audio product listening tests. In a blind listening test, the listener has no specific knowledge of what products are being tested, thereby removing the psychological influence that the product’s brand, design, price and reputation have on the listeners’ impression of its sound quality. While double-blind protocols are standard practice in all fields of science - including consumer testing of food and wine - the audio industry remains stuck in the dark ages in this regard. The vast majority of audio equipment manufacturers and reviewers continue to rely on sighted listening to make important decisions about the products’ sound quality.
An important question is whether sighted audio product evaluations produce honest and reliable judgments of how the product truly sounds.
...
In summary, the sighted and blind loudspeaker listening tests in this study produced significantly different sound quality ratings. The psychological biases in the sighted tests were sufficiently strong that listeners were largely unresponsive to real changes in sound quality caused by acoustical interactions between the loudspeaker, its position in the room, and the program material. In other words, if you want to obtain an accurate and reliable measure of how the audio product truly sounds, the listening test must be done blind. It’s time the audio industry grow up and acknowledge this fact, if it wants to retain the trust and respect of consumers. It may already be too late according to Stereophile magazine founder, Gordon Holt, who lamented in a recent interview:
“Audio as a hobby is dying, largely by its own hand. As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal. [This refusal] is a source of endless derisive amusement among rational people and of perpetual embarrassment for me..”