• The move to the new server is done. There are some software and database maintenance updates in process. This has us passing the hat around to help out. We appreciate any donations. Seriously, even a dollar helps. The payment page may be found here - https://www.audiokarma.org/support.html

Absolute Proof that blind testing is flawed !!!

gvalent

gvalent

Well-Known Member
Below is a post from someone on another site that I found entertaining. His conclusion says it all.


“I was part of a baffling experiment. This was a system with VTL monoblocks, a VTL Reference preamp, and Wilson speakers. Everyone was floored by how amazing the VTL pre-amp sounded, I had to agree, it sounded pretty damn good. Then someone in the group made a comment that listeners would not be able to detect in a blind listening test when the VTL was in our or out of the system.

So we set up an experiment substituting the VTL preamp with a $200 NAD preamp while people listened blindfolded. Not a SINGLE person in the group was able to consistently tell which was the VTL or the NAD. One is around $10,000 and the other is $200.

This brings up the inevitable conclusion that listening evaluation is probably extremely flawed in these types of tests no matter how you slice it or dice it.”



.
 
Last edited:
Register to hide this ad
If you could qucikly A/B something like with a switch, to where there wasn't a gap in the sound, you would probably have a much easier time telling one part from the other..

When I try to hear the differences in something at home, at times it's hard, if not impossible to tell, since I had to stop the music, swap the part, then listen again... When you have to go from memory, I believe sound is something that doesn't last near as long in our minds like smell or sight does...

As an example, I can easily remember what an old friend looks like, but I can't remember what thier voice sounds like, although I can remember thier accents and flucuations, but pitch is hard to pull...
 
gvalent said:
This brings up the inevitable conclusion that listening evaluation is probably extremely flawed in these types of tests no matter how you slice it or dice it.”
Click to expand...

Hehehehe.

Yes, that's exactly what it means. On the planet Wrong.
 
gvalent said:
Below is a post from someone on another site that I found entertaining. His conclusion says it all.


“I was part of a baffling experiment. This was a system with VTL monoblocks, a VTL Reference preamp, and Wilson speakers. Everyone was floored by how amazing the VTL pre-amp sounded, I had to agree, it sounded pretty damn good. Then someone in the group made a comment that listeners would not be able to detect in a blind listening test when the VTL was in our or out of the system.

So we set up an experiment substituting the VTL preamp with a $200 NAD preamp while people listened blindfolded. Not a SINGLE person in the group was able to consistently tell which was the VTL or the NAD. One is around $10,000 and the other is $200.

This brings up the inevitable conclusion [Emphasis added] that listening evaluation is probably extremely flawed in these types of tests no matter how you slice it or dice it.”
Click to expand...

I find nothing inevitable about this conclusion. I would interpret the data as demonstrating that, under the conditions of the experiment, the panel could not distinguish between the VTL and NAD preamps to a level that was statistically significant. This could indicate that the experiment was flawed or that the preamps under test actually sounded the same.
 
Last edited:
Forgive me if I sound painfully stupid here;

I get the whole "garbage in, garbage out" thing.....

But, aren't the differences in pre-amps going to be negligible anyway, dependent upon the power amp and speakers which appear to remain consistent in this experiment?

Which is to say, the lionshare of the voicing/colouration is going to be from the amp and speakers..... no?
 
similost said:
If you could qucikly A/B something like with a switch, to where there wasn't a gap in the sound, you would probably have a much easier time telling one part from the other..

When I try to hear the differences in something at home, at times it's hard, if not impossible to tell, since I had to stop the music, swap the part, then listen again... When you have to go from memory, I believe sound is something that doesn't last near as long in our minds like smell or sight does...

As an example, I can easily remember what an old friend looks like, but I can't remember what thier voice sounds like, although I can remember thier accents and flucuations, but pitch is hard to pull...
Click to expand...

That is why I expect no groundbreaking difference between my new Shure cart or my older Grado when I switch. Plus I don't have an extra headshell to make a fast switch.
Heck, I couldn't tell the difference between my slightly modded PAS 2 to a $1000 preamp/phono stage and it took only a couple minutes to swap.
 
I remember taking a course in college called the "Physics of High Fidelity" given for one term by a visiting professor from South Africa. He claimed that all competently designed amplifiers sound the same. We all laughed at him. We knew all about the Pioneer, Marantz, Yamaha etc sound and we could easily tell them part. Being college students, we had plenty of free time and spent a substantial amount listening to music and equipment (This was the 70's). Just because his 60 year old ears couldn't tell the difference wasn't our problem!
He proposed what is now known as a double blind test. He had access to some fine university equipment (Big East School) and the switching between amps was nearly instantaneous. Bottom line-- none of our 20 year old ears could do better than random chance!
He said we were all victims of marketing BS and that any competent audio engineer knew that. Now that he had proved to us that he may know something about the topic he was teaching, we gave him a lot more respect.

He claimed that the quality of the recording, the acoustics of the room and the speakers interaction with the room acoustics were the only important variables for good sound. Everything else was marketing propaganda.

I'm sure a lot of people here will disagree and I'm not saying he's right, I just relay the story as it seems to fit this thread.

Ray
 
There is only one inevitable conclusion about blind, DBT, and ABX tests as they relate to audio.

The conclusion is there will never be agreement about them. If you pass, the test was flawed. If you fail, the test was flawed. Just depends on the perspective. :D

My opinion? Fully understand the conditions and constraints of the test, and don't bet anything you can't stand to lose.
 
Last edited:
One other point to be made.... perhaps further to my prior.....

Were the listeners actually floored by how amazing the pre-amp sounded? Really? Or were they floored by the way the VTL monoblocks and the Wilson speakers sounded?

I mean, how many of them actually had prior experience hearing the VTL monoblocks.... or the Wilson speakers? Much less the VTL monoblocks and Wilson speakers together? What about the source components and material?

So many variables...... I suppose that's what keeps us (well me, anyway) addictively perusing this site.....
 
The only absolute is that when I hear the words "Absolute Proof", I know absolutely nothing will be proven.
 
Bogframe said:
The only absolute is that when I hear the words "Absolute Proof", I know absolutely nothing will be proven.
Click to expand...


Well....depends, I think.

In the context of the test something may be proven. Whether or not that result applies to the population with a high degree of confidence is the million-dollar question, and crux of the matter. :D
 
gvalent said:
So we set up an experiment substituting the VTL preamp with a $200 NAD preamp while people listened blindfolded. Not a SINGLE person in the group was able to consistently tell which was the VTL or the NAD. One is around $10,000 and the other is $200.
Click to expand...

Why do you mention the retail price of the two preamps? To me, this just proves that a certain percentage of people believe that spending more money always yields better sound. In business school that mindset in people is called a "marketers dream". :D

There are also people who believe that spending more money only yields a much thinner wallet, yet no guarantee of better sound.

Not that there's anything wrong with that......
 
sparky1116 said:
But, aren't the differences in pre-amps going to be negligible anyway, dependent upon the power amp and speakers which appear to remain consistent in this experiment?
Click to expand...

Well, if it does, then it begs the question as to why one preamp is worth $9800 more than the other, no?

bs
 
Give me a month with both of those preamps in MY house, my listening room, with them in identical indistinguishable boxes, with MY turntable, amps, and speakers, who's sound I am intimately familiar with, and I bet anybody here $1000 that I could tell the difference. Hell just give me the time it takes to play a few familiar albums.
 
bshorey said:
Well, if it does, then it begs the question as to why one preamp is worth $9800 more than the other, no?
Click to expand...

'Can't argue with that.... Were I to come up with $9800 (or so) that was entirely disposable..... who am I kidding...... I'd like to find out.... or not.

It's all academic to me anyway. Were I to own the "budget" NAD preamp described, purchased at the MSRP described of $200, it would represent the most expensive piece of kit I own for two-channel. Hopefully not the greatest quality piece, but certainly the most expensive.
 
JohnVF said:
Give me a month with both of those preamps in MY house, my listening room, with them in identical indistinguishable boxes, with MY turntable, amps, and speakers, who's sound I am intimately familiar with, and I bet anybody here $1000 that I could tell the difference. Hell just give me the time it takes to play a few familiar albums.
Click to expand...

Everybody who has said that and tried to prove it has failed. Everybody.
 
juncers said:
He said we were all victims of marketing BS and that any competent audio engineer knew that. Now that he had proved to us that he may know something about the topic he was teaching, we gave him a lot more respect.

He claimed that the quality of the recording, the acoustics of the room and the speakers interaction with the room acoustics were the only important variables for good sound. Everything else was marketing propaganda.
Ray
Click to expand...

90% agree. The rest 10% is a choice of individual components, which sometimes are badly designed or poorly built.
 
You must log in or register to reply here.
Back
Top Bottom