• The move to the new server is done. There are some software and database maintenance updates in process. This has us passing the hat around to help out. We appreciate any donations. Seriously, even a dollar helps. The payment page may be found here - https://www.audiokarma.org/support.html

Absolute Proof that blind testing is flawed !!!

DonQuixote99 said:
If the task is to say whether X = A, or X = B, the result will tend to be random.
Click to expand...

Why? If A=B and B=C, then A=C. Same basic principle. And, if it's a learned skill necessary to pass the ABX test, then wouldn't the listener need the same skills to hear the differences in their livingroom? You can't really say it takes new/additional learned skill to identify a difference in a test if one claims already to be able to identify a difference in a non-test condition. The listener, by saying they hear a difference, has already implied they have the skills to identify a difference, yes?

If I set up an ABX test on my computer with say a WAV file and a 320K MP3 made with LAME I score very poorly on the test...basically guessing.

If I do the same test with the WAV but instead compare to a 96K MP3 I get it right every time. Doesn't seem to be any randomness about it...other then when I can't tell a difference.
 
Last edited:
Tom Brennan said:
That's why I usually make it clear my views on gear are opinions not facts.
Click to expand...

Heh...stating it's your opinion isn't even good enough for the 1%'ers on a certain other forum. To them, you're not really even entitled to an opinion if the opinion is a falsifiable statement. Sometimes I just have to shake my head...
 
whoaru99 said:
Why? If A=B and B=C, then A=C. Same basic principle.
Click to expand...

Yes, but the trick is telling what = what by listening.

whoaru99 said:
And, if it's a learned skill necessary to pass the ABX test, then wouldn't the listener need the same skills to hear the differences in their livingroom?
Click to expand...

Yes.

whoaru99 said:
You can't really say it takes new/additional learned skill to identify a difference in a test if one claims already to be able to identify a difference in a non-test condition. The listener, by saying they hear a difference, has already implied they have the skills to identify a difference, yes?
Click to expand...

All true, but that doesn't necessarily mean they actually have the claimed skill.

whoaru99 said:
If I set up an ABX test on my computer with say a WAV file and a 320K MP3 made with LAME I score very poorly on the test...basically guessing.

If I do the same test with the WAV but instead compare to a 96K MP3 I get it right every time. Doesn't seem to be any randomness about it...other then when I can't tell a difference.
Click to expand...

OK. Now there are two basic hypothesis as to what it may mean in those cases where you can't tell a difference. See Fisherdude's post.
 
DonQuixote99 said:
All true, but that doesn't necessarily mean they actually have the claimed skill.
Click to expand...


That's the whole point. Lots of people make the claim, few try to back it, fewer yet actually do.

Now, I'm not really an advocate of the tests because I don't really care too much what other people claim to hear. I make up my own mind based on my experience. That said, I don't discount the validity of such tests to illustrate whether people can or cannot identify differences.
 
Last edited:
Even a skilled listener is not without bias, and could effect the outcome.
Plus differences in ear design, hearing etc can may differences more or less apparent.
After you reach a certain quality of sound I think the differences are a matter of taste.
Even if you had a random set of skilled listeners, each with little bias, and a true carefully, double blind test adminsitered-what real benefit would be gained?
A statistically significant correlation? Perhaps, but it would probably be more usefull in marketing a said product (if even this) than for the average listener.
 
Last edited:
There's a pile of money to be gained or lost depending on the outcome. Personally, I think if the odds of favorable outcome were high, thus money to be gained, you'd be seeing more about tests of this nature.
 
Last edited:
whoaru99 said:
There's a pile of money to be gained or lost depending on the outcome. Personally, I think if the odds of favorable outcome were high, thus money to be gained, you'd be seeing more about tests of this nature.
Click to expand...

I agree.
A number of companies seem to stick with mere casual relationships as far a marketing goes, with the implication (or perhaps by omission) their research found a statiscally significant correlation.
In part, they may do this because of the makeup of the groups they are marketing the products to.
 
Wigwam Jones said:
Everybody who has said that and tried to prove it has failed. Everybody.
Click to expand...

Wiggy, if there is little problem distinguishing between SS and tube AMPs, there like wise should be little problem doing the same with preamps at my house on my gear.

I'll schedule a test session for a Sunday in late January, and anyone who posts in Thinking out Lout will be welcome(again) to attend another blind testing session.

We'll run the same format schedule as last time. We are in St. Louis, immediately off a major freeway down south.

Any takers? Which weekend would work best?
 
sheltie dave said:
Wiggy, if there is little problem distinguishing between SS and tube AMPs, there like wise should be little problem doing the same with preamps at my house on my gear.

I'll schedule a test session for a Sunday in late January, and anyone who posts in Thinking out Lout will be welcome(again) to attend another blind testing session.

We'll run the same format schedule as last time. We are in St. Louis, immediately off a major freeway down south.

Any takers? Which weekend would work best?
Click to expand...

You mean I have to go out to my icebox of a garage to work on that damned Citation IV again? Arrrgh! :lmao:
 
whoaru99 said:
There's a pile of money to be gained or lost depending on the outcome. Personally, I think if the odds of favorable outcome were high, thus money to be gained, you'd be seeing more about tests of this nature.
Click to expand...

While it wasn't double-blind, a friend of mine who's a rep for various audio companies did a blind test at JBL a while back. He said that it was very positive in regards to JBLs most recent high=end speakers, against a lineup of the usual suspects in the field (like Wilson, etc). I don't know the exact details of it except that he was very impressed by their new speakers, and said he might not have been had he known he were listening to JBLs. Which I'm sure was the point of the test...to take away prejudice.
 
Harman is big into blind testing for at least some of their speakers, and are using it to zero in on what people prefer in terms of sonic charcteristics of speakers. Sean Olive has written about it on several occasions. It would be very cool to check out their testing center with the automatic speaker switcher and such.
 
gvalent said:
While browsing for preamps I came across this on the Van Alstine site:

http://www.avahifi.com/index.php?option=com_content&view=article&id=191&Itemid=181&74a0ad6b5f7a1df0ef4ab98b8fffbb41=0d5d0ac3fdbb988b1ec0e2d52f7207a2
Click to expand...

Oh my, what a sensible fellow!

+1 and two big thumbs up! :thmbsp: :thmbsp:

Ok, I'll add a quick precis (that does this noble essay scant justice):

1. Measuring without listening can give bogus results, but this is not a good reason not to measure, because listening without measuring can give highly bogus results.

2. Don't be dogmatic.
 
Last edited:
Judging from the response level, it looks like no one is interested in a blind testing session. :D

A lot of people can comment on them, but few want to be exposed to such evil.:scratch2:
 
sheltie dave said:
Judging from the response level, it looks like no one is interested in a blind testing session. :D

A lot of people can comment on them, but few want to be exposed to such evil.:scratch2:
Click to expand...

You mean the evil of blind testing, or the evil of traveling to St. Louis?
 
You must log in or register to reply here.
Back
Top Bottom