• The move to the new server is done. There are some software and database maintenance updates in process. This has us passing the hat around to help out. We appreciate any donations. Seriously, even a dollar helps. The payment page may be found here - https://www.audiokarma.org/support.html

Absolute Proof that blind testing is flawed !!!

Wigwam Jones said:
........'our thing'......................................

........people like validation........

.....we're all motivated by different ideals......

And everyone has the vague notion that their motivating factors are also the motivating factors of everyone else.........

Merry Christmas, friends. Good listening.
Click to expand...

I would say that this post qualifies you to be "one of the thought leaders", particularly the thoughts I've quoted above.

We ARE all the same, and we are all different, and we all THINK we are the same.

I'm particularly guilty of the last one, sometimes to the extent that I assume I can read the other fellow's mind and know his motivations.
 
I came across the original post eight or ten months ago and pasted it into a folder of things I find interesting, informative, or humorous.

As stated previously, my post was meant to draw attention to the original poster’s “inevitable conclusion” (and yes, just a hint of sarcasm). The title was to draw attention to the thread.

What I was expecting from the thread was some AK wit and humor, which I’ve always appreciated; and of course it was there.

What I did not anticipate was the direction the thread took, the cross section of AK members who posted and quality and passion expressed in the posts. I’ve always had a great respect for AK members and even more so now.

I’ve learned a great deal.

Merry Christmas!
 
Last edited:
The man that first introduced me to Sansui and Klipsch was blind, he did incredible feats with his hearing ability. I suggest real blind people may have the kind of developed hearing, that the rest of us don't. Maybe they could be used for a test.
 
hotroady said:
The man that first introduced me to Sansui and Klipsch was blind, he did incredible feats with his hearing ability. I suggest real blind people may have the kind of developed hearing, that the rest of us don't. Maybe they could be used for a test.
Click to expand...

Then the same people would be arguing about how the blind person lost their sight, how long have they been blind, and have they been blind long enough to have developed "better" hearing.
 
Well my two cents, is that if you are into blind testing, like anything humans do, one would become somewhat more developed at hearing a difference than others, who might do it occasionally.
The more you do it, the better at it you become. I critical listen for a living and its an ability I have developed and refined over 25 years of doing live audio and tuning systems..........
I also find that, I need more time than switching between two sounds trying to decide which is better, I know I could hear a difference, but to decide which is best for me takes time.
Like when I swap an amp out, it could be a week before I really appreciate its differences and fully tune in to which I actually prefer..............

Just a couple of things I thought of reading though the posts here.....

In summary, it depends what the objective of the blind test is.
Can you hear a difference? Which sounds better? Is it tubes or solid state?
Is to choose equipment or just hear a difference like a competition to see who has the best ability?....

Nice to see this thread going for so long............
 
kevzep said:
In summary, it depends what the objective of the blind test is.
Can you hear a difference? Which sounds better? Is it tubes or solid state?
Is to choose equipment or just hear a difference like a competition to see who has the best ability?....

Nice to see this thread going for so long............
Click to expand...

In general, such tests have been done to tell if a person can tell the difference between two audio items; one well-regarded but consumer mainstream and inexpensive, the other nose-bleed audiophile territory. It doesn't matter what they are - receiver, amp, speakers, etc.

The subject is not asked to tell which unit is which, or even what the two units are. The subject is not asked to judge their relative merits or assign a ranking in terms of quality. The subject is played two samples, then played a third and asked if it sounds like the first or like the second. Just tell them apart, that's all. And that is what testers consistently fail to do for the most part.

The testing is valid; the exact same process has been used to determine if their is a testable difference between things like beer and wine and television picture tubes. It's been done for years. If there is a testable difference, it shows up. If not, then the units are considered functionally identical for the purposes of testing.

Remember the old "Pepsi Challenge?" That's what people think such testing is, but that's not what it is. A true double-blind test would require that the tester not know which was which, but it would also not test to see which one consumers preferred; it would test to see if they could tell the difference after taking a sip from A, a sip from B, and then a sip from X. The tester would be asked if X was A or B. If they could reliably tell the difference then THERE IS A DIFFERENCE. If they cannot, then there IS NO DIFFERENCE - for the purposes of testing. Clearly there are still differences; just not differences that the test being used can isolate.

Testing is good. It reveals things, sometimes startling things. It's how we figure things out. It is not correct to assume that testing methodology is flawed just because it does not demonstrate what a person believes to be true. It may demonstrate that there really isn't an audible difference; or it may demonstrate that the test itself is not capturing the essence of that difference. But the methodology itself is superb.
 
Testing is interesting - it shows that a lot of people can't hear the difference between CD's and MP3's...
 
We conducted a fair blind testing of tube vs solid state amps, where we found there was little difficulty for a majority of QUALIFIED listeners listening under IDENTICAL conditions, music, and all other equipment.

While the Wilsons and VTL may not be the best tools for the job, I would question if the listeners, music, and listening period were appropriately designed and screened to render a reproducible outcome. Hey, based on the confused outcome, I would suspect the VTL preamp to be similar in performance and sound to the NAD, or a lack of testing design rigor was present, or the Wilsons are good at hiding actual measurable differences..

Most of the preamps I have have a readily apparent signature, clarity, soundstage, and extension at both ends. Very few of the SS pres lack the grain and biting edge. Then again the modded KHorns on the firing line deliver the goods with ruthless abandon, so it is quite easy to hear differences the Wilsons may smooth over.
 
Wigwam Jones said:
In general, such tests have been done to tell if a person can tell the difference between two audio items; one well-regarded but consumer mainstream and inexpensive, the other nose-bleed audiophile territory. It doesn't matter what they are - receiver, amp, speakers, etc.

The subject is not asked to tell which unit is which, or even what the two units are. The subject is not asked to judge their relative merits or assign a ranking in terms of quality. The subject is played two samples, then played a third and asked if it sounds like the first or like the second. Just tell them apart, that's all. And that is what testers consistently fail to do for the most part.

The testing is valid; the exact same process has been used to determine if their is a testable difference between things like beer and wine and television picture tubes. It's been done for years. If there is a testable difference, it shows up. If not, then the units are considered functionally identical for the purposes of testing.

Remember the old "Pepsi Challenge?" That's what people think such testing is, but that's not what it is. A true double-blind test would require that the tester not know which was which, but it would also not test to see which one consumers preferred; it would test to see if they could tell the difference after taking a sip from A, a sip from B, and then a sip from X. The tester would be asked if X was A or B. If they could reliably tell the difference then THERE IS A DIFFERENCE. If they cannot, then there IS NO DIFFERENCE - for the purposes of testing. Clearly there are still differences; just not differences that the test being used can isolate.

Testing is good. It reveals things, sometimes startling things. It's how we figure things out. It is not correct to assume that testing methodology is flawed just because it does not demonstrate what a person believes to be true. It may demonstrate that there really isn't an audible difference; or it may demonstrate that the test itself is not capturing the essence of that difference. But the methodology itself is superb.
Click to expand...

Right, very informative..............Thank you.
 
Wigwam Jones said:
The subject is not asked to tell which unit is which, or even what the two units are. The subject is not asked to judge their relative merits or assign a ranking in terms of quality. The subject is played two samples, then played a third and asked if it sounds like the first or like the second. Just tell them apart, that's all. And that is what testers consistently fail to do for the most part.
Click to expand...

Yes, this is ABX testing.

Two devices, A and B, are the devices in the test. At random, for each test, either device A or B is assigned as the variable "X".

"X" is played, and then the listener can switch between X and A or X and B or A and B.

The listener is merely asked to choose if X=A or if X=B for each sample/test, they are not asked to choose which they prefer. If one scores low it indicates the listener cannot reliably differentiate A from B (or, of course, vice versa).
 
Last edited:
Trained perceptions are a vital point. Visually, if we're 'paying attention' looking for particular stuff, we completely ignore other stuff. A famous test had subjects watching a tape of a basketball game. They were told the object was to see how accurately they could count the shots made by each team. Midway in the tape a guy in a gorilla suit came into the picture and clowned around. Afterwards, the shot counters were asked "How about that gorilla?" They mostly replied "What gorilla?"

Part of our perceptual appratus is a filter that takes everything coming in, and filters out most of it. This is the capability that enables you, for example, to follow a conversation at a party despite being surrounded by a loud babble of other conversations. One can easily imagine that audio 'differences,' that become very important to some who learn to pay attention to them, might simply be 'filtered out' by most.

benb said:
I downloaded a file of two 30-second recordings, one of a preamp with an input transformer (one of those $50 to $100 Jensen mic audio transformers), and one without, and I didn't "hear" the difference between the two until after several weeks or months of trying. The transformer apparently adds a LOT more change or 'color' to the sound than does going between different preamps. Part of it was "learning to hear" the difference - once I heard what the difference was, it became easier to distinguish. I think it's the same with the preamps. If only I knew just what to listen for between them, I could tell the difference. It's not quite something that can be put into words, as in "this one's brighter than that one" - it may be that after you hear the difference, but it doesn't quite help if you can't already hear the difference.

We CAN "train our ears" to hear things we don't already notice, but it's not easy if you don't know what your listening for.
Click to expand...
 
DonQuixote99 said:
Trained perceptions are a vital point. Visually, if we're 'paying attention' looking for particular stuff, we completely ignore other stuff. A famous test had subjects watching a tape of a basketball game. They were told the object was to see how accurately they could count the shots made by each team. Midway in the tape a guy in a gorilla suit came into the picture and clowned around. Afterwards, the shot counters were asked "How about that gorilla?" They mostly replied "What gorilla?"

Part of our perceptual appratus is a filter that takes everything coming in, and filters out most of it. This is the capability that enables you, for example, to follow a conversation at a party despite being surrounded by a loud babble of other conversations. One can easily imagine that audio 'differences,' that become very important to some who learn to pay attention to them, might simply be 'filtered out' by most.
Click to expand...

Nice post!
 
DonQuixote99 said:
Trained perceptions are a vital point. Visually, if we're 'paying attention' looking for particular stuff, we completely ignore other stuff. A famous test had subjects watching a tape of a basketball game. They were told the object was to see how accurately they could count the shots made by each team. Midway in the tape a guy in a gorilla suit came into the picture and clowned around. Afterwards, the shot counters were asked "How about that gorilla?" They mostly replied "What gorilla?"

Part of our perceptual appratus is a filter that takes everything coming in, and filters out most of it. This is the capability that enables you, for example, to follow a conversation at a party despite being surrounded by a loud babble of other conversations. One can easily imagine that audio 'differences,' that become very important to some who learn to pay attention to them, might simply be 'filtered out' by most.
Click to expand...


Your post is a good example of how focus on one thing makes the brain filter out other things, but I'm not sure it relates to what is asked of a listener in an blind / ABX test.
 
Last edited:
If I think something sounds better I'm content to use it even though it actually may not; I don't worry about it.

That's why I usually make it clear my views on gear are opinions not facts.
 
whoaru99 said:
Your post is a good example of how focus on one thing makes the brain filter out other things, but I'm not sure it relates to what is asked of a listener in an blind / ABX test.
Click to expand...

If the listener has not learned to pay attention to the particular distinguishing differences, they won't be able to tell the difference between A & B. If the task is to say whether X = A, or X = B, the result will tend to be random.
 
It strikes me that one can save a lot of money be avoiding that learning-period exposure to high-priced items. This is easily acomplished in my case.... :D
 
DonQuixote99 said:
If the listener has not learned to pay attention to the particular distinguishing differences, they won't be able to tell the difference between A & B. If the task is to say whether X = A, or X = B, the result will tend to be random.
Click to expand...

There are two possible explanations for random outcomes:

1. The listener is not paying attention to the distinguishing differences.

2. There are no distinguishing differences.



There is no possible way of determining which is the correct explanation from a listener sample of one. Otherwise, the test must be done multiple times with different listener groups.
 
You must log in or register to reply here.
Back
Top Bottom