Absolute Proof that blind testing is flawed !!!
- Thread starter gvalent
- Start date
No, they haven't. To the best of my knowledge, there haven't been tests like I described, at least in any sort of number.
I'm not talking about a room full of people in an unfamiliar environment hearing a ton of gear for the first time, trying to pick out differences in one, to satisfy somebody ax grinding. I couldn't do that, not in any sort of statistically repeatable manner. Too many variables, too little time to really get what I enjoy and notice out of hi-fi differences. And regardless, almost without fail, the things I notice as 'positives' in a quick A-B in store, become things that annoy me over time.
HI FI is NOT about instantly noticeable differences. It is not about what you notice in a strange room with strange people, your mind in a different (unrelaxed) state where you are concentrating on somebody else's agenda. It is about personal enjoyment.
If you honestly think there are no differences in our gear, why are you here? Just go buy an all-in-one and be done with it.
There's a British Hi Fi magazine that routinely does blind testing and somehow they always manage to point out very detailed, well thought out differences in the products that their group is reviewing.
I wonder why there are differences in those tests and not the ones where there's an agenda?
I wonder why there are differences in those tests and not the ones where there's an agenda?
clydeselsor
Lunatic Member
:lmao:
+1 on both of the above.
I wouldn't go as far as to say that everything boils down to marketing propaganda, but assuming that we're dealing with equal-quality products from different manufacturers, the other factors are going to be more important than the amplification equipment, IMHO.
I've been in one blind test, and learned that I'm very good at picking out tube and solid-state amplifiers.
good point.
also humans have very poor tonal memory. that said i would also have likes to know what the source was.
Wigwam Jones
Caesar non supra grammati
You mean the kind of tests where nothing is really being tested at all, just a person has a preference and expresses it? Oh yeah, I have no doubt you'd be correct at that kind of test. Double-blind? Nope.
I'm not talking about that either, nor was the O/P, as far as I can determine.
I absolutely agree with that statement.
Did I say anything resembling that statement? I don't think I did. I noted an historical fact; people can't tell the difference between some types of audio gear when subjected to double-blind tests. I further noted that it has been my experience that everyone who claimed later that *they* could tell the difference because *they* have some special ability as regards their own equipment has also failed utterly to prove anything of the sort. It's actually amusing that people still step forward to make that bold claim.
I did NOT say that there are no differences in gear.
As to why I am here; it is because I enjoy listening to music, and I enjoy audio gear, and I enjoy tinkering, and I enjoy fellowship, and I enjoy the vintage equipment, banter, the exchange of information, learning new things, and so on. Am I only allowed to be here if I agree that Product A can be distinguished from Product B if only one has a sensitive enough set of ears?
Sensitive, are we?
The blindfold must have been tied over their ears 
Actually I am not. The one or two blind tests that are always referred to in the 'everything sounds the same' camp don't really worry me against the mountain of evidence (including specs/measurements and ABX tests that aren't quoted because they led to, um, differences) and personal experience that tells me the tests proved nothing beyond what happened in those rooms, at that time, with those people.
If somebody can't tell the difference between a high end tube preamp, and a budget transistor one (not making judgement on which sounds better...just that they would sound different), that says more to me about the test than the products. I don't find the quote in the original post to be off base at all...it does suggest to me that the test is flawed. And I'm not saying that I could tell in that test...something was making those two pieces of gear sound alike, to those people. That's it. I don't for a minute think they would sound alike in my home, because I have enough real-world experience with dozens of preamps that have been through here to tell me that they're all, each and every single one not just slightly different but VERY different. If I didn't believe that, I'd find another hobby. But I do believe.
I switch gear around, as my hobby. And there are easily identifiable sonic signatures that come back every time I put a different preamp in the path (if somebody couldn't tell the Phase Linear 3000 from my ARC tube preamp it would suggest to me that they are deaf). That's why I have a half dozen preamps...from low to high end. They all sound quite unique. The idea that they don't is like somebody trying to convince me the sky is red. It says more to me about their vision or the tint of their glasses than the color of the sky.
Last edited:
They also tell you a lot about the person or persons conducting the test.
I deal with quantitative testing at work as part of my job. While we can't completely control the 'unbiased' outcome, we are familiar enough with the tests and who is going to be exposed to them to significantly sway the results in whatever direction we want.
Welcome to AK...hope your stay here is more fun than our tri-annual ABX 'discussion'
Wigwam Jones
Caesar non supra grammati
The 'mountain' is no mountain. And it does not support YOUR assertion, which is that YOU can tell a difference. Those few studies which have shown that a group can tell a difference under some circumstances apply only to the group. Like life insurance, which can predict when a person like you might kack it, but cannot possibly predict when you personally will go sneakers-up.
My assertion that you're a bit sensitive about this refers to your invitation for me to buy an all-in-one amp and leave AK if I don't agree with you. I find that a touch on the sensitive side.
Not when it is a controlled randomized study. That's the reason they do controlled, randomized studies.
There's a world of difference between 'suggest' and 'absolute proof'. But that does seem to be the issue. Superlative trouble. I recommend a dictionary and a thesaurus and call me in the morning.
And I laugh at that assertion. Lots of people have made it, none have been able to back it up. It's really quite amusing.
That's actually a very good analogy, but it proves the opposite.
You see, I am color-blind. I've got it quite badly.
I didn't know I was color-blind until I was in grade school, when we were presented with crayons without labels for the first time, and I proceeded to color my skies purple and my grass brown.
You see, I can argue with you all day and tell you that blue and purple are THE SAME. And I promise you, they are. To me. BUT, I have a deficiency.
How do I know that what I have is a deficiency and that the rest of the world isn't lying to me? Testing. I'm the one in the minority, so I have the deficiency.
The world of science has shown us that double-blind testing demonstrates differences. Properly-controlled, randomized tests demonstrate truths.
I absolutely agree with you that there are lots of differences between, say, a Umpty-Ump Mark IV amplifier and a Yada-Yada Series Two amplifier. Distinct differences, which can be experienced, felt, observed, even described to some extent. I'm with you!
But if a randomized double-blind study demonstrates that the average listener (for that is what a randomized study is) cannot tell the difference between them, I conclude what? Not that there IS NOT A DIFFERENCE, but that the average Joe cannot tell the difference in a randomized, double-blind test.
We audiophiles are at a loss - still - as to how to prove that there are quantifiable differences, or to explain how it is that there are differences that cannot be quantified. I find it curious, but I'm in the group that believes such differences exist. But I won't argue that it makes double-blind randomized testing invalid. That's just crazy-talk by people who have a lot of emotional baggage tied up in not being wrong.
http://www.stereophile.com/content/blind-leading-deaf
steerpike2
Super Member
This colour thing is rather interesting: I have no obvious colour perception problems - my eye tests show normal. But I disagree with almost everyone on the quality of television pictures. What most people like and think looks realistic, to me looks like a stained-glass window.
I can only assume sound is the same - what is accurate may also be a matter of personal interpretation.
gvalent
Well-Known Member
When I started this thread my intention was to show the absurdity of the poster’s ”inevitable conclusion” drawn from the test (thinking it was rather funny regardless of which side of the fence one is on regarding blind testing).
I named the thread “Absolute Proof that blind testing is flawed !!!” with tongue firmly in cheek.
I could have named the thread “Absolute proof of a faulty sense of logic”……but then nobody would have read it.
The OP
I named the thread “Absolute Proof that blind testing is flawed !!!” with tongue firmly in cheek.
I could have named the thread “Absolute proof of a faulty sense of logic”……but then nobody would have read it.
The OP
Last edited:
Truce..truce..!
We're not in as complete disagreement as either of us thought. I'd reply at length but basically, what J. Gordon Holt wrote there about the subject sums up my opinion almost exactly. Where you and I do differ is that I don't agree that audiophiles are at a loss. I think that most of them just don't care, and I'd put myself in that camp. I really don't give a rat's butt about what ABX testing suggests because I don't believe in the findings of the test. We're all different...testing proving to you that there are colors, for example, doesn't change the fact that you don't see all of them. Since there are colors, infrared colors, etc, that none of us sees..who's right? You or me? Well, neither of us are but since you have to live with your eyes, anything I tell you about how you should enjoy the colors you see would be pointless on my part. You might subjectively like an artist that I don't, because of the differences in your personal perception of the colors used. In audio, I might like an amp that you don't, because I hear differently. What would a test of random people's opinion on the art have to do with our personal choices? Well, that's how I pick audio gear. I either like it or not, and could care less about what a group of random people think.
ABX testing is, in my mind, a design tool. A lab tool. Beyond that, the fact that I can point out -to myself- the differences in a way a preamp handles room reverb, attack and decay, spacial information like imaging, and timbre..and hear those things each time I put it in the system..is enough for me. And its enough for almost every 'audiophile' I've ever met.
You hear those things, too, as you said..and seem to be bothered or at least curious that there's not proof of them. I'm not bothered. That might be the only difference we have here. I'm not trying to change your mind, just trying to explain my opinion on the matter.
Art K.
The Voodoo You Do!
Well said, John and I couldn't agree more.
Wigwam Jones
Caesar non supra grammati
We were 100% up to that last bit. I believe in the findings of the test - as does Holt, clearly. I believe they demonstrate what they demonstrate - not what people *think* they demonstrate. They demonstrate that in randomized double-blind tests, people can't tell the difference. That's all. It does not demonstrate that there *is* no difference.
The first key is to understand what the results are saying. Hard to do when one interprets what appears to be negative results as an attack.
But it is valuable to know that there are colors I cannot see. I may have difficulty believing it, since *I* can't see the differences, but the testing shows it to be a fact that others can see a difference.
By the same token, if everyone was color-blind except me, I could claim that blue and purple were different colors all day long - but since testing would show that people cannot tell the difference, I'd be 'wrong' from the point of view of science. The testing only proves what it proves - that the majority of people tested can't tell the difference.
I am curious, yes, but not bothered. I understand what the testing is saying, and I don't question it. It's clear that there is something at work that is not being isolated in the tests done. Perhaps we'll stumble upon it. Perhaps not. It could be a placebo effect; just because I am certain I can hear a difference, that doesn't mean there is one. Or that there isn't one. The testing only proves what it proves - that under the current conditions, people can't normally tell a difference.
I was just reading about the placebo effect, in fact. Seems they just did a study (those damned studies!) that shows that yes, there is an actual REAL effect when patients take a placebo drug, even if they KNOW it is a placebo. A real effect. That means that the patients themselves get better because they believe the drug made them better, even if (weirdly) they know the drug isn't a drug at all. How wild is that?
So yeah, perhaps music sounds better to me on a highly-optimized system because I believe it does. And just like the placebo effect, that is REAL and not fake. But apparently not objectively testable - at least not now.
Wigwam Jones
Caesar non supra grammati
Blind testing means the subjects do not know which equipment they are listening to. Double-blind testing means the testers do not know which equipment is being played for the subjects; for the reasons you specified. However, the test is controlled so that the actual equipment that was played will be known when the results are gathered.