Absolute Proof that blind testing is flawed !!!

[JohnVF]

That's pretty crazy about the placebos, Wigwam. There must be a certain amount of placebo effect in all of our systems, a little self-suggestion about how good our choices have been. But I don't know that it's nearly as prevalent as some say, at least not for everybody. It's probably different for each person, we're all suggestible to one degree or another.

Random story, but tonight I went shoe shopping and watched this poor guy trying to pick out shoes. He'd get all excited about a pair, bring them over to his wife, and she'd say "toes are too pointy" or something. And he'd frown, and then put them back. This repeated about 10 times, she always had something bad to say about his choice. Obviously...he was a very suggestible person. Open to any opinion but his own. There must be some audio people like that...who love an amp then don't because the press doesn't like it, or somebody on some board, or maybe in a group of people blindly evaluating gear. I wonder if the people in the ABX test were isolated from each other, especially in that test in the original post? We've probably all been in a room with a bunch of audio guys, with a really awful system playing (in our opinion..), where one guy says "this is amazing!" and then a bunch of people nod their heads. Maybe they thought it sounded amazing...or maybe they weren't sure enough in their opinion of what they were hearing to confidently say it didn't sound amazing. Who knows.

The fact that we're all variable is very interesting to me. I know the tests try to take that out of it, but I don't know if that's the right way. Like..I wonder if they could do a bunch of tests, and then pick the people who actually got it right (some by chance, others might have heard differences), then put THOSE people in an ABX test, and see if the results change. Maybe do this a couple of times until finally you are left with a core group of people who could consistently say that this and that were different. At least it would say that some people could tell most of the time, instead of 'in this one instance, the average person couldn't tell'.

 

[Wigwam Jones]

That's pretty crazy about the placebos, Wigwam. There must be a certain amount of placebo effect in all of our systems, a little self-suggestion about how good our choices have been. But I don't know that it's nearly as prevalent as some say, at least not for everybody. It's probably different for each person, we're all suggestible to one degree or another.

Here's a link to one version of the placebo story:

http://abcnews.go.com/Health/placebos-work-deception-study/story?id=12462093

As to self-suggestion, I have no doubt it works. I've run into people who swore up and down that 'Head On' works to relieve headaches. However, from what I understand, it's just wax. And even the ads themselves never claimed the product cured headaches. It just said "Apply directly to the forehead" and "your product works!" Never said what it worked for...

Random story, but tonight I went shoe shopping and watched this poor guy trying to pick out shoes. He'd get all excited about a pair, bring them over to his wife, and she'd say "toes are too pointy" or something. And he'd frown, and then put them back. This repeated about 10 times, she always had something bad to say about his choice. Obviously...he was a very suggestible person. Open to any opinion but his own. There must be some audio people like that...who love an amp then don't because the press doesn't like it, or somebody on some board, or maybe in a group of people blindly evaluating gear. I wonder if the people in the ABX test were isolated from each other, especially in that test in the original post? We've probably all been in a room with a bunch of audio guys, with a really awful system playing (in our opinion..), where one guy says "this is amazing!" and then a bunch of people nod their heads. Maybe they thought it sounded amazing...or maybe they weren't sure enough in their opinion of what they were hearing to confidently say it didn't sound amazing. Who knows.

I don't know how the tests were performed in that respect, but I do know that my own biased experiences echo yours; even here on AK. Let some guy whose opinion is respected suddenly post "Hey, you know what? I just bought a pair of vintage El Crappo speakers - you know, the ones that go unsold for $5 on Craig's List - and they sound GREAT if you wrap some silly putty around the terminal screws!" Next thing you know, people are picking the shelves clean nationwide and posting photos of their modded El Crappos.

The fact that we're all variable is very interesting to me. I know the tests try to take that out of it, but I don't know if that's the right way. Like..I wonder if they could do a bunch of tests, and then pick the people who actually got it right (some by chance, others might have heard differences), then put THOSE people in an ABX test, and see if the results change. Maybe do this a couple of times until finally you are left with a core group of people who could consistently say that this and that were different. At least it would say that some people could tell most of the time, instead of 'in this one instance, the average person couldn't tell'.

I have some customers at work who want some new servers. They say their build software is too slow, the machines are taking too long. They want bigger hardware, faster builds. But the tools I have access to show that the machines are loafing.

In my world, you are what you measure. If the tools don't show that the machines are being hammered, then they're not being hammered and you get no new servers.

But I believe my customers. I am sure there is a way to isolate the performance issues they're experiencing and measure it. So I'm still looking for testing tools that might give me what my customers and I both believe is out there.

When it comes to audio equipment, there are clearly differences between equipment - they test differently on sensitive testing equipment. The question is whether or not listeners can perceive those differences. If they can, then the cost to achieve those differences is more readily acceptable to many consumers. If they can't, then some consumers will tend to shy away from it.

I know I have some high-range hearing loss in one ear, due to gunfire when I was serving in the military years ago. So perhaps I am unable to hear what others can. I can buy that. However, I know that whatever they are hearing that I'm not, the kind of double-blind testing that has been done to date doesn't demonstrate it. You are what you measure. If you can't measure it, the raw data says it isn't there; but like my customers, I am pretty sure it is. I just don't think we've stumbled on the right way to measure it yet.

 

[JohnVF]

I just don't think we've stumbled on the right way to measure it yet.

The answer is probably somewhere between 'everything sounds different' and 'everything sounds the same'. It has to be. Things probably sound more alike than I think they do, and more different than others say. Someday we'll figure it out. In the meantime, I have to pack for the holidays! Cheers to all.

 

[Art K.]

So perhaps I am unable to hear what others can. I can buy that. However, I know that whatever they are hearing that I'm not, the kind of double-blind testing that has been done to date doesn't demonstrate it. You are what you measure. If you can't measure it, the raw data says it isn't there; but like my customers, I am pretty sure it is. I just don't think we've stumbled on the right way to measure it yet.

I've said it many times. We simply aren't measuring the right things, or we haven't learned how to measure the right things, not yet. We may never, especially when it comes to how we perceive art (unless we are listening to random noises we are listening to art). Mood may have as much to do with what we hear than anything we can measure.

 

[stereofanboy]

These guys that write for the audio magazines do this for a living, yes? Why doesn't someone set up a double blind test where these guys spend a forty hour work week doing it? That should take care of at least one or two of their objections. They could probably find a college kid to do the switching for minimum wage. They could bring their own speakers, or cables, or their own whatever isn't being tested.

Someone interested in setting something like this up, should spend a while noting all the objections to the veracity of double blind testing and address them. Then at the end of the test, if a listener unhappy with the results says that something was wrong with the test, let them point out what is wrong, correct it, and try again.

We shouldn't only pick on the journalists; what about the manufacturers of super expensive power cords, or interconnects? I'd love to witness or read about those tests.

I'm not claiming that none of these things sound different from others. I'm saying that I doubt the differences are as common, or as great as many (whose financial interests are involved), would have us believe. I'm saying that if you can't hear the difference without peeking....you can't hear the difference. All these excuses remind me of the medium who says, "the spirits won't come, because there is an unbeliever in our midst". How can you argue with that?

 

[JohnVF]

These guys that write for the audio magazines do this for a living, yes? Why doesn't someone set up a double blind test where these guys spend a forty hour work week doing it? That should take care of at least one or two of their objections. They could probably find a college kid to do the switching for minimum wage. They could bring their own speakers, or cables, or their own whatever isn't being tested.

Someone interested in setting something like this up, should spend a while noting all the objections to the veracity of double blind testing and address them. Then at the end of the test, if a listener unhappy with the results says that something was wrong with the test, let them point out what is wrong, correct it, and try again.

We shouldn't only pick on the journalists; what about the manufacturers of super expensive power cords, or interconnects? I'd love to witness or read about those tests.

I'm not claiming that none of these things sound different from others. I'm saying that I doubt the differences are as common, or as great as many (whose financial interests are involved), would have us believe. I'm saying that if you can't hear the difference without peeking....you can't hear the difference. All these excuses remind me of the medium who says, "the spirits won't come, because there is an unbeliever in our midst". How can you argue with that?

Its sounds like a good idea when you state it like that but I highly doubt you'd find enough people who really care that much about it to participate. There is absolutely no movement, not even one brewing, to debunk high-end, beyond a few corners of the internet. The audio magazine writers have no interest in being debunked. The people who buy the gear feel they can judge it by listening with an open mind and open eyes (as like me they've heard plenty of crappy expensive gear...it isn't just dollar signs that make these people choose gear, as much as certain people would like to think). The people that make the gear measure it and test it to whatever standards they choose...some are high, some are low. It is in none of these people's interest to have some test try to prove that their passion, their way of life, is pointless. And because some of us are so passionate about it, is why some of us are so passionate about it when others, via these tests or whatever, try to tell us that it's all, for lack of a better word, a scam.

 

[gvalent]

I was just reading about the placebo effect, in fact. Seems they just did a study (those damned studies!) that shows that yes, there is an actual REAL effect when patients take a placebo drug, even if they KNOW it is a placebo. A real effect. That means that the patients themselves get better because they believe the drug made them better, even if (weirdly) they know the drug isn't a drug at all. How wild is that?

Very interesting stuff.

The article "Placebo Alters Brain Function" was a real mind bender.

http://abcnews.go.com/Health/Depression/story?id=117057&page=1

One of my favorite sites is Science Daily. It's easy to get lost in here.

http://www.sciencedaily.com/


.

 

[Mr. Lin]

Well said, John and I couldn't agree more.

I agree too, and he pretty much had the same reaction I did to the original post, and the spirit of this thread.

However...

I've said it many times. We simply aren't measuring the right things, or we haven't learned how to measure the right things, not yet. We may never, especially when it comes to how we perceive art (unless we are listening to random noises we are listening to art). Mood may have as much to do with what we hear than anything we can measure.

I've been saying this too, and I think it's falling on deaf ears for the most part.

Someone can't tell the difference between entry-level and high-end gear and you guys don't think there's something else going on there?

 

[JohnVF]

Someone can't tell the difference between entry-level and high-end gear and you guys don't think there's something else going on there?

If you start to believe there's a difference then suddenly bragging about that silver-face score down at the thriftstore doesn't give the hit it used to. Next thing you know you're into the heavy stuff. The wallet goes first, the mind second. One day you find yourself reading Stereophile and Absolute Sound and the day after that you're running your hands down the side of a naked Magico, and ogling the profile of a nude stylus, for which you'd sell that first love of yours, the one with the great knobs, the one you once thought was as good as it gets. It's ugly stuff. A slippery slope...best to keep believing it all sounds the same.

 

[Art K.]

If you start to believe there's a difference then suddenly bragging about that silver-face score down at the thriftstore doesn't give the hit it used to. Next thing you know you're into the heavy stuff. The wallet goes first, the mind second. One day you find yourself reading Stereophile and Absolute Sound and the day after that you're running your hands down the side of a naked Magico, and ogling the profile of a nude stylus, for which you'd sell that first love of yours, the one with the great knobs, the one you once thought was as good as it gets. It's ugly stuff. A slippery slope...best to keep believing it all sounds the same.

:lmao::lmao::lmao: Excellent.

 

[stereofanboy]

.......Someone can't tell the difference between entry-level and high-end gear and you guys don't think there's something else going on there?

The original poster was actually poking a little fun at the guy he quoted. The funny part is that the fellow quoted automatically assumed that when the answer wasn't what he expected, the test must have been flawed. If you get the humor, I'm sorry for playing Captain Obvious, but I suspect that some don't get it.

Now I haven't seen the full version of the source, but from the information supplied, the guy quoted didn't offer a hypothesis of what was wrong other than the outcome. The testing in which that fellow was involved probably WAS flawed, but that doesn't mean that double blind listening tests can't be done properly.

Jon, I understand that a guy that sells $5000 interconnects doesn't wish to be the listener in a double blind comparison between his cables and some $500 interconnects, especially if the results were to be published. The same goes for the reviewer who believes that his living is based on his reputation as a golden-eared audiophile. I wonder if they would be confident enough to do it if they made all the rules except the one that made it a double blind listening test.

I do sometimes read TAS and Stereophile and some others, but not for help making buying decisions. Honestly, I have read about very little in those publications that has made or will make its way into my hands. I read them for entertainment and writing ability scores as high for me as listening ability.

 

[benb]

That is why I expect no groundbreaking difference between my new Shure cart or my older Grado when I switch. Plus I don't have an extra headshell to make a fast switch.
Heck, I couldn't tell the difference between my slightly modded PAS 2 to a $1000 preamp/phono stage and it took only a couple minutes to swap.

I've got the impression cartridges are markedly different compared to electronics. Going from an $80 Grado Blue to a $100 AT440ML eight years ago made a world of (positive!) difference for me. But then I just bought the newer AT440MLa, it's supposed to be not as bright as (or somehow sound different from) the original AT440ML, and I can't tell the difference (not that I'm complaining, as I really like the sound).

Here's something people can buy and compare with. I've got both the mic CD and the pre CD from this page:
http://www.3daudioinc.com/
The mics are generally distinguishable, but all the preamps sound identical to me. I'd seen people posting on rec.audio.pro for many years about the 'sound' of different mic preamps, and I trust that they can hear such differences, but I haven't heard any differences.

I downloaded a file of two 30-second recordings, one of a preamp with an input transformer (one of those $50 to $100 Jensen mic audio transformers), and one without, and I didn't "hear" the difference between the two until after several weeks or months of trying. The transformer apparently adds a LOT more change or 'color' to the sound than does going between different preamps. Part of it was "learning to hear" the difference - once I heard what the difference was, it became easier to distinguish. I think it's the same with the preamps. If only I knew just what to listen for between them, I could tell the difference. It's not quite something that can be put into words, as in "this one's brighter than that one" - it may be that after you hear the difference, but it doesn't quite help if you can't already hear the difference.

We CAN "train our ears" to hear things we don't already notice, but it's not easy if you don't know what your listening for.

 

[Mr. Lin]

The original poster was actually poking a little fun at the guy he quoted. The funny part is that the fellow quoted automatically assumed that when the answer wasn't what he expected, the test must have been flawed. If you get the humor, I'm sorry for playing Captain Obvious, but I suspect that some don't get it.

I've been around here long enough to know that a thread title such as this one is most likely to be sarcasm, and yes I get it. What I said before was that a test in which no one could tell the differences between cheap gear and expensive gear sets off an alarm in my head that the test must have been flawed, because that kind of conclusion flies in the face of decades of listening by people with trained ears, and even those of us without trained ears.

I've got the impression cartridges are markedly different compared to electronics. Going from an $80 Grado Blue to a $100 AT440ML eight years ago made a world of (positive!) difference for me. But then I just bought the newer AT440MLa, it's supposed to be not as bright as (or somehow sound different from) the original AT440ML, and I can't tell the difference (not that I'm complaining, as I really like the sound).

You're right, it's easy to hear the differences between most cartridges. Grado and Shure have very different house sounds, so the sound won't be the same.

 

[JohnVF]

The original poster was actually poking a little fun at the guy he quoted. The funny part is that the fellow quoted automatically assumed that when the answer wasn't what he expected, the test must have been flawed. If you get the humor, I'm sorry for playing Captain Obvious, but I suspect that some don't get it.

The funny part to me is that so many people thought it was so funny. I understand the humor, but wouldn't say I get it...seems to be just another example of the wishful thinking that often swirls around this place in regards to gear that costs more than a tank of gas. Money doesn't automatically mean a better product, and these kind of threads are often viewed as insulting by a number of us because that is the gist of them..that self-described 'audiophiles' can't listen beyond their wallets, that if you run expensive gear you're a fool duped by dollar signs. There's plenty of terrible sounding expensive gear, and I've come to that conclusion by listening to it with my eyes open. How on Earth did I do that? How was I not swayed by my desire to make up for some personal inadequacy through the cost of my stereo?

 

[whoaru99]

I've been around here long enough to know that a thread title such as this one is most likely to be sarcasm, and yes I get it. What I said before was that a test in which no one could tell the differences between cheap gear and expensive gear sets off an alarm in my head that the test must have been flawed, because that kind of conclusion flies in the face of decades of listening by people with trained ears, and even those of us without trained ears.

True, Mr. Lin, but the dilemma is that exactly the reverse is said, and appears to be substantiated in the tests conducted.

You're right, it's easy to hear the differences between most cartridges. Grado and Shure have very different house sounds, so the sound won't be the same.

No doubt some components can and do sound different. However, components obviously deviating from flat response, such as those intentionally voiced, are not items typically subject to blind tests because doing so serves no purpose. Blind tests are typically reserved for components that "should" sound the same, such as amplifiers whos purpose is to change a small signal into an exact as possible larger signal.

 

[bshorey]

When I started this thread my intention was to show the absurdity of the poster’s ”inevitable conclusion” drawn from the test (thinking it was rather funny regardless of which side of the fence one is on regarding blind testing).

I named the thread “Absolute Proof that blind testing is flawed !!!” with tongue firmly in cheek.

I could have named the thread “Absolute proof of a faulty sense of logic”……but then nobody would have read it.

The OP

Having not read the entire post that you cut and pasted from, I was under the impression that the original author was probably being a little tongue and cheek as well. If I were him, and reading this thread, I'd be getting a pretty good chuckle out of the debate it launched. No, then I'd be a troll.

bs

 

[Wigwam Jones]

In my humble opinion, much of 'our thing' is complicated by the same things that complicate the lives of collectors. At times in my life, I've been a minor (read 'low budget') collector of vintage cameras, mechanical wrist-watches, and the like. There are some distinct similarities.

First, people like validation. That is why we post here about a 'scroe' from time to time. We not only want to share our triumph about a low-priced deal on a bit of vintage (or occasionally modern) kit, we want validation from our fellows that this particular item is, indeed, worthy.

Second, there are, as in every group of aficionados, pack leaders and trend-setters. There are the budget vintage audio connoisseurs, as well as the top-end high-dollar mavens. They know their gear, and when they talk about this model or that circuit or this flat measurement or that open and airy sound, it (forgive the pun) resonates with those who admire them and mental notes are made. Prices, as has been wryly noted many times, go up right away on eBay and other sites where such gear can be acquired. The nod of a critic means much, as every restaurateur knows.

Third, people are competitive. Many go from item to item, looking for that one-up, that pinnacle. We know, for example, that the TOTL in every category commands a price that is well above that of the second item down the ladder, generally a ridiculous sum. And experienced technicians can tell us that the mechanical and electronic differences are minimal, sonically they should be so close as to be indistinguishable; but yet we strive to get the monster receiver from the days of the receiver wars and reject the next one down the line. If music reproduction were the issue, we'd probably not be so quick, especially if budgets were truly important.

Fourth, people are acquisitive. We're fans of the chase, many of us hoard, we love the searching and the haggling and the restoring and the overall experience of adding a new trophy to the mantle. Not much interest in the prize after it's won, except for the bragging rights that come with it. Oh, many of us do listen to and enjoy the equipment we obtain; but it's not often the last place we stop before moving on.

And finally, we're all motivated by different ideals, admixes of the notions above as well as actual pursuit of musical reproduction excellence. One guy wants great sound, but it also has to look a certain way. One guy wants great sound, but it has to work with a given speaker or meet with financial constraints. One guy prefers a type of music that sounds better with a particular type of equipment playing it. And everyone has the vague notion that their motivating factors are also the motivating factors of everyone else.

This is why we have threads about setting balance controls, or tone controls, or using EQ, or hanging expensive room treatments or raising speaker wire off the floor or what-have-you. Each of us is somewhat convinced that if we think it's important, it is important; even as we seek each other's approval at the same time. It is like a never-ending game of "Here's what I found, is it good, is it good? It's not? OK, then, I hate it. Here's another thing, is this good? It is? Scroe!"

I'm not dissing or making fun of anyone. This is human nature here, folks. We all have many complications and many factors which motivate us in the pursuit of 'our thing'. The amazing thing to me is that despite all of this, most AK'ers have a generosity of spirit and a genuine desire to share information with each other that transcends our occasional forays into thinking each other insane (or at the very least, confused and in need of enlightenment).

I would still love to see if one of the thought leaders on AK (and many of us know who they are) could suddenly seize upon a P.O.S. bit of kit, declare it transcendent in beauty and sonic goodness, and see how many people jump on it and likewise find the heretofore unnoticed electronic beauty and luminosity lurking within. But I admit, I have a bit of the imp of the perverse in me.

Merry Christmas, friends. Good listening.

 

[Fisherdude]

...What I said before was that a test in which no one could tell the differences between cheap gear and expensive gear sets off an alarm in my head that the test must have been flawed, because that kind of conclusion flies in the face of decades of listening by people with trained ears, and even those of us without trained ears...

True, Mr. Lin, but the dilemma is that exactly the reverse is said, and appears to be substantiated in the tests conducted...

Fwiw, I think this entire thread is one of the most interesting, most entertaining, most literate, well-written threads in the history of AK.

Particular kudos to JohnVF and Wiggy. I love to read you guys. Seriously.:yes:

 

[bshorey]

I've been around here long enough to know that a thread title such as this one is most likely to be sarcasm, and yes I get it. What I said before was that a test in which no one could tell the differences between cheap gear and expensive gear sets off an alarm in my head that the test must have been flawed, because that kind of conclusion flies in the face of decades of listening by people with trained ears, and even those of us without trained ears.

Well, that is the point that the OOP (original, original poster) was trying to make, wasn't it? That if nobody could discern a difference in the equipment being measured, then that was proof that the testing must be flawed?

I still think the OOP was being sarcastic too.

Personally, I don't think we should discount the fact that there may very well be some very good mid-fi stuff that can compete with some of the high end stuff.

I'd also like to point out that if we're testing preamps, and we're using some high end CD player as a source, then we're only testing a very small set of the circuitry in the respective preamps, and arguably the simplest set of circuitry. In that case, why wouldn't they sound similar, or be virtually indistinguishable from each other? Perhaps the $10,000 preamp has a phono stage that simply blows away that of the $200 preamp.

Blind testing, or double blind testing,the CD section of various preamps really only gives you a taste of the most simple, basic circuitry in the preamps. I don't really consider it a useful indicator of which preamp is better, as for a preamp, value includes things other than simply signal processing, it's also a function of switching, versatility, controls, etc.

bs

 

[MikeCh]

Fwiw, I think this entire thread is one of the most interesting, most entertaining, most literate, well-written threads in the history of AK.

Particular kudos to JohnVF and Wiggy. I love to read you guys. Seriously.:yes:

+1. The way you guys can articulate your thoughts and feelings in written form is truly impressive.


Merry Christmas gentlemen,
Mike