• The move to the new server is done. There are some software and database maintenance updates in process. This has us passing the hat around to help out. We appreciate any donations. Seriously, even a dollar helps. The payment page may be found here - https://www.audiokarma.org/support.html

Double-blind testing - why are we not using it?

Spasticteapot

Spasticteapot

Nuclear Liberal
I'm personally a big fan of unusual audio designs. Dammar and bits of foam can go a long way twoards a better sound system.

However, I'd like to see more objective and less subjective testing on this forum.

Here's a few suggestions:

1. For cables, provide inexpensive (but decent) cables (Canare Star Quad?) in addition to those reviewed when testing. Use identical high-end connectors. Cover both sets in a LOT of heat-shrink, so they appear identical. Put a stripe of red tape one one pair, and a stripe of white tape on the other.

2. For the Brilliant Pebbles, opaque bottles should be used - one empty, one full of assorted-sized small polished rocks.

3. For the "Intelligent Chip", two will be issued : One normal one, and one that is simply an empty box with a bit of PCB inside, ideally etched so as to appear identical. One will have a red dot on the top, the other a white dot. (I live in Wisconsin. Sue me on the colors.)



The results would then be tablulated by a third party, and the results would be published. Just like any other study.
 
Register to hide this ad
Not because there is anything inherently wrong with the process, but because it 100% of the time leads to stupid arguments.
 
The hearing of a sound, "Yes, I hear the tone", or "No, I don't hear anything", is objective. The act of listening to music is one of life's most SUBJECTIVE activities, right up there with the viewing of art, and of dining.
 
I'm Trying to be careful

luvvinvinyl said:
The hearing of a sound, "Yes, I hear the tone", or "No, I don't hear anything", is objective. The act of listening to music is one of life's most SUBJECTIVE activities, right up there with the viewing of art, and of dining.
Click to expand...
Treading lightly here...
A double blind test would be interesting to me not to prove which audible difference is more pleasing, which would follow your concern regarding objectivity/subjectivity, but to determine if there was any statisitically significant audible difference. For various reasons, there have been few reliable double blind tests done on interconnects or speaker wires.

When I was in the market for speakers, I narrowed the search down to two finalists. I did my best to come up with a blind test. I set up the speakers side by side. My amp has an A/B switch and I hooked up the two pair. There was an efficiency difference. I measured the difference and placed a marker for the "A" and a marker for the "B" speakers.

I had my son and wife help me with the test. He would switch from "A" to "B", but not tell me which was which. He did not know how the speakers were connected. After setting the volume and returning, I'd click the remote to play a selection and we'd score the performance. We took about an hour listening to all types of music. My son and wife scored the same pair better 80% of the time and I scored the speakers about 50/50 (they most likely have better ears). The losing speakers cost three times as much as the winners.

Doug
 
I too, am trying to be careful...

luvvinvinyl said:
The hearing of a sound, "Yes, I hear the tone", or "No, I don't hear anything", is objective. The act of listening to music is one of life's most SUBJECTIVE activities, right up there with the viewing of art, and of dining.
Click to expand...

Hopefully, this will be taken in the non-confrontational way I offer it...your analogy got me to thinking about why I have my particular view of the world. :scratch2:

I agree that listening, viewing, and tasting are very subjective experiences. :yes: It's what makes me a fan of Rush's music, which does nothing for my better half. Why she likes turnip greens, and I don't. Why she likes Thomas Kincade pieces, while I prefer Rembrandt.

However, the engineer in me holds that the quality of a reproduction (after all, that's all our hobby is - the reproduction of works of art) of any of the above is highly objective and measurable. That's why I've always felt that we can measure and compare things (colors, edge transitions, levels of detail, constitutent ingredients, frequency response, waveforms, etc.) in a scientific manner and have a pretty good, if not absolute, measure of the accuracy of the reproduction.

Now, I will admit, sometimes accuracy is not what we prefer. That's why I like to use those tone and loudness controls, to tune the reproduction of the music I like to what I prefer subjectively. :yes: :D

That said, to each his own, I guess. :music: I just felt the urge to share the line of thought that your response prompted. :thmbsp:
 
You run into another problem, especially with speakers, doing a test like this. The better sounding speaker might not be better. I know that sounds weird, but let me explain. Ideally, you don't want a speaker to add its own influence to the sound. You want it to be flat. When switching back and forth between speakers, one that has an accentuated bass and/or treble will almost always be percieved as sounding better. The manufacturers knew this and so, built boosted bass and treble into the design. JBL was one that was terrible for this in my audio sales days. (As were others.) So, you'ld A/B a JBL and a Yamaha, and everyone would pick the JBL. The problem is, if you lived with that speaker awhile, the boosted bass could get tiring.

I don't want to belabor the point, but I think most would agree that you don't want the speaker to have built-in EQ. Yet many speakers do. It made it awful hard to sell the "better" speaker at times.
 
Frankly (from someone who works in a field where double-blind testing is the sine qua non for hypothesis testing) single-blind tests should be adequate for judging the superiority of component (or cable) A vs. B.

Even more frankly, one's own ears (or, in fact, one's own psyche) are/is the ultimate judge of superiority! :-)
 
mhardy6647 said:
Frankly (from someone who works in a field where double-blind testing is the sine qua non for hypothesis testing) single-blind tests should be adequate for judging the superiority of component (or cable) A vs. B.

Even more frankly, one's own ears (or, in fact, one's own psyche) are/is the ultimate judge of superiority! :-)
Click to expand...

I agree completely with your logic. The problem with most of the protocols I've seen for DBT in audio has such a small sample size to make the testing irrelevant. I would love to see a study where hearing baselines were established with participants and a large enough sample size to make the study meaningful. Instead all I see are poorly designed tests that are used to prove a point.
 
nbourbaki said:
I agree completely with your logic. The problem with most of the protocols I've seen for DBT in audio has such a small sample size to make the testing irrelevant. I would love to see a study where hearing baselines were established with participants and a large enough sample size to make the study meaningful. Instead all I see are poorly designed tests that are used to prove a point.
Click to expand...

I think you did a much better job of making the point that I was trying to make: in a "quickie" A/B test, an inferior product can appear to be better.

You need to spend some time with a speaker to get a real feel for it. I realize that this isn't always practical to do.
 
nbourbaki said:
I agree completely with your logic. The problem with most of the protocols I've seen for DBT in audio has such a small sample size to make the testing irrelevant. I would love to see a study where hearing baselines were established with participants and a large enough sample size to make the study meaningful. Instead all I see are poorly designed tests that are used to prove a point.
Click to expand...

I guess that would be helpful for the consumer if the manufacturer's would spend the money on large samples to get meaningful results, but to be well designed would probably be cost prohibitive for such a niche product. Think about it; what would one measure? Bass? Detail? Imaging? Transparency? Highs? Smoothness? Warmth? The list goes on with intangible or near intangible qualities the are subjective in many cases.

These qualities are difficult enough to measure in speakers where the measures are probably the most pronounced, or even in the tube v. SS debates. Personally, I find them near impossible to distinguish in cables, and am very skeptical on rocks, wooden discs, and other magical tweaks. But hey, I'll give them a shot.

I'm pretty much in the camp that has cables way down on the list of tweaks, but can see where a good single blind test would be interesting. Consider the following: Design a set of connector blocks with female connections for up to say 6 sets of speakers and one set of outputs to the speakers. connect that up to something like my Adcom 6 speaker selector so you could switch between not 6 sets of speakers, but six sets of cables. Then one could switch between the cables on the same system with the cable in use known only to the tech who hooked them up. thsi would provide ease of switching with little or no info being distributed to the listeners, who could then make their choices based upon what differences they hear. Don't know what to do with the other tweaks though, although good scientific method would require all other variables to remain constant throughout the testing, including the number of people in the room, their positions and placement.

We can discuss this ad nauseum, but the bottom line is still personal taste; T-bone or NY Strip, Chocolate or vanilla, Coke or Pepsi and finally, Ginger or Mary Ann???
 
Normally, I never, and I mean never, post in these discussions. It's usually exactly the same as getting into discussions about whose kids are better looking. The answer is "mine" btw. Sorry!

Like lots of AK'ers, I have some small education in the engineering/testing area, from long, long ago. And, I love my toys, and I really do find these discussions interesting, so I'm going to forge ahead.

First, how will we judge component "A" vs component "B"? Obviously, it can't be that one sounds "better" than the other. One listener will say that A sounds better, and the other listener will say that he thinks B sounds better. At that point the discussion ends.

So, we must simply try to determine if the group of listeners/testers can tell if A sounds "different" than B. Without going into which one is better, can you even tell them apart? In other words, is it A, or is it B? A very simple example would be to test two interconnects, perhaps one expensive pair and one inexpensive pair. First, the listeners are allowed to know which one is being used, and they are allowed to listen to both, one at a time. Naturally, we would assume that some listeners will feel that not only can they tell a significant difference between the two, but that one will be, in their opinion, "better". Now that all the testers have had an opportunity to listen to both sets openly, we move to the "blind" part.

Without allowing the listeners to know which pair is being used, the listeners must decide if they are listening to A or to B. If they correctly identify which pair is being used approximately 50% of the time, which is the "chance" response, then no significant difference exists. If they correctly identify the one being used more than 50% of the time, then there is a difference. Right? No.

In fact, and my memory is a little weak here, the testers must correctly identify a difference more than approximately 65-70% of the time in order to move significantly past the chance response. Surprising, huh?

Well, here's the real kicker, and I'm choosing my words very, very carefully: (For those of you who have ever studied symbolic logic, this will be fun! If you've never studied symbolic logic, it will make your brain hurt.)

"It is impossible to tell that two things are the same."

This is NOT the same as being able to tell that two things are different.

Two things are either the same, or they are different.
If they are the same, well, then they are the same.
If they are different, you can either tell they are different, or you can't tell that they are different because the difference is so small as to be below your ability to perceive.

If you look at two colors, and they appear to be different, then they are different. If they appear to be the same, they are either the same, or the difference is smaller than the human eye can differentiate.

And, the final thought, and you know this is true...

If you set up a blind test between two different components, but, you sneaky bastard, you just keep using the same one over and over again, asking: "Is it A, or is it B?", you know damn well most of the testers will end up with plenty of A's and B's.

And THAT is the problem with using the human brain to "test" anything.

And now, to the mancave with a drink and a pile of music!!

Ciao,
Clay
 
Last edited:
Well, Clay, I actually followed that ...

I'm an experimental psychologist, and agree with some of the very good points made here, which I'd put this way: you need more observations than are typically found in a double-blind test of audio gear, bias of all kinds matters, and it is easy to miss a real difference.

But there's something that makes this even more complicated - in many situations, people can reliably express preferences when they can't reliably identify differences - that evaluative/emotional coding seems to come first. As one psychologist put it, it appears that "preferences need no inferences" - the theory that we first identify differences, then assign values to them, is just that, a theory.

OK, that should kill the thread :) I'll be joining you in that drink now, Clay :yes:
 
What a terrific response! Thanks!

Believe it or not, I've actually written that post half a dozen times. I've just never had the guts to push the "Post" button, due to the obvious fears...

Your input is spot-on to the points I was trying to make, which is that you really can't ever remove the "human" aspect to all of this. It's built in!
 
Fisherdude said:
What a terrific response! Thanks!

Believe it or not, I've actually written that post half a dozen times. I've just never had the guts to push the "Post" button, due to the obvious fears...

Your input is spot-on to the points I was trying to make, which is that you really can't ever remove the "human" aspect to all of this. It's built in!
Click to expand...

Well, duh. The "Bigger is Better" has a frequency response more spastic than a ferret on crack in an open environment. However, if positioned properly, it can be flattened out - and the 40hz bass is worth it.

However, truly blind testing is not difficult. All that's needed is to make the two cables visually indistinguishable - just give the tester both pairs, and ask them which one they like better. Repeat again, and again, and again, and again.

Results are rarely repeatable. However, a psychological lab technician (the definition of "subjective") has verified that giving someone the real deal and a "sugar pill" is a good way to determine wether something works.

And, for things like cables, it's pretty easy to make them indistinguishable. You have red cables, and you have white cables - you just don't know which is a 500$ pair, and which is a 5$ pair.
 
Artie said:
...The better sounding speaker might not be better.Ideally, you don't want a speaker to add its own influence to the sound. You want it to be flat. .. The problem is, if you lived with that speaker awhile, the boosted bass could get tiring.
Click to expand...
Coloration was a consideration. We're all musicians, and my son has two Master's in music. The losing speaker scored higher with rock music because of its coloration, but this same coloration worked against it on female vocals and piano. We thought about listener fatigue. The speakers I kept were more neutral.

I'm sorry if I led the conversation away from the point I was after. I did a poor job of making my point, mainly because I was trying to be careful. I think I would have scored the more expensive speakers higher if I'd known which was which, just because I thought they should sound better because they are more expensive. Double blind testing would take "wishful thinking" out of the equation for what I consider more "objective" componants, such as speaker wire and interconnects.

Doug
 
Ah! <revealing light from above> See, I thought the burning questions were about judging the relative accuracy of the reproduction, the quest for the perfect reproduction. Instead, I now realize the whole cables/tweaks/mods thing is about judging the resulting 'relative attractiveness' of the reproduction. DOH!

I'm really not trying to be funny, now I think I understand TOL better now - its sort of like a place where you can talk about whether you like your music with the tone controls centered, or loudness on and EQ switched in. With 16 ga. zip cord, or 10 ga. audiophile cables. Or, 'I think it sounds better because...it just does!'. Its about individuals' subjective response to whatever the tweak/mod does to their listening experience. Thanks guys!
 
Y'all realize, I hope, that this may be the single most civil thread on this topic ever posted on the internet.

:-)
 
You must log in or register to reply here.
Back
Top Bottom