Double Blind Testing not appropriate for Audio?

[Pauln]

Now this is interesting! I encourage every DBT advocate to read this carefully and thoughtfully reflect on its arguments. Those who have not accepted DBT may enjoy reading this, too.

Double Blind Testing

 

[eman]

I don't know

The gear is the instrument(s) to me and the human can be flawed for sure whether it is a pharmaceutical trials or a sound system test ultimately we listen to the subjects (listener, trial member) feedback. It can be right or wrong depending on the human in both cases (pharma trial, DBT)

Sure a Pharma Trial is going to be more repeatable but stereo system DBT's are not meant to be repeatable in the same way. We don't A/B the same gear right? We A/B different gear.

Now all that said I lean much more toward subjective and don't believe DBT's are a must. Also sometimes DBT's introduce sound degradation via a switchbox or extra interconnects etc.

So I have added no concrete evidence either way which is oddly kind of my point.


E
T

 

[JP]

Interesting.

I find that my mood and emotions don't change in the short period of time between A and B, so I don't find that to be a valid argument. I'm used to being able to take my time and flip between DUT/samples at will so that I may focus on specific parts of what I'm listening to.

I don't find the photo analogy to hold water. While you can have both photos in front of you, you can only focus your vision on one small area at a time. This is what I do with ABX tests - focus on something in the audio, and flip back and forth as often and as fast as I want.

Due to lots of variables often not controlled, my ABX or DBT results may not be valid to someone else. This is fine, as I really only care if I can hear a difference.

 

[datafone]

I gave up the idea of AB or ABX testing being more valid than just listening sighted ages ago.
I believe it is appropriate in Pharmaceutical testing, but is flawed in audio testing as it in itself is flawed. Why? due to the fact that our brains operate in a different state when under test conditions.

Be it right or wrong it is just my view.

But I still do a bit of back and forth sighted testing if the difference is subtle...sometimes.

 

[ConradH]

IMO, the article is very well thought out. DBT is flawed, but not using DBT is even more flawed. IMO again, audio is a cosmic joke, with everything stacked against making accurate comparisons or ever getting agreement on how anything sounds. That's why I take more of a "don't worry, be happy" approach. Even the car radio sounds good if the music is good.

 

[cwall99]

It's interesting that the author compares audio ABX testing to pharmaceutical ABX (double-blind) studies. In pharmaceutical studies, participants are not placed into both the experimental and the control groups.

Furthermore, he also makes the claim that unlike pharmaceutical studies, in which you can use any number of instruments to assess the impact of medications on study participants, and then claims that no analog for these instruments exists in the audio world. I suspect anyone who's ever used an SPL meter or any other instrument that can determine a speaker's frequency response may have a word or two to say about that.

So, if you really wanted to design an audio-based ABX test, you wouldn't have study participants listening to both signals. Instead, you'd have lots of participants listening to one signal or another, and then you'd compare the results of those two groups.

 

[WobblySam]

IMO, the article is very well thought out. DBT is flawed, but not using DBT is even more flawed. IMO again, audio is a cosmic joke, with everything stacked against making accurate comparisons or ever getting agreement on how anything sounds. That's why I take more of a "don't worry, be happy" approach. Even the car radio sounds good if the music is good.

Yes!

 

[olson_jr]

That's why I take more of a "don't worry, be happy" approach. Even the car radio sounds good if the music is good.

That is how I have gotten. If I don't notice any obvious distortions or abnormalities I am usually good to go.

Both of my stereos sound pretty good to me and that is all it takes to get me into the music.

 

[gslikker]

from the article
"I think the people promoting DBT show a lot of misunderstanding of neuroscience, memory, and perception."

The whole AIM of the article is to "prove" "they" are wrong.
The subject is NOT ABX testing being valid or not but the people thinking ABX testing is good.

Myself, I question the OP's intentions.

 

[racca]

Dr. Toole would disagree with him, to say the least:

https://www.youtube.com/watch?v=zrpUDuUtxPM

Yes, it's a long video, but it's well worth the time to watch.

 

[Pauln]

racca,

That is an excellent video, just enjoyed the whole thing (it is raining outside).

 

[chicks]

racca,

That is an excellent video, just enjoyed the whole thing (it is raining outside).

+1. :thmbsp: Great stuff, thanks for posting!

 

[juncers]

Dr. Toole would disagree with him, to say the least:

https://www.youtube.com/watch?v=zrpUDuUtxPM

Yes, it's a long video, but it's well worth the time to watch.

Brilliant!! Thanks for posting.

Ray

 

[E-Stat]

Dr. Toole would disagree with him, to say the least:

It was a good video. I've also followed the blog of his protege, Sean Olive for a long time. Click here to view.

Note that Toole and Olive are testing for preference among a narrow range of loudspeakers using mono signals. Does anyone disagree that hearing differences between speakers is pretty evident to most everyone? Olive, for example, has found that preferences for speakers is very consistent within the selection of speakers tested using their test criteria. Toole mentioned two criteria that are likewise important to me:

1. Bass linearity below 200 hz. I have spent considerable time experimenting with speaker, bass trap and listening couch distances to get the most linear response in that region. Which is why I'm convinced my speakers would do poorly in the "shuffler" environment given the proximity to back wall and lack of diffusion.

2. Consistent directivity. This has long been an important criteria for me since I was a teenager and why I prefer full range electrostats.

And yet, when you observe what kinds of speakers listeners actually choose for their music system, the results are more diverse because there are so many other factors involved when the goal is listening to music in stereo at home: size, appearance, efficiency, coherency, soundstaging, bandwidth, etc. Not everyone shares the same set of tradeoffs. Planar folks like me prefer one way floor to ceiling dipolar line sources. "Hornies" prefer the superior dynamics and ability to use flea powered amps.

 

[E-Stat]

It's interesting that the author compares audio ABX testing to pharmaceutical ABX (double-blind) studies. In pharmaceutical studies, participants are not placed into both the experimental and the control groups.

So, if you really wanted to design an audio-based ABX test, you wouldn't have study participants listening to both signals. Instead, you'd have lots of participants listening to one signal or another, and then you'd compare the results of those two groups.

Exactly. I've made similar observations before about the differences. You touch on a few:

1. In medicine the training, experience and test-taking abilities of the subjects do NOT affect the results of the test.

In audio they all do. So it is as much of a test of the listeners than differences between the components alone.

2. In medicine, the participants don't compare anything. There are administered either the the control or a placebo.

In audio, one must make a forced choice between two different DUT.

3. In medicine the tests have been scientifically validated for this use.

In blind audio component comparisons, there has been NO validation. The sensitivity of the tests for different sonic parameters has not been determined. That's because the sensitivity changes with each subject (see #1).

DBTs can detect gross differences (as one finds with speaker response and directivity), but are not sensitive enough to identify the kinds of subtle differences found in audio gear.

 

[TubeHiFiNut]

Personally, I never found ABX or DB testing to be a reliable means of determining differences in audio components.

My preference is for long term listening and comparisons in my system and room.

I respect the views of those who feel that it works for them.

Each to his/her own.

YMMV.....

 

[Tom Bombadil]

Due to lots of variables often not controlled, my ABX or DBT results may not be valid to someone else. This is fine, as I really only care if I can hear a difference.

I agree completely.

I also have no problems in relaxing and just listening to the music in an ABX test. And sometimes I run my tests over days of listening. As in having someone else connect my cables and then having a switch by my listening position. Sometimes listening for an hour or longer between switching. Just relaxing and enjoying the music. Throwing the switch to see if there is any noticeable difference, even at the emotional level.

I've passed ABX and/or DBT. Where I did clue in on differences, over time. Where I could differentiate nearly 100% of the time.

And then there have been many, many times where I couldn't detect even the slightest difference. Even after days of comparisons.

I find them to be an invaluable tool towards spending my money wisely to build the best sounding system to my ears.

These arguments about it inducing too much stress are puzzling to me. Why is there stress? Vs doing sighted tests which nearly everyone finds to be fun and relaxing. When I perform a sighted test, I'm trying to find which device/component sounds better. That is exactly the same thing I'm doing in an ABX/DBT. And in an ABX/DBT, I'm letting my ears do all of the evaluation, instead of my eyes and my mental perceptions.

I would not make a significant audio purchase without running some type of blind comparison.

 

[chicks]

It was a good video. I've also followed the blog of his protege, Sean Olive for a long time. Click here to view.

Note that Toole and Olive are testing for preference among a narrow range of loudspeakers using mono signals. Does anyone disagree that hearing differences between speakers is pretty evident to most everyone? Olive, for example, has found that preferences for speakers is very consistent within the selection of speakers tested using their test criteria. Toole mentioned two criteria that are likewise important to me:

1. Bass linearity below 200 hz. I have spent considerable time experimenting with speaker, bass trap and listening couch distances to get the most linear response in that region. Which is why I'm convinced my speakers would do poorly in the "shuffler" environment given the proximity to back wall and lack of diffusion.

2. Consistent directivity. This has long been an important criteria for me since I was a teenager and why I prefer full range electrostats.

And yet, when you observe what kinds of speakers listeners actually choose for their music system, the results are more diverse because there are so many other factors involved when the goal is listening to music in stereo at home: size, appearance, efficiency, coherency, soundstaging, bandwidth, etc. Not everyone shares the same set of tradeoffs. Planar folks like me prefer one way floor to ceiling dipolar line sources. "Hornies" prefer the superior dynamics and ability to use flea powered amps.

Interesting comments on Toole's lecture here:

https://therationalaudiophile.wordpress.com/2015/06/12/that-floyd-toole-lecture/

 

[racca]

Interesting comments on Toole's lecture here:

https://therationalaudiophile.wordpress.com/2015/06/12/that-floyd-toole-lecture/

He's basing his complaints on the presentation itself, which attempts to roll a lifetime of research into a fairly short presentation. If he wants to discount Toole's methods (which I believe are sound) then he should start by reading the published research papers, rather than relying on a generalized presentation intended for non-scientists.

 

[E-Stat]

Interesting comments on Toole's lecture here:

He's not the only one who thinks there is more to evaluating speakers than using mono signals. Siggy Linkowitz (among others) would rather judge more than just tonality:

Linkwitz test procedure for Accurate Stereo Performance

Naturally, the Harman speaker shuffler would not work optimally for Siggy's preferred use of dipoles.