TAM13 Live Challenge

[gerdbonk]

Oddly, no official claimant. A test of high-end "audio" ethernet cable. Submitted by a writer at Ars Technica in response to a raving review for bazillion dollar ethernet cable for digital audio applications.

20 tests, 15 hits required to succeed. 6 misses and you're out.

One at a time, random volunteers each listen to music from a digital system that uses:
1) high end ethernet cable (including arrows indicating which way the data should flow);
2) cheap ethernet cable purchased from Amazon;
3) a randomly selected replay of either 1) or 2).

Testees had to match the 3rd play to one of the first two. If the high end cable made such an obvious difference, this should be an easy match to make. The calim (in the review) was that even a non-audiophile could discern it.

Result:

1 hit. 6 misses. Game over.

Open questions after the test criticized some aspects of the protocol, though there was no disagreement with the result. The complaints were mainly that possible holes in the protocol gave the cable manufacturer an out to dismiss the test altogether.

Issues:

Volunteers were not sequestered, but saw and heard everything until they entered the listening booth -- i.e. they were aware of results as they happened;

Most volunteers claimed there was no difference; complaints were that this should not have been an option; Jamy Ian Swiss explained "no difference" was the equivalent of an incorrect match -- the claim was that the cable made a "substantial, obvious" difference.

Some complaints about the audio/computer components used; the Ars Techinca writer explained the decision for the choices made.

 

[wardenclyffe]

I guess it's better than nothing. I wonder how close they came to having a real applicant.

Ward

 

[Paul2]

Result:

1 hit. 6 misses. Game over.

Was there only one volunteer who did the test? Did one volunteer do all 7 guesses?

I recall fondly the challenge I saw at my last TAM (power bracelet guy). Boring but great anyway.

 

[SkepticScott]

They had 21 volunteers, maybe more. Each did one guess. 20 for the test plus one to do the "open" trial, showing how the test will work.

 

[Sherman Bay]

I am confused by this thread and the other one on the same subject. Is the claim one that says that identical data packets encapsulating audio information will sound different once converted from digital to analog audio?

If the digital data is not identical after being sent through different cables, why are we comparing the converted analog audio?

It seems to me that the first thing should be to analyze the data digitally. If two streams are not identical digitally, bit by bit, something is wrong with the transmission medium and we need go no further.

 

[SkepticScott]

AFAIK, the claim was that the expensive cable "shielded" the system from EMI that would enter through the RJ45 jack, travel through the system, affect the DAC, and thus affect the analog output.

 

[elipse]

Oddly, no official claimant. A test of high-end "audio" ethernet cable. Submitted by a writer at Ars Technica in response to a raving review for bazillion dollar ethernet cable for digital audio applications.

20 tests, 15 hits required to succeed. 6 misses and you're out.

One at a time, random volunteers each listen to music from a digital system that uses:
1) high end ethernet cable (including arrows indicating which way the data should flow);
2) cheap ethernet cable purchased from Amazon;
3) a randomly selected replay of either 1) or 2).

Testees had to match the 3rd play to one of the first two. If the high end cable made such an obvious difference, this should be an easy match to make. The calim (in the review) was that even a non-audiophile could discern it.

Result:

1 hit. 6 misses. Game over.

Open questions after the test criticized some aspects of the protocol, though there was no disagreement with the result. The complaints were mainly that possible holes in the protocol gave the cable manufacturer an out to dismiss the test altogether.

Issues:

Volunteers were not sequestered, but saw and heard everything until they entered the listening booth -- i.e. they were aware of results as they happened;

Most volunteers claimed there was no difference; complaints were that this should not have been an option; Jamy Ian Swiss explained "no difference" was the equivalent of an incorrect match -- the claim was that the cable made a "substantial, obvious" difference.

Some complaints about the audio/computer components used; the Ars Techinca writer explained the decision for the choices made.

The main problem with this protocol boils down to:

The volunteers were given the option of responding in a way that they knew would guarantee failure ("no discernable difference") at a skeptics convention where the volunteers were pulled from people who didn't believe the test would work. That's an invalid test by any definition. I think I actually saw the change in Richard Saunders' face when he realized what they'd inadvertently done.

 

[Squeegee Beckenheim]

The main problem with this protocol boils down to:

The volunteers were given the option of responding in a way that they knew would guarantee failure ("no discernable difference") at a skeptics convention where the volunteers were pulled from people who didn't believe the test would work. That's an invalid test by any definition. I think I actually saw the change in Richard Saunders' face when he realized what they'd inadvertently done.

Yes, I agree with this. Even with an analogue signal there's a very low point beyond which cable quality makes little to no difference (or, to put it another way, even on tour grade systems you can pretty much get away with using power flex), so with digital it's patently nonsense, but this test wasn't an effective way to test these cables as it allowed bias in the results.

 

[Gord_in_Toronto]

The main problem with this protocol boils down to:

The volunteers were given the option of responding in a way that they knew would guarantee failure ("no discernable difference") at a skeptics convention where the volunteers were pulled from people who didn't believe the test would work. That's an invalid test by any definition. I think I actually saw the change in Richard Saunders' face when he realized what they'd inadvertently done.

The claim being tested was that even an idiot could hear the absolutely amazing improvement in sound quality with the use of the magic cables.

If I was to claim that my Magic Socks would protect you from the pain when hit on your toes with a large hammer, would it make any difference what you believed or did not believe?

 

[RecoveringYuppy]

The main problem with this protocol boils down to:

The volunteers were given the option of responding in a way that they knew would guarantee failure ("no discernable difference") at a skeptics convention where the volunteers were pulled from people who didn't believe the test would work. That's an invalid test by any definition. I think I actually saw the change in Richard Saunders' face when he realized what they'd inadvertently done.

Yes, there should also be cases where the two renditions were different but due to reasons other than the cables used. There have to be options where "this one is better" is the correct answer.

 

[CynicalSkeptic]

Any details on the sound system used?

From the highly rated review of these cables on Amazon:

if you can't hear the difference than your system either is incapable of revealing the difference or your hearing has become impaired.

The review, of course, contains all the typical audiophile nonsense terms:

The Cinnamon sounded noticably better than the Forest, more high frequency detail and even wider soundstage. The Vodka sounded very different out of the box, in fact the soundstage did not sound as wide as the Cinnamon, but therefore instruments just sounded better, more fluid and natural, less digital artifacts, less grainy. After a 2 weeks burn in phase the Vodka started to expand in soundstage and became my prefered choice over the Cinnamon.

 

[jaydeehess]

Any details on the sound system used?

From the highly rated review of these cables on Amazon:


The review, of course, contains all the typical audiophile nonsense terms:

Wow, a lot of words to say practically nothing , like describing wine tasting.

"Burn in" of a passive device, that's rich.

I suppose it's too pedantic to expect audiophiles to know the difference between "then", and " than".

 

[Sherman Bay]

AFAIK, the claim was that the expensive cable "shielded" the system from EMI that would enter through the RJ45 jack, travel through the system, affect the DAC, and thus affect the analog output.

Then wouldn't the EMI have to "jump over" or bypass the digital circuit and join the analog signal near the output? If EMI was injected into the digital processing component, either the signal would not be affected, or it would introduce random bit errors. If there were bit errors in the signal, it could be detected by a digital analyzer and either the packets would be discarded or error-corrected.

 

[Squeegee Beckenheim]

If I was to claim that my Magic Socks would protect you from the pain when hit on your toes with a large hammer, would it make any difference what you believed or did not believe?

Potentially yes, actually. If the test was for you to hit yourself with the hammer then report back to people whether or not you hurt yourself then you could simply lie, if you had a mind to. If the result you wanted was for the socks to not be magical you could claim that you felt pain, even if you didn't. If the result you wanted was for the socks to be magical you could claim you didn't feel pain, even if you did.

It's the same problem here. If there are audience members who want the cables to fail the test, then allowing them to simply say "that's failed the test for me" without it being possible to objectively measure whether or not that's true, then it's not a useful test.

Imagine it had been set up the other way round. The test took place at an audiophile convention, with volunteers from the audience. If the volunteers said that they heard a difference, then that would be counted as a hit. Do you think that the JREF would have accepted that as a protocol? Why not?

I know this isn't an official MDC challenge, but considering where it was held and under what circumstances, it should follow the same rules as the MDC, and the rules of the MDC clearly state that the results should be self-evident. This result is not self-evident as the alternate explanation of "the volunteers lied" cannot be ruled out.

 

[gerdbonk]

Then wouldn't the EMI have to "jump over" or bypass the digital circuit and join the analog signal near the output?

I believe the Ars Technica writer said, yes, the manufacturer claims an analog EM signal travels externally along the ethernet cable to the converter, where it can magically shape the output.

"Not with MY converter", chimes every competent DAC designer in the world.

Note that the high-end cable had directional arrows to indicate which way the signal should flow from source to pre-amp.

For best results have the arrow pointing in the direction of the flow of music.

--AudioQuest web site

Turn it the wrong way and, what, the signal has to roll uphill? Against the grain?

 

[Sherman Bay]

I know what an Ethernet cable is, and there is no "wrong" way, as signals travel in both directions, but just what is an "audio Ethernet" cable? Is that like a "video Ethernet" cable? If I try to send audio over my Ethernet cable from my computer, and it's the wrong kind, what will happen? The bits get stuck?

 

[zooterkin]

The report is up on the arstechnica site.

 

[elipse]

The claim being tested was that even an idiot could hear the absolutely amazing improvement in sound quality with the use of the magic cables.

If I was to claim that my Magic Socks would protect you from the pain when hit on your toes with a large hammer, would it make any difference what you believed or did not believe?

Of course not. The problem isn't whether the cables would have done better under some other test. The problem is the protocol as designed allowed the participants to rig the test for failure on purpose if they so chose (or subconsciously.) That should not have been an option.

Requiring participants to pick one of the options would have removed this problem. So would having participants who were totally blind to the purpose of the test, which would have required getting non-tam attendees as participants.

Simpler just to require an answer a or b, imo.

Eta: already addressed by someone else, much better than I said it.