• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

A Deadly Flaw in a Test Protocol

Patricio Elicer

Obsessed with Reality
Patricio Elicer
Joined
Aug 6, 2001
Messages
4,633
Location
Santiago, Chile
Someone started a thread a time ago about whether the million can be won by chance. Seems the answer is yes, especially when there are flaws in the protocol.

It appears that Kramer and The Amazing himself made a gross statistical mistake in the protocol design to test a challenger who claims to be able to improve/change the sound quality of a music CD by applying a special "treatment" to it. That's bad news, and a serious concern, IMO.

The good news is that the error was detected by forum members, so it's pretty sure that the million is safe. BTW, it proved that plublicizing challenge's details in the forums is well worth the effort.

For details, go to http://www.internationalskeptics.com/forums/showthread.php?s=&threadid=53829, an explainatory post by Randi is in there.

For those who don't want to go through countless posts, here's a summary of the proposed protocol:

1) Two identical sealed CDs would be purchased. One of them is to be treated by one person from the claimant party, by whatever method he chooses to

2) Following strict double blind procedures, a second subject from the claimant team would attempt to identify, by listening, which of the two CDs is the one treated.

3) Step 2 is to be repeated a number of times, say 10, with the same two CDs (the claimant says he can have success ten times in a row).

This protocol looks fair at first glance, since the double blind procedure would neutralize any cheating attempt by the claimant "marking" one CD by means of a scratch or a distinctive smell, for example. This way, the odds of getting 10 hits in a row are 1 in 1024, pretty small.

But there's a trick to the case (here the credits go to a number of posters, Timothy being the first, I think). A flaw in the protocol is introduced by using the same pair of CDs in all the trials. The "treatment" can in effect introduce a distinctive "mark" (known to the tester) in one CD. So, all he has to do is pick the marked one (or the other one for that matter) in all ten trials, and the odds of being right will rise to 1 in 2. So a good safeguard is to use different pairs of CDs for each trial, so that we'll end up with a set of ten independant events.

Pretty clever folks!
 
Re: murder she wrote

webfusion said:
And this deserved a thread of it's own, why?
Click to expand...
I wanted to point out and bring up for discussion the fallibility of the JREF on protocol design. The subject deserves a thread of its own, IMO. Also, the importance of discussing tests details in the forums.
 
whatever

"Also, the importance of discussing tests details in the forums."

That's pretty much a given.

As for Randi's tests and JREF protocol design, so far we haven't seen any evidence there are flaws, since no tests were passed in the first instance.

In the meantime, my feeling is that this thread is going nowhere, based on a flawed OP that deserved to be posted in the Audio Critic file.

See ya over there, OK?
 
Re: whatever

webfusion said:
As for Randi's tests and JREF protocol design, so far we haven't seen any evidence there are flaws, since no tests were passed in the first instance.
Click to expand...
Eh?, are you suggesting that a protocol can be proved to be flawed only after a test is passed?.

The point is that Randi outlined a protocol that he and Kramer were pretty sure was OK. But it was flawed, the claimant could've easily won the prize by chance alone.


EDIT: needless to say, you are free to post in whatever thread you want.
 
no tests passed means the protocols are fine

The deadly flawed thread.

What are you going to say next to keep this topic alive?
 
Re: no tests passed means the protocols are fine

webfusion said:
The deadly flawed thread.

What are you going to say next to keep this topic alive?
Click to expand...
Look dude, I'm not interested in keeping this thread alive. I started it because I honestly thought it was a sensitive subject. If no one shares my concerns and interest about it, and the topic dies promptly, so be it. I can live with that.

Interestingly enough, it is you who is adding fuel to keep it alive.

If no one has been able to pass a test by cheating, then the protocol is fine. Yes, probably. And I mean probably.

But you're missing the point. The fact that Randi has not made a mistake so far is no guarantee that he won't in the future. In the draft protocol for the "audio treatment" test, he made a mistake, and a serious one. And the worst part is that he and Kramer were pretty confident that the protocol was OK.

We won't wait till someone wins the prize by cheating to admit that the JREF made a mistake on the protocol, will we?
 
not interested

Look dude, I'm not interested in keeping this thread alive.
Click to expand...

Yes, you are. You keep asking more questions. You know, those sentences ending with these funny squiggly marks and dots under them ????

Here's the real point you don't want to face:
This same topic has been discussed at length previously in other threads (Winning by Chance)

and also

(
Screening Trickery)


and your opening a new thread on basically the same subject seemed to be unneccessary, dude.

But you go ahead, it's an open forum, so try and make whatever case you want to.

Have a nice day.
 
Patricio,

While I agree that there was a flaw in the original protocol, I doubt if it would have been "deadly". I'm sure that the JREF, before committing to the final protocol and signing on it, would have passed it by a statistician for approval. Terming the flaw as "deadly' is being a touch too alarmist.
 
I don't believe anybody is going to die if Mr. Anda wins his challenge.
 
Vikram said:
I'm sure that the JREF, before committing to the final protocol and signing on it, would have passed it by a statistician for approval. Terming the flaw as "deadly' is being a touch too alarmist.
Click to expand...
Alright, that's a good point, maybe I overreacted. However, I'm not sure that the protocol for the prelim is that tight. The Yellow Bamboo's is one example of it.

No one has passed the prelim so far, so just someone passing it would suffice for the woos to cry out loud victory over James Randi.
 
Re: not interested

webfusion said:
This same topic has been discussed at length previously in other threads
Click to expand...

So what?

Personally, I'm glad this thread, with it's summation, was started. I was trying to follow the other thread but got lost in the details. As for looking up topics previously discussed, who has the time? It's not as if there's an index, or anything.

And, when you get right down to it, why on earth should you care? Do you have any authority or liability in the matter, at all?
 
Vikram said:
Patricio,

While I agree that there was a flaw in the original protocol, I doubt if it would have been "deadly". I'm sure that the JREF, before committing to the final protocol and signing on it, would have passed it by a statistician for approval. Terming the flaw as "deadly' is being a touch too alarmist.
Click to expand...

Yeah, but having a thread entitled "protocol for prelim not that tight" would have been less exciting.

Your point about a final confirmation from a statistician is a good one, but the danger was that Randi was making so many positive noises about the 2 discs and freedom to treat, that if it had gone much futher down the line he would have been criticised for "going back on a deal". You know the way some people operate. They'll try to twist every tiny thing to make it seem like the challenge is rigged, even if it was an honest mistake.
 
Patricio Elicer said:
No one has passed the prelim so far, so just someone passing it would suffice for the woos to cry out loud victory over James Randi.
Click to expand...

Again, I have to ask "So What?"

It may be appropriate at this point to ask what the exact purpose of the Challenge might be. Is it bait? Is it a "dare"? Is it an encouragement?

If the $1 million were ever to be awarded, would JREF see it as a defeat? Why? Just supposing somebody did win the Challenge, and did it honestly, wouldn't this broaden scientific horizons? Is Randi's ego really that big, that he would take it as a personal or professional defeat? If so, then I don't see how maintaing the Challenge could be considered honorable.
 
TheBoyPaj said:
Yeah, but having a thread entitled "protocol for prelim not that tight" would have been less exciting.

Your point about a final confirmation from a statistician is a good one, but the danger was that Randi was making so many positive noises about the 2 discs and freedom to treat, that if it had gone much futher down the line he would have been criticised for "going back on a deal". You know the way some people operate. They'll try to twist every tiny thing to make it seem like the challenge is rigged, even if it was an honest mistake.
Click to expand...
You're right about that. Maybe the JREF (if it wishes to) should make it a standard policy to pass all protocol proposals by a statistician before offering them to the claimant. After all, there's a lot of money at stake.
 
Beady said:
Again, I have to ask "So What?"

It may be appropriate at this point to ask what the exact purpose of the Challenge might be. Is it bait? Is it a "dare"? Is it an encouragement?

If the $1 million were ever to be awarded, would JREF see it as a defeat? Why? Just supposing somebody did win the Challenge, and did it honestly, wouldn't this broaden scientific horizons? Is Randi's ego really that big, that he would take it as a personal or professional defeat? If so, then I don't see how maintaing the Challenge could be considered honorable.
Click to expand...
Beady,

It would be a pity if someone were to win the challenge simply because of faulty experimental design. That certainly wouldn't in any way "broaden scientific horizons" and all that would happen would be that the woos would get the opportunity to claim that the paranormal exists.

I personally am a great fan of Randi and the work that he does and I wouldn't want to see the Million Dollar Challenge undermined by a tiny statistical error.

I don't know why you're trying to bring Randi's ego into the issue. He has repeatedly admitted whenever he's made a mistake. In fact, when the flaw in the protocol was pointed out, Kramer acknowledged it. He could have very well said, "Yes. Yes. We know that already. We realized this by ourselves. You didn't need to point it out. We'd already corrected the protocol." and could have tried to save face. But he didn't do that.

By bringing Randi's ego and the honor of the challenge into the picture, are you just trying to fan some non-existent flame?
 
You must log in or register to reply here.

Back
Top Bottom