[Split Thread] How are MDC protocols designed and carried out? (Split from FAQ thread)

[shuttlt]

Bryan,

1. Why shouldn't the JREF turn away claims that they regard as time wasting. I mean, really.... If you have a power that defies explanation, but is barely detectable above chance, how do you know you have the power?
2. Do you have any evidence that the JREF is turning away such claims?

Tom

 

[Garrette]

That's EXACTLY what I'm saying!

I think that James Randi was a bit unfair on that TV show, when he demanded that the "psychics" pass TREMENDOUSLY difficult tests. Of course, if they themselves had made such claims before the start of the show, then that's their own problem; one could hardly criticize Randi for demanding that they pass a test which they themselves claimed to be able to pass.

But what if their only claim is that they can "sense" the result of a coin flip only 55% of the time? Is James Randi (or an "officer" of the JREF) going to throw them out of the office, or are they going to give them a legitimate test for the one million dollar prize, as long and difficult as that would obviously be?

The fact that the JREF apparently doesn't make the answers to these questions abundantly clear beforehand (before somebody actually walks in the door and submits a proposal for a test) is what bothers me...

And I call baloney on this. You accuse Randi of cooking the numbers yet you present none. In fact, by your own admission you don't recall the specifics. You saw the tests. What was the actual accuracy required an dwhat is the accuracy you feel should have been required? And how did you calculate those numbers. Until you show your math you are just going on an emotional reaction that has no value.

 

[zooterkin]

That's EXACTLY what I'm saying!

I think that James Randi was a bit unfair on that TV show, when he demanded that the "psychics" pass TREMENDOUSLY difficult tests. Of course, if they themselves had made such claims before the start of the show, then that's their own problem; one could hardly criticize Randi for demanding that they pass a test which they themselves claimed to be able to pass.

Since that's what happened, then your other points are moot. The protocol is agreed with the claimants, based on what ability is being claimed. The JREF is not unilaterally imposing anything.

 

[Pup]

But what if their only claim is that they can "sense" the result of a coin flip only 55% of the time? Is James Randi (or an "officer" of the JREF) going to throw them out of the office, or are they going to give them a legitimate test for the one million dollar prize, as long and difficult as that would obviously be?

Check out the interminable discussions with Pavel Ziborov on this forum. He had a claim similar to that, which was going to be just above the odds for chance, but had trouble pinning down exactly what would demonstrate his ability.

 

[Pixel42]

The unofficial rule of thumb seems to be that applicants should be required to beat chance odds of around 1:1000 in the preliminary test. That means that for every 1000 applicants JREF tests one would be expected to pass even if there is no such thing as a paranormal ability, which seems a reasonable place to put the bar to me. But as has already been pointed out the claims tested vary considerably, making hard and fast rules impossible to set in advance.

Here's a typical protocol from the archive:

http://www.internationalskeptics.com/forums/showthread.php?t=118952

Perhaps Mr Shelton can tell us if he considers the protocol and success criteria used to test this applicant acceptable, and if not what exactly his problem with them is.

 

[TSR]

The fact that the JREF apparently doesn't make the answers to these questions abundantly clear beforehand (before somebody actually walks in the door and submits a proposal for a test) is what bothers me...

No, JREF *negotiates* the answers to these questions beforehand (before the applicant agrees that the protocol is fair and the test something they can do -- and only *then* is the test scheduled).

Why does this bother you?

 

[Bryan Shelton]

Bryan,

1. Why shouldn't the JREF turn away claims that they regard as time wasting. I mean, really.... If you have a power that defies explanation, but is barely detectable above chance, how do you know you have the power?

It seems perfectly reasonable to me that the JREF would first do a "preliminary test" (as "Pixel42" talked about in a later post this morning) which would require approximately a 1 in 1,000 chance (or less) of passing purely by chance. If the applicant passes it, a somewhat more rigorous test could then be part of the "formal test", which determines the fate of the million dollar prize. The first "preliminary test" would be the obvious way to avoid "wasting time" by applicants who don't have any real ability.

2. Do you have any evidence that the JREF is turning away such claims?

No, but I don't know that much about how these tests have been conducted. It troubles me that from talking to many of you folks here on this forum, it's been like pulling teeth to get some of you to speak about the tests clearly and concisely!

 

[wardenclyffe]

I think the JREF would accept a test where someone could predict a coin flip 55% of the time. This would take a tremendous number of trials. Assuming that the applicant never gets tired, this could probably done in a day. Ususally though, psychic powers tend to be exhausting after just a few trials, so this would require weeks or months. Keep in mind that the applicant is responisble for all the costs of the test. Eventually, the people administering such a long long test will want to be paid for their time. If the applicant never gets tired or if he or she is willing to pay for the time of the administrators, I would expect the JREF to accept a success rate of 55% in a coin flip test.

I don't speak for the JREF, though.

Ward

 

[Pope130]

No, but I don't know that much about how these tests have been conducted. It troubles me that from talking to many of you folks here on this forum, it's been like pulling teeth to get some of you to speak about the tests clearly and concisely!

To find out how the tests were conducted you should read the individual threads about each test, they are all different.

No single description of a test protocol exists. Even in cases where more than one applicant claims the same ability the details of their ability may vary,and the conditions under which they are able to demonstrate it are different. You'll find several different protocols mentioned for telepathy, remote viewing and dowsing, for example. The applicant defines his ability, and makes the initial suggestion of the test protocol. JREF then accepts, or suggests alterations to the protocol before an agreement is reached.

 

[Bryan Shelton]

Check out the interminable discussions with Pavel Ziborov on this forum. He had a claim similar to that, which was going to be just above the odds for chance, but had trouble pinning down exactly what would demonstrate his ability.

And what exactly does "pinning down" mean in this context? You mean PROVING his ability? Or simply explaining what his ability is supposed to be?

 

[Bryan Shelton]

The unofficial rule of thumb seems to be that applicants should be required to beat chance odds of around 1:1000 in the preliminary test. That means that for every 1000 applicants JREF tests one would be expected to pass even if there is no such thing as a paranormal ability, which seems a reasonable place to put the bar to me. But as has already been pointed out the claims tested vary considerably, making hard and fast rules impossible to set in advance.

I'm becoming more and more puzzled that so many of you talk so much about the difficulty of setting "hard and fast rules". I don't doubt that watching-over the applicants and setting rules to maintain honesty and prevent larceny can be difficult, but merely calculating the odds of success through the laws of probability should be a relatively simple matter.

 

[Bryan Shelton]

I think the JREF would accept a test where someone could predict a coin flip 55% of the time. This would take a tremendous number of trials. Assuming that the applicant never gets tired, this could probably done in a day. Ususally though, psychic powers tend to be exhausting after just a few trials, so this would require weeks or months. Keep in mind that the applicant is responisble for all the costs of the test. Eventually, the people administering such a long long test will want to be paid for their time. If the applicant never gets tired or if he or she is willing to pay for the time of the administrators, I would expect the JREF to accept a success rate of 55% in a coin flip test.

The only thing you haven't talked about here is HOW MUCH statistical significance you think should be required in such a successful test; but other than that one missing factor, I think you wrote a very lucid and sensible post!

 

[Pixel42]

I don't doubt that watching-over the applicants and setting rules to maintain honesty and prevent larceny can be difficult, but merely calculating the odds of success through the laws of probability should be a relatively simple matter.

And yet when someone gave you an actual example:

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test?

you replied:

I have no idea what they are, or what they should have been.

I have no idea how they should have been figured out.

In cases where a protocol is agreed where the odds of chance success can be calculated (the majority of cases), a success criteria of beating odds of around 1:1000 is typically used. But such a calculation is not always "a simple matter"; in some cases it is impossible, so another way of reaching agreement on the success criteria must be found, which is why no hard and fast rules can be set.

 

[Baloney]

Bryan,

And yet when someone gave you an actual example:...

And in that case, it was only 5 out of 10 correct to move beyond the preliminary steps. How, again, is that unfair, when the claimant agreed to all of the protocols?

 

[Dumb All Over]

The fact that the JREF apparently doesn't make the answers to these questions abundantly clear beforehand (before somebody actually walks in the door and submits a proposal for a test) is what bothers me...

What will you do if the JREF changes nothing and continues on without making answers to your questions abundantly clear beforehand to your satisfaction. Will this continue to bother you for the rest of your life? If so, how will you possibly deal with this? Is your level of botheredness something psychological therapy might be able to help with? How will you cope?

 

[EdG]

Bryan, if you have a specific paranormal ability you would like tested, just state your ability clearly and succinctly. Forums members would be pleased to give their judgement on what they would consider success to be and JREF itself would be very pleased to help you work out a test protocol.

Here's an example:

The PowerBall is a lottery game played in the United States in which 5 white balls are drawn without replacement from a set of 59 and 1 red ball is drawn from a set of 35. The odds of correctly guessing 3 white balls plus the red ball are 1 in 12,245.

Joe Lucky states that he can pick the 3+1 combination using his innate paranormal ability. JREF arranges a preliminary test and Joe successfully picks the winning combination while being observed.

However, even though the odds are 1 in 12,245, it is still possible for a person to choose 4 correct numbers through random chance. After all, hundreds of people do just that every drawing.

The question for Joe becomes how often and consistently can he pick his winning sets? Suppose he can pick a 3+1 winner for each drawing held over the next two weeks, i.e. 4 independent drawings. That becomes a much more solid demonstration of ability than a single instance and could be part of a protocol for winning the MDC.

The question for you becomes how can JREF possibly devise a "hard and fast" set of rules that cover every possible paranormal claim? Is dowsing the same as picking lottery numbers? Is being a card guesser the same as being a pee-starter? Is a premonition that an airliner will crash the same as picking up psychic vibes that the hot chick at the bar digs you?

If you have an ability, spit it out and let people help you figure out how to demonstrate it.

 

[TSR]

Bump for Bryan, who apparently missed this:

The fact that the JREF apparently doesn't make the answers to these questions abundantly clear beforehand (before somebody actually walks in the door and submits a proposal for a test) is what bothers me...

No, JREF *negotiates* the answers to these questions beforehand (before the applicant agrees that the protocol is fair and the test something they can do -- and only *then* is the test scheduled).

Why does this bother you?

 

[xterra]

Point 1:

Having re-read this thread from the beginning, I note that there has not been any discussion of this:

[...] When random choices are a clear and obvious element of such a test (as many of them truly are) [....]

Mr. Shelton, could you please explain what you mean by the highlighted statement?

Do you, for instance, mean that the applicant makes random choices?

That the JREF makes random choices?

That during a test, some random elements are introduced by the applicant or the JREF or just good old Ma Nature?

Point 2:

In any case, I am confused about why it is unfair to set a standard that in most cases is much lower that the applicants claim to be able to meet.

I say, "I can detect Substance Y, by whatever method, infallibly, 100% of the time. Never miss. Never. Not even when I have a hangover and a bad cold. Never."

JREF says, "That's fine. But we will be generous to you. We will allow 1 miss per 10 trials, for 10 10-trial sequences." *

How is that unfair? Does the answer relate to Point 1, above?


(My very poor statistical skills say that should be odds of 1 in 1,000. If I am wrong, someone please correct me. However, I think the idea should be clear.)

 

[Bryan Shelton]

[snip]In cases where a protocol is agreed where the odds of chance success can be calculated (the majority of cases), a success criteria of beating odds of around 1:1000 is typically used. But such a calculation is not always "a simple matter"; in some cases it is impossible, so another way of reaching agreement on the success criteria must be found, which is why no hard and fast rules can be set.

I recommend that the JREF _not_ test people who claim to be able to make people have to urinate. Stick to matters of a more purely probabilistic nature, where more time is spent merely determining the odds of numerical success, and far less time is spent making sure that the rest of the test is fair.

 

[Bryan Shelton]

Bryan, [snip] And in that case, it was only 5 out of 10 correct to move beyond the preliminary steps. How, again, is that unfair, when the claimant agreed to all of the protocols?

I don't think it's unfair (although I haven't looked closely at the numbers of that test).

 

[Bryan Shelton]

What will you do if the JREF changes nothing and continues on without making answers to your questions abundantly clear beforehand to your satisfaction. Will this continue to bother you for the rest of your life? If so, how will you possibly deal with this? Is your level of botheredness something psychological therapy might be able to help with? How will you cope?

I'd be a lot happier if they simply made it clear at the start (before any testing at all) what the requirements for success actually are; somelike like: "You must beat a 1-in-1,000 odds test in the Preliminary, followed by a 1-in-5,000 odds test (or whatever) in the Final."

 

[Rasmus]

I recommend that the JREF _not_ test people who claim to be able to make people have to urinate.

I am sure every employee or supporter of the JREF has been waiting on your valuable advise on what to test. How did they ever get on without your wisdom?

Stick to matters of a more purely probabilistic nature, where more time is spent merely determining the odds of numerical success, and far less time is spent making sure that the rest of the test is fair.

And why the ******* should they or anyone else listen to you?

 

[Bryan Shelton]

Bryan, if you have a specific paranormal ability you would like tested, just state your ability clearly and succinctly...If you have an ability, spit it out and let people help you figure out how to demonstrate it.

I don't have anything I'd like tested. I'm sorry if my general skepticism about JREF led you to suspect otherwise.

 

[Pup]

And what exactly does "pinning down" mean in this context? You mean PROVING his ability? Or simply explaining what his ability is supposed to be?

Explaining what his ability was supposed to be, as in how many different pictures, how many guesses, how many trials, whether runs would be counted as so many separate incidents or whether all guesses would be lumped together, that kind of thing.

He started out with a simple claim to be able to see hidden things, at well above chance levels, but gradually lowered his claim down to closer to chance, so it became more like the 55% ability you're talking about, thus requiring more trials to get an acceptable significance level.

I haven't read the threads since they were new, a few years ago, but as I recall, it was a lot of negotiations over probability theory, in response to self-tests that gave varying results. The threads should be all there to read, by searching on his name.

 

[Bryan Shelton]

Bump for Bryan, who apparently missed this: [snip] No, JREF *negotiates* the answers to these questions beforehand (before the applicant agrees that the protocol is fair and the test something they can do -- and only *then* is the test scheduled).

Why does this bother you?

I'll be happy if they make the odds of success (by somebody with no "special" abilities at all) perfectly clear before every test. Perhaps even more importantly, such odds should remain the same for ALL such tests.

 

[Baloney]

I'd be a lot happier if they simply made it clear at the start (before any testing at all) what the requirements for success actually are; somelike like: "You must beat a 1-in-1,000 odds test in the Preliminary, followed by a 1-in-5,000 odds test (or whatever) in the Final."

I'll have to look it up, but I read recently about the MDC that the preliminary was 1:1000 and the final was 1:1000000. If I have time I'll see if I can find the reference.

[ETA -- Here's a guy whining about it here: http://dailygrail.com/features/the-myth-of-james-randis-million-dollar-challenge

What everyone complaining about the odds is forgetting, is that the odds provided apply to success by chance! If the odds of randomly rolling 6 a die (d6) 10 times out of 100 is 1 in 10, the odds are significantly different for someone manipulating the die as it is being rolled (i.e. telekinetically.)]

 

[Bryan Shelton]

Point 1:

Having re-read this thread from the beginning, I note that there has not been any discussion of this:

"[...] When random choices are a clear and obvious element of such a test (as many of them truly are) [....]"

Mr. Shelton, could you please explain what you mean by the highlighted statement?

Do you, for instance, mean that the applicant makes random choices?

That the JREF makes random choices?

That during a test, some random elements are introduced by the applicant or the JREF or just good old Ma Nature?

I was referring to the applicant. Since I don't believe in "special abilities" by any of them, I think ALL of their choices are random.

Point 2:

In any case, I am confused about why it is unfair to set a standard that in most cases is much lower that the applicants claim to be able to meet.

I say, "I can detect Substance Y, by whatever method, infallibly, 100% of the time. Never miss. Never. Not even when I have a hangover and a bad cold. Never."

JREF says, "That's fine. But we will be generous to you. We will allow 1 miss per 10 trials, for 10 10-trial sequences." *

How is that unfair? Does the answer relate to Point 1, above?

I want the JREF to clearly state the chances of success (by ALL people) right up front. This business of "negotiation" is for the birds.

 

[Bryan Shelton]

I am sure every employee or supporter of the JREF has been waiting on your valuable advise on what to test. How did they ever get on without your wisdom?

And why the ******* should they or anyone else listen to you?

You're off my Christmas list!

 

[Bryan Shelton]

I'll have to look it up, but I read recently about the MDC that the preliminary was 1:1000 and the final was 1:1000000. If I have time I'll see if I can find the reference.

Surely they realize, don't they, that the COMBINED chance of winning 1:1000 in the preliminary and 1:1000 in the final is 1:1000000?? Are they just playing it SUPER-safe with their money?

 

[zooterkin]

Surely they realize, don't they, that the COMBINED chance of winning 1:1000 in the preliminary and 1:1000 in the final is 1:1000000?? Are they just playing it SUPER-safe with their money?

You still seem to be missing the point that those odds only apply to winning the prize by chance. The applicants are claiming to have an ability that means they perform a lot better than chance. If they can do anything remotely like what they claim, they will pass the test easily, because of the way the test is designed and agreed between the JREF and the applicant.

 

[Dumb All Over]

I'd be a lot happier if they simply made it clear at the start (before any testing at all) what the requirements for success actually are; somelike like: "You must beat a 1-in-1,000 odds test in the Preliminary, followed by a 1-in-5,000 odds test (or whatever) in the Final."

Thanks for your reply, Bryan.

Please understand that your happiness is something the JREF is probably not too interested in. Mr. Randi has challenged those who make extraordinary claims. It is his challenge and he gets to make the rules, as he and his organization have put up the substantial amount of money for the prize. That you don't like the terms is something you'll have to deal with. Good luck.

 

[jsfisher]

I recommend that the JREF _not_ test people who claim to be able to make people have to urinate. Stick to matters of a more purely probabilistic nature, where more time is spent merely determining the odds of numerical success, and far less time is spent making sure that the rest of the test is fair.

So, telekinesis, clairvoyance, and a host of other paranormal abilities shouldn't be testable under your ruling?

I don't have anything I'd like tested. I'm sorry if my general skepticism about JREF led you to suspect otherwise.

I haven't notice much general skepticism, just mostly kvetching from you. I'm not sure why you'd mistake one for the other.

 

[TSR]

I'd be a lot happier if they simply made it clear at the start (before any testing at all) what the requirements for success actually are; somelike like: "You must beat a 1-in-1,000 odds test in the Preliminary, followed by a 1-in-5,000 odds test (or whatever) in the Final."

Which is exactly what happens.

What doesn't happen is making clear what the requirements are before they even know what the paranormal claim is.

 

[TSR]

I'll be happy if they make the odds of success (by somebody with no "special" abilities at all) perfectly clear before every test.

Which they do.

Perhaps even more importantly, such odds should remain the same for ALL such tests.

All tests of the same claimed ability *do* have, *must* have the same odds.

You appear to want the same odds regardless of what the ability is, and that is not possible.

If the claimed ability is to be able to levitate unassisted 12 feet off the ground indefinitely, the odds approach unity against. The odds of knowing a 2 digit number, randomly chosen are 1:100 against. What would be fair in the first case would be grossly unfair in the second.

 

[wardenclyffe]

Which they do.

All tests of the same claimed ability *do* have, *must* have the same odds.

You appear to want the same odds regardless of what the ability is, and that is not possible.

If the claimed ability is to be able to levitate unassisted 12 feet off the ground indefinitely, the odds approach unity against. The odds of knowing a 2 digit number, randomly chosen are 1:100 against. What would be fair in the first case would be grossly unfair in the second.

What Bryan Shelton seems to be saying is that levitation should not be tested. Only tests that are essentially guessing games should be allowed.

Please correct me if I'm wrong.

Ward

 

[Gr8wight]

It seems perfectly reasonable to me that the JREF would first do a "preliminary test" (as "Pixel42" talked about in a later post this morning) which would require approximately a 1 in 1,000 chance (or less) of passing purely by chance. If the applicant passes it, a somewhat more rigorous test could then be part of the "formal test", which determines the fate of the million dollar prize. The first "preliminary test" would be the obvious way to avoid "wasting time" by applicants who don't have any real ability.

Well, Bryan, this would be entirely reasonable if not for the fact that... no, wait... YOU JUST DESCRIBED THE EXACT JREF TESTING PROCEDURE!

 

[xterra]

I was referring to the applicant. Since I don't believe in "special abilities" by any of them, I think ALL of their choices are random.

But surely you see that the applicants do believe in their "special abilities." That means that for them these are not random choices. They think they are being guided by their special abilities.

That's why they claim -- at least initially -- that they never miss an identification of a photo or of a substance, or that they can dowse for water reliably (by which they usually mean much higher than 55% of the time), or can always make a spoon bend by mental ability alone.

The goal of the statistical design of the test is to eliminate success by pure luck or chance. The goal of the physical design of the test is to eliminate success by cheating.

This is the equivalent of saying that the JREF wants any success to be a clear demonstration of the claimed ability. Looked at in that light, one could say that the JREF is giving the applicant every opportunity to clearly and unequivocally demonstrate his or her ability.

That's different from saying that the JREF is setting out to ensure that the candidate fails the test.

If I have misrepresented your meaning or if I have not been clear, please correct me or ask for clarification.

Xterra

 

[ronin688]

But surely you see that the applicants do believe in their "special abilities." That means that for them these are not random choices. They think they are being guided by their special abilities.

That's why they claim -- at least initially -- that they never miss an identification of a photo or of a substance, or that they can dowse for water reliably (by which they usually mean much higher than 55% of the time), or can always make a spoon bend by mental ability alone.

The goal of the statistical design of the test is to eliminate success by pure luck or chance. The goal of the physical design of the test is to eliminate success by cheating.

This is the equivalent of saying that the JREF wants any success to be a clear demonstration of the claimed ability. Looked at in that light, one could say that the JREF is giving the applicant every opportunity to clearly and unequivocally demonstrate his or her ability.

That's different from saying that the JREF is setting out to ensure that the candidate fails the test.

If I have misrepresented your meaning or if I have not been clear, please correct me or ask for clarification.

Xterra

Well said, that's about as clear as I think anyone could possibly make it.

 

[Bryan Shelton]

"Surely they realize, don't they, that the COMBINED chance of winning 1:1000 in the preliminary and 1:1000 in the final is 1:1000000?? Are they just playing it SUPER-safe with their money? "

You still seem to be missing the point that those odds only apply to winning the prize by chance. The applicants are claiming to have an ability that means they perform a lot better than chance. If they can do anything remotely like what they claim, they will pass the test easily, because of the way the test is designed and agreed between the JREF and the applicant.

Of course! That's obvious!! I personally don't believe that the applicants have any special ability; if their claim is actually correct, then they DESERVE to win the prize and collect the million dollars. So what exactly is YOUR point?

 

[Bryan Shelton]

"I'd be a lot happier if they simply made it clear at the start (before any testing at all) what the requirements for success actually are; somelike like: 'You must beat a 1-in-1,000 odds test in the Preliminary, followed by a 1-in-5,000 odds test (or whatever) in the Final.' "

Thanks for your reply, Bryan.

Please understand that your happiness is something the JREF is probably not too interested in. Mr. Randi has challenged those who make extraordinary claims. It is his challenge and he gets to make the rules, as he and his organization have put up the substantial amount of money for the prize. That you don't like the terms is something you'll have to deal with. Good luck.

Why don't they specify (and reveal) the exact terms beforehand? Why do they turn this into a guessing-game?