Split Thread How are MDC protocols designed and carried out? (Split from FAQ thread)

Bryan Shelton

Scholar
Joined
Aug 14, 2006
Messages
56
snuffkin wrote "I know nothing about the tests or how they would measure the abilities?" Each protocol has to be custom-designed for the claimant. Claims of dowsing, mind-reading, and telekinesis have to be tested differently. In each case, however, there would be an objective pass/fail criteria.

I've read the JREF Challenge FAQ, but I don't see any clear discussion of how these "tests" are actually designed and carried out. When random choices are a clear and obvious element of such a test (as many of them truly are), who determines whether or not the results are significant enough to win the million dollar prize? Is that up to James Randi? Do people here consider that to be fair?
 
I've read the JREF Challenge FAQ, but I don't see any clear discussion of how these "tests" are actually designed and carried out.

All of this information is included in the application:

http://www.randi.org/site/images/stories/mdc-rules-and-application-2011-03-09.pdf

If the application is accepted, testing protocols will then be developed in consultation with the Applicant, who must provide clear guidelines so that the test(s) may be properly designed and carried out to the mutual agreement of the JREF and the Applicant. The JREF may consult with experts, including statisticians, magicians, and others with specialized knowledge relevant to the claim. James Randi may or may not be present at these tests, but he will not interact with the materials used nor interfere with the protocol once a test is underway.
 
I've read the JREF Challenge FAQ, but I don't see any clear discussion of how these "tests" are actually designed and carried out. When random choices are a clear and obvious element of such a test (as many of them truly are), who determines whether or not the results are significant enough to win the million dollar prize? Is that up to James Randi? Do people here consider that to be fair?

See above... it is decided on in advance and in agreement with the applicant. For example, JREF says, "Do you think you could achieve it sixteen times out of twenty?", applicant says, "Oh yeah, that's a cinch!". JREF: "Let's make that the success threshold then?", applicant says "Agreed."
 
Sorry, but just saying that it has to be "to the mutual agreement of the JREF and the Applicant" doesn't answer my question. What is the MINIMAL statistical significance that such a test must provide, and who determines it? Is it James Randi, or is it somebody else?
 
See above... it is decided on in advance and in agreement with the applicant. For example, JREF says, "Do you think you could achieve it sixteen times out of twenty?", applicant says, "Oh yeah, that's a cinch!". JREF: "Let's make that the success threshold then?", applicant says "Agreed."
Using the simple laws of probabilities, I could correctly guess the identity of a card in a shuffled deck of playing cards exactly 1 out of every 52 attempts. But what if I claimed that I could do it 1 out of every 45 times? How many times would I have to demonstrate that ability, before I win the JREF million dollar prize? Who determines how many times I have to do, before I win the prize? Is it James Randi, or somebody else? I strongly suspect that my requirement for such a successful test would strongly differ from that of James Randi.
 
Using the simple laws of probabilities, I could correctly guess the identity of a card in a shuffled deck of playing cards exactly 1 out of every 52 attempts. But what if I claimed that I could do it 1 out of every 45 times? How many times would I have to demonstrate that ability, before I win the JREF million dollar prize? Who determines how many times I have to do, before I win the prize? Is it James Randi, or somebody else? I strongly suspect that my requirement for such a successful test would strongly differ from that of James Randi.

then you and the JREF would continue negotiating until your protocol had something that would satisfy both of you, and the test would not take place until it did.

The protocol is an agreement. It's the meeting point of what you are willing to say you can do and what Randi is willing to give you money for doing. Both sides have to agree or there is no test.
 
It depends on the claim. There is no one number than can be used for all experiments. In biology, for example, we often look for a confidence interval of 95%. You choose an n (number of experiments) suited to the question you are addressing. You could compare data in a variety of ways including student's t-test, anova, or simple comparrison of standard error between groups. You pick the form of analysis in advsnce based on what is appropriate for the data, and you cant change it after to get the result you want. Thats why they ask you to agree in advance- so you cant reanalyse the data in a context which is inappropriate to the data typr,but which gives a false positive.
 
So who would I be negotiating with? James Randi, or another member of the JREF? Why is it so hard to pull this information out of everybody?? :)
 
So who would I be negotiating with? James Randi, or another member of the JREF? Why is it so hard to pull this information out of everybody?? :)

it's not hard to get the information, it's just that you don't seem to have read anything.

per the application that I linked you to earlier:

This offer is made and administered by the JREF, and no one may negotiate or make any changes, except as set forth in writing by an officer of the JREF.
 
It depends on the claim. There is no one number than can be used for all experiments. In biology, for example, we often look for a confidence interval of 95%. You choose an n (number of experiments) suited to the question you are addressing. You could compare data in a variety of ways including student's t-test, anova, or simple comparrison of standard error between groups.
So who determines the ultimate success of my ability to identify random cards from a shuffled deck? James Randi, or somebody else? What is the required degree of success OVER what would be expected purely from chance? Why is all this material (the exact rules of the test) left completely unexplained by the JREF? Who exactly would I be dealing with?? :confused:
 
So who determines the ultimate success of my ability to identify random cards from a shuffled deck? James Randi, or somebody else?

You agree on what the criteria for success would be before the test. If you meet these criteria, you pass.

What is the required degree of success OVER what would be expected purely from chance?

You agree on what the criteria for success would be before the test.

Why is all this material (the exact rules of the test) left completely unexplained by the JREF?

The exact rules are explained and you have been linked to them.

Who exactly would I be dealing with?? :confused:

You would be dealing with appointed officers of the JREF.

I'm not sure what part of this is so confusing to you.
 
Last edited:
it's not hard to get the information, it's just that you don't seem to have read anything.

per the application that I linked you to earlier:
So any single "officer of the JREF" would be sufficient, as long as I could talk him into agreeing with my own view of the test?
 
I'm not sure what part of this is so confusing to you.

A few years ago when I first joined the forum here, I got a similar run-around about exactly how the test is performed, and the required statistical significance of the test. Nowadays, I'm less inclined to let this important issue slide by! ;)
 
So who determines the ultimate success of my ability to identify random cards from a shuffled deck? James Randi, or somebody else? What is the required degree of success OVER what would be expected purely from chance? Why is all this material (the exact rules of the test) left completely unexplained by the JREF? Who exactly would I be dealing with?? :confused:

Have you emailed the JREF this question? People here (well, the vast majority of them) are mere mortal forum users. We do not speak for the JREF.

I think I read somewhere that a general rule of thumb was that the preliminary and the final test should only be passed by mere chance in about 1/1000 cases each - so your chances of submitting the protocol about predicting the card without any paranormal ability would be at about one in a million.

Personally, I think they should be much higher.

People are essentially claiming they can do the impossible.

Say, e.g. I was posing my challenge to a group of tragically blind-from-birth three year olds. They cannot see, and they have no concept of seeing and they cannot read or have a concept that such a thing as letters exist.

Now I propose that i can tell what is on one of their audio-CDs without any of the usual methods: I will not play it, or even have access to a player, I will not know how or where the CD has been sorted, etc.

Of course, all I have to do is look at the CD and read what it says on top of the disk. How reliably would you think I could identify a CD that way - if it was assured that the selection that each CD was chosen from would be heterogeneous enough to make mix-ups and confusions highly unlikely?

Say there would be 20 CDs. Bringing them into some pre-defined order or naming each one correctly by chance alone would be 1:20! or 2.43290201 × 1018.

My one-in-a-million chance would allow me to try and get only 10CDs sorted right, and I would get three tries and I would only have to get it all right once.

So, that being said, whenever someone starts arguing about their chances here, I can only conclude one of two things:

They do not understand the test procedures or they want to cheat.
 
So any single "officer of the JREF" would be sufficient, as long as I could talk him into agreeing with my own view of the test?

At the point where the protocol is being negotiated, I'd imagine you would be speaking to whoever oversees these tests for the JREF. I believe that this is currently Banachek. The reason for the wording is so that if Banachek is not available, someone else within the JREF can perform these duties in his absence. Also, so the whole protocol doesn't have to be rewritten every time staffing changes slightly.
 
A few years ago when I first joined the forum here, I got a similar run-around about exactly how the test is performed, and the required statistical significance of the test. Nowadays, I'm less inclined to let this important issue slide by! ;)

the only run around is happening in your own head.

I'm sorry that you aren't able to read documents and comprehend the information contained therein, but have you considered that this might be your problem rather than the JREF's?
 
Using the simple laws of probabilities, I could correctly guess the identity of a card in a shuffled deck of playing cards exactly 1 out of every 52 attempts. But what if I claimed that I could do it 1 out of every 45 times? How many times would I have to demonstrate that ability, before I win the JREF million dollar prize? Who determines how many times I have to do, before I win the prize? Is it James Randi, or somebody else? I strongly suspect that my requirement for such a successful test would strongly differ from that of James Randi.
It's probably best to start with what ever the remarkable thing is that you can do, and then work on a way of making it as remarkable as possible without taking up an inordinate amount of time. If you really do have a remarkable power, then you can presumably show it to an arbitrary level of certainty. If you don't, then of course how lucky the JREF demand you be is very important.
 
A few years ago, there was a fascinating program on TV, and I bet many of you know what I'm talking about: several people who claimed to be "psychics" (or whatever they called themselves) were put to a direct test by James Randi, with the possibility of winning a pile of money (probably the million dollars in the JREF test)! Each one had a different claimed ability, so their tests all differed a bit in what they did. It was a long time ago and I'm forgetting the vast majority of them, but the one that I remember the best was this very pretty woman who was given a bunch of keys (or something similar) on a chain; she had claimed that she could "sense" something about the keys (I'm vague nowadays about the details), just by looking at them, or feeling them, etc. So as part of the test, she was required to "sense" those keys, and reveal some specific information about them. The woman FAILED to do so, of course, as did all the other people who took part in the tests on that show. The look on the poor woman's face, though, distressed me very deeply: she appeard to be SHOCKED that she was unable to identify the keys! She just couldn't believe that she had failed!!

But another thing that made an impression on me was that all the people on the show had all been given ENORMOUSLY difficult challenges, for each of their "tests". Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route? What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it? It just didn't seem very fair to me, especially that poor woman who was so SHOCKED at her lack of success! I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.
 
A few years ago, there was a fascinating program on TV, and I bet many of you know what I'm talking about: several people who claimed to be "psychics" (or whatever they called themselves) were put to a direct test by James Randi, with the possibility of winning a pile of money (probably the million dollars in the JREF test)! Each one had a different claimed ability, so their tests all differed a bit in what they did. It was a long time ago and I'm forgetting the vast majority of them, but the one that I remember the best was this very pretty woman who was given a bunch of keys (or something similar) on a chain; she had claimed that she could "sense" something about the keys (I'm vague nowadays about the details), just by looking at them, or feeling them, etc. So as part of the test, she was required to "sense" those keys, and reveal some specific information about them. The woman FAILED to do so, of course, as did all the other people who took part in the tests on that show. The look on the poor woman's face, though, distressed me very deeply: she appeard to be SHOCKED that she was unable to identify the keys! She just couldn't believe that she had failed!!

But another thing that made an impression on me was that all the people on the show had all been given ENORMOUSLY difficult challenges, for each of their "tests". Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route? What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it? It just didn't seem very fair to me, especially that poor woman who was so SHOCKED at her lack of success! I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

Success is being able to do what they say they can do. Failure is not being able to do what they say they can do. The reasons why are immaterial.

Was it fair to give them a test with long odds? Sure. It's JREF's money, they can do what they want with it.

What if someone had a genuine psychic ability that they couldn't manifest on a TV show? Then they shouldn't be tested on a TV show.

I don't get what you are finding unfair about this.
 
Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route?

How is that at all relevant? They weren't claiming to do it by the normal route, so the chance they'd have if they were is utterly irrelevant.

What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it?

What if they did? Giving them a good chance of passing without actually using their "genuine" magic powers wouldn't do anything to address that, it would simply mean they could pass without actually having magic powers at all. If people think being nervous will reduce their success rate, then they should only agree to a test that takes that into account. If they fail to do so it's entirely their own stupid fault for agreeing to be tested for something they can't actually do.

Out of interest, where else do you think this argument applies? I've taken part in quite a few competitions in various sports and music. Should I have been claiming it was all unfair because I might have been nervous during my performances? Given how incredibly difficult you seem to think it is for people to simply do what they claim, I'm frankly amazed that it's possible for professional sports, musicians, comedians, actors, and so on to exist at all. What if their natural nervousness and trepidation prevents them from performing when they're supposed to?

Of course, it's worth bearing in mind that most applicants are absolutely certain of having a 100% success rate and actually have to be argued down to a lower rate for a test precisely to take this sort of thing into account.

I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

Perhaps it would have helped if you'd been determined to actually listen to the answers as well.
 

ISF - Join now!

Every member here is approved by hand. No bots, no spam, just people who care about evidence and honest debate.

Membership is free!

Create your free account

Back
Top Bottom