Split Thread How are MDC protocols designed and carried out? (Split from FAQ thread)

A few years ago, there was a fascinating program on TV, and I bet many of you know what I'm talking about: several people who claimed to be "psychics" (or whatever they called themselves) were put to a direct test by James Randi, with the possibility of winning a pile of money (probably the million dollars in the JREF test)! Each one had a different claimed ability, so their tests all differed a bit in what they did. It was a long time ago and I'm forgetting the vast majority of them, but the one that I remember the best was this very pretty woman who was given a bunch of keys (or something similar) on a chain; she had claimed that she could "sense" something about the keys (I'm vague nowadays about the details), just by looking at them, or feeling them, etc. So as part of the test, she was required to "sense" those keys, and reveal some specific information about them. The woman FAILED to do so, of course, as did all the other people who took part in the tests on that show. The look on the poor woman's face, though, distressed me very deeply: she appeard to be SHOCKED that she was unable to identify the keys! She just couldn't believe that she had failed!!

A few likely explanations:

She was deluded.
She was trying to keep face.
She thought she could get away with some trick.

But another thing that made an impression on me was that all the people on the show had all been given ENORMOUSLY difficult challenges, for each of their "tests".

How were they difficult?

Did you read my above post?
Would it have been difficult for them to perform if they really had the abilities they claimed to possess? Would it be difficult for a person to sort a few CDs if they could read the labels?


Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route?

Yes.

They said they weren't going to go the normal route! The odds would make it difficult to cheat or win by chance, they would not make it difficult to pass the test with the claimed ability.

Arguably, it might still be difficult to perform the ability, but the odds assume that the ability does not exist at all. If I claimed I could keep 20 apples in the air, I could chose to juggle them. It would still be difficult, but the difficulty of juggling 10 apples is unrelated to the difficulty of levitating 10 apples.

If a challenger claimed he could levitate apples, the test would only make it difficult for him to juggle them.

What if some of them had a genuine "psychic" ability, but their natural nervousness and trepidation (especially on a TV show) prevented them from fully demonstrating it?

Then what? Get some public exposure, and practice a little more. come back after a year and grab the million dollars.

It just didn't seem very fair to me, especially that poor woman who was so SHOCKED at her lack of success! I was determined that if I ever had a chance to talk to people on a forum like this, I would have them explain exactly how such tests are designed and conducted, and how their "success" is determined.

How is any of this the JREF's problem?

Again, let us assume I claimed I could juggle 10 apples, and you bet me 100$ that I can't.

We talk about the bet and agree what it means for me to successfully juggle 10 apples:

At no time can I ever hold more than 2 apples in any one hand, no apple can be in one of my hands for longer than 3 seconds at a time, an apple my be nowhere else other than in my hands or the air, and specifically, an apple may not touch the ground. I have 20 seconds to get all the apples moving from their initial position oin the ground.

We agree that there will be 5 observers, each video-recording my performance on a camera. If I can show that I followed the rules, I get your money; if you can show that I failed to meet the requirements, i do not get the money.

Now, on the day of the test, I end up being terribly nervous, I keep dropping apples, totally fail to ever pick up apple number 7. etc. etc.

Do you owe me the money? Certainly not!
Are you in any way to blame for my inability to foresee that I would not be able to perform under such stressful conditions? No!

Should I whine about being treated unfairly? Neither!

Now, why don't I just spend a week or two on practising, in front of cameras and people. Maybe ask you to adjust the rules - give me a few more moments to get the apples into the air, a second longer to hold on to an apple, etc.

Should I be allowed to reduce the number of apples to 6? No, not if my claim is "I can juggle 10 apples".

Should I be allowed to have up to three apples lying on the ground at any time? No, because then I'd be juggling 7 apples and not 10.

And maybe, just maybe, I should come to the realisation that I am not able to juggle apples ...
 
Was it fair to give them tests that had a chance of success of only 1 in HUNDREDS OF THOUSANDS, by the normal route?

Yes it was fair, because their claim is usually much, much much better than that in accuracy.
How many psychic out there you see claiming accuracy lower than 1/10 ? Usually they claim much better accuracy than that.

As for nervousness and so forth : my impression is that JREF test are not made into public tv but with very limtied public. It smells like an excuse to explain away not doing better than chance.
 
Last edited:
I don't get what you are finding unfair about this.

I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.
 
I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.
Since they don't know what the remarkable occurrence is that they are going to have to adjudicate on, it's kind of hard to come up with hard and fast rules. Also, up against them are a bunch of professional con artists and maybe, possibly, but probably not, somebody with genuine ability. If they don't keep a free hand to try to rule out fraud, they'll be conned out of the million long before anybody with a real ability, if such a person exists, comes along.

By the way, how come the folks don't get nervous on TV and in front of big crowds until Randi puts talcum powder on their hands, or what ever it might be that would stop them cheating?
 
I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

They are offered a million dollars! A million! I'd have to work some 20 years and not eat or pay rent or buy as much as toothpick during that time to accumulate that much money.

If anyone was offering me a million dollars to simply demonstrate one of my (more or less) amazing abilities, I would happily submit myself to those tests. And if, actually having an ability, I'd fail to demonstrate it, I would work out what caused me to fail in my demonstration.

And then I would spend a year on getting around those factors. And then I would kindly ask the JREF if I could be tested again. I might actually be more than happy to use the exact same test protocol, because it might still be a good protocol. Maybe I had to get over my nervousness, or get used to the bright lights, or I got distracted and will simply wear earplugs ... or anything, really!

I would not waste my time whining on a forum about how the test was unfair. (Unless, of course, it was unfair. In which case I could explain precisely why that was and how it should be changed. And chances are I wouldn't blame the JREF, and I can guarantee you that my requests would not make it easier for me to pass the test in case I didin't actually have my ability!)

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.

What's unfair?

Both parties are supposed to work out a protocol that specifies the answers to these questions!

For every claim, these answers will be different, so it always, always, always has to start with the applicant stating what exactly it is they are able to do.

Judging from the various discussions I've read here so far, I am forced to conclude that paranormal abilities are tied to the inability to construct or understand a few coherent sentences, though. If the test required applicants to pass an English Language Test of any kind, there would be no challenge.
 
Since they don't know what the remarkable occurrence is that they are going to have to adjudicate on, it's kind of hard to come up with hard and fast rules.

Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

Also, up against them are a bunch of professional con artists and maybe, possibly, but probably not, somebody with genuine ability. If they don't keep a free hand to try to rule out fraud, they'll be conned out of the million long before anybody with a real ability, if such a person exists, comes along.

Sure, preventing FRAUD from con artists would be a normal and expected thing for Randi and his officers to do. But that shouldn't have anything to do with them "cooking" the numbers, or deliberately making the tests incredibly difficult to pass!

By the way, how come the folks don't get nervous on TV and in front of big crowds until Randi puts talcum powder on their hands, or what ever it might be that would stop them cheating?

I don't know. Maybe some of them DO get nervous, even before that happens?
 
Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

Yes. But that requires them to first claim what they can do. And since that is not known, the JREF simply leaves that point open until after someone claims what it is they can do.

I can accept that you might like it better otherwise, or if there was a fixed claim that you'd have to beat odds of precisely 1:n. But surely, this couldn't e a problem, right?

Sure, preventing FRAUD from con artists would be a normal and expected thing for Randi and his officers to do. But that shouldn't have anything to do with them "cooking" the numbers, or deliberately making the tests incredibly difficult to pass!

I am not aware of a single instance of that happening.

The "numbers" are only about how likely it would be to pass the test if you performed randomly. But your claim is that you do not perform randomly!

I don't know. Maybe some of them DO get nervous, even before that happens?

Of all the examples I am aware of - and I know I haven't followed every test, or even the majority - I am not aware of any one test where I'd say your accusations might have merit.

the one and only thing I am aware of, is allowing a magician (Banachek, I think) handle sealed envelopes during a test. In that case I would support a re-test where only people that are not famous magicians get to touch, move, or otherwise interact with the objects that are needed during the test. But that one example does not involve any obvious cooking of numbers or making things harder than neccesary.
 
Sure, preventing FRAUD from con artists would be a normal and expected thing for Randi and his officers to do. But that shouldn't have anything to do with them "cooking" the numbers, or deliberately making the tests incredibly difficult to pass!
This is where you are going wrong.

If you have the magical power to "see" the face of upturned playing cards, we might pick 5 cards from 5 randomly (we'd have to agree on how to achieve this) shuffled decks and get you to tell us what they were. If you have no powers, the probability of you getting all of them (call it P1) should be, I think, (1/52)^5 = 0.00000000263. If you can actually "see" the face of the upturned cards, the probability (call it P2) should be 1.

I would expect the JREF to seek to arrive at a test that would keep P1 as low as possible while maximizing P2. They only want to make it difficult to pass if you have no powers.
 
This is where you are going wrong.

If you have the magical power to "see" the face of upturned playing cards, we might pick 5 cards from 5 randomly (we'd have to agree on how to achieve this) shuffled decks and get you to tell us what they were. If you have no powers, the probability of you getting all of them (call it P1) should be, I think, (1/52)^5 = 0.00000000263. If you can actually "see" the face of the upturned cards, the probability (call it P2) should be 1.

Here I disagree.

there is no reason to assume that the magical seeing of the turned card should work at all. Surely, if it does work, there is no reason to assume that it will work 100% of the time, or 70%, or 7% or 3% or just 1%.

Who's to say? It's magic, after all!

Just because visually seeing the card works reliably (under ideal conditions), doesn't mean magical seeing has to be the same.

Even real stuff doesn't always work like that, and there are probabilities involved. After what time does a particular element decay? We know it's a particular time on average, but we can never predict - precisely - when it will happen the next time, right? (Forgive me if i have this totally wrong, though.)

Maybe there are 52 card-spirits that whisper the next card to the performer in turns, and maybe some of them like to cheat occasionally? And maybe their frequency of cheating averages out to a success rate of 45 cards per deck?
 
Here I disagree.

there is no reason to assume that the magical seeing of the turned card should work at all. Surely, if it does work, there is no reason to assume that it will work 100% of the time, or 70%, or 7% or 3% or just 1%
In my example, he could "see" the cards, so it is as close to 1 as makes no odds. Normally people's powers seem to be closer to 1 than 0 probability until they get tested.

Who's to say? It's magic, after all!
What is magic? I don't mean anything specific by it other than winning by some means that isn't a cheap (or expensive) trick based on outsmarting the JREF. Telling homeopathic water from nonhomeopathic water would do it for me.

Just because visually seeing the card works reliably (under ideal conditions), doesn't mean magical seeing has to be the same.

Even real stuff doesn't always work like that, and there are probabilities involved. After what time does a particular element decay? We know it's a particular time on average, but we can never predict - precisely - when it will happen the next time, right? (Forgive me if i have this totally wrong, though.)

Maybe there are 52 card-spirits that whisper the next card to the performer in turns, and maybe some of them like to cheat occasionally? And maybe their frequency of cheating averages out to a success rate of 45 cards per deck?
Sure, and then you can build an experiment to give you any probability of success you like. Keep repeating it and it becomes easier and easier for somebody with a real power to demonstrate it and harder and harder for somebody to fluke it. For some reason Byan seems to be complaining that it is unfair to make the test really hard to fluke, or am I misunderstanding?
 
Here I disagree.

there is no reason to assume that the magical seeing of the turned card should work at all. Surely, if it does work, there is no reason to assume that it will work 100% of the time, or 70%, or 7% or 3% or just 1%.

Who's to say? It's magic, after all!

Just because visually seeing the card works reliably (under ideal conditions), doesn't mean magical seeing has to be the same.

Even real stuff doesn't always work like that, and there are probabilities involved. After what time does a particular element decay? We know it's a particular time on average, but we can never predict - precisely - when it will happen the next time, right? (Forgive me if i have this totally wrong, though.)

Maybe there are 52 card-spirits that whisper the next card to the performer in turns, and maybe some of them like to cheat occasionally? And maybe their frequency of cheating averages out to a success rate of 45 cards per deck?

well, in those cases you don't ask for perfection, but you ask for consistently better than chance over long periods. If I claim to be able to guess a coin flip correctly 55% of the time, obviously I won't be asked to guess the coinflip 100% of the time, but I would be asked to guess a huge number of coinflips so the chance of such a result occurring by chance is as low as possible.
 
In my example, he could "see" the cards, so it is as close to 1 as makes no odds. Normally people's powers seem to be closer to 1 than 0 probability until they get tested.

I certainly agree that the claims that usually come up have no reservations or margins of error that would not be comparable with other abilities.

But, of course, even in your example, the person does not "see" the cards as such - no light is reflected from the cards in different colours towards his retina. So whatever it is, and whatever it feels like to the participant, it would not be "seeing".


What is magic? I don't mean anything specific by it other than winning by some means that isn't a cheap (or expensive) trick based on outsmarting the JREF. Telling homeopathic water from nonhomeopathic water would do it for me.

I was just using the term to describe all or any happenings of stuff that would be classified as "paranormal".

I agree that telling which water has been treated homeopathically would qualify for a test.

But we cannot expect the challenger to do this at a success-rate of 100%, or 80% or anything. We "expect" that he performs within the boundaries of random chance, after all. We have no explanation for any other result, so it seems unjustified to expect any other result to be at 100%.

and, yes, I agree that they will usually claim as much - in which case there is no need to expect anything less than that, or significantly less.


Sure, and then you can build an experiment to give you any probability of success you like. Keep repeating it and it becomes easier and easier for somebody with a real power to demonstrate it and harder and harder for somebody to fluke it. For some reason Byan seems to be complaining that it is unfair to make the test really hard to fluke, or am I misunderstanding?

As far as I can tell, this is what his complaint boils down to - but I am not sure if he means and understands that, or if he thinks this would actually influence the chances of a truly gifted applicant. (And I agree that it does, as long as we assume the applicant has a 100% chance and as long as that is not actually the case.)

For any ability of the applicant to perform less than perfectly but above chance, we can find a test-design that makes it arbitrarily difficult for a cheater to pass by mere chance alone yet would let us expect a genuinely gifted applicant to pass.

Usually, this would require a ridiculous number of trials, and of course it calls into question how the applicant ever discovered his ability in the first place to make claims about with any confidence.

It is easy to define my superpower as follows:

I have the ability to predict the next card drawn from a deck of cards not with the expected 1:52, but 1:51 - if and only if the card happens to be the trey of clubs.

If I genuinely had this truly amazing power ... I would never ever notice it, if I lived a million years and did nothing else but play with card decks.
 
I don't find the example of that TV show all that "unfair", although I do think that Randi was being rather over-the-top in making the requirements of the tests so incredibly stringent. It's his money, though (although I believe I've heard that it actually came from somebody else), so he can be as tough with it as he wants. All that TV show really did for me was to make me wonder if possibly, just POSSIBLY, there really was a bit of truth to the "psychic" claims from some of those people.

Something else that really bothers me, though, just like I mentioned in my original posts from several years ago, is the confusion and uncertainty on the part of the JREF officers as to what they will allow, and what they will REQUIRE during the tests that they monitor. That really troubles me, and strikes me as unfair.

Bryan Shelton, this link might be of some help to you:
Aesop's Fables - The Fox and the Grapes
 
Bryan Shelton,

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test? What should they have been? How should they have been figured out? If there's no easy way to figure out those odds, should she have been prohibited from participating?

Ward
 
Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

After the JREF is told what the ability is, yes, and that's why the odds are agreed proposed during the negotiations, after the applicant specifies their ability.

Are you saying the JREF should be able to figure the odds, even before an applicant announces whatever their ability is?
 
After the JREF is told what the ability is, yes, and that's why the odds are agreed proposed during the negotiations, after the applicant specifies their ability.

Are you saying the JREF should be able to figure the odds, even before an applicant announces whatever their ability is?

then they would be eligible to win their own challenge...
 
Baloney. There's no doubt in my hand that a relatively simple examination of what the subjects are doing, along with (possibly) a little help of some statisticians and/or some mathematicians could most assuredly prove what the expected results would be.

Hey, stop using my name in vain!
 
Bryan Shelton,

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test? What should they have been? How should they have been figured out? If there's no easy way to figure out those odds, should she have been prohibited from participating?

Ward

http://www.internationalskeptics.com/forums/showthread.php?t=89877

Brilliant example.

I don't know if a background discussion is available, but you can see very exact claims being made, no expected success rate for random occurrences, a quick test, and surprisingly, no winner.

To be fair, though, we would expect that a healthy person who just emptied their bladder would be found wetting themselves within 15 or 30 minutes would be almost exactly 0.0%. It is almost impossible that this should happen by mere chance.

But I agree there is a chance this could happen randomly - so it would have made things very difficult if a percentage had been pre-defined.
 
well, in those cases you don't ask for perfection, but you ask for consistently better than chance over long periods. If I claim to be able to guess a coin flip correctly 55% of the time, obviously I won't be asked to guess the coinflip 100% of the time, but I would be asked to guess a huge number of coinflips so the chance of such a result occurring by chance is as low as possible.

That's EXACTLY what I'm saying! :)

I think that James Randi was a bit unfair on that TV show, when he demanded that the "psychics" pass TREMENDOUSLY difficult tests. Of course, if they themselves had made such claims before the start of the show, then that's their own problem; one could hardly criticize Randi for demanding that they pass a test which they themselves claimed to be able to pass.

But what if their only claim is that they can "sense" the result of a coin flip only 55% of the time? Is James Randi (or an "officer" of the JREF) going to throw them out of the office, or are they going to give them a legitimate test for the one million dollar prize, as long and difficult as that would obviously be? :confused:

The fact that the JREF apparently doesn't make the answers to these questions abundantly clear beforehand (before somebody actually walks in the door and submits a proposal for a test) is what bothers me...
 
Bryan Shelton,

There was a recent (at least in terms of MDC tests) test of a woman who claimed she could make someone have to urinate. What are the odds for that test? What should they have been?

I have no idea what they are, or what they should have been.

How should they have been figured out? If there's no easy way to figure out those odds, should she have been prohibited from participating?

I have no idea how they should have been figured out. Personally, I would have had nothing to do with such a test.
 

ISF - Join now!

Every member here is approved by hand. No bots, no spam, just people who care about evidence and honest debate.

Membership is free!

Create your free account

Back
Top Bottom