• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

JREF Challenge Statistics

T'ai Chi

Penultimate Amazing
Joined
May 20, 2003
Messages
11,219
I've been thinking about this for quite some time, and finally put up a webpage on it

http://www.statisticool.com/jrefchallengestats.htm

Does anyone believe that after 1,000+ tests that are statistical in nature are carried out, that anyone will win by chance?

This applies not only to the JREF preliminary tests, but also similar tests by other skeptical organizations.
 
Hypothetically, how about after 10,000 tests?

At what point would the not winning by chance be 'odd' ?
 
Chance is only involved as it pertains to the actual tests, as in measuring results against those expected by chance. In uncontrolled tests we might expect someone to "win" by chance, but these are controlled tests presumably designed to eliminate chance. In these tests, "win" is equivalent to "demonstrating paranormal ability" as defined in each test. It's not like playing slot machines. Given that, I would not expect anyone to "demonstrate paranormal ability" by chance, especially if the test conditions are properly designed. We're talking about abilities that violate the laws of physics, after all, and there is no good reason, under proper, scientifically controlled conditions as is done with the JREF Challenge tests, to expect a chance observance of such a violation.
 
Obviously, that depends on the alpha levels of each test, which will vary depending on claim. Remember, each test is the test of a specific claim, and to arbitrarily assign a different alpha level to your overall analysis is unfounded. If a claimant says he can do something 9 times out of 10, the test can end in failure with even an average of 8 out of 10; the test can thus end considerably earlier than it would if we were merely testing something against chance alone. Those data are not then appropriate to test against chance alone.

Combining many short runs is not the same as taking one long run--Rhine's lab made that mistake, and (at least in his casual talks) so does Sheldrake.

In sum, your proposed test is not a very good method of analyzing the test data.

ETA: Pyrrho beat me to it.
 
Hypothetically, how about after 10,000 tests?

At what point would the not winning by chance be 'odd' ?

Each test is a "Stand Alone" event. Like throwing dice or flipping a coin, the current test is not dependent on the previous test. The probability of "Success by chance" does not change after any number of tests--unless the current testee has learned new tricks from the previous tests...
 
In addition, to win the million, they have to be able to pass the preliminary test AND the final test. So even if they managed to pull off 10,000/1 odds once, they'd have to do it again.
 
In uncontrolled tests we might expect someone to "win" by chance, but these are controlled tests presumably designed to eliminate chance. In these tests, "win" is equivalent to "demonstrating paranormal ability" as defined in each test. It's not like playing slot machines. Given that, I would not expect anyone to "demonstrate paranormal ability" by chance, especially if the test conditions are properly designed. We're talking about abilities that violate the laws of physics, after all, and there is no good reason, under proper, scientifically controlled conditions as is done with the JREF Challenge tests, to expect a chance observance of such a violation.

I think you're misunderstanding something. In controlled tests one would still expect someone to win by chance. Ask yourself what alpha means. It involves probability.
 
I think you're misunderstanding something. In controlled tests one would still expect someone to win by chance. Ask yourself what alpha means. It involves probability.

Most of the test protocols I've read here are specifically designed to ensure that the odds of winning by chance are around 1 in a million. This is easily achieved, given that most tests are also specifically designed to consist of a series of judgement-free hit-or-miss actions by the testee. For example, if a dowser is trying to identify one pot containing water from twenty, the odds of getting it right by guessing are 1 in 20. That's too easy, so one could ask him to repeat the task to give 1 in 400 odds, then again three more times for total odds of 1 in 3.2 million.

So, assuming odds of 1 in a million for an average preliminary protocol, it would still be 100 to 1 against someone winning by chance after 10,000 tests have been done.

One would hope that, long before 10,000 tests have been done, society will have given up this sort of superstitions anyway... :)

(My glass is half-full)
 
Most of the test protocols I've read here are specifically designed to ensure that the odds of winning by chance are around 1 in a million.

I'm specifically talking about the preliminary tests. I'ev been informed that the alpha for these is typically .001.
 
What percentage of the applicants have a claim that is statistical in nature?
 
How many preliminary tests were conducted last year (2005)? I may have missed or forgotten some, but I think it was three. Achau Nguyen, Angela Patel, and Deja Gateward. (Somebody correct me if I missed any.)

Given the rate that applicants devise a protocol that can be agreed upon, it will take a very long time before 10,000 tests are conducted and somebody beats the challenge on a fluke.
 
it will take a very long time before 10,000 tests are conducted and somebody beats the challenge on a fluke.

I agree. 10000 was just a hypothetical number to illustrate, because one would expect, on average, someone to pass by chance after 1000, if alpha is typically .001 for such tests.

I start talking about the JREF preliminary tests that are statistical in nature, because the JREF tests are the most well known, but also consider similar tests done by many skeptical organizations for my argument.
 
I think you're misunderstanding something. In controlled tests one would still expect someone to win by chance. Ask yourself what alpha means. It involves probability.
Perhaps you could explain what alpha means, for those who do not know, and how that definition supports the expectation that someone would "win" by chance. I don't accept the chore of explaining alpha. You're the statistician; educate us, please.

In this example:

http://www.randi.org/jr/032902.html

We numbered ten JREF coffee-mugs from 1 to 10 on the outside bottoms. For the baseline part of the test (20 "open" trials in which all those present would know in which cup the target had been placed) Mike was first asked to choose one of ten face-down shuffled cards bearing numerals from 1 to 10, and that choice would designate where the target would be placed, each time. I had asked him to carefully "scan" the floor area of our library in advance to make sure there were no distracting elements present, and he himself carefully chose the positions of each of the ten cups on the floor. He was encouraged by me to adjust the placement of the cups as many times as he needed to, during this phase. He'd told us, first, that at least five feet of separation was required between each cup, but that he could work with just three feet between them. I immediately insisted that he must use at least five feet, since I did not want to allow an excuse later on that the spacing had been inadequate. As it turned out, he chose to have some cups within a foot of one another. But we could not interfere with his choice, since he assured us that all was sufficient for his needs.

Mike also asked that several metallic objects (trophy cups, plaques, steel devices) be removed from the bookshelves nearby. At his request, a teaspoon was taken to the next room because he said that the silver could also attract his stick; that spoon was made of aluminum. But, again, we did not correct his statements.

For the "open" phase of the preliminary test procedure, the target package was placed in the designated cup, which was then openly placed in the spot Mike had chosen for it, mouth-down. He then scanned all ten cups, and declared — both by pointing and verbally — where he believed that his stick had detected the target. Another number was then selected, and the procedure was repeated, twenty times in all. His score was 100% in these "open" tests.

Pause. Let me explain here the purpose of the baseline test of twenty "open" detections, in which the location of the target is known in advance. It served five distinct purposes, which is why we always use such a procedure:

(1) The performer has the opportunity to try out the setup, and make any necessary changes, adjustments, or re-locations that he thinks are needed. Mr. G. changed the location of the ten cups on the floor many times before the "open" detection trials were completed, and finally declared his total satisfaction with the placements, and with the conditions.

(2) The process of randomizing numbers, etc., which is sometimes unfamiliar or unknown to the performer, becomes clear. For Mike, we prepared ten cards bearing numbers from one to ten, shuffled them face-down, and asked him to choose one for each test.

(3) The performer becomes familiar with the sequences and rules of the test. With Mike, we changed only one factor: we began with plastic cups, but because of the bulk of the target package, we switched to using the JREF coffee mugs.

(4) The performer has the opportunity of deciding for himself — in the "open" tests — whether it's his powers, or just his foreknowledge of the answer, that is actually at work. Mike was convinced of the former.

(5) After the "blind"test is done, following the "open" series, the performer cannot offer the excuse that his powers were not working at this time. Mike obtained 100% results during the "open" test, quickly and positively, showing that he was quite able to use his powers.

Following the "open" sequence, for each of the "blind" tests, Mr. G. and I stepped out of the library area, and two other persons randomly (by choosing a face-down card, as before) placed the target package in position, then they left the area and informed us that the target was in place. Mike and I re-entered, alone, and he made his determination while I watched carefully to be sure that he did not nudge any cups, or otherwise attempt to use any means but the movements of his forked stick, to make his guess; at no time was any such procedure observed. After Mike made his guess on each trial, the other two persons were invited back in, and we recorded the results. That procedure was repeated ten times.

...

The results were that when Mike G. knew the location of the concealed target (the "open" tests), he obtained 100% results. When the test procedure was double-blinded, he obtained exactly what chance alone would call for: one out of ten correct.

What's the alpha for these tests, and how it is calculated? I think there's enough information there for a calculation to be made.
 
What's the alpha for these tests, and how it is calculated? I think there's enough information there for a calculation to be made.

One doesn't calculate alpha. You set alpha before the test is done.

It is the probability of making a type 1 error, that is, the probabilty of rejecting the hypothesis tested when the hypothesis is in fact true.

Typically it is .05, but for more extreme claims it is typically set lower, to .01 or lower. For the preliminary tests it is typically .001.

As far what alpha in that specifc preliminary test was set to, you'd have to ask the JREF. They don't make such numbers readily available, so I don't know.
 
Last edited:
rwguinn made the best point, I think. The tests are seperate events. Someone losing one does not increase the chance of someone else winning. If you flip a coin 100 times and get all heads, the probability of you getting heads the next time is still 1/2.
 
If alpha = .001 for each test, and the tests are sufficiently similar, you'd expect someone who doesn't have paranormal powers to pass the test 1 in 1000 times on average.
 
Last edited:

Back
Top Bottom