The Amazing Test makes me hopeless

No. why should they?

And what does that have to do with fairness, even?

Here's what happens in a nutshell:
Applicant: "I can do x, even though it should not be possible for anyone under any conditions to do x."

Randi: "Well, I bet you a million dollars that you cannot do x."

The only fair thing then is for the applicant to demonstrate that he can actually do x.

Why should it matter to the applicant how difficult it would be for anyone to pass the test who isn't able to do x?


Example:

Applicant: "I can speak Chinese!"

Randi: "Really? Well, listen to this Chinese audio-book and then tell me the following things:

Is it a love story, a thriller, a whodunnit or a comedy?
Is the female lead character between 10 and 20 years, 21 and 30 years or between 31 and 40 years?
The book mentions a painting. Is it of a dog, a windmill or a woman?
Does the grandmother get married?
Where/how does the milk man die? An accident while mountain climbing, a car crash on the motorway, in hospital due to kidney failure, ran out of oxygen diving or parachute failure?
What colour is the airplane - blue, red, yellow, green or white?


You can listen to it as much as you like, and you can rewind or fast forward as much as you please. If you answer all these questions correctly, I'll give you a million dollars."

Applicant: "But that's not fair. I only have a chance of 1 in 1800. Why does it have to be so difficult? Wouldn't it be just as good if I got half the questions right? Whaaaaaaaaaaaaaaaaaaa!"

-> It doesn't matter, and it is not unfair. If the applicant is actually able to speak Chinese, then getting every question right could not possibly be a more trivial task.

We're not doing the test to see if it is likely or possible that the applicant maybe speaks Chinese. We just want fairly reasonable proof.

We could discuss the chances of getting an answer wrong accidentally. To compensate, we could add three or four more questions and allow for one or possibly two answers to be incorrect. The basic facts would remain the same: It will be almost impossible to pass the test unless you actually speak Chinese. And it will be trivially easy to pass the test, if you do.

How is it unfair that someone unable to speak Chinese would have a very hard time passing the test?

And why should it be any different for paranormal claims?

This misses the obvious objection that psi powers may not work with 100% accuracy. That doesn't mean they don't exist. Getting 70 out of 200 Zener cards correct is not even 40% accurate, but someone who could do that should win the challenge.
 
This misses the obvious objection that psi powers may not work with 100% accuracy. That doesn't mean they don't exist. Getting 70 out of 200 Zener cards correct is not even 40% accurate, but someone who could do that should win the challenge.

No it doesn't. If 35%, 15% above chance, is what a claimant says they can do, then that is what will be tested. You can test for lower success rates, but to make certain it's not just chance, you need to increase the number of tests. IIRC, the preliminary test needs only to be one in 10,000 to win by chance, and thankfully, claimants very often make bold claims. Otherwise it could get very tedious.
 
This misses the obvious objection that psi powers may not work with 100% accuracy. That doesn't mean they don't exist. Getting 70 out of 200 Zener cards correct is not even 40% accurate, but someone who could do that should win the challenge.

We could discuss the chances of getting an answer wrong accidentally. To compensate, we could add three or four more questions and allow for one or possibly two answers to be incorrect. The basic facts would remain the same: It will be almost impossible to pass the test unless you actually speak Chinese. And it will be trivially easy to pass the test, if you do.

How is that any different?

Besides, it is up to the claimant to define what they can do. That includes their accuracy, does it not? It is their own fault if the claim they make does not take into account any limitations of their power.

At least the challenge FAQ make that very clear as well.
 
Correction: Thanks Dancing David.

Your welcome either way.

The meaning of statistcs is often obscure to begin with. Take correlation, people will report 'a tweny five percent' correlation between X and Y, well that means that overall there are 75% that are not related. So it is best to seek out 68% or greater depending on the method. And that correlation does not equal causation.

As players of DnD and other dice based games know, you can roll a twenty or a 100. Often to disaster for your character.
 
Last edited:
This misses the obvious objection that psi powers may not work with 100% accuracy. That doesn't mean they don't exist. Getting 70 out of 200 Zener cards correct is not even 40% accurate, but someone who could do that should win the challenge.

I would want to see the protocol, I am not sure they would take that one, you have five cards and 200 trials so just choosing the same card would be forty hits. So really 70 would not require that great an amount of random variation. I think they would want to raise the bar to at least 50-60%%.

But that is on a single trial of 200.

If it were my protocol I would lower the bar, the more trials they were, but I would think a decent effect should be at least twice that expected from random selection. So say 1,000 cards then 48%, with each 1000 being worth 2% so if they choose 5000 cards and get 40% I would agree to that.
 
No it doesn't. If 35%, 15% above chance, is what a claimant says they can do, then that is what will be tested. You can test for lower success rates, but to make certain it's not just chance, you need to increase the number of tests. IIRC, the preliminary test needs only to be one in 10,000 to win by chance, and thankfully, claimants very often make bold claims. Otherwise it could get very tedious.

1/10000= p-value of .0001 , I think.
 
No it doesn't. If 35%, 15% above chance, is what a claimant says they can do, then that is what will be tested. You can test for lower success rates, but to make certain it's not just chance, you need to increase the number of tests. IIRC, the preliminary test needs only to be one in 10,000 to win by chance, and thankfully, claimants very often make bold claims. Otherwise it could get very tedious.

But the test is to prove the existence of the paranormal, correct? Getting 70+ out of 200 Zener cards is either extremely lucky or proof of some psi ability. It is well above the 5% statisitical significance level usually used in clinical trials. The odds are about 1 in a million? Unless the online calc I'm using is totally off base.
 
It is about 1 in a million; I suspect Dancing David was eyeballing it and not expecting it to be that difficult. This is heading down a path we've taken before about ability vs. chance and the guy who can predict coin flips with 50.00001% accuracy.

Ultimately, it boils down to the fact that the JREF has a hard success number, which is 0.001 for the preliminary test*, and a soft requirement on how long the test will take. For the zener card example, JREF would likely consider 10 out of 20 to be a successful preliminary test. If I can predict the card 95% of the time, I'm virtually guaranteed to succeed. If I can predict the card 50% of the time, I'm going to lose by chance almost 60% of the time! I can protect myself by increasing the number of trials.

If I go to 200 trials, and have to hit 58, I'm again >99.999% chance to succeed if I can predict 50% of the cards. But what if I'm only good to 25%. That's still better than chance, right! At that rate, I lose the 200 trial test >90% of the time due to random chance!

So if i'm at 25%, I might ask for 2000 trials, to show I can get 456 right. That I can do almost 99% of the time. But now the trials are getting longer to execute. Ultimately, if I'm only just above chance, how useful is my ability anyway?


*generally speaking. I remember a dowsing test in which the candidates were given a 1/150ish chance of success!
 
It is about 1 in a million; I suspect Dancing David was eyeballing it and not expecting it to be that difficult. This is heading down a path we've taken before about ability vs. chance and the guy who can predict coin flips with 50.00001% accuracy.

Ultimately, it boils down to the fact that the JREF has a hard success number, which is 0.001 for the preliminary test*, and a soft requirement on how long the test will take. For the zener card example, JREF would likely consider 10 out of 20 to be a successful preliminary test. If I can predict the card 95% of the time, I'm virtually guaranteed to succeed. If I can predict the card 50% of the time, I'm going to lose by chance almost 60% of the time! I can protect myself by increasing the number of trials.

If I go to 200 trials, and have to hit 58, I'm again >99.999% chance to succeed if I can predict 50% of the cards. But what if I'm only good to 25%. That's still better than chance, right! At that rate, I lose the 200 trial test >90% of the time due to random chance!

So if i'm at 25%, I might ask for 2000 trials, to show I can get 456 right. That I can do almost 99% of the time. But now the trials are getting longer to execute. Ultimately, if I'm only just above chance, how useful is my ability anyway?


*generally speaking. I remember a dowsing test in which the candidates were given a 1/150ish chance of success!

So let's say the person makes the claim that they have some low-grade psi ability that allows them to score higher than random chance on a Zener card test. The preliminary test is given, and they get 35 out of 100 right. Very low odds of that happening. So the person is tested again: they get 30 out of 100. Again, they beat the odds, though not as much as the first time. Now, if that were a drug trial, the pharma company would be breaking out the champagne. So does the guy get the million?
 
That's why there needs to be an agreement before the test. If JREF says 30 out of 100 is acceptable to them, you win. (They're more likely to say 33, for the 1/1000).

In a lot of the threads involving applicants here, you'll see a lot of effort put into trying to get them to identify exactly what they can do, and how good they are. If you're confident you can get 33 out of 100, the million is probably yours.

The difference between this and a drug trial is that this is "two" and done. You succeed twice, you get the million, you don't ever have to perform again if you don't want to. The drug company will continue to be responsible for the effectiveness of the drug (I'm assuming safety isn't entering into this). It is technically possible for the drug trials to have been lucky "hits", and an ineffective drug will be introduced to the market. They will continue to profit from it until it is shown that the drug is not truly effective. In fact, I'm sure there are examples of this happening, although I don't know of any personally. It's a lot less likely to be discovered if the drug is safe.
 
So let's say the person makes the claim that they have some low-grade psi ability that allows them to score higher than random chance on a Zener card test. The preliminary test is given, and they get 35 out of 100 right. Very low odds of that happening. So the person is tested again: they get 30 out of 100. Again, they beat the odds, though not as much as the first time. Now, if that were a drug trial, the pharma company would be breaking out the champagne. So does the guy get the million?

What evidence do you have for your claim that a pharma would break out the champagne for a 10% and 5% above the mean result?

Seriously, data please?
 
But the test is to prove the existence of the paranormal, correct? Getting 70+ out of 200 Zener cards is either extremely lucky or proof of some psi ability. It is well above the 5% statisitical significance level usually used in clinical trials. The odds are about 1 in a million? Unless the online calc I'm using is totally off base.

If one looks at the odds it always depends.
70/200=.35=35% , so only tem percent above the mean.

So my idea would be two trials of ten thousand by lot of 200 (don't worry a computer will do it).

One program the machine just says card 'stars' and then it measures the results for each of the fifty trials of 200 cards.

Another program generates a random card choice and measures the results for each of the fifty trials of 200 cards.

Then you have two control samples of 50 trials of 200 card runs. This allows you to generate a mean, mode and standard deviation.

then i would randomly chose ten subject and have them randomly paired as rearedr and sender. Each pair would do 200 cards for 50 trials. Then you can again get a mean, mode and standard deviation.

Now, we can seriously take a stab , given the limited control samples, at determining what the occurrence of 70/200 cards it compared to the three control groups. If it falls within one standard deviation, that is not very remarkable as 68% of the population falls in one standard deviation, if it is two standard deviations, then that is more remarkable at 28% of the population, and so on.

Now it is two standard deviations away and they can do it on 5 runs of 200 cards each (at 35% each of 5 runs), I would give them the million.
 
Dancing David said:
So let's say the person makes the claim that they have some low-grade psi ability that allows them to score higher than random chance on a Zener card test. The preliminary test is given, and they get 35 out of 100 right. Very low odds of that happening. So the person is tested again: they get 30 out of 100. Again, they beat the odds, though not as much as the first time. Now, if that were a drug trial, the pharma company would be breaking out the champagne. So does the guy get the million?

What evidence do you have for your claim that a pharma would break out the champagne for a 10% and 5% above the mean result?

Seriously, data please?

Your odds are wrong.
 
If one looks at the odds it always depends.
70/200=.35=35% , so only tem percent above the mean.

So my idea would be two trials of ten thousand by lot of 200 (don't worry a computer will do it).

One program the machine just says card 'stars' and then it measures the results for each of the fifty trials of 200 cards.

Another program generates a random card choice and measures the results for each of the fifty trials of 200 cards.

Then you have two control samples of 50 trials of 200 card runs. This allows you to generate a mean, mode and standard deviation.

then i would randomly chose ten subject and have them randomly paired as rearedr and sender. Each pair would do 200 cards for 50 trials. Then you can again get a mean, mode and standard deviation.

Now, we can seriously take a stab , given the limited control samples, at determining what the occurrence of 70/200 cards it compared to the three control groups. If it falls within one standard deviation, that is not very remarkable as 68% of the population falls in one standard deviation, if it is two standard deviations, then that is more remarkable at 28% of the population, and so on.

Now it is two standard deviations away and they can do it on 5 runs of 200 cards each (at 35% each of 5 runs), I would give them the million.

Again, you're way off on your odds. A Zener card deck has five different cards. 70 or more out of 100 is not like winning the lottery, but it certainly is higher than "ten percent above the mean". It's about a million to one.

ETA: You can't just divide 70 by 200!
Use this: http://stattrek.com/Tables/Binomial.aspx
 
Last edited:
One program the machine just says card 'stars' and then it measures the results for each of the fifty trials of 200 cards.

Another program generates a random card choice and measures the results for each of the fifty trials of 200 cards.

Can someone explain the purpose of doing these two separate things?

If the machine producing the cards that need to be guessed, is producing them truly randomly, what would be the difference between guessing stars every time and guessing a random guess every time? Wouldn't the odds be the same?

The only way I can see it making a difference is if the machine was biased for or against producing stars. Is it testing for that?
 

ISF - Join now!

Every member here is approved by hand. No bots, no spam, just people who care about evidence and honest debate.

Membership is free!

Create your free account

Back
Top Bottom