What's the required p-value to beat?

[FluffyPersian]

A significance test does not simply test the claim, "this succeeds 80% of the time" The p-value tests the null-hypothesis, not the alternative.

Actaully, "this succeeds 80% of the time" can be the null hypothesis But in that case, you'd have a 2-tailed test with an alternative hypothesis of

HA: Pi<>.8

In most MDC tests, on the other hand, you'd want to test whether something succeeds MORE THAN X% of the time.

However, as I said aboavae, the significance level and the population proportion being tested are two different parameters ("inputs").

 

[xtifr]

No, it doesn't. The significance level, which is what you're apparently referring to, is separate from the threshold proportion and the sample size.

Yes, sorry. The point is, a claim "this works 100% of the time" will require a different protocol from the claim "this works 80% of the time".

 

[GnaGnaMan]

Not necessarily. It actually depends on how good is the psychic. A telekinetic, for example, can be a professional of getting 55% heads/tails. However, it is most likely that the psychic will not reach a p-value of 0.001 in the small run. If the JREF wants to test his/her claim with an 80% chance of getting a p-value of 0.001, JREF would need a sample-size of 1704 coin tosses. Of course this isn't a problem with claimants who have high hit rates and claim they are professional of doing so.

Not sure how this relates to what I was saying.
It should be possible to perform 2000 coin tosses in less than 2 hours so there does not seem to be a problem with getting tiny p values if someone can really get 55% right on a coin toss.

OTOH how would one notice that ability in the first place? That would take 1000s of recorded coin tosses as well.

Also, if someone is genuinely telekinetic then surely one would be able to test this in a better way than coin tossing. It's seems gratuitous to introduce a random element. Why not just use a force sensor? I guess that's not the point, though.

 

[Marcus]

There is only one way to succeed in the MDC. One must make a claim that can be tested sufficiently in a single day to prove that the odds of chance success of 1 in 1000 have been exceeded (for the preliminary).

This would be simple indeed if the various practitioners could do what they say they can do.

The fact that no one has passed the preliminary speaks volumes.

 

[Musibrique]

There is only one way to succeed in the MDC. One must make a claim that can be tested sufficiently in a single day to prove that the odds of chance success of 1 in 1000 have been exceeded (for the preliminary).

This would be simple indeed if the various practitioners could do what they say they can do.

The fact that no one has passed the preliminary speaks volumes.

You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power.

What you're saying is nothing but absurd.

 

[Pixel42]

the probability of an applicant reaching such p-value actually depends on the applicant's hit rate.

Which is why JREF always ask the applicant what their hit rate is, so they can take that into account as they help them design a suitable protocol.

 

[GnaGnaMan]

You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power.

Correct enough but it is unclear what you think this implies.
Nothing about this makes what Marcus said "absurd"-

 

[Marcus]

You don't understand. You may think an applicant can easily reach a p-value of 0.001 or 0.000001 if the applicant could do what the applicant claims. However, it's not that simple. Statistically speaking, the probability of an applicant reaching such p-value actually depends on the applicant's hit rate as well as the sample-size of the test. It's a process known as statistical power.

What you're saying is nothing but absurd.

It's all about applicants being able to perform the paranormal feats they claim that they can.

If someone can really communicate telepathically or dowse for gold, the exact probabilities necessary to satisfy JREF are not that important, they just have to demonstrate their claimed abilities.

In fact, the odds are just a guideline. Not all claims can easily have probabilities assigned to them. If JREF decides a claim is paranormal, and can be tested in a reasonable amount of time, it is a potential candidate.

 

[xterra]

Musibrique and Marcus,

I believe that my post

http://www.internationalskeptics.com/forums/showthread.php?postid=8392067#post8392067

is relevant to your disagreement.


xterra

 

[Diploid]

Some folks have argued that the Challenge is unfair because the p-values and effect sizes called for are too extreme.

I think the JREF is looking for people with genuine ability, not people who are lucky. Even a fair coin will flip heads 20 times in a row if you try it long enough and the JREF prize has been on the table for some 50 years, if I remember correctly.

Besides, why should it be any easier for someone to guess a number between 1 and 10 than between 1 and a trillion? You either can or can't do it. The range is irrelevant.