Merged Odds Standard for Preliminary Test

[jojonete]

The odds standards shouldn't be in the rules because the rules are binding: applicants must agree to them, sign them and notarize them. All applicants, all rules. As the Challenge FAQ puts it:

2.6 I’d like to change a rule.
No. The Challenge rules are in place for a reason. When you fill out the JREF Challenge application, you are entering into a contract. The JREF doesn’t cut corners or make exceptions. Not even for you.
[...]
2.7 This particular rule shouldn’t apply to me.
It does. Again, the application is a contract.

That said, it is true that the results of many tests consist of two lists of numbers to compare, and standard odds for these tests do make sense. I feel the Challenge FAQ (not the rules) is the right place to put this information, always making it clear that p-values are only orientative, and that some tests (e.g. the UFO summoning test) do not require p-values at all.

In fact, Randi said in Swift February 29, 2008:

... So, as of now, we will require that applicants beat a one-in-one-hundred chance of success – by dumb luck or co-incidence – for the preliminary test, and then a one-in-one-hundred-thousand chance in the formal test – a point that has not yet been reached in the past ten years of our trying…

This paragraph, with a relevant question and very little editing, could be easily included in the Challenge FAQ. With this, p-values would be somehow more "official" and we in the forums would have something to quote from instead of saying "it's been said somewhere that the JREF usually asks for 1-in-1000 odds".

 

[Rodney]

In any case, I can't quite figure out what your complaint is. Is it that Randi is not willing to spend an unlimited amount of time on any one applicant? Or, perhaps, that he doesn't give, in the official rules, an indication of how much time he is willing to spend?

My complaint is that the JREF Challenge Rules discourage most serious applicants from applying. If the JREF wants to specify in those Rules that no more than X hours will be devoted to any application, that would at least clarify that certain protocols, such as the Ganzfeld, are not realistically amenable to being tested by the JREF.

 

[drkitten]

My complaint is that the JREF Challenge Rules discourage most serious applicants from applying.

Yes, that's certainly the lie that frauds tell their devoted believers.

Your "complaint" has no evidence behind it and no validity.

If the JREF wants to specify in those Rules that no more than X hours will be devoted to any application, that would at least clarify that certain protocols, such as the Ganzfeld, are not realistically amenable to being tested by the JREF.

But they don't want to specify that, because someone might have a legitimately testable but time-consuming test. For example, "I can speed up continental drift," which might require a fifteen-year test protocol, but will not require minute-to-minute attention on the part of Randi & Co. A similar test was accepted a few years back, where a claimant promised to produce unusual weather (I think the actual terms were "snow in Berkeley, California on the Fourth of July" or something like that) if given six months lead time.) The protocol was duly accepted (and duly failed, as proven by the accepted neutral arbiter, the Weather Bureau, who is going to publish weather statements anyway.

The JREF has no interest in publishing any rules that will be in any way subject to negotiation. P-values, time, and so forth, are legitimately the subject of negotiation, and as such must be negotiated instead of dictated by the JREF.

And your p-value complaint is, frankly, silly. Anyone who says that Randi would simply set the bar arbitrarily and unreasonable high is either lying or misinformed. He has never done that to my knowledge; in fact, standard procedures seems to be for him to offer the claimant substantially better odds than s/he claims in the application. For example, if the claimant says she can detect buried gold "without error," he will usually accept eighteen out of twenty or thereabouts.

 

[shadron]

I don't see why Randi would have a problem setting up a protocol for someone who claimed to be able to make a consistent 51% correlation in a Ganzfeld. What Randi would insist on is that there be enough trials that making a 51% success in a truly random control test would be very small (smallness commensurate with risking $1M), so that scoring 51% would be a significant achievement. That may make the protocol long - but that's what you buy when you want small differences to be significant.

The point of the whole procedure is, like any legal contract, you and Randi come to an agreement on the protocol a priori - nothing more. Obviously, Randi is going to assure himself that the protocol pretty much guarantees that winning randomly will not occur; but similarly you will assure yourself of your capability of being able to win even under the handicap of not being allowed to "luck out". The protocol is the agreement. If you cannot come to a meeting of the minds on the protocol, then one of the two of you, or both, want the odds tipped into their favor - he doesn't want a random happening to win, and you don't want to spend the rest of your life in trial after trial. That will result if you have a small confidence of being able to do what you say you can do. It is the ultimate fair test, if the protocol can be agreed upon.

Randi makes no guarantees that such an agreement can happen - he only pledges a good faith effort on his part, and assumes one on yours.

An analogy of this working is if you and your wife/husband are getting a divorce, and you have to split the common estate. An equitably fair arrangement can always be guaranteed by having one ex-partner define the division, and the other gets to choose which side of the division to take. Simple.

 

[Rodney]

Yes, that's certainly the lie that frauds tell their devoted believers.

The person who told me that the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test" is a highly respected Ganzfeld researcher.

Your "complaint" has no evidence behind it and no validity.

In your totally objective opinion.

But they don't want to specify that, because someone might have a legitimately testable but time-consuming test. For example, "I can speed up continental drift," which might require a fifteen-year test protocol, but will not require minute-to-minute attention on the part of Randi & Co. A similar test was accepted a few years back, where a claimant promised to produce unusual weather (I think the actual terms were "snow in Berkeley, California on the Fourth of July" or something like that) if given six months lead time.) The protocol was duly accepted (and duly failed, as proven by the accepted neutral arbiter, the Weather Bureau, who is going to publish weather statements anyway.

Obviously, what I was referring to is time spent by the JREF. Are you saying that the JREF would accept a time-consuming Ganzfeld protocol?

The JREF has no interest in publishing any rules that will be in any way subject to negotiation. P-values, time, and so forth, are legitimately the subject of negotiation, and as such must be negotiated instead of dictated by the JREF.

P-values "are legitimately the subject of negotiation"??? So, in one case an applicant achieving P=.00001 might win an applicant the million, and in another case an applicant achieving P=.000001 might not?

And your p-value complaint is, frankly, silly. Anyone who says that Randi would simply set the bar arbitrarily and unreasonable high is either lying or misinformed. He has never done that to my knowledge; in fact, standard procedures seems to be for him to offer the claimant substantially better odds than s/he claims in the application. For example, if the claimant says she can detect buried gold "without error," he will usually accept eighteen out of twenty or thereabouts.

That's sporting of him, but it doesn't answer the question I'm raising regarding the lack of an odds standard in the JREF Challenge Rules.

 

[Bob Klase]

The person who told me that the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test" is a highly respected Ganzfeld researcher.

But he obviously has not spent much time researching the MDC. The criteria for success is established by agreement between the challenger and JREF. It's not arbitrary and once an agreement is reached it's no more changeable at Randi's whim than it is at the challenger's whim.

One would expect a highly respected researcher to actually do some research before stating conclusions.

 

[GzuzKryzt]

...
That's sporting of him, but it doesn't answer the question I'm raising regarding the lack of an odds standard in the JREF Challenge Rules.

What is so difficult for you to understand about this statement: There is no standard because there are no standard claims. The claims vary, the protocols vary, hence the odds vary. It's that simple.

 

[Paul2]

It seems there are two types of MDC claims: those that involve odds and for which JREF would set a p-value, and those that don't (make a UFO appear). I think Rodney's point is that, for those claims that do involve a p-value, shouldn't it be established beforehand? Otherwise, it seems arbitrary for each applicant.

"For some claims, the JREF will require a p-value for a successful test. For those claims, applicants must achive results greater than p=X in order to be successful."

Can someone explain why a p-value might be different for applicant A compared to applicant B (assuming both A and B require some p-value level of success)?

 

[shadron]

It seems there are two types of MDC claims: those that involve odds and for which JREF would set a p-value, and those that don't (make a UFO appear). I think Rodney's point is that, for those claims that do involve a p-value, shouldn't it be established beforehand? Otherwise, it seems arbitrary for each applicant.

"For some claims, the JREF will require a p-value for a successful test. For those claims, applicants must achive results greater than p=X in order to be successful."

Can someone explain why a p-value might be different for applicant A compared to applicant B (assuming both A and B require some p-value level of success)?

Why should it matter? If Randi and the claimant come to an agreement, then, presumably, they do, in fact, agree that the criteria, whatever they are, are definitive as to whether the claim is met or not. Why should it matter that one person claims 100% accuracy and the next claims 75% accuracy, and that both can be accommodated within the rules to each person's, and Randi's, full satisfaction before the testing is carried out?

 

[shadron]

That's sporting of him, but it doesn't answer the question I'm raising regarding the lack of an odds standard in the JREF Challenge Rules.

In the final analysis, it's the foundation's money and the foundation makes the rules. They are fair rules, because the claimant and Randi agree that the protocol will be a definitive determinant before the test is started. Inability to come to such an agreement is not either's fault. Your point is not an issue, except as a means of casting a lame blame on Randi for failure in negotiations, and as such, just too darned bad.

 

[drkitten]

The person who told me that the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test" is a highly respected Ganzfeld researcher.

Well, since Ganzfeld researchers are ispo facto frauds, I'm happy to stand by my statement.

Obviously, what I was referring to is time spent by the JREF. Are you saying that the JREF would accept a time-consuming Ganzfeld protocol?

Probably not. The problem in this case isn't the the time so much as the opportunity for cheating (and the time spent by the JREF). If the researcher could propose an iron-clad protocol that was impossible to cheat (and bearing in mind that Randi does not know enough about cryptography to accept most of the standard cryptographic protocols that would commonly be used -- with good reason), there's no reason that he woudn't accept such a protocol.

P-values "are legitimately the subject of negotiation"??? So, in one case an applicant achieving P=.00001 might win an applicant the million, and in another case an applicant achieving P=.000001 might not?

Absolutely. The winning and losing criteria are evaluated on individual bases.

That's sporting of him, but it doesn't answer the question I'm raising regarding the lack of an odds standard in the JREF Challenge Rules.

It does. The answer is, "your question is irrelevant and the one posing it is a fool."

 

[drkitten]

Can someone explain why a p-value might be different for applicant A compared to applicant B (assuming both A and B require some p-value level of success)?

Because applicant A and applicant B negotiate different protocols with different success criteria, obviously.

Can some explain why a salary might be different for applicant A compared to applicant B (assuming both A and B require some salary)? Why am I paid more than the department secretary, but less than the football coach?

 

[Rodney]

Because applicant A and applicant B negotiate different protocols with different success criteria, obviously.

If the odds of success can be readily calculated, permitting different applicants to negotiate different success criteria -- as measured by P values -- makes no sense. What the JREF Challenge is supposed to be about is setting the bar high enough that it cannot be hurdled by random chance, but not so high as to make hurdling it unlikely even if an alleged paranormal phenomenon is real. So, it would be illogical, for example, for the JREF to apply a more stringent P value standard to playing card guessing than to number guessing.

Can some explain why a salary might be different for applicant A compared to applicant B (assuming both A and B require some salary)? Why am I paid more than the department secretary, but less than the football coach?

Because that's what market forces dictate. How is your example in any way analogous to using different P value criteria for different Challenge applicants?

 

[Rodney]

Well, since Ganzfeld researchers are ispo facto frauds, I'm happy to stand by my statement.

Evidence?

 

[Moochie]

My complaint is that the JREF Challenge Rules discourage most serious applicants from applying. If the JREF wants to specify in those Rules that no more than X hours will be devoted to any application, that would at least clarify that certain protocols, such as the Ganzfeld, are not realistically amenable to being tested by the JREF.

Can you point to a few dozen of these "most serious applicants"? And do "most serious applicants" believe, as you seem to, that "paranormal" ability is so subtle that it requires a statistician to determine its existence?

M.

 

[GzuzKryzt]

Before the mudflinging starts: Rodney, do you now have a better understanding why there are no "standard odds"?

Perhaps an inquiry at challenge@randi.org would clear up any remaining questions. Please ask them to reply in this thread.

 

[Rodney]

Can you point to a few dozen of these "most serious applicants"?

I believe that most Ganzfeld experimenters would be interested in applying for the Challenge, if they knew they were eligible and were confident that the test would be conducted fairly.

And do "most serious applicants" believe, as you seem to, that "paranormal" ability is so subtle that it requires a statistician to determine its existence?

With regard to controlled tests, I think that most do. However, I think that most believe, as I do, that the paranormal often manifests itself spontaneously in a much less subtle fashion.

 

[Rodney]

Before the mudflinging starts: Rodney, do you now have a better understanding why there are no "standard odds"?

Not with regard to those applications that involve readily calculable odds.

Perhaps an inquiry at challenge@randi.org would clear up any remaining questions. Please ask them to reply in this thread.

Done.

 

[Speed of Light]

This is from the Carina Landin thread

Hi
There is something that no-one has mentioned.
It has often been said on the forum that for the preliminary test, the probability of succeeding by chance alone, must be smaller than 1 in 1000, and for the final test, 1 in a million.

By my calculations, for this test, the probability of getting 16 or more correct out of 20, purely by chance alone, works out as (approx) 1 in 169.234.

I agree with you Rodney, that there ought to be a set standard to aim at 'officially', so that any applicant knows where they stand.

However, I'm not too sure that this is the reason why there are no 'serious' applicants.

I feel it has more to do with Randi's attitude to 'psychics'.
The whole mood of the JREF is against it.
He is vey impatient with anything paranomal - Not what you'd call open minded. He is really interested only in clear obvious tests and results which are readily apparent to anyone, making better publicity.
I don't believe Randi has the patience for 'subtleties'.
He surrounds himself with similar people, which is probably the reason why, in Golfy's thread, the idea of a polygraph was immediately rejected.
I don't believe they have the patience, unless forced to, to seriously consider the subtleties - They want clear cut rules - immediately understandable by everyone.
Randi is not a statistician - I think he said that in his February 29th monologue - He is more a 'people' person, interested only really in the public aspect of it.

Having said that, I believe that if a serious applicant was determined to enter the challenge, and brave the waters, so to speak, it would not be too difficult to design a protocol.
If they claimed for example 54% success rate in guessing the colour of a randomly chosen card, this could easily be done by a computer simulation, using some sort of random number generator. The whole test could easily be carried out by email, and the 'guesses' made in the applicant's own time.
So if 3500 trials were needed, the applicant would then be free to carry out the test in their own time, perhaps at times when they felt most 'psychic', however long it takes, or even divide it between several 'psychics'. It would be extremely simple for the JREF to set up such a test by email.
Also, now that Randi has mentioned 1 in 100 for the preliminary, we (more or less) know what odds to aim at, now.

 

[Cuddles]

I think your answer helps explain why there are so few non-delusional challengers for the prize. The idea that, for example, anyone can ALWAYS paranormally pick out a red card from a selection of face up cards is preposterous on the face of it (so to speak ). However, it may be possible for someone to pick out a red card at a significantly above chance rate, even if that rate is only 51-52% over thousands of trials. Would such a person apply for the prize, when there is not even a hint in the official rules of what level (s)he must perform at even to pass the preliminary test, let alone win the million dollars? The point is that, by not specifying an odds standard, serious challenges for the prize are discouraged. As a person involved in Ganzfeld experiments told me, the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test."

All of which is utterly irrelevant. It makes no difference at all whether someone claims to be 100% accurate or 51% accurate. If they can do what they claim, it doesn't matter in the slightest what the possibility of them passing by chance is. The only way chance can matter is if they are actually relying on chance and not their claimed ability. I agree that this explains why there are no non-delusional challengers, but not in the way that you seem to think.