Merged Odds Standard for Preliminary Test

[steenkh]

That minimum standard should be an intellectual, substantive issue, related to what is necessary to ensure results beyond chance, not related to the pragmatics of protocal negotiations.

Why? The MDC, while using scientific methods, is not a scientific study. It is simply a challenge for people to do what they claim they can do. The JREF is betting a million dollars that this will not happen, and it is only reasonable to have low odds for a false positive.

Now it also seems a problem for some that the JREF sometimes accepts higher odds for false positives. Why?

It seems that rather than concentrate on the keeping the odds for a false negative low, people are trying to arrange for increasingly higher odds for false positives. Is this a sign of desperation?

The million is for the JREF to give, and the rules are clear: just perform as claimed. I have little respect for people who claim they can do something, and then it turns out that they can actually not do it with any certainty, and I do not see why they should be catered for with extra tests and more chances for success.

 

[Cuddles]

Firstly, what Steenkh said. This has been said many times already, but it bears repeating as often as is necessary. The probability of winning by chance alone is irrelevant to anyone who genuinely believes they have a paranormal ability. If they can actually do what they claim, chance doesn't come into it. The only time chance is relevant is if they can't do what they claim, in which case the JREF wants there to be as small a chance of a win as is practical. As has been pointed out, what is practical varies from test to test. Carina Landin was allowed a very high chance of winning by chance because her claim relied on fairly rare items, so much so that there were problems with the test and the JREF will allow her to retake it. On the other hand, Pavel's test uses easily obtained envelopes and photos, and it is therefore very easy to greatly reduce the probability of winning by chance.

I think one problem here is that there seems to be some confusion about the different probabilities involved. There are at least three very different probabilities:
1) The success rate of the applicant's ability;
2) The chance of a false positive;
3) The chance of a false negative.

People seem to regularly conflate 1) with both 2) and 3). The 70% (or whatever) rate given by Pavel is not the chance of him winning. It is simply the chance, given a single envelope, that he will identify what is inside. The 1:1000 odds (or whatever the actual value is for a test) is the chance of a false positive. The test can be arranged to have a 1:1000 chance no matter what the rate claimed in 1) actually is. 70%, 50%, 1%, whatever. Of course, the lower the success rate, the bigger and longer a test needs to be, so there reaches a point where it is no longer practical to test, even if you accept a higher chance of a false positive.

Obviously the JREF is most concerned about 2), since they don't want to give their money to someone who only won by chance. 1) is the applicants claim, and as long as it is testable, the JREF really doesn't care what they claim about it. 3) is important for the applicant and should be what they focus their attention on, but from the applications I've read, it seems only the JREF ever thinks about this, hence suggesting that an 80% success rate would win when 100% was claimed.

The point is, these are three very different things. 1) is completely independent of 2) and 3), and it is in the interest of both the JREF and any honest applicant to have both 2) and 3) as low as possible. If you are truly interested in deomstrating an ability you believe you have, you should want the chance of winning without an ability to be as low as possible, as long as that doesn't make the chance of losing despite the ability too high. Exactly what "as low as possible" and "too high" are wil vary depending on circumstances, but general arguments that the JREF is being unfair by trying to reduce both of them, as Rodney argues, just make no sense at all from either the JREF's or the applicants' point of view.

 

[drkitten]

Even though it's JREF's challenge, they set the rules, and they must agree on the protocol (as must the applicant), I don't see why an odds-based test should have different standards for different applicants, as a *substantive* matter.

That minimum standard should be an intellectual, substantive issue, related to what is necessary to ensure results beyond chance, not related to the pragmatics of protocal negotiations.

Don't do much bench science, do you? I believe it was Sir Peter Medawar (a Nobel laureate) who pointed out that "if politics is `the art of the possible,' science is `the art of the soluble.'" If you know beforehand that practical considerations prevent you from running an experiment, there's not much point in wasting time and energy on optimizing the theoretical design.

Scarcity of data, for example, is a very real problem for any experiment; if there are only 100 people in the world with a particular disease, it will be impossible to conduct large-scale clinical trials of your shiny new drug. If you need to run an experiment on the ground at an elevation of more than 7000 meters, there are fewer than 500 places world-wide that you can do it. If you need to run an experiment under 15,000 meters of water, you can't do it at all.

If you know that you can only get 200 points of data, you design the test to squeeze as much out of those 200 points as you can --- and if you need to accept slightly larger error bars, that's a substantive limitation.

 

[Paul2]

Why? The MDC, while using scientific methods, is not a scientific study.

I didn't say it was scientific.

It is simply a challenge for people to do what they claim they can do. The JREF is betting a million dollars that this will not happen, and it is only reasonable to have low odds for a false positive.

I didn't say low odds were a problem. I'm only arguing for the *same* minimum odds for any applicant.

It seems that rather than concentrate on the keeping the odds for a false negative low, people are trying to arrange for increasingly higher odds for false positives. Is this a sign of desperation?

That's not me, that's not what I said.

The million is for the JREF to give, and the rules are clear: just perform as claimed. I have little respect for people who claim they can do something, and then it turns out that they can actually not do it with any certainty, and I do not see why they should be catered for with extra tests and more chances for success.

I agree, I'm not arguing for extra tests or more chances for success.

 

[EHocking]

That's an evasion -- there should be a clear standard applicable to all odds-based applications.

I noticed that you ignored the rest of my post on this issue. Perhaps because it explains the situation?.

Let me repeat it.

Rule 1.
Applicant must state clearly in advance, and applicant and JREF will agree upon, what powers and/or abilities will be demonstrated, the limits of the proposed demonstration (so far as time, location and other variables are concerned) and what will constitute both a positive and a negative result.

Rule 3 qualifies the above with.
We will consult competent statisticians when an evaluation of the experimental design, is required

The rules are not meant to be taken in isolation (quote mining) as you are doing, they are to be taken in their entirity as they explain the process and JREF's stance on protocols and applications.

Up to this point, every such application appears to have been handled on an ad hoc "make it up as you go along" basis.

The Rules are not meant to be taken in isolation but referred to as a whole explanation of the Challenge.

You are quote mining here. Read ALL the rules in proper context.

Try reading (Pavel's) post #281 on this thread:

"When I really tried to find out the odds and things, by emailing to JREF I was refused any answers, reasoning that I need to apply firs and after that we start any negotiation.. even when I have asked .. if I claim sirtain % of a minimum rate (that JREF ask to state in application and description of the claim) even that was ignored." Pavel then inquired: "Will it be fine for the claim as the accuracy to state 70% minimum from my side?"

Fie on your post #281, I have already quoted his posts from the start of this thread before he applied for the Challenge.
From my post (#269) which prompted Pavel's response in #281:

He stated this in a number of posts prior to applying to the Challenge.

Quote:
Post 51 from his first thread.
WHAT would be self evident prove? how many times i have to perform it.. let say its an 1 hour test session..
so ill know what is minimum has to be performed..

Quote:
Post 241 after much discussion
As to the accuracy with which I can perform it, I think I have no choice but to claim that I can perform the results that beat 1 to 1000 odds as that what JREF want to be beaten. If I am not mistaken, or I have a choice? Like to claim I can beat 1 to 300 or even 400 odds? 200-300 odds that is way more than just by chance but that will not be considered as a success isn’t it? Or I misunderstood something from requirements?

It's quite clear that he came to the Forum to determine at what level he needed to perform his skill to, to pass the Challenge.

 

[Paul2]

The probability of winning by chance alone is irrelevant to anyone who genuinely believes they have a paranormal ability. If they can actually do what they claim, chance doesn't come into it. The only time chance is relevant is if they can't do what they claim. . . .

I can claim that I can hit a bullseye with a bow an arrow, but I can't do it 100% of the time. And, Pavel doesn't claim he can see inside envelopes with 100% accuracy. There are many activities that we commonly think people can do but they can't do 100% of the time.

. . . in which case the JREF wants there to be as small a chance of a win as is practical.

Huh? Can you explain this further? It sounds like something I don't think you mean.

As has been pointed out, what is practical varies from test to test. Carina Landin was allowed a very high chance of winning by chance because her claim relied on fairly rare items, so much so that there were problems with the test and the JREF will allow her to retake it. On the other hand, Pavel's test uses easily obtained envelopes and photos, and it is therefore very easy to greatly reduce the probability of winning by chance.

But surely there must be some minimum odds *that are independent of the particulars of a test* that, by themselves, argue that the results were or were not due to chance.

 

[Cuddles]

I can claim that I can hit a bullseye with a bow an arrow, but I can't do it 100% of the time. And, Pavel doesn't claim he can see inside envelopes with 100% accuracy. There are many activities that we commonly think people can do but they can't do 100% of the time.

Did you even bother reading my post? You've done exactly what I said people were doing and conflated points 1) and 3). As I said, that is utterly irrelevant. The whole point of my post was that the claimed success rate of an ability is independent of the probability of a false positive or negative. A test can be designed to give any false positive chance you like, no matter what success rate is claimed. Of course, whether such a test is practical or not depends on the specific claim.

Huh? Can you explain this further? It sounds like something I don't think you mean.

It means exactly what it says. If an applicant doesn't have an ability, the JREF doesn't want them to win. What part of that do you find hard to understand?

But surely there must be some minimum odds *that are independent of the particulars of a test* that, by themselves, argue that the results were or were not due to chance.

Why? I see people saying that there should be the same chance of false positive for all tests, but I don't see anyone giving a reason. From the applicant's point of view, the chance of false positive is irrelevant. From the JREF's point of view, the chance of false postive should be as small as possible, but since it's the JREF's money, they are free to accept whatever odds they like.

 

[Paul2]

Did you even bother reading my post? You've done exactly what I said people were doing and conflated points 1) and 3).

I see your point.

Why? I see people saying that there should be the same chance of false positive for all tests, but I don't see anyone giving a reason. From the applicant's point of view, the chance of false positive is irrelevant. From the JREF's point of view, the chance of false postive should be as small as possible, but since it's the JREF's money, they are free to accept whatever odds they like.

I think the issue crops up with the inevitable interpretation of someone passing the challenge. Admittedly, interpreting a successful challenge is beyond the terms of the challenge. But it sure would be inevitable.

Passing the challenge would not prove the existence of the paranormal, but it would be one piece of evidence, good or bad. In order to consider how good that piece of evidence would be for arguing that the paranormal existed, one question would be the odds of a false positive. So should there be different standard for one test versus another *in these terms, in this context?*

So arguing for some inherent standard for a false positive versus the pragmatic, variable approach is the difference between looking at the challenge narrowly (just one protocol between JREF and an applicant, on such and such terms) versus the larger implications of the challenge, especially if someone ever passed it.

I would accept the argument that the narrow, pragmatic approach is perfectly fine, and let interpretation take care of itself. But I think that's where the inherent standard idea comes from.

 

[drkitten]

Passing the challenge would not prove the existence of the paranormal, but it would be one piece of evidence, good or bad. In order to consider how good that piece of evidence would be for arguing that the paranormal existed, one question would be the odds of a false positive. So should there be different standard for one test versus another *in these terms, in this context?*

Absolutely, for several reasons.

First, it is in the JREF's best interest to have the test be as conclusive as practical. That's not the same thing, at all, as "as conclusive as possible." If the costs or effort involved in making an absolutely ironclad test is unreasonable, then they will (justifiably) settle for a merely zinc-clad or copper-clad test, on the grounds that they can't get ironclad. And they're certainly not going to go for unobtainium-clad when unobtainium is, by definition, not available.

In the case of the Landin test, for example, the number of diaries available limited the strength of the test that could be performed. I'm not sure that the Landin test would even qualify as "duct-tape-clad," but it was the best that could be arranged, and both Landin and and the JREF were willing to accept that limitation. When we're dealing with things as omnipresent as photographs and envelopes, it's hard to argue about material limitation.

Second, the JREF is not a scientific institution, but an educational one -- and if there is educational value in running a more lightly controlled test (as there would be, for example, if one of the "name" psychics were to attempt the challenge), then the JREF can, quite justifiably, tell the scientific purists to go hang. Especially since the difference between 1:200 and 1:1000 is not likely to make a difference in any one particular instance. But again, I don't see any particular educational value in testing Pavel per se; he hasn't got the huge following of Sylvia or Uri, and showing him to be mislead would not have nearly as much the impact.

 

[Paul2]

Absolutely, for several reasons.

First, it is in the JREF's best interest to have the test be as conclusive as practical.

I think I get it now. Sorry, Rodney, I tried to see your side, but it just ain't gonna work.

 

[GzuzKryzt]

Absolutely, for several reasons.

First, it is in the JREF's best interest to have the test be as conclusive as practical. That's not the same thing, at all, as "as conclusive as possible." If the costs or effort involved in making an absolutely ironclad test is unreasonable, then they will (justifiably) settle for a merely zinc-clad or copper-clad test, on the grounds that they can't get ironclad. And they're certainly not going to go for unobtainium-clad when unobtainium is, by definition, not available.

In the case of the Landin test, for example, the number of diaries available limited the strength of the test that could be performed. I'm not sure that the Landin test would even qualify as "duct-tape-clad," but it was the best that could be arranged, and both Landin and and the JREF were willing to accept that limitation. When we're dealing with things as omnipresent as photographs and envelopes, it's hard to argue about material limitation.

Second, the JREF is not a scientific institution, but an educational one -- and if there is educational value in running a more lightly controlled test (as there would be, for example, if one of the "name" psychics were to attempt the challenge), then the JREF can, quite justifiably, tell the scientific purists to go hang. Especially since the difference between 1:200 and 1:1000 is not likely to make a difference in any one particular instance. But again, I don't see any particular educational value in testing Pavel per se; he hasn't got the huge following of Sylvia or Uri, and showing him to be mislead would not have nearly as much the impact.

Well said.

 

[Rodney]

I think I get it now. Sorry, Rodney, I tried to see your side, but it just ain't gonna work.

You were right the first time. But let's hope that the JREF can finally muster the energy to test Pavel, even it it can't explain its odds standard.

 

[Moochie]

Did you even bother reading my post? You've done exactly what I said people were doing and conflated points 1) and 3). As I said, that is utterly irrelevant. The whole point of my post was that the claimed success rate of an ability is independent of the probability of a false positive or negative. A test can be designed to give any false positive chance you like, no matter what success rate is claimed. Of course, whether such a test is practical or not depends on the specific claim.



It means exactly what it says. If an applicant doesn't have an ability, the JREF doesn't want them to win. What part of that do you find hard to understand?



Why? I see people saying that there should be the same chance of false positive for all tests, but I don't see anyone giving a reason. From the applicant's point of view, the chance of false positive is irrelevant. From the JREF's point of view, the chance of false postive should be as small as possible, but since it's the JREF's money, they are free to accept whatever odds they like.

I get really annoyed by the specious arguments put forward to re-jig the stats. It's perfectly obvious to me that what we are after is an actual demonstration of a "paranormal" ability, one that we can confidently say did not happen by chance.

If I were the JREF, I wouldn't even countenance testing the kind of "ability" being claimed here, any more than I would test the ability of a master gambler. It's looking more and more like there is no actual "paranormal ability" on offer here, but only a calculated risk of successful guesswork.


M.

 

[GzuzKryzt]

I get really annoyed by the specious arguments put forward to re-jig the stats. It's perfectly obvious to me that what we are after is an actual demonstration of a "paranormal" ability, one that we can confidently say did not happen by chance.

If I were the JREF, I wouldn't even countenance testing the kind of "ability" being claimed here, any more than I would test the ability of a master gambler. It's looking more and more like there is no actual "paranormal ability" on offer here, but only a calculated risk of successful guesswork.


M.

...which would of course perfectly illustrate one reason why the MDC was discontinued: It has served its purpose. By attracting people making claims like the one Pavel makes.

 

[catbasket]

You were right the first time. But let's hope that the JREF can finally muster the energy to test Pavel, even it it can't explain its odds standard.

Strange, could have sworn I recently read a thread which showed there is no odds standard. If only I could recall where that was ...

 

[EHocking]

Strange, could have sworn I recently read a thread which showed there is no odds standard. If only I could recall where that was ...

er... I think it was the very first response to Rodney's OP in this thread?

Post 2 Quote: GzuzKryzt
Experience suggests that in most cases, one in 1000 chances are reflected in the protocols.
Since every claim seems to be different, however, the JREF has not set the standard at a certain probability mark. It depends on the claim.

No idea why it has taken 332 subsequent posts for Rodney not to see this point.

 

[drkitten]

I get really annoyed by the specious arguments put forward to re-jig the stats. It's perfectly obvious to me that what we are after is an actual demonstration of a "paranormal" ability, one that we can confidently say did not happen by chance.

I think I have more sympathy for the pro-paranormalists (in general) than you have. I think it's obvious to anyone with the sense of a cucumber that if the paranormal exists, it's a very subtle thing. I can't simply scratch names on onions and see whom I'm going to marry or spin a set of coins to tell me if I will get a better job next year.

But it's also obvious to those same cucumbers that the real world is very subtle; I can't just walk into a pharmacy and grab a drug to make me feel better, which is why medical school takes several years. So it's obvious to any thinking paranormalist (which I admit is a very small group) that demonstrating that the paranormal exists will require a very deft touch and a very sensitive experiment.

A major research project, in fact.

The flip side of that is that such projects are expensive to run. I could easily burn through a million bucks in a year testing remote vision. Well, I could , if I had that million bucks -- but the NSF wouldn't touch that proposal with a hayfork. The only group that appears willing to front large amounts of research money for this purpose is the JREF.

Having said that, that's not what the JREF does, and it's no more willing to shell out a million for psychic research than the NSF. But I think people get more upset about their misunderstanding of the JREF than their (true) understanding of the NSF.

I think Rodney, in particular, is misunderstanding what the JREF does. And it ticks him off that an organization run by a magician is more into showmanship and fraud detection than it is into bench science. Poor thing.

 

[petre]

So what do you suppose is the lowest standard that the JREF would accept in an odds-based preliminary test? In Carina Landin's case, it was 1:170. Would they go as low as 1:100? 1:50? 1:10? Or what???

Though I have little in the way of evidence (most of what is available in that direction has been presented already), I suggest it is quite likely the JREF is willing to lower the bar (either on odds for the prelim test, or skipping it entirely) based on the media presence and employment history of the applicant. For someone like Pavel, who has been pretty honest and fair in dealing with other people, the bar's a bit rigid and unlikely to move. He appears to be almost unknown (at least as far as his ability) outside of people he's had personal contact with. Ms. Landin (from memory) did a fair bit of reading for folks and had a small, but established, following. For someone like Sylvia Browne, I imagine JREF would be willing to go as far as skipping the preliminary and maybe even accepting 1:100 odds, though only they could say for sure (Sylvia has only to ask...).

If Pavel had spent the last 10 or 30 years attempting to make a living from his ability, perhaps even overstating it often in order to dupe foolish people into giving him lots of money, writing (or ghost-writing) several "best-selling" books with a lot of pages but not much content, and appearing on many media outlets that feel using words like "maybe" and "perhaps" absolves them of any journalistic responsibility when reporting sensationalism as fact, Pavel might find it easier now to get JREF to give a little for the chance to expose such a history for the fraud it is.

You may find it an unfortunate inequality, but when the world looks at Sylvia Brown more like it regards Pavel, the JREF may be more willing to treat them as equals as well.

 

[Rodney]

I think Rodney, in particular, is misunderstanding what the JREF does.

The JREF contributes to that misunderstanding by stating:

"The Foundation is committed to providing reliable information about paranormal claims. It both supports and conducts original research into such claims.

"At JREF, we offer a one-million-dollar prize to anyone who can show, under proper observing conditions, evidence of any paranormal, supernatural, or occult power or event."

If you're right, that language should be changed to:

"The Foundation is committed to providing reliable information about some paranormal claims, such as the claims of psychic celebrities who appear on TV. It both supports and conducts original research into such claims, but does not have the time or inclination to more seriously investigate the paranormal.

"At JREF, we offer a one-million-dollar prize to anyone who can show, under proper observing conditions, evidence of a limited number of paranormal, supernatural, or occult powers or events."

 

[Rodney]

For someone like Sylvia Browne, I imagine JREF would be willing to go as far as skipping the preliminary and maybe even accepting 1:100 odds, though only they could say for sure (Sylvia has only to ask...)

Let's suppose the JREF were dumb enough to do that. What would it prove that someone managed to beat 1:100 odds?