Information on Protocol for Pavel Ziborov Applicant

[rjh01]

I think pavel_do wins the award for having the longest time between the first post on this forum about him and JREF closing the file, without actually getting anything close to agreeing on a protocol. It is over two years since this post http://www.internationalskeptics.com/forums/showthread.php?postid=2597233#post2597233

 

[GzuzKryzt]

I apologize for not reading all of the available information, so please excuse me if this has been addressed before:

What does it have to be a binary test? As I understand the protocol, which could be wrong, he gets pairs of pictures from a pool of 20 possible photos. He cannot see the pictures. He is told what the two pictures are (boat/dog, cheesburger/toaster, etc). He then chooses one picture to identify and states which one it is of the pair. Lather, rinse, repeat.

So why not start with all 20 photos in a set? Pick one to identify. Lather, rinse, repeat a whole lot fewer times. Is there something about the way the ability works that makes this unworkable?

Don't rock the boat, UY.

But of course you're right. Once Pavel knows what he can do and tells the people what he can do in clear concise language, we could take it from there.



I dare you to browse through the two - and as of yesterday: three available threads discussing Pavel and his claims to count the changes in Pavel's claims, if he ever came to state something of a precise claim. Yes, some tinkering happened after he was told what would work for the JREF and what wouldn't.

 

[Uncayimmy]

Don't rock the boat, UY.

LOL! Sorry.

But of course you're right. Once Pavel knows what he can do and tells the people what he can do in clear concise language, we could take it from there.

I do have another question, though. The JREF now requires academic affidavits that state that they saw the claimant do what he said he could do. Well, what exactly did the university pyschology department see him do?

I dare you to browse through the two - and as of yesterday: three available threads discussing Pavel and his claims to count the changes in Pavel's claims, if he ever came to state something of a precise claim. Yes, some tinkering happened after he was told what would work for the JREF and what wouldn't.

Did it have anything to do with kidneys? I only understand kidneys and whether they are missing or not.

I went back through some stuff. Quite frankly, I keep getting lost. It seems like he holds the envelopes and gets some sort of mental picture. That is, except when he doesn't. Then again, sometimes his brain tries to draw pictures, which are not "real" like the other times when his brain draws the pictures. The images are not always clear, so the photos need to be as dissimilar as possible. Sorta like if I looked at a blurry photo I could distinguish a kangaroo and a giraffe but not a small dog and a big rat.

He also seems to get "feelings" about what the images are...almost a Vision From Feeling, if you will. Other times (or maybe all the time) he feels a sensation of warmth from the envelope, but I'm not sure how that ties into which image it is.

And apparently despite the various perceptions he has, they don't seem to be strong enough to get a reading when there are more than a few choices available at the time. It seems he can make a prolonged series of of binary choices, just not one choice out of the same number of photos.

Am I understanding that correctly? If so, what's the hold-up?

 

[fromdownunder]

Here is your basic problem Pavel. You have already asked for your challege file request to be closed. I am quoting exactly what you said from the main thread relating to your claim, and highlighting exactly what you asked for:

I do not agree with the protocol that from the beginning guarantees my failure. I have nothing more to say but to close my file as I see no more point in negotiations with JREF.

You asked, they did. Pretty basic really.

Norm

 

[GzuzKryzt]

...
I do have another question, though. The JREF now requires academic affidavits that state that they saw the claimant do what he said he could do. Well, what exactly did the university pyschology department see him do?

I wish I could Jack-Bauer-it-up to find out the answer to that. However, I - and the world - will survive without it.

Did it have anything to do with kidneys? I only understand kidneys and whether they are missing or not.

I went back through some stuff. Quite frankly, I keep getting lost. It seems like he holds the envelopes and gets some sort of mental picture. That is, except when he doesn't. Then again, sometimes his brain tries to draw pictures, which are not "real" like the other times when his brain draws the pictures. The images are not always clear, so the photos need to be as dissimilar as possible. Sorta like if I looked at a blurry photo I could distinguish a kangaroo and a giraffe but not a small dog and a big rat.

He also seems to get "feelings" about what the images are...almost a Vision From Feeling, if you will. Other times (or maybe all the time) he feels a sensation of warmth from the envelope, but I'm not sure how that ties into which image it is.

And apparently despite the various perceptions he has, they don't seem to be strong enough to get a reading when there are more than a few choices available at the time. It seems he can make a prolonged series of of binary choices, just not one choice out of the same number of photos.

Am I understanding that correctly? If so, what's the hold-up?

Apparently, the JREF wanted to proceed in this case and Randi basically set an ultimatum. Understandable? Perhaps. Sensible, congruent and fair? Perhaps not.

When will we see a JREF response in the forum?



Don't anybody get me wrong: Pavel can waffle with the best of them and he did his fair share of it for two years. I can't shake the suspicion that he doesn't show his full deck of cards - and in stark contrast to some other applicants he does have one.

 

[Cuddles]

It doesn't make sense to suggest your average success rate, because that limits you to an approximately 50% chance of passing the test. It makes sense to suggest the lowest number that you think the JREF will accept and one that is lower than your success rate. Then you might have more like a 90% chance of passing the test, assuming you are right about your success rate. That is how Starz came up with 67%

Actually, it's not. In fact, I just went and had a bit of a more thorough look through the past stuff, and I was wrong earlier. The 67% claim was not actually the average of the tests I mentioned. In fact, Pavel's reported accuracy at that point was only 60% (out of 140 pairs). In other tests, reported here and here, he has claimed 66% and 70% accuracy. Overall, his average accuracy out of all the tests I'm aware of him reporting, he has achieved slightly less than 62% accuracy.

It may not make any sense to only suggest your average success rate, but Pavel is doing worse than that and actually suggesting a success rate that he has not achieved at all. If he wanted to propose something significantly below his average success rate to have a reasonable chance of winning, he would have to claim 50% accuracy. On guessing pairs. I don't see the JREF going for that personally.

 

[fls]

Actually, it's not. In fact, I just went and had a bit of a more thorough look through the past stuff, and I was wrong earlier. The 67% claim was not actually the average of the tests I mentioned. In fact, Pavel's reported accuracy at that point was only 60% (out of 140 pairs). In other tests, reported here and here, he has claimed 66% and 70% accuracy. Overall, his average accuracy out of all the tests I'm aware of him reporting, he has achieved slightly less than 62% accuracy.

I agree, and that's what I said in my post (the part you snipped). The 67% seemed to have been chosen as a number which would be acceptable odds-wise to the JREF. And it was accepted by Pavel because he had achieved higher than that on some of his tests (rather than representing an average which he calculated).

Linda

 

[steenkh]

Actually, it's not. In fact, I just went and had a bit of a more thorough look through the past stuff, and I was wrong earlier. The 67% claim was not actually the average of the tests I mentioned. In fact, Pavel's reported accuracy at that point was only 60% (out of 140 pairs). In other tests, reported here and here, he has claimed 66% and 70% accuracy.

So he has actually achieved results that are not statistically far from the expected 50%, but because it gives him a remote shot at a million dollars, he graciously accepts 67%.

Now he is miffed that this has not impressed the JREF?

 

[Startz]

Actually, it's not. In fact, I just went and had a bit of a more thorough look through the past stuff, and I was wrong earlier. The 67% claim was not actually the average of the tests I mentioned. In fact, Pavel's reported accuracy at that point was only 60% (out of 140 pairs). In other tests, reported here and here, he has claimed 66% and 70% accuracy.

So he has actually achieved results that are not statistically far from the expected 50%, but because it gives him a remote shot at a million dollars, he graciously accepts 67%.

Now he is miffed that this has not impressed the JREF?

For many folk, the results of probability calculations are nonintuitive. (Or maybe I should just speak for myself. I'm often surprised.)

The probability of randomly getting 60% right in 140 trials is 0.016. While one might not offer a million dollar bet against those odds, by most standards that is "statistically far from the expected 50%."

 

[Uncayimmy]

For many folk, the results of probability calculations are nonintuitive. (Or maybe I should just speak for myself. I'm often surprised.)

The probability of randomly getting 60% right in 140 trials is 0.016. While one might not offer a million dollar bet against those odds, by most standards that is "statistically far from the expected 50%."

First off, your math is wrong. You gave the probability of getting 60 correct, not 60%. The probablity of getting 60%, which is 84 correct, is 0.004.

I don't think either number is statistically far from the expected 50% at all. Look at the chart below. The larger black box is where the probability of getting 60% lies. It's clearly within the meat of curve. If you want to talk about what's "far off" the scale, you need to start looking where you can't even see the line anymore.



You can think of it another way. Suppose you're giving a lecture to 100 freshman about stats. You have them flip a coin 140 times each. You're not going to be very surprised if somebody has 60% of them turn up heads. You might raise an eyebrow, but it's not terribly surprising. If you test this person again, odds are they will get a much lower score. If they get a similar high score, that's when you get excited.

Even then, what does that really mean? Suppose the person says they have a skill that allows them to flip the coin and have it come up heads. Does this mean they can do it 60% of the time? Absolutely not. Remember, we expect to see heads about 50% of the time anyway. Their "influence" on the coin flip is very small. If the ability were real, it would mean the skill only worked about 5% to 10% of the time (a guess...didn't crunch the numbers).

Stated another way, if the person could influence the flip 60% of the time, then 84 of the flips would be due to their ability, not chance. That leaves the other 56 flips to chance. This means another 28 or so flips will come up heads by chance. That means we would expect 112 heads for a total of 80% for someone who could influence the flip 60% of the time.

If that's hard to grasp intuitively, think about a multiple choice (4 options) test on identifying 100 people by name. Suppose you don't know anybody pictured on the test. You'll get about 25% correct. Does that mean you know 25% of the people? No.

Now let's suppose you know 10% of the people pictured. What will your score be? Will it be 10%? No. You'll get 10 correct due to knowledge. You'll also get 25% of the rest due to guessing. 25% of 90 is 23. Thus, your score will be around 33. So someone who knows 10% of the questions on that test should get about 33% correct.

This, of course, brings us back to what is meant by far off stastically. These small effect sizes get lumped together with similar scores in the meat of the curve. You need a lot of trials before you can be confident that you're seeing a small effect rather than chance.

 

[Startz]

First off, your math is wrong. You gave the probability of getting 60 correct, not 60%. The probablity of getting 60%, which is 84 correct, is 0.004.

The probability of getting 84 or more correct out of 140 is 0.011. I thought the "or more" was implicit. Sorry for any confusion.

Geek alert:

I originally wrote 0.0160 rather than 0.011. The small difference was random. No really, it was. I got it from a Monte Carlo simulation and hadn't run enough trials. The 0.011 was verified by both a longer simulation run and by direct evaluation of the incomplete beta function.

 

[fls]

The probability of getting 84 or more correct out of 140 is 0.011. I thought the "or more" was implicit. Sorry for any confusion.

Geek alert:

I originally wrote 0.0160 rather than 0.011. The small difference was random. No really, it was. I got it from a Monte Carlo simulation and hadn't run enough trials. The 0.011 was verified by both a longer simulation run and by direct evaluation of the incomplete beta function.

Show off.



Linda

 

[Uncayimmy]

The probability of getting 84 or more correct out of 140 is 0.011. I thought the "or more" was implicit. Sorry for any confusion.

Using the binomial distribution function in Excel, the probabliity of getting exactly 60 correct is 0.016, which is the number you wrote for getting 60% or more correct. What are the odds of that?

But the larger point is that a 1/100 chance is not a big deal to overcome. If you watch poker on TV you'll see people overcoming those odds on the river about once every 100 hands, which is to say it happens all the time in the tournaments.

 

[Startz]

Using the binomial distribution function in Excel, the probabliity of getting exactly 60 correct is 0.016, which is the number you wrote for getting 60% or more correct. What are the odds of that?

But the larger point is that a 1/100 chance is not a big deal to overcome. If you watch poker on TV you'll see people overcoming those odds on the river about once every 100 hands, which is to say it happens all the time in the tournaments.

Certainly not enough to base a Challenge on!

 

[Uncayimmy]

Certainly not enough to base a Challenge on!

It might work for a $50 challenge where applications cost $1.

If you are trying to decide between random chance and possible effect, 1 in 100 doesn't tell you a damned thing other than "keep on testing."

 

[Cuddles]

The important point is not so much whether 60% success is significantly different from 50%, it is the point fls initially raised that if you only test for the average success rate, you only have a 50% chance of passing even if you have a genuine ability. In Pavel's case, several of his self-tests had a 50% or lower success rate.

As someone mentioned earlier, the JREF is certainly interested more in avoiding false positives than false negatives, but if they want the challenge to mean anything at all they have to pay attention to both. It's not enough to say that there's a 1/1000 chance of Pavel passing due to chance, you also have to ensue that there's a reasonably low possibility of him failing due to chance. The former is fairly easy to work out, the latter is rather a lot harder if Pavel can't tell us whether he is accurate 90%, 70% or 50% of the time.

 

[EHocking]

The important point is not so much whether 60% success is significantly different from 50%, it is the point fls initially raised that if you only test for the average success rate, you only have a 50% chance of passing even if you have a genuine ability. In Pavel's case, several of his self-tests had a 50% or lower success rate.

As someone mentioned earlier, the JREF is certainly interested more in avoiding false positives than false negatives, but if they want the challenge to mean anything at all they have to pay attention to both. It's not enough to say that there's a 1/1000 chance of Pavel passing due to chance, you also have to ensue that there's a reasonably low possibility of him failing due to chance. The former is fairly easy to work out, the latter is rather a lot harder if Pavel can't tell us whether he is accurate 90%, 70% or 50% of the time.

I'm terrible at probability, so rely on these tables at http://www.automeasure.com/chance.html .

For a 1/2 probability outcome, for 100 trials

at odds of
1:100 the expected successes are 37-63
1:10,000 the expected successes are 31-69

Anything outside that range, i.e. 30 right or 70 right is outside the probability attributable to random chance.
(ETA: 1:1,000 is about 34-66?)

The difference is that Pavel claims only to be significantly successful, not significantly UNsuccessful.

It would be interesting to see the JREF reaction to this protocol if I was to say, I CANNOT get more than 30 right out of 100 trials.
Mathematically they would have to accept it - but would they be satisfied that this "ability" is paranormal?

 

[steenkh]

As someone mentioned earlier, the JREF is certainly interested more in avoiding false positives than false negatives, but if they want the challenge to mean anything at all they have to pay attention to both.

The JREF is not trying to provide proof of the existence of the paranormal. The stated purpose is for the claimant to "show, under proper observing conditions, evidence of any paranormal, supernatural, or occult power or event". With this definition they do not need to worry about false negatives; that will be for the claimant to worry about.

 

[Cuddles]

The JREF is not trying to provide proof of the existence of the paranormal. The stated purpose is for the claimant to "show, under proper observing conditions, evidence of any paranormal, supernatural, or occult power or event". With this definition they do not need to worry about false negatives; that will be for the claimant to worry about.

No, it's not just for the claimant to worry about. Under what circumstances does a 50% chance of success constitute "proper observing conditions"? If the JREF actually administered tests that even people with superpowers are likely to fail, they would be guilty of exactly the kind of rigging they are so often accused of.

This isn't normally a problem since the majority of applicants seem convinced that they are never wrong, or at worst only a small proportion of the time. In fact, they often have to be persuaded to accept a lower threshold for success in order to make the test fairer in exactly this way. The only difference here is that Pavel appears to have such a low success rate, and the rate is so variable, that avoiding false negatives becomes much harder and actually has to be considered as a separate point rather than just falling out of the protocol design.

It would be interesting to see the JREF reaction to this protocol if I was to say, I CANNOT get more than 30 right out of 100 trials.
Mathematically they would have to accept it - but would they be satisfied that this "ability" is paranormal?

But that ability is identical to the one being claimed, not just mathematically but in practice as well. All you have to do is take the test but look at the picture that is not chosen and you've got more than 70 correct.

 

[fls]

The JREF is not trying to provide proof of the existence of the paranormal.

I agree. The purpose of the Challenge is to provide an incentive for people to step up to the plate. And the purpose of that is to provide a demonstration of what happens to the sensation of certainty and the influence of chance and bias, when blinding, controls and systematic record-keeping are used. It would be nice if it also served the purpose of persuading the Challenger that they were mistaken, but that never happens. Instead, it serves as more of an educational exercise for the benefit of the audience. To that end, it doesn't matter whether or not it is fair for the claimant. It only needs to be something that the JREF can get the claimant to agree to. Because let's get real here. No matter how fair and conciliatory the JREF is, the claimant will not believe the results of the test when they fail, so there's no point in worrying about false negatives for the claimant's sake. And for the purposes of education, if someone is already knowledgeable enough to realize that it isn't a fair test for the claimant, they are probably already knowledgeable enough to realize why these claims are false in the first place (i.e. they are not sitting on the fence before they watch the test). So it remains that the JREF only needs to give the appearance of fairness to a naive and generally receptive audience. And I suspect that it doesn't take all that much to accomplish this, given that most people do not understand Type II error anyway.

To that end, the rest of us should probably shut up.

Linda