Complaints Dept: Connie Sonne MDC

[fls]

I'm not sure what you are getting at.. Are you claiming that that is what happens? That tests are made so that even if the claimant can do what she claims, she will not pass?

Yes. The power (the probability that an ability will be detected when that ability is present) of many tests is about 0.50 and for many tests, it is even lower. This is the result of setting the threshold for passing at or above the level that can reasonably be assumed based on the claims made. The power isn't at a level which would make it easy for the claimant to pass (such as 0.8 or 0.9).

In that case, I'll beg to differ. All trials where I have followed the details, it has been evident that IF the claimant could perform as advertized, then they would have passed the test. Do you have an example of the other?

Hans

Can you give an example where the power level was greater than 0.60? An example otherwise would be the one I offered at the start of this conversation.

http://www.internationalskeptics.com/forums/showthread.php?postid=5032589#post5032589

I can also go through the Challenge protocols and show you examples where the threshold for passing was set at the same level as was claimed by the challenger.

Linda

 

[MRC_Hans]

Well I can perform and latest protocol with 100 pairs is suitable enough for that and it has been reviews by independent statistician who previously worked with JREF and he found it good enough.. Still.. I am not given FAIR chance to perform what I claim! The JREF sew that seems like I can be a bit of a pain in the neck so to say..and possible could show the results that make every one wonder.. so they wanted me in the end to accept there protocol that guaranties my failure! And they not bothered to explain why the previous one was not good enough, beside mentioning that its too expansive that is not good enough as I pay for it any way.

I haven't followed your protocol negotiations (if public at all), but in some I have followed, the protocol suggested by the claimant was statistically sound, but the way the test was suggested left room for cheating. So the JREF tightened the procedure, whereafter the claimant basically (not saying it directly, of course) found they would not be able to live up to their claimn and backed out (or obfusciated the negotiations till JREF had to close them down).

Hans

 

[fls]

Sorry I'm really not following this, can you try to explain again(!) (in no more than A level maths!) how if I can do what I say I can do the challenge is biased against me demonstrating my ability.

I rest my case.

If this seems a bit nonsensical to a clever and involved person such as Darat, why are we expecting the average claimant to grasp this point all on their own? In fact, this whole conversation makes my case for me. I have linked to a more detailed explanation and I've attempted some elaboration on this point, and yet most people still seem to fail to grasp it. If this is "so easy a caveman could do it", then why am I still having to argue it?

Linda

(I will elaborate on 'power' in a bit for you, Darat.)

 

[Ivor the Engineer]

Sorry I'm really not following this, can you try to explain again(!) (in no more than A level maths!) how if I can do what I say I can do the challenge is biased against me demonstrating my ability.

When people claim to be able to do something at a particular level of performance, what they typically mean is on average they achieve that level of performance. Sometimes they will perform better, other times they will perform worse. E.g., I can drive to work in 20 minutes. Sometimes it only takes me 15 minutes, other times it takes me 25 minutes or longer.

By setting the threshold for passing the challenge at their claimed (average) level of performance, there is a 50% chance they will perform below this level and fail the test.

 

[Startz]

Linda does a great job of explaining this. Probably better than I can, but let me give it a try.

Suppose an applicant says he can recite two lines of Klingon, flap his arms, and fly like a bird. All we'd ask of a protocol would be that he do this once, with appropriate controls in place.

But suppose the applicant admits that his Klingon accent is imperfect, and that he only achieves lift-off about one time out of three. Would it still be reasonable to have a protocol in which he only gets one chance?

Follow Linda's logic here. If the applicant is right, one chance means he'll probably fail. Three shots at flight mean he has a good chance of winning (about 70 percent).

From JREF's point of view, the only limit on the number of tests should be the time involved. Heck, if he can ever fly it's paranormal.

My view is that the purpose of the Challenge is to stand as a rebuke to those who claim paranormal abilities, by saying "prove it." If JREF tells our Klingon flapper that he only gets one chance, then it's a pretty wimpy rebuke.

 

[Darat]

When people claim to be able to do something at a particular level of performance, what they typically mean is on average they achieve that level of performance. Sometimes they will perform better, other times they will perform worse. E.g., I can drive to work in 20 minutes. Sometimes it only takes me 15 minutes, other times it takes me 25 minutes or longer.

By setting the threshold for passing the challenge at their claimed (average) level of performance, there is a 50% chance they will perform below this level and fail the test.

But isn't it in the case of (at least) some claims the JREF and the applicant set the success criteria below their "claim rate"? So if I said "I can move a pen across the table without any physical contact whenever I want"" the challenge would be set so that I'd only have to do it 5 times out of ten. Doesn't that in effect balance out this negative bias?

 

[Darat]

Linda does a great job of explaining this. Probably better than I can, but let me give it a try.

Suppose an applicant says he can recite two lines of Klingon, flap his arms, and fly like a bird. All we'd ask of a protocol would be that he do this once, with appropriate controls in place.

But suppose the applicant admits that his Klingon accent is imperfect, and that he only achieves lift-off about one time out of three. Would it still be reasonable to have a protocol in which he only gets one chance?

Follow Linda's logic here. If the applicant is right, one chance means he'll probably fail. Three shots at flight mean he has a good chance of winning (about 70 percent).

From JREF's point of view, the only limit on the number of tests should be the time involved. Heck, if he can ever fly it's paranormal.

My view is that the purpose of the Challenge is to stand as a rebuke to those who claim paranormal abilities, by saying "prove it." If JREF tells our Klingon flapper that he only gets one chance, then it's a pretty wimpy rebuke.

As far as I am aware the applicant wouldn't only get the one chance if that was their claim rate.

 

[fls]

Sorry I'm really not following this, can you try to explain again(!) (in no more than A level maths!) how if I can do what I say I can do the challenge is biased against me demonstrating my ability.

I'm going to use proportion correct as an example, since that is one of the most frequent types of claims made ("I can tell whether a kidney is missing or absent, what the card is, which envelope contains a photograph of a card, etc.). The outcome assuming they don't have the ability is fairly straightforward - it shouldn't be any better than the outcome due to chance (i.e. the 'odds').

What is more tricky is looking at the outcome assuming that they do have the ability. First we need to determine the effect size of their ability. Many claimants come here stating that they have 100 percent accuracy, but since this usually represents their success under uncontrolled conditions, it is reasonable to suspect that their success rate will be less under controlled conditions, especially considering that normal factors such as nervousness will also influence performance. Since the MDC test will be taking place under controlled conditions, the claimant needs to figure out their success rate under those conditions. Pavel performed some controlled tests and took the average of those tests when he claimed a 67 or 70 percent success rate. Since the claimants tend to be statistically naive, I would suspect that the various success rates quoted would represent other measures as well - the best they've ever done, the worst, the median or mode, a range, etc.

If we were to repeat the Challenge testing over and over again and if we set the threshold at the claimed success rate, whenever the success rate represents the central tendency, about half the time the results will fall below the threshold (as Ivor explained). Taking into consideration that the claimant is more likely to perform a bit more poorly under stressful conditions, or that they have likely given an exaggerated number for their success rate, you can expect that considerably more than half the time the results will fall below the threshold. There is also a tendency for Randi to ask for a somewhat higher threshold than the claimed success rate in order to maintain high odds while keeping the number of trials to a minimum. And claimants seem to judge whether or not that is achievable by whether they have ever performed that well, rather than by whether they usually perform that well.

Linda

 

[fls]

But isn't it in the case of (at least) some claims the JREF and the applicant set the success criteria below their "claim rate"? So if I said "I can move a pen across the table without any physical contact whenever I want"" the challenge would be set so that I'd only have to do it 5 times out of ten. Doesn't that in effect balance out this negative bias?

Yes, one of the solutions I have proposed is that the success rate be set lower than the claimed success rate. It can be worked out just how much lower will give a power of 0.8 to 0.9, which would be very fair. This may require that the number of trials be increased in order to make sure that success is very unlikely due to chance.

Linda

 

[SkepticScott]

I rest my case.

You seemed to have missed Darat's point. If a claimant claims to be able to do something much better than chance, how is a protocol that is designed so there is only a 1 in 1000 chance of the claimant passing by chance if the claimant doesn't have the power you claim biased against the claimant?

For example, if X claims to be able to control a coin flip, having it land the specified way 90% of the time, I could set up a test where:
(a) each trial consists of flipping a coin 15 times,
(b) a trial is a pass if the coin lands heads each of the 15 times
(c) the test consists of 20 trials. If X passes even one trial, X has passed the test.

By chance, there the chance of passing a trial by chance is 0.0000305. The chance of passing any of 20 trials is 0.0006102. This is less than one in 1000.

On the other hand, if X has the power X is claiming, the chance of passing a trial is 0.2058911. The chance of passing any one of 20 trials is 0.9900551. This is better than a 99% chance!

I think the JREF designs their tests this way, so it is very unlikely that someone would pass by chance, but very likely that they would pass if they have the power they claim.

If X claimed to be able to control the coin flip 100% of the time, then the test is even easier. Flip the coin 10 times, if it lands heads each time, X passes. If not, then X fails since X did not demonstrate 100% control.

I X claimed to be able to control the coin flip 51% of the time, then the test becomes much longer, but I'll be someone could design a test that has a 99% chance of being passed with that power, but a 0.1% chance of being passed by chance.

 

[Andrew Wiggin]

When people claim to be able to do something at a particular level of performance, what they typically mean is on average they achieve that level of performance. Sometimes they will perform better, other times they will perform worse. E.g., I can drive to work in 20 minutes. Sometimes it only takes me 15 minutes, other times it takes me 25 minutes or longer.

By setting the threshold for passing the challenge at their claimed (average) level of performance, there is a 50% chance they will perform below this level and fail the test.

I'd be in complete agreement with you if the average claimant here came in saying 'I can do this feat X percent of the time'. Connie said she could see through envelopes. So did Pavel. So did VFF, for a certain value of envelope. So did Edge, for coffee cans and dirt. So do all of the remote viewing and dowsing crowd. It's only after they realize that it's actually going to be rigorously tested that the statistics come out, and the protocol negotiations start to drag. I think this should be binary.

For your example: Don't test the claim 'I can drive to work in 20 minutes'. That's the secondary claim that appears when the primary claim of 'I can drive to work' falls through. If driving to work was a paranormal feat, either you could or you couldn't. If you say 'I can drive to work' and I say 'prove it', then you come back talking about time constraints, probability of getting there, acceptable failure rates that will still result in a 'getting to work' pass, multiple tries just in case, etc., then I'm going to start to doubt you can even drive or own a car...


A

 

[fls]

You seemed to have missed Darat's point. If a claimant claims to be able to do something much better than chance, how is a protocol that is designed so there is only a 1 in 1000 chance of the claimant passing by chance if the claimant doesn't have the power you claim biased against the claimant?

It doesn't. I have said that already multiple times in this thread. It is one of the points I have been trying to drive home. I'll repeat this so it is clear that I agree. A protocol designed so that there is only a 1 in 1000 chance of the claimant passing by chance (if the claimant doesn't have the claimed ability) has no effect on whether the protocol is biased against the claimant. It is possible to make a test that is both in favour of the JREF (if the claimant does not have the ability) and is in favour of the claimant (if the claimant does have the ability), as you go on to demonstrate.

For example, if X claims to be able to control a coin flip, having it land the specified way 90% of the time, I could set up a test where:
(a) each trial consists of flipping a coin 15 times,
(b) a trial is a pass if the coin lands heads each of the 15 times
(c) the test consists of 20 trials. If X passes even one trial, X has passed the test.

By chance, there the chance of passing a trial by chance is 0.0000305. The chance of passing any of 20 trials is 0.0006102. This is less than one in 1000.

On the other hand, if X has the power X is claiming, the chance of passing a trial is 0.2058911. The chance of passing any one of 20 trials is 0.9900551. This is better than a 99% chance!

Yes. This is exactly what I have said could be done.

I think the JREF designs their tests this way,...

Well, they don't. My point is, if you actually look at the tests which have been designed, they rarely give the claimant such a good chance of passing if they have the claimed ability. If the JREF designed tests like the one you outlined above, I would have no complaint.

Linda

 

[fls]

I'd be in complete agreement with you if the average claimant here came in saying 'I can do this feat X percent of the time'. Connie said she could see through envelopes. So did Pavel. So did VFF, for a certain value of envelope. So did Edge, for coffee cans and dirt. So do all of the remote viewing and dowsing crowd. It's only after they realize that it's actually going to be rigorously tested that the statistics come out, and the protocol negotiations start to drag.

I agree. And that's where we (us forum participants) have a chance to be valuable. Because we can help them form a testable claim and we can help them figure out what their succes rate will be under controlled conditions, instead of the uncontrolled conditions they are used to, which can then be used to determine whether or not protocol negotiations will be fruitful (a small effect size will need a large number of trials which Randi is unlikely to agree to unless the time needed for each trial is very small).

Linda

 

[Startz]

As far as I am aware the applicant wouldn't only get the one chance if that was their claim rate.

I spent a considerable amount of time helping Pavel put together a statistically sensible protocol, as did others here. It is certainly true that the details of Pavel's claim evolved a great deal in response to what people here on the forum suggested. But the protocol that was officially submitted was designed precisely to meet the JREF standard of avoiding a false positive while still giving Pavel a reasonable chance of success if his ability was as suggested. And JREF just plain blew him off.

Perhaps there were other reasons than statistical issues for JREF's actions, but there is no evidence to that effect. The only thing we know for a fact is that the response of JREF was to insist on a protocol that did not give Pavel much chance of succeeding if his claim were true.

 

[Paul2]

Well, they don't. My point is, if you actually look at the tests which have been designed, they rarely give the claimant such a good chance of passing if they have the claimed ability. If the JREF designed tests like the one you outlined above, I would have no complaint.

Linda

Can you offer a specific example?

 

[fls]

Can you offer a specific example?

There is the example I have referred to previously (that of Pavel).

http://www.internationalskeptics.com/forums/showthread.php?postid=5032589#post5032589

Linda

 

[fls]

Can you offer a specific example?

I've just been going through the first page (i.e. the most recent page) of MDC claims. Of those who have reached the stage where a protocol has been suggested and the claimant made a statement as to their success rate:

Claimant|Reported Success Rate|JREF Success Rate
Edge/Mike Guska|50 - 70%|70%
Raymond Powell|100%|100%
Chris Cordero|96%|44% (4/9)
Carina Landin|80%|80%
Connie Sonne|?|100%
Patricia Putt|?|50%
Natal'ya V.|100%|100%
Eileen McKusick|?|71% (5/7)
Pavel|67%|~90-100%

If anyone can fill in some of the question marks, please do so. The only reported success rates that were likely based on their own controlled tests were Mike Guska's and Pavel's (as far as I can tell).

Note that for Chris Cordero, the JREF provided a test which would be close to what I am suggesting (with the exception that Chris' reported success rate was probably not determined by him under controlled conditions).

Linda

 

[The Man]

If someone says that they will "test" you, does it not imply that if you can do what you say you can do, you will pass the test, and if you can't, you will fail? Isn't that what "test" means? Instead, most Challenge protocols make it unlikely that you will pass regardless of whether you have the ability. Like you, most claimants seem to think that they understand the Challenge, and like you, they fail to notice that it isn't really a 'test' of their ability. If it is reasonable to make this sort of 'test' and expect claimants to be willing to take up the challenge anyway, then there should be no problem with simply making this explicit. If there is nothing to hide, then there is no need to hide behind obfuscatory language.

Linda

Not “test”, challenge. Just what do you think I‘m failing to notice? If I claim I have a way of reading cards inside envelopes with 100% accuracy then I should not fail at all, unless I can’t really read cards inside of envelopes with 100% accuracy. I imagine that you might claim that you can read a card if I show you it’s face. How many cards do you think you will get wrong under those conditions? Now if I propose a protocol where you will only be shown the face of the card for .0000003 seconds, then perhaps you might not want to agree to that protocol.

When people claim to be able to do something at a particular level of performance, what they typically mean is on average they achieve that level of performance. Sometimes they will perform better, other times they will perform worse. E.g., I can drive to work in 20 minutes. Sometimes it only takes me 15 minutes, other times it takes me 25 minutes or longer.

By setting the threshold for passing the challenge at their claimed (average) level of performance, there is a 50% chance they will perform below this level and fail the test.

How would setting the threshold for your driving time at your average time be in any way a challenge to your driving abilities? Sure you may fail 50% of the time from your average under average conditions (which you don’t really know when you leave your house what the conditions will be, average, better for you or worse). Being given the conditions you will be driving under, if your have no confidence in your abilities to perform as required (perhaps even better than your average time) under those conditions, than I recommend that you do not accept such a challenge. A 50-50 chance at 1 million, how is that bad (particularly when you’re not putting up even money if you fail)?

I've just been going through the first page (i.e. the most recent page) of MDC claims. Of those who have reached the stage where a protocol has been suggested and the claimant made a statement as to their success rate:

Claimant|Reported Success Rate|JREF Success Rate
Edge/Mike Guska|50 - 70%|70%
Raymond Powell|100%|100%
Chris Cordero|96%|44% (4/9)
Carina Landin|80%|80%
Connie Sonne|?|100%
Patricia Putt|?|50%
Natal'ya V.|100%|100%
Eileen McKusick|?|71% (5/7)
Pavel|67%|~90-100%

If anyone can fill in some of the question marks, please do so. The only reported success rates that were likely based on their own controlled tests were Mike Guska's and Pavel's (as far as I can tell).

Note that for Chris Cordero, the JREF provided a test which would be close to what I am suggesting (with the exception that Chris' reported success rate was probably not determined by him under controlled conditions).

Linda

Well there you go. 3 exactly at the claimed rate, 1 at the high end of the claimed rate, 3 unknown claimed rates, 1 higher than the claimed rate and 1 lower. So based on that evidence (excluding the unknowns for now), more are set at the claimed rates than not. How is setting the success rate at what is claimed a test that has been “set up so that even if they have the claimed ability, it is unlikely they will be able to pass”? Do not forget that a claimed success rate is part of the claimed ability. As Ivor the Engineer notes above, if it is your average success rate then it is equally likely that you will pass as fail. If you’ve given a higher than average rate, whose fault is that? If you have given an absurdly low rate (approaching or below chance) then it is not surprising that the protocol would require a higher rate for the challenge.

 

[weirdl]

@Andrew Wiggin

You said Ms. Sonne claimed to be able to dowse through envelops.
She never claimed that! The JREF substituted their protocol for her original claim without making that switch clear. This is one of my major complaints.
Ms. Sonne only claimed the ability to dowse for upside-down cardboard peices that had been freshly written on.
I know she agreed to the changes, but Ms. Sonne did that because the voices in her head told her the JREF was helping her save the world and she should trust them completely.
Before anyone goes there, Connie Sonne would not have won if she had been treated fairly. I don't believe in magic.

 

[Uncayimmy]

Not “test”, challenge. Just what do you think I‘m failing to notice? If I claim I have a way of reading cards inside envelopes with 100% accuracy then I should not fail at all, unless I can’t really read cards inside of envelopes with 100% accuracy. I imagine that you might claim that you can read a card if I show you it’s face. How many cards do you think you will get wrong under those conditions? Now if I propose a protocol where you will only be shown the face of the card for .0000003 seconds, then perhaps you might not want to agree to that protocol.

If you propose such a protocol, that would enormously damage your credibility and be a tacit admission of fear that the ability exists.

What you are "missing" is that the JREF claims to be so incredibly confident that the paranormal cannot be demonstrated that it advertises a challenge with a $1M enticement to prove them wrong. If they even just propose a protocol where someone with a "genuine" ability could easily fail (this includes coin-flip odds), they come across as cowards. It looks like they are "fixing" the test and that their confidence with the $1M is just so much bluster.

Who gives a **** if the claimant is stupid or arrogant enough to accept the protocol? What's important is the message that the worldwide audience receives. Suppose that message includes, "The JREF proposed and the claimant accepted a protocol where if the claimant actually has the ability claimed, there's only a 50% chance he will win the money." The JREF looks like a bunch of asses in the minds of many of the people they want to reach with the challenge.