Complaints Dept: Connie Sonne MDC

[gnome]

I realize that they use the same excuse. What I'm wondering is why the JREF can't just roll up shop and leave them alone? If the actions of the JREF are characterized by "it doesn't matter whether we take advantage of people as long as what we do is legal", why doesn't that same characterization make the actions of psychics acceptable as well?

Linda

Being incompetent to defend a bad claim doesn't mean the bad claim should go unchallenged, in the name of "fairness".

 

[Ivor the Engineer]

Increasing the number of trials does not increase the probability that the claimant will pass due to chance.

If the requirement for passing was set at a particular alpha level (i.e. the probability of a particular result due to chance), then increasing the number of trials would make it easier for them to pass. But it's not. The requirement for passing is set at a particular effect size (i.e. x correct results out of y attempts) instead.

Linda

That’s not the whole story though. While it is true that the probability of passing due to chance does not go up as the number of trials is increased with the null hypothesis effect size held constant, the probability of a real effect being detected (i.e. the power of the test) does go up as the number of trials increases, if the null hypothesis effect size is less than the true size of the underlying effect. However, if the null hypothesis effect size is set higher than the true size of the underlying effect then increasing the number of trials reduces the probability the effect will be detected.

 

[MRC_Hans]

Increasing the number of trials does not increase the probability that the claimant will pass due to chance.

No, and that was not what I meant. What I meant is that IF a claimant states that her predicted success rate is only slightly over random, then it will require a large number of instances to produce a statistically significant result.

So, it is actually the opposite: The large n reduces the probability of a random positive.

However, the bottom line is that you can't have it both ways: If you try to exclude a false negative, you allow a chance of false positives, and vice versa. And, of course, the JREF must opt to exclude a false positive, which means that there is some chance of a false negative.

- And this is exactly the reason a claimant is asked to predict her performance, and required to stick to that prediction.

Hans

 

[MRC_Hans]

What do you think? For the most part, what psychics do isn't illegal. If that's all this is about, why not simply let them use the same excuse you've made available to the JREF and have done with it?

Linda

I think the problem here is that the JREF challenge is really (the way I understand it) directed at the big frauds; the Sylvia Brownes and the Uri Gellers. However, they are not so dumb that they take it, so instead all the small, deluded people line up and are duly whopped. I assume the requirement for media presence and/or affidavits is an attempt to stop the small fishes from coming in, but OTOH they can't very well flatly refuse people, just because they are not big enough frauds.

Hans

 

[fls]

Being incompetent to defend a bad claim doesn't mean the bad claim should go unchallenged, in the name of "fairness".

I agree. I also didn't say that. I said that if we are agreed that they should be challenged, the challenge should be fair, even if we are given the (legal) opportunity to take advantage of their incompetence.

Linda

 

[Ivor the Engineer]

I agree. I also didn't say that. I said that if we are agreed that they should be challenged, the challenge should be fair, even if we are given the (legal) opportunity to take advantage of their incompetence.

Linda

I think the particular level of alpha for a challenge test is irrelevant because I doubt claimants believe their paranormal ability prior to performing the test is only a little above what would be expected by chance. More likely is that people come to the conclusion they have paranormal abilities from casual observation. I.e. the effect size is moderate to large.

If alpha was held constant as the number of trials was increased, thus allowing the detection of ever smaller effects, the risk of non-paranormal systematic biases producing a significant result goes up. It therefore makes sense to test the claim being made (i.e. by specifying a claimant exceed a minimum expected effect size to win), rather than if there is a statistically significant difference from chance at fixed level of alpha.

ETA: Fair in this case would be increasing the number of trials until the power of the test was above some minimum threshold, say 50-90%.

 

[fls]

However, the bottom line is that you can't have it both ways: If you try to exclude a false negative, you allow a chance of false positives, and vice versa.

Again, this is incorrect. False positives and false negatives are not complementary. Reducing the chance of a false negative does not necessarily increase the chance of a false positive.

The chance of a false positive is the alpha level, which can be set independently of any other parameter and is simply a matter of choosing a number.

The chance of a false negative is the beta level, and it is not a single number which is independently chosen, but rather the result of the alpha level, the effect size (which is independent of alpha and of sample size), and the sample size. Increasing the power by changing the effect size or the sample size has no influence on the chance of a false positive (i.e. the alpha level).

And, of course, the JREF must opt to exclude a false positive, which means that there is some chance of a false negative.

This is the part that I really, really wish would stop getting passed around as though it is true. The JREF can both exclude false positives and exclude false negatives. The two are independent, not complementary. The JREF can be fair, if it chooses.

- And this is exactly the reason a claimant is asked to predict her performance, and required to stick to that prediction.

Hans

It is my impression that the claimant is asked to predict their performance only so that the JREF can exclude false positives. I don't think I've ever heard mention of the false negative level from the JREF. I'm willing to give them the benefit of the doubt and consider that this is done purely through ignorance. And even though I've mentioned it numerous times now, including within threads that Alison has at times participated in, I wouldn't assume that they've seen it from me. But I have to wonder why the statisticians that they claim they've consulted haven't mentioned it to them. The other option is that they've chosen to be unfair. Which, like I said, I find disappointing, but I realize others do not.

Linda

 

[fls]

I think the particular level of alpha for a challenge test is irrelevant because I doubt claimants believe their paranormal ability prior to performing the test is only a little above what would be expected by chance. More likely is that people come to the conclusion they have paranormal abilities from casual observation. I.e. the effect size is moderate to large.

If alpha was held constant as the number of trials was increased, thus allowing the detection of ever smaller effects, the risk of non-paranormal systematic biases producing a significant result goes up. It therefore makes sense to test the claim being made (i.e. by specifying a claimant exceed a minimum expected effect size to win), rather than if there is a statistically significant difference from chance at fixed level of alpha.

ETA: Fair in this case would be increasing the number of trials until the power of the test was above some minimum threshold, say 50-90%.

Thank you. Either you are providing independent support or you've been listening to my soapbox rants.

Linda

 

[Ivor the Engineer]

Thank you. Either you are providing independent support or you've been listening to my soapbox rants.

Linda

All of the above and I'm currently trying to set thresholds to extract data and sync pulses from the green waveform in the image below in the presence of noise, so I was thinking about this sort of stuff anyway.

Yellow trace is the extracted sync signal and cyan is the data+sync.

 

[MRC_Hans]

I agree. I also didn't say that. I said that if we are agreed that they should be challenged, the challenge should be fair, even if we are given the (legal) opportunity to take advantage of their incompetence.

Linda

True, but I don't see 'us' doing that. What the JREF does is try to make sure that if someone wins the MDC, then there was indeed something truely extraordinary going on. - Extraordinary claims ....


Hans

 

[fls]

That’s not the whole story though. While it is true that the probability of passing due to chance does not go up as the number of trials is increased with the null hypothesis effect size held constant, the probability of a real effect being detected (i.e. the power of the test) does go up as the number of trials increases, if the null hypothesis effect size is less than the true size of the underlying effect. However, if the null hypothesis effect size is set higher than the true size of the underlying effect then increasing the number of trials reduces the probability the effect will be detected.

I'm having trouble picturing this. Can you elaborate on this (Are you taking advantage of something to do with the analysis of ROC curves?)?

Linda

 

[fls]

True, but I don't see 'us' doing that. What the JREF does is try to make sure that if someone wins the MDC, then there was indeed something truely extraordinary going on. - Extraordinary claims ....

Hans

I object to this ongoing pretense that making sure the claim is extraordinary requires us to place the claimant at a disadvantage. It is entirely possible to both place the claimant at an advantage and make sure the claim is extraordinary. If we were truly interested in evaluating these claims, that is what we would do. That we deliberately choose to do otherwise shows that the condemnation of our actions by those people we call "woos" is legitimate.

Linda

 

[fls]

All of the above and I'm currently trying to set thresholds to extract data and sync pulses from the green waveform in the image below in the presence of noise, so I was thinking about this sort of stuff anyway.

Yellow trace is the extracted sync signal and cyan is the data+sync.

[qimg]http://www.internationalskeptics.com/forums/imagehosting/thum_81954b7d3c8d67abf.jpg[/qimg]

I'm sorry. This is getting off-topic, but I'm confused. I think of cyan as turquoise and I would guess that it is the green trace (C4) that is data+sync and the cyan (C3) that is sync. But I may not understand what you mean by "sync" or "cyan".

Linda

 

[Fnord]

So, after more than one-hundred posts in this thread, has anyone yet managed to argue their way into being awarded any of Mr. Randi's millions?

 

[Ivor the Engineer]

I'm having trouble picturing this. Can you elaborate on this (Are you taking advantage of something to do with the analysis of ROC curves?)?

Linda

Imagine two partially overlapping bell curves. Let’s say the one on the left is the null distribution and the one on the right is the estimated effect size for power calculation. In a conventional hypothesis test a vertical line is draw between the two curves so that the area under the null distribution curve to the line right of the line is equal to alpha. The area under the estimated effect size curve to the left of the line is beta.

If we leave the vertical line where it is and increase the number of samples the two distributions become more pointy, with more area closer to the two means. The area under the null distribution curve to the right of the line (i.e. alpha) gets smaller, as does the area under the estimated effect size curve to the left of the line (i.e. beta).

Now, if the threshold is set higher than the actual effect size, then both the null and actual effect size curves are to the left of the vertical line, which means as the number of samples is increased it becomes less likely the actual effect will be detected.

ETA: Apologies for the sloppy terminology, which no doubt is not aiding understanding.

 

[Ivor the Engineer]

I'm sorry. This is getting off-topic, but I'm confused. I think of cyan as turquoise and I would guess that it is the green trace (C4) that is data+sync and the cyan (C3) that is sync. But I may not understand what you mean by "sync" or "cyan".

Linda

The green waveform (C4) is the signal coming into the detection system. The big bipolar pulses in this waveform are sync(hronisation) pulses, the small bipolar pulses are data pulses.

The detection system extracts a sync signal by looking for pulses bigger than a threshold, which is set somewhere between the level of the sync and data pulses in the green waveform (C4). This produces the yellow waveform (C1). The cyan* channel (C3) is produced by thresholding the green waveform against a scaled version of the threshold used to extract the sync signal. Because this threshold is lower, the cyan channel contains pulses for both the sync and data pulses in the green waveform.

Anyhow, you're right this is off topic. My next problem is finding someone in the company who knows how this data is supposed to be processed and packaged up to be sent to a PC...



*I'm not being pretentious, that's the name given to the colour in computer programming. To me it looks very much like light blue.

 

[fls]

Imagine two partially overlapping bell curves. Let’s say the one on the left is the null distribution and the one on the right is the estimated effect size for power calculation. In a conventional hypothesis test a vertical line is draw between the two curves so that the area under the null distribution curve to the line right of the line is equal to alpha. The area under the estimated effect size curve to the left of the line is beta.

If we leave the vertical line where it is and increase the number of samples the two distributions become more pointy, with more area closer to the two means. The area under the null distribution curve to the right of the line (i.e. alpha) gets smaller, as does the area under the estimated effect size curve to the left of the line (i.e. beta).

Now, if the threshold is set higher than the actual effect size, then both the null and actual effect size curves are to the left of the vertical line, which means as the number of samples is increased it becomes less likely the actual effect will be detected.

Thanks, I got it. I was making it difficult for myself.

ETA: Apologies for the sloppy terminology, which no doubt is not aiding understanding.

Yeah, it's a good thing I know what you mean by "more pointy".

Linda

 

[fls]

The green waveform (C4) is the signal coming into the detection system. The big bipolar pulses in this waveform are sync(hronisation) pulses, the small bipolar pulses are data pulses.

The detection system extracts a sync signal by looking for pulses bigger than a threshold, which is set somewhere between the level of the sync and data pulses in the green waveform (C4). This produces the yellow waveform (C1). The cyan* channel (C3) is produced by thresholding the green waveform against a scaled version of the threshold used to extract the sync signal. Because this threshold is lower, the cyan channel contains pulses for both the sync and data pulses in the green waveform.

Anyhow, you're right this is off topic. My next problem is finding someone in the company who knows how this data is supposed to be processed and packaged up to be sent to a PC...

Okay, that's what I thought. I just didn't know how your terminology worked.

Linda

 

[The Man]

I object to this ongoing pretense that making sure the claim is extraordinary requires us to place the claimant at a disadvantage. It is entirely possible to both place the claimant at an advantage and make sure the claim is extraordinary. If we were truly interested in evaluating these claims, that is what we would do. That we deliberately choose to do otherwise shows that the condemnation of our actions by those people we call "woos" is legitimate.

Linda

In a casino the gambler is at a distinct (but relatively small) disadvantage, the house has a slight advantage. The more the gambler plays the more likely they are going to conform to the ordinary outcome of losing money to the casino. So the casino making money is the ordinary outcome. Someone continually playing and winning would be extraordinary. Change that advantage around so that the gambler now has the advantage and the gambler tending to win the more they play becomes the ordinary outcome. You can not separate the advantage from the ordinary outcome. Someone with an extraordinary advantage (supernatural powers, an effective method of cheating, or even an effective method of playing within the rules) should be able to overcome an ordinary disadvantage. Stacking the deck in the claimants favor only makes the ordinary seem extraordinary. For the most part that is why some people may actually believe they have such supernatural powers. They have been stacking the deck in their favor without even realizing it all along. The MDC is not an “evaluation” it is a challenge, thus is suppose to be challenging for the claimant. Having an advantage is not very challenging, but being at a disadvantage is.

 

[fls]

In a casino the gambler is at a distinct (but relatively small) disadvantage, the house has a slight advantage. The more the gambler plays the more likely they are going to conform to the ordinary outcome of losing money to the casino. So the casino making money is the ordinary outcome. Someone continually playing and winning would be extraordinary. Change that advantage around so that the gambler now has the advantage and the gambler tending to win the more they play becomes the ordinary outcome. You can not separate the advantage from the ordinary outcome. Someone with an extraordinary advantage (supernatural powers, an effective method of cheating, or even an effective method of playing within the rules) should be able to overcome an ordinary disadvantage. Stacking the deck in the claimants favor only makes the ordinary seem extraordinary. For the most part that is why some people may actually believe they have such supernatural powers. They have been stacking the deck in their favor without even realizing it all along. The MDC is not an “evaluation” it is a challenge, thus is suppose to be challenging for the claimant. Having an advantage is not very challenging, but being at a disadvantage is.

I'm not talking about ordinary advantages and disadvantages, or about stacking the deck in the claimants favour (I'm going to start a poll pretty soon to see if anyone can guess how many times I'll have to say that before it sinks in). Everybody is capable of recognizing that it is hard to come up with a correct guess 20 times in a row. Someone with an extraordinary advantage should be able to overcome an ordinary disadvantage. I'm talking about putting in an extraordinary disadvantage without telling anyone, like removing all the 10's from the decks used in 21. Sure, you'll tell them the 10's are gone if someone thinks to asks, but as long as they don't ask, why tell?

Linda