How would you test for a PEAR level effect?

[roger]

Anybody else identify with this?

Zeph

Almost.

The biggest problem, as others have pointed out, is that the proposition is unfalsifiable. Minds can control things, remotely, into the past or future, but only at such tiny amounts that it could easily be noise or experimental error. How do you falsify that? You can't. Because you have mean ole skeptics wishing the PEAR effect won't work.

You cannot, absolutely cannot, design a (good) experiment for an unfalsifiable hypothesis. No amount of mental gymnastics will change that. That is not me being a skeptic that wants to 'win' an argument - it's just fact.


I see no mechanism to explain how this effect could possibly be real (we do a lot of precise experimentation that would out such an effect if it existed), but hey, we discover new things every day, and they are occasionally revolutionary. I would consider that having an "open mind" - but the nature of the claim means it is unfalsifiable. It's not that some 'woo' believer is crying "goat/sheep" - it's the nature of the claim. The possibility of contamination by 'skeptical' minds swamps the small signal that is claimed to exist.


edit: I'll let this stand, even though I don't agree with it. I believe you can prove something to be true even if it is unfalsifiable. Suppose the claimed effect was not 2 in 10,000, but 3,000 in 10,000. I can't falsify it, as you could have a mean ole skeptic surpressing the results, but if I can show that 3,000 effect, well, it's pretty strong evidence that the effect exists (even if we haven't yet proven that my explanation for the cause is correct).

 

[drkitten]

The biggest problem, as others have pointed out, is that the proposition is unfalsifiable. Minds can control things, remotely, into the past or future, but only at such tiny amounts that it could easily be noise or experimental error. How do you falsify that? You can't. Because you have mean ole skeptics wishing the PEAR effect won't work.

You cannot, absolutely cannot, design a (good) experiment for an unfalsifiable hypothesis. No amount of mental gymnastics will change that. That is not me being a skeptic that wants to 'win' an argument - it's just fact.

Yup. Basically, the PEAR claims, as defended by Limbo and people like him, turn into claims like "my coffee cup is made of gingerbread, but it's made of magic gingerbread that is indistinguishable from ceramic by all tests."

On another thread, Limbo said "You act as though the distance would prevent negative "goat" psi from exerting its share of influence on the results. It wouldn't. Neither would time." In other words, as long as any skeptic, anywhere in space or time, ever existed, then there's no reason to accept any psi study that shows a negative result as anything other than an indicator of that skeptic's immense "goat" psi. From beyond the grave, Randi's mighty mind will continue to foil charlatans and poseurs.....

Seriously? I mean,.... seriously? And we're supposed to discuss statistical power and issues of sample size in this context?

 

[Resume]

Yup. Basically, the PEAR claims, as defended by Limbo and people like him, turn into claims like "my coffee cup is made of gingerbread, but it's made of magic gingerbread that is indistinguishable from ceramic by all tests."

On another thread, Limbo said "You act as though the distance would prevent negative "goat" psi from exerting its share of influence on the results. It wouldn't. Neither would time." In other words, as long as any skeptic, anywhere in space or time, ever existed, then there's no reason to accept any psi study that shows a negative result as anything other than an indicator of that skeptic's immense "goat" psi. From beyond the grave, Randi's mighty mind will continue to foil charlatans and poseurs.....

Seriously? I mean,.... seriously? And we're supposed to discuss statistical power and issues of sample size in this context?

In 1967, Gary Roberts told me he received 14 stitches in his ill-advised attempt to slide into home. Invisible stitches as it turned out. Worked out the same as none.

I didn't believe Gary then, and I don't believe Limbo now.

 

[Robin]

Jahn's claim for the meta analysis was that there were 2.6 bits in 10,000 affected for HI trials and 1.6 bits in 10,000 for LO trials. The meta-analysis encompassed half a billion bits for the non control trials and half a billion control bits.

There could be a single experiment with the power of his entire meta-analysis.

For example if there were 200 subjects, each doing 30 3.3 minute sessions this would provide the same number of bits - control and non control - as the meta analysis.

Before the experiment starts 500,000 bits of data are generated by a pseudo random number generator.

When each subject undergoes a session a random number generator selects whether this is a HI, LO or no change trial and whether this is a control or real session is selected randomly. The experimenter is blind to this.

This information is saved, encrypted, to a database along with the subject's identification number.

When the session starts the results of each 200 bit trial of random (or pseudo random) data generated is also saved encrypted to the database automatically with no experimenter intervention.

This would be a fairly expensive experiment probably requiring one or two full time experimenters for a period of 10 - 20 weeks.

And if it produced no statistically significant result then believers would dismiss it as another example of the negative psi of skeptics.

But if it produced statistically significant results and it could be monitored and demonstrated to have been conducted correctly then it would provide compelling evidence for this type of telekinesis.

 

[Zeph]

Jahn's claim for the meta analysis was that there were 2.6 bits in 10,000 affected for HI trials and 1.6 bits in 10,000 for LO trials. The meta-analysis encompassed half a billion bits for the non control trials and half a billion control bits.

There could be a single experiment with the power of his entire meta-analysis.

For example if there were 200 subjects, each doing 30 3.3 minute sessions this would provide the same number of bits - control and non control - as the meta analysis.

Before the experiment starts 500,000 bits of data are generated by a pseudo random number generator.

When each subject undergoes a session a random number generator selects whether this is a HI, LO or no change trial and whether this is a control or real session is selected randomly. The experimenter is blind to this.

This information is saved, encrypted, to a database along with the subject's identification number.

When the session starts the results of each 200 bit trial of random (or pseudo random) data generated is also saved encrypted to the database automatically with no experimenter intervention.

This would be a fairly expensive experiment probably requiring one or two full time experimenters for a period of 10 - 20 weeks.

And if it produced no statistically significant result then believers would dismiss it as another example of the negative psi of skeptics.

But if it produced statistically significant results and it could be monitored and demonstrated to have been conducted correctly then it would provide compelling evidence for this type of telekinesis.

That's the kind of thinking I am looking for.

Small modification: I think it would produce a statistically significant result in either case - if not demonstrating the effect, then statistically bounding the size of effect which could have gone undetected (to some p)

The resultant data (with decryption key) could be released for other scientists to verify the statistical analysis.

Many believers will dismiss negative results in any case; many skeptics will dismiss any positive results. (Just look at the global warming skeptics - they can ALWAYS find some rationale for discrediting something they don't want to believe in). They are not the audience to aim for.

By the way, I don't know if this effect (if it existed) would be "telekinesis". Perhaps it would be more like "manipulation of probabilities". Some of the stranger (and often speculative = unproven) aspects of quantum physics come to mind; as you are probably aware, there are disagreements among serious scientists about the role of consciousness in some quantum effects, with neither side having proven their case (ie: the ball is still in play scientifically, without unscientific woo woo). See quantum mind theories (really hypotheses and speculations with some reference to evidence). My guess is that these are the fields of science which would be called upon someday to help explain any positive results. And that there would be generations of debate within the legitimate scientific community.

Anyway, I appreciate your points.

Zeph

 

[Zeph]

Yup. Basically, the PEAR claims, as defended by Limbo and people like him, turn into claims like "my coffee cup is made of gingerbread, but it's made of magic gingerbread that is indistinguishable from ceramic by all tests."

On another thread, Limbo said "You act as though the distance would prevent negative "goat" psi from exerting its share of influence on the results. It wouldn't. Neither would time." In other words, as long as any skeptic, anywhere in space or time, ever existed, then there's no reason to accept any psi study that shows a negative result as anything other than an indicator of that skeptic's immense "goat" psi. From beyond the grave, Randi's mighty mind will continue to foil charlatans and poseurs.....

Seriously? I mean,.... seriously? And we're supposed to discuss statistical power and issues of sample size in this context?

Agreed. That way be dragons of uncertainty.

Let's keep it down to the simple basic experiment: can a subject mentally influence the outcome of a random generator during the time of the experiment? We can keep the HI, LO and neutral time periods as seen by the subject secret from spatially remote goats (until publishing the data). We won't pretend there's any way to shield the experiment from temporally remote goats, that goes too far.

So, tongue firmly implanted, I can see a convergence of interests. Before the experiment even starts, believers who want the experiment to have a fair shot at success just need to believe very hard that there is no cross-time sheep/goat effect. And skeptics without effort will tend to join them in that belief. So if most everybody disbelieves in a cross-time sheep goat effect, it shouldn't get in the way, either because it doesn't exist or because of all that focused disbelief from both sides!

Zeph

 

[Zeph]

You cannot, absolutely cannot, design a (good) experiment for an unfalsifiable hypothesis. No amount of mental gymnastics will change that. That is not me being a skeptic that wants to 'win' an argument - it's just fact.

Point taken.

However, I do think that this can be framed in a way which is "falsifiable", at least in the statistical sense which we are talking about.

The Hypothesis: A given series of experiments should statistically demonstrate and characterize an effect to within p=0.95 if that effect is larger than 1 in X bits, and if it shows up with the given subjects.

Positive Outcome: An effect of approximately 1 in Y bits (+/- delta) was detected to p=0.95 in this experiment.

Negative Outcome: No effect was detected, so we can say with p=0.95 that the effect did not happen during this experiment, or was below 1 in X bits.

That's about all any experiment with human subjects could expect, and it's enough. There's no point in trying to frame it as "the results will subjectively convince everybody", keep to limited and reasonable scientific results. You would not and cannot and need not prove that another group of subjects (etc) wouldn't have different results - but that's true of any experiment.

It's not unfalsifiable because the scope of the experiment would avoid the sheep/goat question - like many other experiments, the only results it can give is "did the effect happen this time", not "could the effect exist under other conditions".

I see no mechanism to explain how this effect could possibly be real

I agree. If you had a positive result, there would be a whole lot of work needed because it would break many current models. That's why if I were betting, I'd bet on negative results being more likely.

I believe you can prove something to be true even if it is unfalsifiable. Suppose the claimed effect was not 2 in 10,000, but 3,000 in 10,000. I can't falsify it, as you could have a mean ole skeptic surpressing the results, but if I can show that 3,000 effect, well, it's pretty strong evidence that the effect exists (even if we haven't yet proven that my explanation for the cause is correct).

Yep. If in concept it could be demonstrated at the higher level, then doing so at a lower effect level is not outside theory, it just changes the number of trials (and, outside the realm of a thought experiment, the practical cost and fundability).

Zeph

 

[drkitten]

Point taken.

However, I do think that this can be framed in a way which is "falsifiable", at least in the statistical sense which we are talking about.

The Hypothesis: A given series of experiments should statistically demonstrate and characterize an effect to within p=0.95 if that effect is larger than 1 in X bits, and if it shows up with the given subjects.

Well, this gets back to the appropriateness of the experiment.

If the experiment itself is badly designed, then the statistics won't and don't matter. In particular, if there's some effect that can cause a systematic overdetection or underdetection of psi, then the power/sample size calculations I gave you upthread are all so much chin music.

And that's the problem. Too many of the psi proponents do not accept that PEAR's protocols are valid (although they would accept the validity of a real effect, if shown). And too many psi opponents also do not accept that PEAR's protocols are valid (although they would accept the validity of a negative effect, if shown).

If you take stuff like the sheep/goat effect seriously, then you can't just handwave it away and pretend that it won't change anything.

And, from a physics standpoint, I'd need to be convinced that any physical white-noise generator can be calibrated to be balanced and stable to one part in 10,000 over an extended period. Otherwise the statistical baseline isn't valid.

Basically, this like looking for unicorns with a metal detector. If you take the basic position that unicorns are non-metallic, then issues of statistical power don't enter into it.

 

[Zeph]

And that's the problem. Too many of the psi proponents do not accept that PEAR's protocols are valid (although they would accept the validity of a real effect, if shown). And too many psi opponents also do not accept that PEAR's protocols are valid (although they would accept the validity of a negative effect, if shown).

Agreed. Which is why I never conceived that even a meticulously done experiment would be accepted by everybody.

If you take stuff like the sheep/goat effect seriously, then you can't just handwave it away and pretend that it won't change anything.

I understand. It depends on how strong somebody considers the sheep/goat effect. For example, can a goat somehow reach back though time after the results were published and change all copies of the encrypted recorded data, rewriting the historic record retroactively? If one believes that, then no scientific experimentation of any sort is possible. Let's but that level of sheep/goat effect outside the target testing envelope.

(If if the sheep/goat effect is that strong, I think we'd be living in a rather more magical world, but that's another matter. Why would goats be vastly more powerful than sheep? Unless there was some asymmetry, like "goats are aided by the resilience of an objective universe, while sheep are having to painfully and partially override that objective universe").

So here's what we could offer in this thought experiment: believers can have as many believers as they can round up in the next room, sheeping like crazy while the experiments are going on, with no goats nearby (except some but not all of the subjects one at a time). The schedule of the experiment will not be disclosed to goats before publicatio, so even if a goat could affect the 1's and 0's from Africa, they don't know which way to root for until too late.

I'm quite aware that this wouldn't stop objections from true believers. But it *would* define the limitations. A negative result would be "Even in the presence of many sheep who know what results are being tested for, and without goats having advance knowledge of the timing, we found no effect above 1 bit in N". Those disputing the experiment would be left with either "the effect is less than 1 bit in N", or "the goats have so much power that they could overcome even that". It sets an envelope, pushes the remaining believers to stipulate an especially powerful goat effect as the "minimum" threshold.

And, from a physics standpoint, I'd need to be convinced that any physical white-noise generator can be calibrated to be balanced and stable to one part in 10,000 over an extended period.

Suppose it doesn't have to. Why would you use only one contiguous control period, eg: before all the tests for your baseline? Calibrate your baseline before, after, and interspersed with subject testing. If the interspersed "control" periods are statistically like each other but the "test" periods differ, you have a result. Even if there was a small systematic or random drift.

(By the way, this is similar to what I do to compensate for a drifting clock in data aquisition. I can't get clocks within my power and financial budget which do not drift over the years, but I can record time deltas every few weeks and interpolate to well within my accuracy needs).

Suppose the generator slowly drifted from 49.99% 1's to 50.02% ones over the course of the weeks of the experiment. At the early part of the experiment (when the control periods were giving 49.99% 1's before and after subject testing periods) the subjects could tweak that upward or downward by 1 in 10000. In the later end of the experiment, the subjects were tweaking the results up or down by 1 in 10000 from the adjacent "control" periods before and after. I'm just making up numbers, but the point is that one can scientifically and statistically show a or discredit correlation between cloud cover and sunspot count, without either being a perfect random event.

The better the randomness, the easier it becomes, but perfect randomness isn't required (and is hard to even define), just correlation. A better generator just means fewer trials. No, I'm not assuming linear drift - all the interspersed control periods would detect nonlinear drifting as well.

The only reason to do a lengthy calibration period beforehand would be if one wanted to estimate in advance how many trials would be needed to overcome the instability of the random generator - but this "pre experiment" would not be used in the primary analysis of the results - the time interspersed control periods would be.

Zeph

 

[Zeph]

Let's note that there are experiments which cause some to believe in a sheep/goat effect. However, even those experiments do not give the goats infinite "override" power -- like "no results are obtained if there are any goats anywhere on the planet, regardless of the number of sheep". Obviously, if they believe there are sometimes real world effects, then by definition they believe that under the right conditions the sheep can in practice overcome the goats.

So we can load the dice by incorporating conditions in the testing protocol which (rational) believers stipulate in advance would be more favorable to the sheep than the goats, if there IS a sheep/goat effect; while at the same time irrelevant and neutral to the experiment to the satisfaction of the skeptics (if there is no sheep/goat effect). For example, that bunch of meditators can be in the next room. This just gets incorporated into the results (that some effect was found, or not found, under those documented conditions).

A positive results would lead to "let's try it without the extra sheep". A negative result would lead to "no effect larger than 1 in N even when the proposed sheep/goat effect is biased towards one".

Zhahai

 

[Zeph]

And let's not forget the double blind choice between testing intention against a random noise source and a pseudorandom generator. If one of these shows effects (comparing intention periods to control periods) and the other does not, that's significant.

It might suggest that conciousness has some as-yet-unexplained influence on stochastic processes but not deterministic ones. Without harming a cat.

My suggestion was further to have an independent statistician process the data without being informed of which periods used which generator, yet another level of blinding. In fact, that information could remain encrypted until after the initial analysis, so nobody knows it yet. Then you decrypt the information about which generator was used when, and do further analysis (like the above) with that extra data.

If there is a real effect, these measures should not mask it, and should help convince the more rational critics.

Zeph

 

[Robin]

I note that in Jahn and Dunne's paper "Change the Rules", (which I would say is a parody if I did not know better) there is a need to change science to rid it of those features that inhibit anomalous phenomena.

So I guess there will be a need to purge experiments of those nasty psi inhibiting factors like secrecy, lack of choice/personal initiative and lack of trust.

Once these have been removed from I experiments we will see plenty of positive results

 

[mike3]

I see that even you are abusing that easy out for skeptics - the scoff and guffaw factor. I must say I'm disappointed. I had hoped for more intuitive insight from you.
Its not the case that "any negative result" would be rejected as the sheep-goat effect, because not every negative result is strong enough to be statistically significant. In order to be a manifestation of that effect, the results would have to be statistically significant and in the predicted (negative) direction, with the prediction based on the psychological variable of belief or lackthereof. The results would either support a sheep/goat hypothesis or they wouldn't.

So how can you know what negative results are truly negative results and which ones are due to "goat psi"? You need to provide me a rigorous and reliable method to do that. Also, what is a "statistically significant negative effect"? A negative result would be a result equivalent to chance, i.e. what you'd get from guessing randomly. So what does it mean to be "significant"? What -- really close to chance? And if a "weak negative" is indicative of something, vs. a "true negative", i.e. it is further from chance, then it's not a negative at all, but a weak positive! What's going on here?

P.S. Science isn't about "dogmatism" or "materialism", it's about CATCHING ERRORS. The whole point of demanding rigorous testing and standards has nothing whatsoever to do with any kind of "dogmatism", it's there to make sure people aren't fooling themselves!

 

[drkitten]

Suppose it doesn't have to. Why would you use only one contiguous control period, eg: before all the tests for your baseline?

Because that's what the stats demand. If I'm going to be using a binomial test, then I need to be able to assume that each "trial" is independent and that the distribution of ones and zeros is uniform over the experimental regime. If those assumptions are not met, then the test is invalid and no conclusions can be drawn.

So, basically, it's back to the (statistical) drawing board; I would need to come up with a different experimental design, a different set of statistics to collect, and because I'm being forced into a less powerful statistical methodology, I'll probably need to get a lot more data.

 

[Zeph]

Because that's what the stats demand. If I'm going to be using a binomial test, then I need to be able to assume that each "trial" is independent and that the distribution of ones and zeros is uniform over the experimental regime. If those assumptions are not met, then the test is invalid and no conclusions can be drawn.

So, basically, it's back to the (statistical) drawing board; I would need to come up with a different experimental design, a different set of statistics to collect, and because I'm being forced into a less powerful statistical methodology, I'll probably need to get a lot more data.

I don't get that. The following is not an exact proposed protocol, but an abstracted simplification for discussion.

Version one: Record 500,000,000 random bits as a control. Then record 500,000,000 bits during a series of 50 experimental sessions. Worry about whether the RNG drifted between the first and second series.

Version two: Record 10,0000,000 random bits of control followed by and 10,000,000 bits of subject influence for each of 50 sessions.

You wind up with the same number of bits to analyze. If there is some drift in the RNG, it will largely be the same for the controls and the experimental results and can be factored out. In what way is there any statistical advantage to version one?

Zeph

 

[drkitten]

I don't get that. The following is not an exact proposed protocol, but an abstracted simplification for discussion.

Version one: Record 500,000,000 random bits as a control. Then record 500,000,000 bits during a series of 50 experimental sessions. Worry about whether the RNG drifted between the first and second series.

Version two: Record 10,0000,000 random bits of control followed by and 10,000,000 bits of subject influence for each of 50 sessions.

You wind up with the same number of bits to analyze. If there is some drift in the RNG, it will largely be the same for the controls and the experimental results and can be factored out. In what way is there any statistical advantage to version one?

They're both incompetently done experiments; neither would pass skeptical muster, which is why I laid so much stress on calibrating your control set.

The problem with version two is that "it will largely be the same" is exactly what the p-values and whatnot are supposed to be measuring. How far from exact balance is the control set supposed to veer? This, in turn, determines the variance, and this determines the statistical power of the test.

10,000 coin flips of a perfect coin are expected to come within 100 [sqrt(10,000)] of the expected value of 5000 heads. But if the coin is only perfect to, say 1% tolerance, then the coin is expected to come to within 100 of the actual expected value, which could be anywhere from 4900 to 5100 heads. But this, in turn, nearly doubles the actual experimental range we'd expect to see.

So, no. It will not "largely be the same" between the two conditions; we need to actively calculate how close to "the same" it's expected to be and to update our error bars accordingly.

 

[lemurien]

Now, if you'd cut it short and write it up, maybe you could come up with something more coherent than the Men Who Stare at Goats.
Other than that, fat chance.
More than 50 years of research and null result.
But for fun why not.

 

[epepke]

I apologize for the delay in getting back to you. I didn't see this. Nor is an easy was to track the threads one contributes to apparent to me on this site.

I'm less certain whether the difference between thermal noise and radioactive decay (for example) really matter. Certainly a positive result (mental intention can slightly influence random numbers generated by thermal noise/radioactive decay) would be similarly significant in either case. Either would indicate a phenomenon violating current concepts of causation.

We thought that thermal noise was good, until we tried it, and we found all sorts of problems. Most important to this is the fact that thermal noise involves high amplification and then it becomes difficult to exclude outside sources of noise. I would hate to see the source biased in some unexpected way by, say, a particularly punchy cell phone, or whether the air conditioner is on, or even a nearby radio station. You might argue that this is unlikely, but I'd rather just avoid the problem, especially as it is so easy to do by using radioactive decay.

Using both forms of random generators from the start would be interesting in one way - in the case of negative results, there would be no room left for saying "well sure, there was no effect with thermal noise, but there would have been with radioactive decay".

That's a point. Especially as a lot of people have ideas that consciousness affects quantum events. The PEAR researcher I talked to was all about non-mechanistic effects. As close as I could tell, he meant stuff like many worlds.

Likewise I don't see that the experiment depends on extra high quality random numbers. To take an example, if one used a generator which produced 49% ones and 51% zeros, BUT it could be demonstrated that human intentions would modify these ratios by one in 10^4 bits when comparing the same not-smoothly-random streams with and without intentions, it would still be a significant result. It's the unexplained modification of non-deterministic events (or lack thereof) which matters. (I guess that a particularly bad random number generator - like one the kept getting stuck for a while with all 1s or all 0s occassionally - might make it harder to statistically pull the signal out of the noise).

But these are my thoughts. If there is some way in which the purity of the random numbers matters more than I realize, I'd like to understand.

Central to my idea is that the researchers using the boxes not be able to tell the difference between a deterministic source and a non-deterministic source. The problem is not a simple bit bias but long-term patterns. These patterns are so incredibly easy to detect with simple techniques that it leaves open the possibility that human beings, who are pattern-matching creatures, can get a feeling for whether a source is random or pseudo-random by paying attention to it.

You could argue that this is also unlikely, but what we have here are claims of something just about at the level of statistical significance, which are fairly obviously sensitive to unlikely events.

So, at least for the first experiments, I'd like to keep things as foolproof as possible. Two types of boxes, identical save for a single switch. In record mode, radioisotope decay produces the signals which are also recorded on lead-shielded flash memory. In playback mode, the radioisotope decay is ignored, and the signals come from playing back what is on the flash memory. The researchers would be provided boxes in record mode and boxes in playback mode that had previously been in record mode long enough to fill the flash memory with enough bits for the trial.