Pigasus Awards & Sheldrake

[bfinn]

While I am in general support of the aims of JREF, I was unimpressed by the Pigasus Award to Rupert Sheldrake for his telephone telepathy research (sorry, I've only just heard about this) and in particular for the justification provided for it.

The justification provided is a link to the Sept 8 SWIFT, whose sum total of objection to Sheldrake's actual research is:

"The sample size was small on both trials, just 63 people for the controlled telephone experiment and 50 for the email, and only four of the subjects were actually filmed in the phone study and five in the email. This gives cause for some skepticism. In addition, until a completed paper has been thoroughly vetted, the actual protocol will need to be viewed with some doubt, as well."​

I see the sample size objection has been discussed on these forums before - but to summarize, there were 571 trials, with statistical significance of p = between about 10^-8 and 10^-16 on the various experiments (i.e. astronomically significant), and 10^-12 on the videotaped experiments. The implication of Randi's vague and incorrect objection on the grounds of 'sample size' is that he simply didn't read the research (or at least understand the statistics). (The four people videotaped were incidentally the four people being phoned - since collusion would be required at both ends of the call it's not really necessary to video the callers too, but we're getting into details of protocol here.)

Randi's second objection above is that 'until a completed paper has been thoroughly vetted...' - which seems to me to imply that Randi believes (or wants the reader to believe) that there isn't 'a completed' paper yet (note he doesn't say 'the completed paper'). I.e. the experiments have been done but the write-up hasn't been completed (and hence not published), and so there hasn't been the opportunity yet to review the protocol used.

The paper in question was published as long ago as 2003, and I located it in one attempt by typing "Sheldrake telephone telepathy paper" into Google (it's the first result). It even has a helpful abstract with all the key information and statistics.

It seems to me that Randi either didn't do the most elementary research (taking all of 5 seconds) to see that there was a published paper, or he found it and didn't even read up to the 5th line (where he would have found the statistics which wipe out his 'sample size' objection), or he did read it but then decided to suggest that there wasn't yet 'a completed paper' describing the 'actual protocol', when in reality Randi has had nearly 4 years in which he could have 'thoroughly vetted' it.

So if you have actual objections to the research, why not state that the paper has been published and that you have read it (if you have), maybe even provide a link to it (so people can read it for themselves), and raise specific objections to the statistics and protocols set out in the paper? (Hey, like what a proper scientist would do.)

I think it's fair to say that the above represents poor grounds for Randi giving Sheldrake a Pigasus award. Sheldrake has been vastly closer to following an objective scientific method here than Randi has.

 

[blutoski]

I think you're partly correct. However, I would still say that Sheldrake's paper is a joke, scientifically speaking. I routinely reject medical papers that are better (I rejected a coworker's paper last year), so I'm rejecting this one as part of enforcing a single standard for papers that claim to be 'scientific'.

Important weaknesses of the protocol were:

• the dropout incidence is undocumented, despite its importance
• too many ****ing protocols - just do one experiment already. throwing together six different experiments is a red flag as it's a setup for data mining
• the protocol is very poorly documented - they go into great detail about the cup ("For the throw of the die, we used high-quality casino dice and a ribbed casino-style dice cup, purchased in Las Vegas."), but I don't even know if the callers used the same phone or not. It is possible that contacting Sheldrake will resolve these questions, and possibly even to his benefit. This would be a good action item to make any critique complete and fair. Did the callers/recipients have Call Display? That'd explain everything right there. Why did they have to make their guess after answering the phone but before the caller spoke? There's a huge opportunity for leakage in this period where the recipient can probably hear the caller's background noise, perhaps even their voice. The only thing we know is that the caller hasn't announced their name. The protocol also obviously provides the recipient with feedback of the accuracy of their guess. They can learn which background noise is associated with which caller, and future inbound call identification will be more accurate.
• as with the Demkina trials, the comparison against random chance is a red herring: the comparison is against educated guessing from clues, and the attention and effort should be directed toward protocol tightness, dropout analysis, and file-drawer (positive reporting bias), rather than p values

Personally, I'd love to replicate part of this study. It seems pretty straightforward.

The modifications I'd make:

• one protocol
• phone volunteer subjects from one room to another, with control of both phones (the sets will be modified so only the test conductors can hear the caller's voice/background noise) so there can be no leakage
• triple-blind so that none of the experimenters, callers, nor recipients is aware of the running score to avoid the learning concern
• dropouts' results will be recorded

 

[blutoski]

I just want to put numbers on my concern about the dropout rate. If you look at protocol 2, 21 participants dropped out, and 16 completed the 10 trials.

Sheldrake dismisses the possibility that poor running results could be a motivation for dropout. He does not prove that the dropouts' rates were the same as those who completed the experiment, though, which would put this concern to rest.

The fact that the majority of paid participants balied after starting trials is certainly suspicious.



This is how we caught fraud in the prayer study: the running results of the dropout group were not the same as for those who finished, and the chance of this distribution of results in a 'randomized' trial is so small as to be proof of fraud.

 

[Paul C. Anagnostopoulos]

Don't forget the clock synchronization leak. I brought this up with Sheldrake and he admitted that they didn't control for it.

~~ Paul

 

[blutoski]

Another problem is that they pooled their results. Due to the fact that they did different protocols, this is completely inappropriate and utterly contrary to any scientific practice. This is one reason I advise against mixing a bunch of experiments into one paper, and calling them "an experiment". What we have here is a collection of experiments whose data was inappropriately pooled. A sort of DIY-meta-analysis.

The largest number of trials in a single experiment in this paper is 188 trials with 14 recipients.

 

[blutoski]

Don't forget the clock synchronization leak. I brought this up with Sheldrake and he admitted that they didn't control for it.

~~ Paul

You mean that in the 'anytime' protocol, the sender and receiver could have made a prior (off-camera) agreement to phone at specific times?

eg: Emma calls at quarter-past, Gayle calls at half-past, Jayne on the 45, and Kay on the hour?

The protocol I propose will eliminate that by making all the calls very close together. Just sit the five down, put one in the receiving room, make four calls in succession, then rotate places.

I would recruit packs of 5 friends. Shouldn't take more than 15 minutes to do each batch of 20 calls. The question is: should I run participants through the 20-call trial more than once, as Sheldrake did?

 

[tkingdoll]

At one point Chris French was going to redo the experiment with Sheldrake. No idea what happened to that.

I was against it because I hate money being spent on this sort of crap. If the new test shows no effect (which it will), woos will just cry skeptic effect. No-one wins.

Waste of money, time and effort.

 

[CFLarsen]

Don't forget the clock synchronization leak. I brought this up with Sheldrake and he admitted that they didn't control for it.

That is an incredibly huge hole. It is akin to what Houdini did with his wife, during his "mind reading act" days. Simply agree on a code in advance, and mask the information.

It speaks volumes of Sheldrake's (in)ability to design a sound experiment.

At one point Chris French was going to redo the experiment with Sheldrake. No idea what happened to that.

I was against it because I hate money being spent on this sort of crap. If the new test shows no effect (which it will), woos will just cry skeptic effect. No-one wins.

I disagree. Woos will win, because the feeble excuses will be bought, hook, line and sinker.

Waste of money, time and effort.

That I agree 100% with.

 

[bfinn]

>At one point Chris French was going to redo the experiment with Sheldrake. No idea what happened to that.

AFAIK (I have met Sheldrake, who lives not far from me, and asked him about this) they have been trying to organize this, but because they are following a more stringent protocol involving videoing all callers as well as the callee, which for practical purposes involves all callers being in the same location, they have had difficulty recruiting enough people who know each other and can all be in one place at once.

I have also met Chris French, and he also (like Sheldrake) seems sound and genuinely interested in investigating this in further detail, without harbouring extreme prejudice either way (unlike some people on this forum...)

>I was against it because I hate money being spent on this sort of crap. If the new test shows no effect (which it will)

(...like I was saying.)

 

[fls]

While I am in general support of the aims of JREF, I was unimpressed by the Pigasus Award to Rupert Sheldrake for his telephone telepathy research (sorry, I've only just heard about this) and in particular for the justification provided for it.

I think it's fair to say that the above represents poor grounds for Randi giving Sheldrake a Pigasus award. Sheldrake has been vastly closer to following an objective scientific method here than Randi has.

I think this illustrates something that I have seen mentioned here a few times. At some point, all of us will be out of our league when it comes to evaluating whether or not a particular idea is silly. We don't really depend upon our own capabilities, but rather let those people in the field with expertise sort out the good ideas from the silly ideas. And while some people do go on to try and provide their own (often clumsy (sorry)) reasons for derision, underneath you will find the real reason for their derision is that the idea has not drawn any converts from the relevant scientific field.

The concerns about Sheldrake's methodology are legitimate. That Randi's attempt to provide a scientific-type criticism was perhaps awkward and somewhat off the mark, does not mean that he does not understand how silly ideas get identified as such.

Sometimes it bothers me. Some woos are heavily invested in a particular topic and have enough expertise to easily poke holes in amateur attempts, allowing them to give the impression that withholding acceptance of their idea is unreasonable and close-minded. But the value of this forum is the opportunity to hear from people with expertise, and so strengthen our own criticisms.

Linda

 

[CLD]

...they have had difficulty recruiting enough people who know each other and can all be in one place at once.

This isn't an insurmountable difficulty or even a practical difficulty. A dozen or so individuals from a community college class, book club, chess club, swim team, church congregation, pub regulars, office mates, hiking group -- virtually any situation where humans interact on a repeated basis -- would suffice.

 

[bfinn]

I don't even know if the callers used the same phone or not. It is possible that contacting Sheldrake will resolve these questions, and possibly even to his benefit. This would be a good action item to make any critique complete and fair. Did the callers/recipients have Call Display? That'd explain everything right there.

As far as I recall (either from the paper or from a description I read elsewhere) the callers were in different locations (so used different phones), and the recipients definitely did not have call display (this was explicitly stated somewhere).

I can't comment on the other queries you raised - I don't know enough about it.

>Personally, I'd love to replicate part of this study. It seems pretty straightforward.

OK, why not do that? I recommend you contact Sheldrake about this and discuss protocols with him. I get the impression he is only too happy for people to replicate his experiments.

 

[tkingdoll]

>At one point Chris French was going to redo the experiment with Sheldrake. No idea what happened to that.

AFAIK (I have met Sheldrake, who lives not far from me, and asked him about this) they have been trying to organize this, but because they are following a more stringent protocol involving videoing all callers as well as the callee, which for practical purposes involves all callers being in the same location, they have had difficulty recruiting enough people who know each other and can all be in one place at once.

I have also met Chris French, and he also (like Sheldrake) seems sound and genuinely interested in investigating this in further detail, without harbouring extreme prejudice either way (unlike some people on this forum...)

>I was against it because I hate money being spent on this sort of crap. If the new test shows no effect (which it will)

(...like I was saying.)

Extreme prejudice? No.

When someone credible, somewhere, produces one single piece of evidence supporting the existence of telepathy, then I will entertain the idea that it is provable by testing. But IF such a thing exists, it has so far proven itself to be a singularly evasive, unharnessable power under controlled conditions. I therefore see no need to assume the results for a properly controlled test will be anything but negative.

It is not prejudice to bet on the outcome of something which has so many precedents. The test will fail. They ALWAYS do.

Please do go ahead and tell me why you don't think a properly controlled test for telephone telepathy will result in no measurable effect.

 

[bfinn]

You mean that in the 'anytime' protocol, the sender and receiver could have made a prior (off-camera) agreement to phone at specific times?

eg: Emma calls at quarter-past, Gayle calls at half-past, Jayne on the 45, and Kay on the hour?

Er, no, Sheldrake isn't an idiot.

As far as I recall (sorry, haven't re-checked the paper) in each trial the experimenter rolls the die and then calls the corresponding caller and tells them to call the recipient (right away).

I don't know exactly what was meant about the 'clock synchronization' leak above, but note that in one experiment the recipient was videoed throughout, and so presumably was not able to consult a clock if some subtler code was involved (e.g. the precise second at which the caller calls).

 

[tkingdoll]

Er, no, Sheldrake isn't an idiot.

Of course not. Like the time he put out a call asking for people whose pets wake up when stared at. No flaw in that at all, for example the rather gaping question of how one asks the pet why it woke, or how one ascertains whether it would have woken anyway. Or the current request for people who can 'feel' a difference between live TV and pre-recorded. Clearly we are dealing with a genius of science.

 

[bfinn]

Extreme prejudice? No.

When someone credible, somewhere, produces one single piece of evidence supporting the existence of telepathy, then I will entertain the idea that it is provable by testing.

Er, how about the evidence of the experiments under discussion? Even if you have reservations about the protocols used, it is nonetheless 'evidence' (while not conclusive).

Or if you don't think it counts as evidence, how about conducting the experiment under more rigorous protocols in an attempt to find stronger evidence - oh no, we can't do that, that would be a waste of time & money. Because there isn't any evidence already, see!

Please do go ahead and tell me why you don't think a properly controlled test for telephone telepathy will result in no measurable effect.

Because it hasn't been conducted yet?

What's the point of conducting any experiment if you already 'know' the outcome in advance?

 

[bfinn]

Of course not. Like the time he put out a call asking for people whose pets wake up when stared at. No flaw in that at all, for example the rather gaping question of how one asks the pet why it woke, or how one ascertains whether it would have woken anyway.

It's clear from this parody that you haven't read the relevant research (I have, incidentally), in which of course such issues are dealt with. No point my arguing with you.

Is it worth my pointing out that Sheldrake has had fellowships in biology at Cambridge University and the Royal Society? I don't believe those institutions dish fellowships out to just anyone - so I suspect, contrary to the implication above, he has quite a good grasp of scientific principles.

And no doubt he has better scientific qualifications than almost anyone on this forum. Not that I think that makes his research in any way above criticism - but it does make it worthy of serious consideration (rather than uninformed dismissal).

 

[bfinn]

This isn't an insurmountable difficulty or even a practical difficulty. A dozen or so individuals from a community college class, book club, chess club, swim team, church congregation, pub regulars, office mates, hiking group -- virtually any situation where humans interact on a repeated basis -- would suffice.

You also need to get people who are prepared to take part in the experiment, probably for a number of hours, in a particular location, simultaneously. I gather (from the conversation I had) this does represent a practical problem.

Recruiting individuals is much easier than (say) groups of 4 people who know each other well. If an individual drops out or doesn't show up, you can replace them; but if an individual from a group of 4 drops out or doesn't show up then you've lost the whole group. And to get a reasonable amount of data you'll need to recruit many such groups.

 

[bfinn]

I think this illustrates something that I have seen mentioned here a few times. At some point, all of us will be out of our league when it comes to evaluating whether or not a particular idea is silly. We don't really depend upon our own capabilities, but rather let those people in the field with expertise sort out the good ideas from the silly ideas. And while some people do go on to try and provide their own (often clumsy (sorry)) reasons for derision, underneath you will find the real reason for their derision is that the idea has not drawn any converts from the relevant scientific field.

The concerns about Sheldrake's methodology are legitimate. That Randi's attempt to provide a scientific-type criticism was perhaps awkward and somewhat off the mark, does not mean that he does not understand how silly ideas get identified as such.

Unfortunately though, Randi is not some obscure innocent amateur. His word wields power, can severly damage reputations, perhaps end careers. So before ridiculing people and then awarding them widely-publicized prizes which basically label them as frauds/fruitcakes, he ought to make a properly argued case - don't you think?

It is still not clear to me whether Randi bothered to read Sheldrake's telephone telepathy paper (or even the abstract of the paper), given that he gave the impression it hadn't even been published. Anyone know?

I too am an amateur, and much less experienced in these matters than Randi, but at least I did read the research before expressing an opinion. (Hey, like a proper scientist would do!)

 

[davidsmith73]

Why did they have to make their guess after answering the phone but before the caller spoke? There's a huge opportunity for leakage in this period where the recipient can probably hear the caller's background noise, perhaps even their voice. The only thing we know is that the caller hasn't announced their name. The protocol also obviously provides the recipient with feedback of the accuracy of their guess. They can learn which background noise is associated with which caller, and future inbound call identification will be more accurate.

I would like to see a sceptical researcher perform two experiments in parallel -one with the potential flaws eliminated and one with the potential flaws present, perhaps randomising "controlled" and "flawed" trials for good measure. If the potential flaws really are to blame then the better controlled trials will show negative results, but more importantly, the flawed trials should re-introduce the positive results. "Experimenter effects" will be controlled for since its the same nasty sceptic performing the experiment.