-
Security incident: ISF was recently accessed by intruders. Please change your password, and change it anywhere else you used it. Read more
Chris Carter Exposes Richard Wiseman Misdeeds
- Thread starter alex.tsakiris
- Start date
Beth
Philosopher
- Joined
- Dec 6, 2004
- Messages
- 5,598
Thanks for the effort, but it doesn't really help me understand what is going on. But that's my fault, not his. I'll try to reread the paper before I comment again.
My first gut reaction was negative, too. Pooling data from different experiments and treating them as if they were all the same is a general No.
However, after a bit of thinking I find the exact binomial to be as valid as the Stouffer z.
The pooling is only done to calculate a p-value, which of course assumes the null. IE no effect of any kind. The baseline supposedly is the same in all studies and caculated with probability theory.
What's problematic is that it's unclear what exactly leads to a rejection of the null. The exact binomial is more sensitive at detecting something but is it detecting psi or just publication bias?
The reason why is explained in the paper.
Wiseman and Smith tested if the dog did what it was claimed to do. IE whether he went to the window at a certain time.
What do you think Sheldrake did? What do you think the data shows?
In the first trial Wiseman and Smith used as a criterion the first time the dog went to the window. The went stayed only for a few moments, though, and was too early.
After that, the owners suggested that he may have been simply distracted so that he was required to stay in the spot for 2 minutes in the following trials.
Maybe you're remembering that.
The file-drawer thing reminds me of the reasoning used to try to dismiss the apparent fine-tuning of this Universe by invoking a multiverse.
i.e. the data do not fit with your belief system, therefore you imagine-into-existence a load of other data that would fit your belief system... if they existed.
Just as with the multiverse get-out, the file-drawer get-out is pretty much irrefutable... hence its usefulness to a faith-driven soul like Wiseman.
i.e. the data do not fit with your belief system, therefore you imagine-into-existence a load of other data that would fit your belief system... if they existed.
Just as with the multiverse get-out, the file-drawer get-out is pretty much irrefutable... hence its usefulness to a faith-driven soul like Wiseman.
Last edited:
He discusses this in the paper. The family did not think that shorter visits than this represented his signalling behaviour. And remember, Wiseman was testing the claims which were made - that the behaviour of the dog predicted the return of the owner.
As Wiseman said in your interview of him, his design tests the claims of the owners - that changes in the dog's behaviour signals the return of the owner. Sheldrake was testing a different question.
Well time at window vs. time away from window just tells you that the dog's behaviour varies. It doesn't address the claim which started all this, which is that Pam's parents knew when she was returning based on the dog's behaviour. Instead it addresses the idea that patterns in the dog's behaviour are present, including a pattern which would be present if the dog was reacting to the return of the owner, but would also be present for other reasons.
I wouldn't expect Wiseman to defend testing which didn't address the claim, since he seems to have consistently stated that he was interested in the claims made about the dog's behaviour rather than the more open-ended study which Sheldrake performed which was more exploratory than hypothesis testing.
Linda
Yes. This is what I said earlier. It is somewhat valid to combine the studies if there is no difference in effect from study to study. Not necessarily that there is null effect, but rather that the same effect is being measured each time and the only variation between studies is due to sampling. And it is reasonable to pool the results if the baseline is identical. Putting aside the issue that the baseline is not identical (the extent to which uniform numbers are chosen of each target position varies) which would invalidate the premise anyway, it still could be the case that the effect is the same.
However, this is not the claim made by the psi proponents. They claim the opposite - that variations in studies lead to variations in the effect. Creative people exhibit more psi, static targets are different from moving targets, introverts differ from extroverts, standard ganzfeld tests are more reliable than those with altered components, etc. And the claim is that tests with higher hit rates represent tests where more psi effects are present. So their own claims invalidate the assumptions under which the data could be pooled.
But even if they weren't making these claims, tests of the ganzfeld data show that it is overly heterogeneous anyway. One might question whether it is even reasonable to combine the studies (using random effects, rather than fixed effects). It certainly precludes the more strict criteria for pooling studies.
Linda
The paper "A Dog That Seems To Know When His Owner is Returning: Preliminary Investigations" (Sheldrake, Smart, Journal of the Society for Psychical Research 62, 220-232 (1998)) seems to focus on the time when the dog, Jay-tee, began to respond and how that matched when Pamela Smart began to return. Not that he went to the window most in the ten minutes before Pamela Smart arrived - which is the method used in a following paper by Sheldrake.
http://www.sheldrake.org/Articles&Papers/papers/animals/pdf/dogknows.pdf
http://www.sheldrake.org/Articles&Papers/papers/animals/pdf/dogknows.pdf
Attachments
I haven't focused on that issue, but rather the general issue of meta-analysis. The Milton/Wiseman article has a footnote that states: "For each study that used the usual method of measuring its outcome by comparing the number of hits obtained to the number expected by chance, a z score was derived from an exact binomial test . . ." My understanding is that 26 of the 30 ganzfeld studies fell into that category. So, could not the trials and hits in those 26 studies have been aggregated and a binomial test used to determine aggregate statistical significance?
But aren't you concerned that the results obtained by Milton and Wiseman may simply be in error?
Putting aside the issue that they cannot be aggregated without knowing that target choices were uniform, they can be aggregated if you think that any of the differences between experiments had no effect on psi - that is standardness, personality characteristics of the participants, experimenter effects, types of targets, etc. do not modify the effect of psi. You can pool data if you have good reason to think that you are measuring the same effect in the same way in every experiment. Otherwise you need a standardized measure (such as effect size) for each study in order to combine the data.
I would be if I thought their study proved something, but I don't. As I have said on numerous occasions, metanalysis is a very weak method under these conditions, so all of these metanalyses are useless anyway. Discovering whether one metanalysis is slightly more useless than another is of low priority. And it seems to give parapsychologists and believers the impression that overcoming these nitpicks somehow serves to prove that psi is real, when even a perfectly performed metanalysis would still be useless in this regard.
Linda
alex.tsakiris
Scholar
- Joined
- Aug 26, 2007
- Messages
- 63
I don't think so... but the "data" in this case was the video of the dog. So, the "collection of the data" was exactly the same for Wiseman and Sheldrake... especially since Wiseman used Sheldrake's camera
alex.tsakiris
Scholar
- Joined
- Aug 26, 2007
- Messages
- 63
alex.tsakiris
Scholar
- Joined
- Aug 26, 2007
- Messages
- 63
Pam (and the family) never said that... please provide a quote from them.
Wiseman was brought in by Sheldrake to collaborate on Sheldrake's research... Wiseman turned it into a debunking exercise of the worst kind.
You're buying into Wiseman double-speak... ask, what question was Sheldrake asking?
More silliness... correlate dog behavior to owner behavior and you have evidence of telepathy.
alex.tsakiris
Scholar
- Joined
- Aug 26, 2007
- Messages
- 63
Not sure I understand your point... result are very impressive either/any way you slice it.
Also, interesting to note that the 10-minute interval thing (kinda klunky and unnecessary) was introduced by Wiseman during the initial collaboration with Shelrdrake... of course he later abandon it (without telling Sheldrake) and went with the 2-minute cut-off.
.
This is what Utts says that Milton and Wiseman did:
So to calculate the p number, you take into account the sample size. Right? But Chris says they don't do that.
Let’s take a look at the data. First, if what Chris says is true and M&W ignored sample size, then those with the same hit rate should return the same z score. But that isn’t what we see. Looking at the four experiments that scored 37% (the most common hit rate in the database) we find:
Johansson & Parker, Study 2, n=30, z score=1.25
Johansson & Parker, Study 3, n=30, z score=1.25
Broughton & Alexander, Emotionally close series, n=51, z score=1.81
Kanthamani & Broughton, Series 4, n=65, z score=2.01
So, the z scores are getting larger with n, despite the hit rate being the same, which is what we’d expect.
Just to double check, let’s compare M&W’s z scores to the original z scores as reported in the individual papers.
Johansson & Parker, Study 2, M&W z score=1.25, author’s z score=1.26
Johansson & Parker, Study 3, M&W z score=1.25, author’s z score=1.26
Broughton & Alexander, Emotionally close series, M&W z score=1.81, author’s z score=1.81
Kanthamani & Broughton, Series 4, n=65, M&W z score=2.01, author’s z score=2.014
Which would indicate that either M&W’s method of calculating the z scores is fine (at least for those experiments with hit rates) or that they simply use the original author’s z score which is fine, too. Isn’t it?
Just to recap, Chris Carter's claim on this subject is that "The 30 studies that Milton and Wiseman considered ranged in size from 4 trials to 100, but they used a statistical method that simply ignored sample size (N)" which I don't see myself.
Maybe someone with a little more statistical nous could comment.
Last edited:
Just clarifying where Wiseman and Smith's criteria came from.
alex.tsakiris said:
Carter is suggesting that each study be weighted twice based on size. First, he admits that a larger study will give a larger z-score than a smaller study with the same proportion of hits. Since the Stouffer z involves adding the z-scores, a larger z-score adds more to the total than a smaller z-score. In this way, larger studies already contribute more to the results than smaller studies. He doesn't make any sort of case as to why a second weighting is useful, nor does he explain why Ersby's recognition of this effect means "he just does not understand z-scores and p-values", while Carter's recognition of this effect presumably does not.
Utts does demonstrate that you get different results depending upon how you combine the data. Here's the problem though. She makes no attempt to demonstrate that any one of those results is closer to the truth than the others. It makes no sense to simply presume that the exact binomial test gives you the most reliable and valid result, especially since the differences between the two studies looks like more than just sampling differences, on its face. Simply assuming that only those tests which tell you what you want to hear are correct (which means that the 'correct' on one occasion may turn out to be the 'wrong' test the next time around) and the rest are wrong, is part of why parapsychology is in the state it is in.
Linda
When reviewing RW’s videotape PS correctly remarked that Jaytee only stayed at the porch
for a fairly brief period of time during his 19.57 visit (53 seconds) and that a better indicator
of his signal might be him remaining there for a longer period of time. There were three
occasions when Jaytee stayed at the porch for more than 2 minutes (20.09, 20.58, 21.04) and
two of these were close to the departure time of 21.00. For this reason the authors decided
that any future study should not take the first time that he inexplicably went to the porch as
his ‘signal’ but instead, the first time that he inexplicably visited the porch for more than two
minutes . In addition, PS noted that there were many distractions outside the porch and these
may have caused Jaytee to pay an unusually high number of visits to the porch, resulting in
some unusually ‘noisy’ data. It was hoped that this problem could be minimised by running
the next experiment in the afternoon rather than early evening.
http://www.richardwiseman.com/resources/petsBJP.pdf
No, you don't. There may even be a correlation that contradicts telepathy, such as in this case.
If behavior A occurs before behavior B then behavior B cannot have telepathically triggered behavior A.
If behavior B is triggered by some outside factor (like the experimenter throwing a dice) then we know that behavior B is not triggered telepathically by behavior A.