The Ganzfeld - my findings to date

[Ersby]

Posting this for comments/criticisms.

Well, it's been long enough so the other day I chucked out my previous calculations, went right back to the raw data, and dug around in Excel until I found the right formula so it could do the sums properly. I then used a piece of free meta-analysis software to crunch the numbers for me. This is what I got.

Number of experiments=65
Number of trials=2,854
Weighted average z-value=1.4571
Significance (one-tailed) P=0.07 (or 1 in 13)

In the instructions for this piece of software, it recommended transforming the data into r values, so I did that too.

Population effect size=0.02792 (in other words, very small)

Some people may have seen from the filenames in the thumbnail below that this is the homogenous version of the database. The first time I ran the data through the program I got a statistically significant (at odds of 1 in 150,000 approx) but very heterogeneous.

According to the instructions, this is not acceptable, so I trimmed the outliers until I got the homogenous results.

Also, apologies to people I’d sent data/notes to. If they looked over my work, I hope they don’t feel they wasted their time now that I’ve posted my own figures before hearing their recommendations. I’m still interested in what you have to say – if I’ve made any mistakes, let me know!

Next, the inclusion criteria used for putting this database together. Again, any comments are welcome.

No unpublished papers, media demonstrations, or classroom demonstrations
No experiments labelled pilot, informal or exploratory
No papers without pre-defined numbers of trials
No unfinished experiments (one exception – Smith, Savva’s 2008 experiment fell just ten short of its target of 128 trials. It felt churlish to exclude it)
Sargent’s work at Cambridge excluded due to security issues
No experiments that allow contact between someone knowledgeable about the target and the person judging the session
Only experiments that used random number generators or random number tables to chose the target set and target
Only experiments that had duplicate target sets for judging purposes
Only experiments that used red light as the visual stimulus
Only experiments that used white noise or pink noise (ie, static) as the audio stimulus
No experiments of very short duration (15 minutes or under) or very long (45 minutes or longer)
Only visual targets
Only one target per session

All of these are issues which have been raised by parapsychologists with regard to the ganzfeld. I haven't just plucked them out of the air. In fact, between me only choosing criteria according to the writings of other people, and getting the computer to run the numbers for me, I feel like I've removed myself as much from the process as is possible.

Lastly, I have to admit being amazed at the result. Everything I'd done to date had pointed to odds of around one in ten thousand. When it came out with 1 in 13, I was stunned.

(below - scattergraph of p numbers and trials for the database. P=0.5 is chance. Then two result screens from the program I used -
http://web.fu-berlin.de/gesund/gesu_engl/meta_e.htm )

 

[Aepervius]

For those of us which are a bit too dumb to udnerstand your screenshot, can you explain ?

Also

Lastly, I have to admit being amazed at the result. Everything I'd done to date had pointed to odds of around one in ten thousand. When it came out with 1 in 13, I was stunned.

What is the 1 out ten thousand and what is the 1 out of 13 odd on what ?

 

[Ersby]

Basically, I think that the database of all ganzfeld experiments into ESP, when sorted for methodological soundness and standardness of protocol, don't give a statistically significant result. The 65-experiment database has an overall hit rate of 27.4%.

For now I’m limiting myself mostly to p values, because they’re simple.

The result of p=0.07 translates, roughly into odds of 1 in 13, ie, a 7% chance of these results occurring if there was no effect.

The “one in ten thousand” was what I thought the end result would be before I sat down and started again from scratch. That was based on my previous (erroneous) sums and the work of others. It was quite a rough estimate.

But Wikipedia has a decent page on the subject of p values, including the many misunderstandings. I’m sure I’m making at least one of these mistakes. I hope someone here can point out which one(s).

http://en.wikipedia.org/wiki/P_value

 

[cj.23]

According to the instructions, this is not acceptable, so I trimmed the outliers until I got the homogenous results.

There may be an issue here - but ''ll let those who know what they are talking about deal with that. Certainly trimming would be normal in thsi situation, but I'm not sure... It certainly precludes fraud and massive experimenter error to some extent though!

Next, the inclusion criteria used for putting this database together. Again, any comments are welcome.

No unpublished papers, media demonstrations, or classroom demonstrations
No experiments labelled pilot, informal or exploratory
No papers without pre-defined numbers of trials
No unfinished experiments (one exception – Smith, Savva’s 2008 experiment fell just ten short of its target of 128 trials. It felt churlish to exclude it)
Sargent’s work at Cambridge excluded due to security issues
No experiments that allow contact between someone knowledgeable about the target and the person judging the session
Only experiments that used random number generators or random number tables to chose the target set and target
Only experiments that had duplicate target sets for judging purposes
Only experiments that used red light as the visual stimulus
Only experiments that used white noise or pink noise (ie, static) as the audio stimulus
No experiments of very short duration (15 minutes or under) or very long (45 minutes or longer)
Only visual targets
Only one target per session

Here I do know what I am talking about, and yes very much a solid list. What happens if you exclude Smith & Savva (2008)? I can't imagine it will make much difference!

Great research, look forward to hearing more on this. You going to publish it?

cj x

 

[cj.23]

Do you mean p=0.5 is chance? Surely you mean p=0.5 is significance?

cj x

 

[Ersby]

The funny thing about trimming outliers is that I got that technique from the discussion over Milton & Wiseman's meta-analysis. Someone (Radin, I think) said that trimming the outliers made the database (a) homogenous, and (b) significant. But I'd like more input on this topic too.

p=0.05 is significant (at odds of 1 in 20). p=0.5 is chance (odds of 1 in 2).

 

[paximperium]

Do you mean p=0.5 is chance? Surely you mean p=0.5 is significance?

cj x

A p-value means the results are due to chance.

A p=0.5 means a 50% probability that the results were due to random chance.

Most use p=0.05 as the cut-off.

 

[cj.23]

The funny thing about trimming outliers is that I got that technique from the discussion over Milton & Wiseman's meta-analysis. Someone (Radin, I think) said that trimming the outliers made the database (a) homogenous, and (b) significant. But I'd like more input on this topic too.

p=0.05 is significant (at odds of 1 in 20). p=0.5 is chance (odds of 1 in 2).

Sorry, yes I know that - well I should, given my former job! I was being particularly dim and misreading. I have spent the morning working with fractions of fractions since blobru asked about my math on the fundie thread... So yeah, 93% likely to be not chance, so a couple of percent under significance, and quite possibly an artefact. Not significant anyway! Interesting!

Well if you trim the data set, say removing Kathy Dalton's extreme results, you will get a more homogenous data set - and it should work just fine, without distorting at all. Did you use a file drawer correction? If we apply that it may be the whole thing drops to chance.

How however i see a chance to so something really fun. Is there anyt way we can test your data for chronological sequence, ie how do results change over time? I see a continual move towards more and more chance results - perhaps suggesting experimental procedures are improving, or the reality of an expectation Experimenter Effect, or that Loki does not like Ganzfeld anymore. Anyway of checking for a decline effect?

 

[Ersby]

I can't do much with the data at the moment, since the program I use is on my computer at home.

I do have the excel files, though, and it's interesting to note that the "decline effect" has gone. If anything, results get slightly better. (Again, using p values on the y-axis, bearing in mind a low value indicates better results)

 

[Professor Yaffle]

A p-value means the results are due to chance.

A p=0.5 means a 50% probability that the results were due to random chance.

Most use p=0.05 as the cut-off.

Not quite. It means that if there were only chance/random effects (null hypothesis is true) you would expect to see results like this once out of 20 times. You can't reverse this and say given a particular set of results, the chance that the null hypothesis is true is one in twenty.

 

[cj.23]

I can't do much with the data at the moment, since the program I use is on my computer at home.

I do have the excel files, though, and it's interesting to note that the "decline effect" has gone. If anything, results get slightly better. (Again, using p values on the y-axis, bearing in mind a low value indicates better results)

lol! Well that is another myth ended then! I do agree your inclusion criteria are exactly what the Ganzfeld advocates have suggested - have you got Ian Humes 2008 study in there? Might not be published yet? It is complete - email him at Coventry ask for it?

cj x

 

[Aepervius]

So .... That mean you get a ~27-28% and if those result were due to chance alone you would expect result like this in 1 chance out of 13 not a p value that confident if I read between the line...

If i read that corectly , then the infamous ganzfeld experiment are much addo to nothing... Or am I completely misinterpreting you ?

 

[cj.23]

So .... That mean you get a ~27-28% and if those result were due to chance alone you would expect result like this in 1 chance out of 13 not a p value that confident if I read between the line...

If i read that corectly , then the infamous ganzfeld experiment are much addo to nothing... Or am I completely misinterpreting you ?

Chance should give 25%. Over a very large number of trials the result is closer to 27 to 28% above chance, but not at significance - so it may just be "noise", not a real effect. I think that sums it up, but I could be wrong!

cj x

 

[Ersby]

That's how it looks. I'm going over it again, and keep expecting to find a huge mistake that I've overlooked. Nothing yet, but I'll keep looking.

I've not seen the Coventry experiment. I don't know if it'll go in, since I have to draw the line somewhere. As for writing a paper and getting it published, it's possible. I've got something half finished, but it'll need a lot of work to knock it into shape.

 

[fls]

Now that you've made me feel guilty...

(I have a few excuses - my kitchen/family room/flooring renovations have become far more consuming than I imagined, and the family suffered a prolonged bout of illness throughout February. )

Things I hadn't finished working out:

The extent to which your inclusion/exclusion criteria would be found to be acceptable to parapsychologists. Cj.23 may have given you an answer on that one. I have avoided, as much as possible, discovering the results of the studies so as to be uninfluenced by them when considering inclusion/exclusion. I'm not sure about the Sargent study - the problem is that the discovery of the randomization/security problem was happenstance. Unless you apply that same process of discovery to other researchers, it's an arbitrary exclusion. Are papers presented at a conference, but not subsequently published excluded (I'm thinking of Dalton)?

How did you weight your z-scores? With N?

Whether to use random-effects vs. fixed-effects.

What to do about the heterogeneity. How did you trim it? Just throw away outliers or is that how you came to your exclusion/inclusion criteria? I was going to recommend meta-regression. Then groups could be formed on the basis of significant influences that may be more homogeneous.

How to account for publication bias. I'm of the opinion that it's better to describe something, than to pretend that you can fix it after the fact. Did you do a funnel plot? This may be a moot point without significant results, though.

This is what I can remember off the top of my head. I don't have access to my computer right now.

Linda

 

[Ersby]

Now that you've made me feel guilty...

Success!

I'm not sure about the Sargent study - the problem is that the discovery of the randomization/security problem was happenstance. Unless you apply that same process of discovery to other researchers, it's an arbitrary exclusion. Are papers presented at a conference, but not subsequently published excluded (I'm thinking of Dalton)?

Sargent's work was criticised by a number of people, not just Susan Blackmore. When Jessica Utts did a paper using the ganzfeld, she removed Sargent's data. All things considered, I decided to remove them.

But, I should clarify that Sargent's later work, when the method was changed so the security flaw no longer applied, is included.

Papers presented at conferences are included, to guard against successul experiments being written up properly for publicaton, while less successful work is dropped.

How did you weight your z-scores? With N?

According to the program's instruction manual, yes.

Whether to use random-effects vs. fixed-effects.

I haven't got that far into the manual.

What to do about the heterogeneity. How did you trim it? Just throw away outliers or is that how you came to your exclusion/inclusion criteria? I was going to recommend meta-regression. Then groups could be formed on the basis of significant influences that may be more homogeneous.

I just removed outliers until the database became homogenous. According to the program, one index is still showing as heterogenous, but rather than trim a fourth experiment, I decided two out of three indices was good enough.

How to account for publication bias. I'm of the opinion that it's better to describe something, than to pretend that you can fix it after the fact. Did you do a funnel plot? This may be a moot point without significant results, though.

I'm fairly sure this isn't an issue.

This is what I can remember off the top of my head. I don't have access to my computer right now.

Linda

Thanks.

 

[Limbo]

This morning I took the liberty of e-mailing a link to this thread to Jessica Utts, asking her if she could spare a few minutes to glance at it and maybe offer some thoughts. I just got home from work a few minutes ago, and lo! A brief response:

"I didn't read all of the entries, but from the initial analysis it looks like the author made a classic mistake I warn my intro students about on day 3 or so - removing outliers just because they are outliers. In a case like the ganzfeld, it is particularly egregious because there are only likely to be high outliers, not low outliers in the other direction. (Low outliers would mean hit rates much less than 25%, which are not likely to happen.) As an analogy, it's like someone saying that the average fever when people have the flu is only about 100, and not a problem, after removing all temps above 102 from the data set because they are "outliers" and not consistent with the "normal" temperatures. That would, of course, produce an artificially low average. That's what has happened here.

Hope that helps." -Jessica Utts (personal correspondence)

 

[Rodney]

Basically, I think that the database of all ganzfeld experiments into ESP, when sorted for methodological soundness and standardness of protocol, don't give a statistically significant result. The 65-experiment database has an overall hit rate of 27.4%.

For now I’m limiting myself mostly to p values, because they’re simple.

The result of p=0.07 translates, roughly into odds of 1 in 13, ie, a 7% chance of these results occurring if there was no effect.]

If the number of trials was 2854 and the hit rate was 27.4%, that means there were 782 hits. With an expected 25% hit rate, the mean number of hits would be only 713.5. The odds of at least 782 hits occurring, using a binomial test, is only 0.16%, or about one chance in 625. So through what sort of statistical legerdemain did you come up with only one chance in 13?

 

[Rodney]

This morning I took the liberty of e-mailing a link to this thread to Jessica Utts, asking her if she could spare a few minutes to glance at it and maybe offer some thoughts. I just got home from work a few minutes ago, and lo! A brief response:

"I didn't read all of the entries, but from the initial analysis it looks like the author made a classic mistake I warn my intro students about on day 3 or so - removing outliers just because they are outliers. In a case like the ganzfeld, it is particularly egregious because there are only likely to be high outliers, not low outliers in the other direction. (Low outliers would mean hit rates much less than 25%, which are not likely to happen.) As an analogy, it's like someone saying that the average fever when people have the flu is only about 100, and not a problem, after removing all temps above 102 from the data set because they are "outliers" and not consistent with the "normal" temperatures. That would, of course, produce an artificially low average. That's what has happened here.

Hope that helps." -Jessica Utts (personal correspondence)

Even neglecting the problem of excluding inconvenient outliers, if you e-mail her again ask if she can see any way that a 27.4% hit rate in 2854 trials (with a 25% hit rate expected by chance) can produce a statistically insignificant result.

 

[Professor Yaffle]

never mind