Ganzfeld Experiments

[Uncayimmy]

This started over in the MDC forum. I am moving it here to actually discuss Ganzfeld outside of the MDC.

Because ganzfeld experiments have produced 32% hits where only 25% would be expected by chance ("Between 1974 and 2004, 88 ganzfeld experiments were done, reporting 1,008 hits in 3,145 tests" -- see http://en.wikipedia.org/wiki/Ganzfeld_experiment#cite_note-EntangledMinds-10), I think somewhere in that range would be an appropriate threshold to test. If a 30% threshold were used and the experiment were to achieve 600 hits in 2000 trials (as opposed to the expected 500 hits), the odds against would be 4.3 million to one, according to the binomial distribution.

You seem to be assuming that, if Ganzfeld experiments are showing a real psi effect, that effect can be refined and the hit rate increased to 50% or more. Why do you assume that?

I assume no such thing. I said the numbers you cited would indicate to me that my study was flawed in terms of blinding, judgment or randomization. I said that if, and it's a big if, I were to form a psi theory and saw my results improve as I tested this theory, I would think I was on to something. In other words if there is something there, I would be looking for a pattern. What would you do?

 

[Hitch]

Can someone explain this a little better than that wikipedia article. I must have misunderstood it.

It looked like to me they were doing a fairly straightforward psychic sending/reading test (with a bit of woo thrown into the isolation of subjects). Then they'd do a open ended series of tests and keep going until random chance had given them a lucky run that pushed the result to the limits of standard deviation (however long that took) then stopped the test at whatever point gave them the most anomalous reading.

Did I read that wrong? Or did they actually decide on a set number of iterations before starting the test?

 

[Rodney]

I assume no such thing. I said the numbers you cited would indicate to me that my study was flawed in terms of blinding, judgment or randomization. I said that if, and it's a big if, I were to form a psi theory and saw my results improve as I tested this theory, I would think I was on to something. In other words if there is something there, I would be looking for a pattern. What would you do?

I would also look for a pattern, but you seem to think that a hit rate in the 32% range is suspicious, whereas a hit rate in the 50% range would be less so. Why is that?

 

[Rodney]

Can someone explain this a little better than that wikipedia article. I must have misunderstood it.

It looked like to me they were doing a fairly straightforward psychic sending/reading test (with a bit of woo thrown into the isolation of subjects). Then they'd do a open ended series of tests and keep going until random chance had given them a lucky run that pushed the result to the limits of standard deviation (however long that took) then stopped the test at whatever point gave them the most anomalous reading.

Did I read that wrong? Or did they actually decide on a set number of iterations before starting the test?

I believe that the number of iterations is usually decided in advance, but what makes you think you can get an anomalous reading by stopping early or late?

 

[Uncayimmy]

I would also look for a pattern, but you seem to think that a hit rate in the 32% range is suspicious, whereas a hit rate in the 50% range would be less so. Why is that?

To be clear, we're working off a 25% chance assuming everything is completely random. Given the same number of trials, the farther away we get from 25%, the more significant it is. I think that's patently obvious to anyone with a rudimentary grasp of statistics.

In the example I gave, which was not quoted, I said that the first thing I would do is double and then triple check my protocol to ensure that there wasn't some flaw. If I didn't find anything, I'd bring other experts in to check my work. If they couldn't find anything, then I would repeat my experiment. If I got similar numbers, I would think I was on to something.

Then I would analyze my data looking for patterns in the ones that were right. Suppose I think I see a pattern. For the sake of argument let's pretend that it looks like the room temperature might be having an effect.

My next step would be to repeat my study, but controlling for room temperature rather than just noting it down. If I saw my results come in at 50% above a certain temperature and 29% below a certain temperature, I would be very excited.

Of course, I would again look for flaws in my protocol. Maybe at a certain temperature something else happens that I never noticed before.

What I am illustrating here is an approach. The Ganzfeld folks don't seem to have taken the same caution that I, your UncaYimmy, would take.

 

[RoboTimbo]

I believe that the number of iterations is usually decided in advance, but what makes you think you can get an anomalous reading by stopping early or late?

This was discussed in a thread in which you participated with fls about pavel's MDC challenge. If you don't recall it, it was fortunate that I did and provided a link to the specific post.

Early and late stopping are definitely ways to introduce anomolies into a sloppy protocol.

 

[Uncayimmy]

I believe that the number of iterations is usually decided in advance, but what makes you think you can get an anomalous reading by stopping early or late?

From what I have read of Ganzfeld, they took numbers from a bunch of different studies. Right off the bat you have the problem of not knowing if the studies they heard about were just the ones where some "positive" result happened. For example, recently I did some research about distant healing. I found studies that were funded and conducted, but the result not given or it was just mentioned that no positive results were found. Had I not known about these studies in advance and *only* looked at studies that were published with positive results, my numbers would be skewed.

But to answer your question, look at the following table. We expect a 50% hit rate. In 20 trials it would not be unusual to get a 55% hit rate. But we don't actually hit 55% until the 20th trial. Stopping at different points gives us different results.

Trial Hit Pct
1 1 100.0%
2 1 100.0%
3 1 100.0%
4 0 75.0%
5 1 80.0%
6 0 66.7%
7 1 71.4%
8 0 62.5%
9 1 66.7%
10 1 70.0%
11 0 63.6%
12 0 58.3%
13 1 61.5%
14 1 64.3%
15 0 60.0%
16 1 62.5%
17 0 58.8%
18 0 55.6%
19 1 57.9%
20 0 55.0%

Random events will have streaks. Over the long run those streaks even out. So one might assume that with Ganzfeld having 3,000 trials (or whatever it was) is sufficient to iron out those streak. Nope.

They gathered data from a bunch of different tests. If those tests did not decide in advance how many trials to do, then it is conceivable that the people doing the tests may have chosen to stop while on a hot streak or right after it started to turn cold. It's human nature, especially among those who are not smart enough to set the number of trials in advance.

Combine this possibility with people only choosing to report studies that supported their belief in psi and what do you think might happen?

It's good research to know in advance about 50 studies of 60 trials each under the same conditions and combine the results. It's bad research to take only the studies you heard about after the fact that were conducted under varying conditions and without a number of trials set in advance.

Do you understand how this could result in numbers appearing to be above random chance? Non-random elements were introduced, so of course the numbers did not appear to be random.

 

[AntiTelharsic]

I believe that the number of iterations is usually decided in advance, but what makes you think you can get an anomalous reading by stopping early or late?

I can influence die-rolls. To prove it, I will roll a die ten times and make it come up "6" at least five times. Here we go: 6, 6, ... actually, let's just stop there, because that was enough to demonstrate my ability, right? In fact, that's 100% accuracy -- I'm even better than I'd thought!

 

[Ersby]

From what I have read of Ganzfeld, they took numbers from a bunch of different studies. Right off the bat you have the problem of not knowing if the studies they heard about were just the ones where some "positive" result happened.

That's what Radin did. He took the most famous results and collected them together in his meta-analysis. Of course, the most famous results are the best results, so it's hardly surprising he got highly significant findings.

 

[cj.23]

That's what Radin did. He took the most famous results and collected them together in his meta-analysis. Of course, the most famous results are the best results, so it's hardly surprising he got highly significant findings.

Indeed. To be fair, Radin did apply a fairly robust file drawer analysis as I recall? I wonder what happens to his figures if you remove outliers like the Dalton experiments? Anyway, I'd better have another look at Entangled Minds I guess, see what he says he did. Ersby, you found a significant result in your vastly more inclusive study didn't you?

cj x

 

[cj.23]

Incidentally anyone know if the Smith/Savva Liverpool Hope experiments or Ian Hume's Coventry one been published yet?

cj x

 

[Ersby]

Indeed. To be fair, Radin did apply a fairly robust file drawer analysis as I recall? I wonder what happens to his figures if you remove outliers like the Dalton experiments? Anyway, I'd better have another look at Entangled Minds I guess, see what he says he did. Ersby, you found a significant result in your vastly more inclusive study didn't you?

cj x

Why do a calculation regarding a putative file-drawer when you actually have details of the things he left out?

I've been working on a new meta-analysis with properly defined criteria (re standardness and methodological rigour) and am sending the data to people to see what they think. Ideally I'd like to send the data to a statistician, removing any reference to the nature of the studies, to see what they make of it (if anyone knows of someone who'd do this, let me know). The data is still statistically significant, using a stouffer z measure, but plotting the results on a graph shows no funnel shape - it's just a bit of a mess.

 

[Rodney]

This was discussed in a thread in which you participated with fls about pavel's MDC challenge. If you don't recall it, it was fortunate that I did and provided a link to the specific post.

Early and late stopping are definitely ways to introduce anomolies into a sloppy protocol.

Early and late stopping is relevant only if a low odds standard is used. It is completely irrelevant to the Ganzfeld results summarized by Dean Radin and reported in the Wikipedia article that I cited.

 

[Rodney]

Why do a calculation regarding a putative file-drawer when you actually have details of the things he left out?

What, exactly, are you talking about? In his 2006 book, Entangled Minds, Dean Radin states at page 120: "From 1974 through 2004 a total of 88 ganzfeld experiments reporting 1,008 hits in 3,145 trials were conducted."*

"*This excludes a few of the earliest ganzfeld studies that couldn't be evaluated with a hit vs. miss type of analysis."

I've been working on a new meta-analysis with properly defined criteria (re standardness and methodological rigour) and am sending the data to people to see what they think. Ideally I'd like to send the data to a statistician, removing any reference to the nature of the studies, to see what they make of it (if anyone knows of someone who'd do this, let me know). The data is still statistically significant, using a stouffer z measure, but plotting the results on a graph shows no funnel shape - it's just a bit of a mess.

If you do that, I would suggest asking the statistician about the validity of using the Stouffer Z vis-a-vis the binomial distribution. You might also try e-mailing Beth, who participates here. She says that she's a professional statistician.

 

[Uncayimmy]

What, exactly, are you talking about? In his 2006 book, Entangled Minds, Dean Radin states at page 120: "From 1974 through 2004 a total of 88 ganzfeld experiments reporting 1,008 hits in 3,145 trials were conducted."*

You realize that's an average of just 35 trials per experiment, right?

Can you provide any specifics about the studies? Ideally I like to see a list of all 88 with the number of trials and hits for each. That would be a good starting point.

 

[cj.23]

You realize that's an average of just 35 trials per experiment, right?

Can you provide any specifics about the studies? Ideally I like to see a list of all 88 with the number of trials and hits for each. That would be a good starting point.

Are you a student? If so for about $25 you can get access to a database that allows access to titles that include the JSPR, PSPR, EJP, and the JoP as I recall. I have a feeling it is available to non-students at amuch higher price - I'll try to find out... Anyway that should give you a large number of the studies to read???

cj x

 

[Uncayimmy]

Are you a student? If so for about $25 you can get access to a database that allows access to titles that include the JSPR, PSPR, EJP, and the JoP as I recall. I have a feeling it is available to non-students at amuch higher price - I'll try to find out... Anyway that should give you a large number of the studies to read???

cj x

I'm not a student, but my wife works at a college. I'm definitely interested.

 

[Ersby]

What, exactly, are you talking about? In his 2006 book, Entangled Minds, Dean Radin states at page 120: "From 1974 through 2004 a total of 88 ganzfeld experiments reporting 1,008 hits in 3,145 trials were conducted."*

"*This excludes a few of the earliest ganzfeld studies that couldn't be evaluated with a hit vs. miss type of analysis."

If you do that, I would suggest asking the statistician about the validity of using the Stouffer Z vis-a-vis the binomial distribution. You might also try e-mailing Beth, who participates here. She says that she's a professional statistician.

Radin says there are 88 experiments in that timeframe, but that doesn't alter the fact that there are about 142. The "hit/miss" reason for leaving experiments out does not make any sense if you read the original papers.

I used the stouffer z because that was the measure used most often by parapsychologists. Of course if the statisician decides a different measure is more apt, I'd bow to their knowledge.

 

[Ersby]

I'm not a student, but my wife works at a college. I'm definitely interested.

I think he's talking about www.lexscien.com

It gives you a free week's trial, with limited downloads, so take a look.

 

[fls]

Indeed. To be fair, Radin did apply a fairly robust file drawer analysis as I recall?

No, he used a fail-safe N, which assumes no bias in the selection of unpublished papers. I hope that you can see this assumption would be unwarranted. Dropping this assumption downgrades the number of studies sitting in file-drawers by several orders of magnitude.

http://www.csicop.org/sb/2002-12/reality-check.html

Linda