Near Death Experiences Explained... Debunked

[Maia]

(laces up running shoes to get a headstart)

Now, before everyone starts chasing me out of town waving virtual torches, y’all should know that what follows is not an argument that NDE’s represent proof for God and angels and the Second Coming of Jesus bringing us all back to heaven. But I can’t stand what the popular press does to scientific research, and to be perfectly honest, I was not impressed by this research anyway. Here’s why. Please read all the way to the end before commenting. I did a lot of original work on this, ALL BY MYSELF. Take a deep breath, and let’s dive in!

Here's the actual real-live study yapped about in National Geographic:

Right here.

On careful analysis, the results are so far from anything that was reported in National Geographic (or anywhere else) that it’s simply embarrassing. The differences are so significant. But you really do have to do some digging.

The claim was that:

Patients with higher petCO2 had significantly more NDEs. Patients with higher pCO2 had significantly more NDEs. Patients with previous NDEs had significantly more NDE's. The NDE score was positively correlated with pCO2 and with the serum level of potassium. Patients with lower pO2 had more NDEs, although the difference was not statistically significant.

I decided not to try to chase down info about potassium levels, etc., because those weren't the main claims that had been passed down through the popular press (although the part about the patients with previous NDE's having more was very interesting.) Here we go:

Table 2
Correlation of independent variables with the presence of NDEs
Variable NDEs group (mean ± SD) Non-NDEs group (mean ± SD) P
________________________________________
Age (years) 57.9 ± 13.8 51.8 ± 14.6 0.217
Time until ROSC (minutes) 8.3 ± 6.7 8.8 ± 5.3 0.772
petCO2 (kPa) 5.7 ± 1.1 4.4 ± 1.2 < 0.01
pO2 (kPa) 16.4 ± 11.1 25.3 ± 15.1 0.108
pCO2 (kPa) 6.6 ± 2.3 5.3 ± 1.4 0.041
Serum sodium (mmol/l) 139.2 ± 6.1 140.4 ± 4.0 0.439
Serum potassium (mmol/l) 4.6 ± 1.2 4.1 ± 0.8 0.118
________________________________________
NDE, near-death experience; petCO2, initial partial end-tidal pressure of carbon dioxide; pCO2, partial pressure of carbon dioxide; pO2, partial pressure of oxygen; ROSC, return of spontaneous circulation; SD, standard deviation.
Klemenc-Ketis et al. Critical Care 2010 14:R56 doi:10.1186/cc8952

If you have a TI-83 plus calculator, you may want to get it out and follow along. First, I made little charts of NDE and non-NDE-rs, with n, xbar (sample mean), and standard deviations. For PCo2, for example, the NDE group had an n of 11, a sample mean of 6.6, and a standard deviation of 2.3; the non-NDE group had an n of 41, a sample mean of 5.3, and a standard deviation of 1.4. The null hypothesis was that the population means were equal. The claim was that population mean one was greater than population mean two. Here's where the TI-83 comes in.

STAT-TESTS-2SAMPTTEST. More about why this test was chosen later.

Now enter in your data. Your p value is .049469,but this is NOT WHAT THE RESEARCHER SAY. They claim that the p value is .041. It is not. Get out your stats calculator and enter the numbers yourself. The alpha value they chose is the very common one of .05. This means that they've barely, BARELY inched over into showing an effect for that pCo2. This is just barely statistically significant, and it has no practical significance by any definition. If you round up at 2 decimal places rather than 3 (which you're not supposed to do, of course, but let's just say for the sake of argument...) you cannot reject the null hypothesis.

The numbers get more robust for the petCo2 levels, because the p value is .0017. But there's a tremendous problem here. The multivariate analysis found that:

Higher pCO2 was an independent predictor of NDEs. The logistic regression model explained 46% of the variation (Table 3). A higher NDE score was independently associated with higher pCO2, higher serum levels of potassium, and previous NDEs. The linear regression model explained 34% of the variation (Table 4).

It's all very easy to say that pCO2 was an independent predictor of NDEs, but its effect was so extremely minor that the independent predictor effect couldn't have been anything but negligible. Nothing plus nothing equals nothing.

Now, a stats expert I am not. But think about this: they said that they first used univariate analysis, considering each variable separately. By the time they got to the information in Table 2, what else could you use for those particular results besides an independent samples t-test for 2 means? What they actually said in the study was:

To identify statistically significant differences between different variables, we used an independent samples t-test, chi-squared test, and a Wilcoxon rank sum test.

They used SPSS 13.0, which means this wasn't done by hand. I'm not so sure this is always a good idea, because you can really miss some things. And for this particular set of relationships between variables, nothing else really makes any sense besides a t-test that I can see, because you’re testing two groups separated by one defined characteristic (NDE vs. non-NDE) to see if you can come up with good evidence that they come from a population whose “averages” are different on this, that, and the other defined property (various blood gas levels). (You wouldn’t use ANOVA, for instance, because you only have 2 groups. You wouldn’t use a chi‐square test because it isn’t nominal data, you wouldn’t use a Kolmogorov–Smirnov test because it isn’t ordinal data.)Maybe… I don’t know, I’m just trying to come up with something here… they used some adjustment factor or other to come up with the differing numbers (.041 rather than .04596), in order to compensate in some way for the fact that n most certainly did not equal 30 for the NDE group (more about that later)… a stats expert might know more about if this is even possible, but then I think they would have to have said something more specific about it than they did. A straight t test for 2 means gives a result of 049469, not .041.

The only obvious way to get this number that I can see would be to artificially conflate the sample sizes. For instance, if you say that you had 100 subjects in each group (n1 and n2 each = 100,) then you’ll end up with p < .0001. The effect would be very robust if you could just get a large enough sample size. But that is just about the worst methodology that anybody could ever even imagine coming up with. The sample only consisted of 52 people, 11 (yes, eleven) of which had NDE’s. This was from an original pool of 400 who had cardiac arrest, and only 76 survived! Ack. Seven of the 11 NDE’ers were atheists. I’m not sure if this was unusual or not, though, because I have no idea what the religious makeup of Slovenia is.)

I don’t know. I’m really, really confused about this point, because I basically just do not see how they came up with a figure of .041 based on their data. Even so, a P-value of .041 is not the least bit impressive compared to an alpha of .05 either.

Overall, higher petCo2 was not an independent predictor of NDE's, and higher PCo2 was but just barely, barely managed to cross over into even statistical significance. The only conclusion to really come to is that this study simply does not mean what it has been ballyhooed to mean.

When all is said and done, of course, there’s also a fatal flaw that totally and completely invalidates the kind of conclusions that have been drawn from this study by the popular press, and it's that in order to draw any real conclusions from these stats methods, the sample sizes absolutely must be at least n = thirty for each group. The NDE group was n = 11. The researchers said clearly that this was a prospective observational study. That was the responsible thing to say because of its limitations, but particularly because of the size problem.

The final lesson to take away from all of this, I think, is to just not believe the version you read in the popular press. Complex neurobiological phenomena do not have simplistic explanations. I’ll be sending all of this information to National Geographic, and I’ll let them know what I think of their fact-checking quality. All of this took me half an hour, and really, most of the people on this board could have done it too.

 

[jhunter1163]

Impressive work, Maia. I couldn't have done it if you gave me 30 months. Two virtual thumbs up.

 

[devnull]

now, where did I put my virtual torch?

 

[Baby Nemesis]

now, where did I put my virtual torch?

I've got mine.

On the face of it, that seems impressive, Maia. However, I'm in no position to judge, not having had a science education and all that. I think there may be cause not to accept what you say at face value though, since you do have a history here of making suspect and even extraordinary claims. Take the study you enthused about in your A Rational Exploration of NDE-Related Research thread. Naturally, making dubious or mistaken claims about near death experience research in one thread isn't in itself serious cause to doubt your analysis in this one. It might be highly accurate. Still, it might be cause for anyone who actually understands what you wrote to cast an extra-critical eye over it.

You said, in the other thread:

First of all, I have absolutely no patience with the topic of NDE research somehow getting confused with anything related to the "paranormal." I think this has been a huge factor in why skeptical people have had so much trouble taking this research seriously.

Indeed. But you then go on to cite research by someone who appears to have an agenda to promote the idea that near-death experiences prove there's something spiritual going on:

But here's a good example of why real NDE research interests me so much:

Beauregard, Courtemanche, & Paquette. (2009) Resuscitation. June 30 [Epub ahead of print]
(It's not letting me copy and paste the entire abstract, for some reason...)

Anyway, in 2 separate experiments, brain activity was measured in NDE experiencers with functional MRI and EEG during a "meditation" condition in which experiencers visualized the "being of light" which they had allegedly encountered during their NDE, and during a control condition in which they mentally visualized the light emitted by a lamp. During the meditation condition, marked hemodynamic and neuroelectric changes were noted in brain regions normally
involved with profound meditative experiences. This was not found to be the case in the control condition.

That would presumably be because of the meaning attributed to the light seen in the NDE - i.e. the promise of a peaceful afterlife, happiness to come, and so on, being recalled when visualising it. People just wouldn't get that with looking at a lamp. Also, you appear to think it's significant that their brain activity mimicked that of experienced meditators. How many of them were experienced meditators?

From an article called * Decoding The Mystery Of Near-Death Experiences

..."And it seems that these people have a different sort of brain," Beauregard says in his soft French accent. "It's like there's a shift in their brain, and this shift will allow these people to stay in touch with the spiritual world more easily, on a daily basis."

Beauregard recruited 15 people who had a near-death experience. One of those was Gilles Bedard. In 1973, Bedard's heart stopped, and in the moments before he was resuscitated, he was greeted by what he describes as 12 beings of light.

"And I felt it was like the breath of the universe. Because it was like …" he says as he blows out his breath, slowly, like a low wind, "very, very peaceful."

Since then, Bedard has meditated every day, and he says he often reconnects with the light. The research question is, how will his brain respond when he does?

A Permanent Change In Brain Activity?

For the experiment, Bedard is shut into an isolation chamber at Beauregard's Montreal lab. Bedard's head sprouts 32 electrodes, which will record his brain wave activity. He's told to relax for a few moments. Then he'll be instructed to imagine his near-death experience. ...

Afterward, the researcher asks Bedard if he was able to connect with the light.

"Yeah, it was coming from within," he says. "It was loving, intelligent … very powerful." ...

The article goes on to once again quote Beauregard as mentioning a spiritual connection. So it appears he may have an agenda. And given at least one of the subjects was already an experienced meditator, it isn't in the least bit surprising that his brain waves should have been like those of an experienced meditator. You may well have attributed significance to this in error.

Also, Beauregard used a sample size of ... Fifteen? In this thread, you criticise the study reported in National Geographic as being unlikely to be able to tell us anything significant because of its small sample size. But in this other NDE thread of yours, you're enthusing about research where the sample size isn't much different, it seems.

The article I quoted goes on to say that skeptics have drawn completely opposite conclusions from the exact same study results.

"The brain function of many of these people who have undergone a near-death experience is altered," Woerlee says. "That's correct. It is altered. Extreme oxygen starvation does change brain function — because it causes brain damage to the larger cells in the brain."

It's brain chemistry, he says, not a trip to heaven.

In other words, Woerlee and Beauregard looked at the same images and came to opposite conclusions.

Still, this must be where the lamp light comes in. If the brain looks like any other when its owner's looking at a lamp, perhaps there is something else going on. ... But what of significance did the study really find?

It turns out that Beauregard has written a book called The Spiritual Brain: A Neuroscientist's Case for the Existence of the Soul. Here's an excerpt:

... But I belong to a minority — nonmaterialist neuroscientists. I do not doubt in principle that a contemplative might contact a reality outside herself during a mystical experience. In fact, I went into neuroscience in part because I knew experientially that such things can indeed happen. I simply sought to study what the neural correlates — the activity of the neurons — during such an experience might be.

Of course, you may well ask, can neuroscience studies of contemplative nuns demonstrate that God exists? No, but they can — and did — demonstrate that the mystical state of consciousness really exists. In this state, the contemplative likely experiences aspects of reality that are not available in other states.

Do you have an agenda in promoting this stuff that you're ostensibly denying? After all, it seems strange to say you have absolutely no patience with people who propose paranormal explanations for near death experiences, and then to go on to enthuse about someone who is obviously inclined to attribute them to mystical phenomena.

You then go on to state in the other thread:

The meaning of an NDE, on the other hand, is impossible to "prove." (i.e., does it prove life after death, does it prove there isn't life after death, does it prove there's a flying spaghetti monster...) I don't know what it is, and I don't think there's any point in arguing about it, so I would rather
kind of take that out of the equation.

This appears to be a strange thing to say, since the cause of the experience will very much determine the meaning.

You then make the extraordinary claim in your next post in the thread:

... Any experience as traumatic as one that would cause an NDE (in any of its distinct categories-- citations about this later, I have to get up really early!) absolutely should cause PTSD, and there's every reason to believe that it would also cause pathological dissociative symptoms in at least a significant percentage of experiencers. But from the research to date, NDE's don't do this. For an experience like an NDE to lead to an increased ability to meditate doesn't make any sense whatsoever from the viewpoint of trauma and its normal aftereffects. For it to lead to increased positive feelings, a sense of peace, oneness with the world, etc.,etc... this goes against everything we know about how trauma affects the human brain. Do NDE's offer some kind of neuroprotective effect against trauma? If so, what is it? Can it be quantified? Can we show what literally happens in the brain? How? Could it be replicated? Who might benefit from it? Is there any way in which it could be perhaps used to help PTSD survivors who haven't had NDE's? (Kind of unethical to induce an NDE in an attempt to get the protective effect.

This is bizarre, for more than one reason. Why assume that an experience that would cause an NDE "absolutely should" cause PTSD and often dissociative symptoms? And why don't you have any idea of why NDE's would cause feelings of peace? For one thing, being near death won't always be traumatic by any means. For example, someone dying of cancer might be very glad to go. And if you meant the trauma should be caused by the chemical changes in the brain, if you don't know the mechanism by which the brain causes NDE's, how could you possibly be sure that would cause trauma symptoms itself?

And for another thing, why haven't you taken into account the conventional wisdom that if you see relatives or peaceful beings of light in your NDE for whatever reason, you'll likely come back with a feeling of reassurance and optimism that is in fact highly likely to mitigate any trauma you might otherwise feel, and quite likely to entice one to meditate on the experience since it was so enjoyable.

Also, if you would like to cause trauma survivors to have the same meditative state as the NDE experiencers in the study, and that's the reason these findings interest you so much, just why do you imagine this could ever soothe their trauma long-term, and how would you propose to get them to have the experiences? If there's no practical way of doing it other than half-killing them in the hope they have an NDE, what's the point of exploring this?

Lastly, though you're of course right to be dubious of the interpretation the popular press puts on study findings, it seems you aren't cautious enough about findings documented in abstracts on places like PubMed. For instance, here's the Beauregard study on PubMed: Brain activity in near-death experiencers during a meditative state. It says nothing about the apparently small sample size, and the fact that the findings have been disputed.

As I illustrated in my latest post of a couple of days ago in the What's the appeal of anti-psychiatry thread, relying too heavily on what you read in even reputable journals could possibly be tragically dangerous, or so it seems. The post makes several mentions of infamous trauma psychiatrist Colin Ross, million dollar challenge applicant and writer of 140 peer-reviewed papers, who incidentally is actually alleged to have carried out research into NDE's by deliberately nearly killing patients. Whether the allegations are true or not, I can't of course know.

 

[Maia]

Impressive work, Maia. I couldn't have done it if you gave me 30 months. Two virtual thumbs up.

Thanks jhunter!

It's important to know that I'm not saying that this proves any other explanation for NDE's, because clearly it doesn't. It just means that on the basis of this evidence, this null hypothesis (that people who reported NDE's as corroborated by the Greyson scale did not actually have meaningful statistically or practically higher levels of pCo2 than those who did not report NDE's) cannot be rejected. Because only pC02 levels were supposedly independently correlated with both higher levels of reported NDE's and a higher NDE score, this means that the argument that higher Co2 levels in general cause NDE's or are solely responsible for the subjective and objective effects which correspond to NDE's cannot be supported by this study. Essentially, we need better studies.

 

[jhunter1163]

This particular area of study has always interested me (near-death and out-of-body experiences). My feeling has always been that there is some area of the brain that, when stimulated, produces this type of experience. I don't subscribe to any paranormal explanation for the phenomenon (I know, that's no fun). Hopefully, there'll be a more definitive study at some point.

Again, excellent work

 

[RSL's better half]

I must confess that I understood none of the OP, but thanks for your work, and for sharing it with us!

 

[Floyt]

I don't think the issue of p=.041 vs .049 deserves the quibbling. The 5% p-value convention is a convention, nothing more; one false positive out of twenty versus one false positive out of twenty-one might matter for FRST funding applications, but not for hypothesis formulation. It shows a trend worth following up, with plenty of interpretational leeway in either direction.

However, I agree that the deciding factor here are the sample sizes. N=11 in that hell of a multi-variate system? This can only be purely exploratory.

 

[Uncayimmy]

I'm not sure what you're all worked up about. Your post reads somewhat frantic, and I'm not clear what conclusions you want me to draw.

You seem to be criticizing the 0.05 level of statistical significance, but I think it's appropriate it in this case. The correlation to CO2 levels does not seem to be an extraordinary claim. From what I read in your link, there already exists research that indicates it could be a factor:

...it is known that CO2 changes the acid-base equilibrium in the brain, which can provoke unusual experiences in the form of bright light, visions, and out-of-body or even mystical experiences [3,5]. Some earlier studies have shown that inhaled CO2, used as a psychotherapeutic agent, could cause NDE-like experiences [33,34]. Therefore, we can conclude that CO2 might be one of the major factors for provoking NDEs, regardless of when NDEs occur

What we have here is a plausible physical mechanism backed up by research that says something sometimes happens when we employ that mechanism. They then tried to see it "in action" and saw it looks like a 1 in 20 chance this was just a statistical variation. When looked at as a whole it's not wrong to say there appears to be something there.

It's a matter of understanding statistics and how they apply to real life. Suppose we measure the friction and tackiness of a die. We find that a wet die has more friction and is more "sticky" on a wood table. We then lick one side of a die and roll it a bunch of times. We might find that our target number comes up more frequently, and the p-value is 0.049, which you call "just barely" statistically significant. I'm comfortable with saying licking the die influences the outcome of the roll.

Contrast that against a study where (say) some psychic, let's call her RodneyVision, says she can make the die come up to whatever side she chooses every time, all through the sheer force of her mind. There's no plausible mechanism or evidence that she can physically influence the die under any conditions. We run the same trial and get the same p-value as in the prior test. Is that significant? No, not really, because there's no explanation for what happened except ordinary statistical variation.

 

[Tatyana]

(laces up running shoes to get a headstart)

Now, before everyone starts chasing me out of town waving virtual torches, y’all should know that what follows is not an argument that NDE’s represent proof for God and angels and the Second Coming of Jesus bringing us all back to heaven. But I can’t stand what the popular press does to scientific research, and to be perfectly honest, I was not impressed by this research anyway. Here’s why. Please read all the way to the end before commenting. I did a lot of original work on this, ALL BY MYSELF. Take a deep breath, and let’s dive in!

Here's the actual real-live study yapped about in National Geographic:

Right here.

On careful analysis, the results are so far from anything that was reported in National Geographic (or anywhere else) that it’s simply embarrassing. The differences are so significant. But you really do have to do some digging.

The claim was that:



I decided not to try to chase down info about potassium levels, etc., because those weren't the main claims that had been passed down through the popular press (although the part about the patients with previous NDE's having more was very interesting.) Here we go:



If you have a TI-83 plus calculator, you may want to get it out and follow along. First, I made little charts of NDE and non-NDE-rs, with n, xbar (sample mean), and standard deviations. For PCo2, for example, the NDE group had an n of 11, a sample mean of 6.6, and a standard deviation of 2.3; the non-NDE group had an n of 41, a sample mean of 5.3, and a standard deviation of 1.4. The null hypothesis was that the population means were equal. The claim was that population mean one was greater than population mean two. Here's where the TI-83 comes in.

STAT-TESTS-2SAMPTTEST. More about why this test was chosen later.

Now enter in your data. Your p value is .049469,but this is NOT WHAT THE RESEARCHER SAY. They claim that the p value is .041. It is not. Get out your stats calculator and enter the numbers yourself. The alpha value they chose is the very common one of .05. This means that they've barely, BARELY inched over into showing an effect for that pCo2. This is just barely statistically significant, and it has no practical significance by any definition. If you round up at 2 decimal places rather than 3 (which you're not supposed to do, of course, but let's just say for the sake of argument...) you cannot reject the null hypothesis.

The numbers get more robust for the petCo2 levels, because the p value is .0017. But there's a tremendous problem here. The multivariate analysis found that:



It's all very easy to say that pCO2 was an independent predictor of NDEs, but its effect was so extremely minor that the independent predictor effect couldn't have been anything but negligible. Nothing plus nothing equals nothing.

Now, a stats expert I am not. But think about this: they said that they first used univariate analysis, considering each variable separately. By the time they got to the information in Table 2, what else could you use for those particular results besides an independent samples t-test for 2 means? What they actually said in the study was:



They used SPSS 13.0, which means this wasn't done by hand. I'm not so sure this is always a good idea, because you can really miss some things. And for this particular set of relationships between variables, nothing else really makes any sense besides a t-test that I can see, because you’re testing two groups separated by one defined characteristic (NDE vs. non-NDE) to see if you can come up with good evidence that they come from a population whose “averages” are different on this, that, and the other defined property (various blood gas levels). (You wouldn’t use ANOVA, for instance, because you only have 2 groups. You wouldn’t use a chi‐square test because it isn’t nominal data, you wouldn’t use a Kolmogorov–Smirnov test because it isn’t ordinal data.)Maybe… I don’t know, I’m just trying to come up with something here… they used some adjustment factor or other to come up with the differing numbers (.041 rather than .04596), in order to compensate in some way for the fact that n most certainly did not equal 30 for the NDE group (more about that later)… a stats expert might know more about if this is even possible, but then I think they would have to have said something more specific about it than they did. A straight t test for 2 means gives a result of 049469, not .041.

The only obvious way to get this number that I can see would be to artificially conflate the sample sizes. For instance, if you say that you had 100 subjects in each group (n1 and n2 each = 100,) then you’ll end up with p < .0001. The effect would be very robust if you could just get a large enough sample size. But that is just about the worst methodology that anybody could ever even imagine coming up with. The sample only consisted of 52 people, 11 (yes, eleven) of which had NDE’s. This was from an original pool of 400 who had cardiac arrest, and only 76 survived! Ack. Seven of the 11 NDE’ers were atheists. I’m not sure if this was unusual or not, though, because I have no idea what the religious makeup of Slovenia is.)

I don’t know. I’m really, really confused about this point, because I basically just do not see how they came up with a figure of .041 based on their data. Even so, a P-value of .041 is not the least bit impressive compared to an alpha of .05 either.

Overall, higher petCo2 was not an independent predictor of NDE's, and higher PCo2 was but just barely, barely managed to cross over into even statistical significance. The only conclusion to really come to is that this study simply does not mean what it has been ballyhooed to mean.

When all is said and done, of course, there’s also a fatal flaw that totally and completely invalidates the kind of conclusions that have been drawn from this study by the popular press, and it's that in order to draw any real conclusions from these stats methods, the sample sizes absolutely must be at least n = thirty for each group. The NDE group was n = 11. The researchers said clearly that this was a prospective observational study. That was the responsible thing to say because of its limitations, but particularly because of the size problem.

The final lesson to take away from all of this, I think, is to just not believe the version you read in the popular press. Complex neurobiological phenomena do not have simplistic explanations. I’ll be sending all of this information to National Geographic, and I’ll let them know what I think of their fact-checking quality. All of this took me half an hour, and really, most of the people on this board could have done it too.

Are you using a calculator to do p-values? Did you double check your calculations and data input?

It is a good idea to do statistics by hand? Seriously?





For biomedical experiments, p = 0.05 is significant, it isn't 'barely significant'.

 

[PixyMisa]

As Floyt says, the problem isn't so much the statistics as the sample size itself. It's a useful indicator to direct further research, but it's hardly conclusive.

And Maia, when you say

Complex neurobiological phenomena do not have simplistic explanations.

That's far from universal. You're drunk is a simplistic explanation for a complex (and common) neurobiological phenomenon.

 

[Maia]

I'm not sure what you're all worked up about. Your post reads somewhat frantic, and I'm not clear what conclusions you want me to draw.

Now UY, really! Nobody is frantic or "all worked up." You KNOW that you don't have to descend to these kinds of comments. Just draw the conclusions that you decide to draw. This is a discussion.

You seem to be criticizing the 0.05 level of statistical significance, but I think it's appropriate it in this case. The correlation to CO2 levels does not seem to be an extraordinary claim. From what I read in your link, there already exists research that indicates it could be a factor:

No, alpha = .05 is a common and correct level to use. I'm not interested in "research that already exists." We don't know about the details of how that research was done, what the claims were, what was found, what the methodology was, or anything else about that research from the published results of this study. I'm interested in the claims made here, in this published study. The ones regarding pCo2 levels are not supported by the data within the study itself, and the TI-83plus stats T-tests, run repeatedly, show very different figures than what the researchers had. If anybody has the TI83 or another stats calculator, try it yourself.

What we have here is a plausible physical mechanism backed up by research that says something sometimes happens when we employ that mechanism. They then tried to see it "in action" and saw it looks like a 1 in 20 chance this was just a statistical variation. When looked at as a whole it's not wrong to say there appears to be something there.

It's a matter of understanding statistics and how they apply to real life. Suppose we measure the friction and tackiness of a die. We find that a wet die has more friction and is more "sticky" on a wood table. We then lick one side of a die and roll it a bunch of times. We might find that our target number comes up more frequently, and the p-value is 0.049, which you call "just barely" statistically significant. I'm comfortable with saying licking the die influences the outcome of the roll.

I would certainly be comfortable too, IF n equaled 30 for each set of tests. If n does not equal 30, then no, and that's just what happened here. Here's what I mean. Let's run the same set of T-tests on the same data in this study for the same pCo2 levels with n = 30 for each group (instead of n = 11 for NDE'ers adn n = 41 for non- NDE'ers.) Now, our p value is .006. Now, that's statistically significant. If n= 100, the p value becomes <.001. On the other hand, if n = 11 for both NDE and non-NDE groups, our p value is now .064. That would not be statistically significant at all, and it would simply not be possible to even come close to rejecting the null hypothesis.

Do you see what the central problem is here? If we pretend that the sample size is what it should have been for both groups, then it looks like there could be a very statistically significant correlation between pCo2 levels and NDE experiences. But there's not a lot of difference between that and saying, "well, if I'd only known all 6 of the Powerball numbers, then I could have won $100,000,000 on Thursday." If we start doing that, then we might just as well start arguing for licorice teapots orbiting Jupiter. This is exactly how statistics do indeed apply to real life. T-tests are not reliable when n doesn't equal 30 for both groups, and that's why. I don't know exactly what they did to come up with such different numbers (some kind of weird adjustment?) but if it wasn't a straight T-test, then in my opinion, it just wasn't honest. Running a straight comparision T-test when both n's don't equal 30 does show why conclusions can't honestly be drawn.

Contrast that against a study where (say) some psychic, let's call her RodneyVision, says she can make the die come up to whatever side she chooses every time, all through the sheer force of her mind. There's no plausible mechanism or evidence that she can physically influence the die under any conditions. We run the same trial and get the same p-value as in the prior test. Is that significant? No, not really, because there's no explanation for what happened except ordinary statistical variation.

In this case, the highlighted portion is the reason why I would set the alpha to quite a bit lower and more demanding level in this study (which I wouldn't have anything to do with, personally!) Let's say .01. Now, if RodneyVision actually manages to come up with a p value of less than .01 over at least n = 30 trials, yes, that would be impressive. If that actually happens, I expect to be riding on the licorice teapot.

 

[Maia]

One more thing and then off to work:

With any rolling-the-dice experiment, another major difference is that we wouldn't do a T-test, we'd do a Z-test, because we already know the standard deviation of the entire population instead of the population of the sample. We know that certain rolls of the dice will come up a certain number of times, and that this will not change. With pCo2 levels, all we know is the standard deviation of the sample. So with the Z-test, our p value is going to be lower to begin with. We can be a whole lot more confident about how likely or unlikely an event is to occur. Because of that, I don't think the dice analogy is the most accurate one to use. If we could run the pCo2 levels test knowing what our population SD's were (which we clearly can't), for instance, we'd get a p value of .037, even with n=11 for one of the groups (which is what we have.) But we can't play games like that. We can speculate about what would happen if certain facts were known or if our n values were different, but we can only come to conclusions based on what actually exists and according to certain rules. For another example, I can't come to the conclusion that higher pCo2 levels are not associated with more NDE's or with a higher NDE score on the Greyson scale, only to the conclusion that based on this study, the null hypothesis (that they are not) cannot be rejected.

 

[Uncayimmy]

Now UY, really! Nobody is frantic or "all worked up." You KNOW that you don't have to descend to these kinds of comments. Just draw the conclusions that you decide to draw. This is a discussion.

I'm telling you how I read it, and I read it twice. To me it read like somebody breathless from an exciting event trying to relay a story in halting sentences assuming the other person knows what's going on. I'm telling you that I had a hard time following your points and due to the less than analytical way it was presented, question the numbers you put forth.

No, alpha = .05 is a common and correct level to use. I'm not interested in "research that already exists." We don't know about the details of how that research was done, what the claims were, what was found, what the methodology was, or anything else about that research from the published results of this study. I'm interested in the claims made here, in this published study. The ones regarding pCo2 levels are not supported by the data within the study itself, and the TI-83plus stats T-tests, run repeatedly, show very different figures than what the researchers had. If anybody has the TI83 or another stats calculator, try it yourself.

You're making a very strong accusation by saying the data does not support the conclusion. Are you referring to you arriving at .049469 and them arriving at 0.41? At best that sounds like a discrepancy where a self-proclaimed non-expert should ask what's going on rather than accuse those with far more experience of getting it wrong. This is where your apparent state of mind while writing the post comes into play. If (say) you had written in the style of Linda/fls, we wouldn't be at this point.

I would certainly be comfortable too, IF n equaled 30 for each set of tests. If n does not equal 30, then no, and that's just what happened here. Here's what I mean. Let's run the same set of T-tests on the same data in this study for the same pCo2 levels with n = 30 for each group (instead of n = 11 for NDE'ers adn n = 41 for non- NDE'ers.) Now, our p value is .006. Now, that's statistically significant. If n= 100, the p value becomes <.001. On the other hand, if n = 11 for both NDE and non-NDE groups, our p value is now .064. That would not be statistically significant at all, and it would simply not be possible to even come close to rejecting the null hypothesis.

You are stating the obvious: a higher population in the samples allows for a higher p-value with the same relative numbers. We have have a lot more confidence in 600 out of 1,000 coin flips being something special than we do with 6 out 10. That's not a revelation. They worked with what they had.

Do you see what the central problem is here? If we pretend that the sample size is what it should have been for both groups...

What do you mean by should have been? They worked with what they had.

 

[fls]

I would drop the criticism about the p-value. It's merely a convention anyway with no intrinsic meaning. And it gives the false impression that a 'significant' p-value makes a statement about whether a hypothesis might be true (a fallacy we should be trying to dispel).

Linda

 

[Kuko 4000]

I cannot contribute anything original to this discussion, but I just bumped into this analysis while going through my usual 'favourites':

http://www.nhs.uk/news/2010/04April/Pages/Near-death-experiences-and-CO2.aspx

Maybe some of you will find it interesting, I really like the site:

http://www.nhs.uk/News/Pages/NewsIndex.aspx

 

[Maia]

You are stating the obvious: a higher population in the samples allows for a higher p-value with the same relative numbers. We have have a lot more confidence in 600 out of 1,000 coin flips being something special than we do with 6 out 10. That's not a revelation. They worked with what they had.

The difference here, I think, is that anyone can flip a coin 600 times rather than 6. If someone does flip a coin 6 times rather than 600 and gets an inaccurate p-value, then it would be pretty bizarre to not flip it the additional 594 times. There's no way to do this with the extremely small sample of the population they had. We don't know what was different about the 374 people in the original population of 426 they started with. We do know what's different about all the unflipped coins: nothing. We know the population standard deviation of coin flipping; there's no way to know what the population SD of the 426 was in many, many ways. A big part of working with what you have is admitting the limitations of what you ended up with, not playing numbers games or whatever it was that actually happened with that sample of 52, n=11 in one group and n=41 in the other. I will be emailing those researchers at the addresses provided, because I do want to know what went on here. The media version is so different from the original study that I think there's a particular need for any inaccuracies to be corrected.

Kuko's linked story was a better summary than the National Geographic story for sure and provided more info (thanks Kuko!), but it still gives some very misleading impressions. Popular press, eh. Again, there was no indication of the fact that the link between pCo2 levels and reported NDE's does indeed just barely manage to inch over that statistical significance line in the study. Put it this way: if someone claimed that they demonstrated that they could affect a phenomenon in some kind of "paranormal" study with a p value of .049469 and an alpha of .05 in one prospective observational study where n=11 in one of the groups, how impressed would you be? Also, there was this statement:

The model showed that a higher level of petCO2 was an independent risk factor for the number of near-death experiences and for the score on the near-death experience scale.

Not true.

Here's the statement in the study itself:

Higher pCO2 was an independent predictor of NDEs. The logistic regression model explained 46% of the variation (Table 3). A higher NDE score was independently associated with higher pCO2, higher serum levels of potassium, and previous NDEs. The linear regression model explained 34% of the variation (Table 4).

This is a significant error because only petCo2 levels were indeed significantly higher in the study, not pCo2 levels. I do appreciate what fls says about that p value issue.... but the way this was reported in the media was so inaccurate, and if they're going to even try to explain these things and then they get that part so wrong, too...

Anyway, the media will find something else to jump on next week and will be just as inaccurate about it.

ETA: The potassium levels, if anything, seem more interesting to follow up on. It's hard to imagine what that could possibly mean, though.

 

[Beth]

STAT-TESTS-2SAMPTTEST. More about why this test was chosen later.

Now enter in your data. Your p value is .049469,but this is NOT WHAT THE RESEARCHER SAY. They claim that the p value is .041. It is not. Get out your stats calculator and enter the numbers yourself.

Have you checked that you are both running either a one-tailed test or a two-tailed t-test? If they used a one-tailed t-test and your calculator gives the result for a two-tailed t-test, that might account for the discrepancy.

 

[fls]

ETA: The potassium levels, if anything, seem more interesting to follow up on. It's hard to imagine what that could possibly mean, though.

It would be an indicator of the extent of tissue anoxia/damage.

Linda

 

[Robo Sapien]

Does it really require a NASA calculator to determine that people are making this crap up? [/cynicism]