Check my methodology - prayer study

[digithead]

I haven't used Matlab in years so I don't know....

In SAS, to generate a random bivariate normal from two N(0,1) random variables with a correlation of 0.5 it's:

data a;
keep x y;
mu1=0; mu2=0; sigma1=1; sigma2=1; rho=.5;
do i = 1 to 10000;
x = mu1+sqrt(sigma1)*rannor(0);
y = (mu2+rho*(sqrt(sigma2)/sqrt(sigma1))*(x-mu1)) +
sqrt(sigma2*(1-rho**2))*rannor(0);
output;
end;
run;

The rannor function with a seed of 0 generates the random number off of the computer clock at the time of calculation so it's somewhat the "most random" of the pseudorandom number generators...

One can also adapt the above to any other mu, sigma or rho they wish and I'm sure it can be translated into Matlab or any other stats/math package that's used...

But it's a moot point because as I've pointed out above, studies have shown that intercessory prayer has no clinical effectiveness. In fact, when Masters, et al (2006) removed the discredited Cha and Wirth (2001) study, the overall effect of intercessory prayer not just not statistically insignificant it was nearly nil...

So I'd just leave Saizai to his wishful thinking that he'll somehow demonstrate that prayer works...

 

[Startz]

I haven't used Matlab in years so I don't know....

In SAS, to generate a random bivariate normal from two N(0,1) random variables with a correlation of 0.5 it's:

data a;
keep x y;
mu1=0; mu2=0; sigma1=1; sigma2=1; rho=.5;
do i = 1 to 10000;
x = mu1+sqrt(sigma1)*rannor(0);
y = (mu2+rho*(sqrt(sigma2)/sqrt(sigma1))*(x-mu1)) +
sqrt(sigma2*(1-rho**2))*rannor(0);
output;
end;
run;

The rannor function with a seed of 0 generates the random number off of the computer clock at the time of calculation so it's somewhat the "most random" of the pseudorandom number generators...

One can also adapt the above to any other mu, sigma or rho they wish and I'm sure it can be translated into Matlab or any other stats/math package that's used...

But it's a moot point because as I've pointed out above, studies have shown that intercessory prayer has no clinical effectiveness. In fact, when Masters, et al (2006) removed the discredited Cha and Wirth (2001) study, the overall effect of intercessory prayer not just not statistically insignificant it was nearly nil...

So I'd just leave Saizai to his wishful thinking that he'll somehow demonstrate that prayer works...

I translated your suggestion into Matlab and ran 50,000 simulations. The simulated false positive rate was 4.908 percent, just as one would expect. So for this little piece of the puzzle, Saizai would appear to have a perfectly valid procedure...which I'm sure will therefore fail to win the million dollars.

%{
confoundedPrayer.m
Monte Carlo to illustrate testing for prayer efficacy
in presence of confounder
Dick Startz
August 2006
%}
rand('state',0);  %% reset random number generators
randn('state',0);
nMonte = 50000
n = 400
rejectSum = 0;
mu1=0; mu2=0; sigma1=1; sigma2=1; rho=.5;

for iMonte = 1:nMonte
    prayFor = rand(n,1)>0.5;
    notPrayFor = ~prayFor;
    x = mu1+sqrt(sigma1)*randn(n,1);
    y = (mu2+rho*(sqrt(sigma2)/sqrt(sigma1))*(x-mu1)) + sqrt(sigma2*(1-rho^2))*randn(n,1);

    outcome = x + y + 0*prayFor;
    meanDif = mean(outcome(prayFor)) - mean(outcome(notPrayFor));
    stdErr = sqrt(var(outcome(prayFor))/sum(prayFor) + var(outcome(notPrayFor))/sum(notPrayFor));
    rejectSum = rejectSum + (abs(meanDif/stdErr)>1.96);
end
disp(['False rejections occurred ',num2str(100*rejectSum/nMonte),' percent of the time']);

 

[saizai]

In addition to my feeling that this "experiment" does not belong in this venue,

In what way exactly? It is a potential MDC app, albeit of a claim somewhat different from the usual "I can do it every time" sort.

I don't see how any study of prayer can possibly be considered valid as it is impossible to effectively set up a control group. How do you tell people not to pray for someone who is sick? How do you know they are telling the truth if they claim not to have prayed. How do you know how many people outside of the study's purview did or did not pray for the subjects in the study? I don't see how randomisation can be an effective control in this area unless the sample size is huge.

I am not telling anyone not to pray. The people who are praying do not know the people they are praying for personally. I simply tell them to pray for those people.

If prayer works in such a way that one prayer is enough for everything, and more is not additive, then no prayer should work at all because people have prayed that everyone in the world be healthy for ever and ever amen. I do track how many outside people the recipient knows to be praying for them, but I expect that to be irrelevant.

You need to explain how your claimed lack of controls would result in a false positive for this to be a flaw - unless you are taking a believer's stance and saying that the study is *too* tightly controlled for how you believe prayer works, and therefore will show a false *negative*.

Which is it?

 

[saizai]

startz - Thank you for your examples; they demonstrate exactly what I was thinking.

digithead - I am aware of the previous literature on the subject. As I have said elsewhere, previous studies have either:
* been flawed in methodology
* had very small sample sizes
* shown no effect
* shown marginal effect
* been inconclusive

I simply want to run something that is big enough to find a smallish effect. It is relatively easy for me to do this, so I see no reason not to. Plus I'm using it as an excuse to build up my Ruby on Rails skills. ;-)

Your advice to read the literature seems to simply be a suggestion that I not do it at all, not that I do it better.

 

[saizai]

digithead - Thanks for the reference to SF-36; I think I may end up using it. Do you know of any similar standard survey that is cancer specific (or adds cancer specific questions)?

I'm also considering requesting monthly reports from Healers as well. It wouldn't impact a result as far as the MDC is concerned, but would still be potentially interesting data from a research perspective.

 

[Gr8wight]

I am not telling anyone not to pray. The people who are praying do not know the people they are praying for personally. I simply tell them to pray for those people.

If prayer works in such a way that one prayer is enough for everything, and more is not additive, then no prayer should work at all because people have prayed that everyone in the world be healthy for ever and ever amen. I do track how many outside people the recipient knows to be praying for them, but I expect that to be irrelevant.

You need to explain how your claimed lack of controls would result in a false positive for this to be a flaw - unless you are taking a believer's stance and saying that the study is *too* tightly controlled for how you believe prayer works, and therefore will show a false *negative*.

Which is it?

My point is that the numbers have to be compared against a control group that has not received prayer. However, the fact that you have no way of determining if any or all persons in the control group were prayed for by someone outside of the experiment, you cannot actually say the group is an effective control. Any results you get, positive, or negative will be meaningless because you plain don't know who was really prayed for, and who wasn't.

 

[Ladewig]

To do a rigorous test of this "normal" prayer would require deceiving people: you would have to have some people believing that they were being prayed for by their families, where in fact you've, say, deliberately kept the family misinformed about that person's illness (you say that they're on an extended work assignment overseas, say).

Thanks. I had absolutely no idea what you were talking about until this paragraph.

 

[petre]

My suggestion to saizai:

1. Take an existing study
2. Identify any flaws saizai exist in the methodology
3. Propose the specific changes made to address said flaws

Introducing new elements (online participation, self-selection, etc) without providing a full analysis of their effect on the study only increases the opportunity for flawed methodology, and therefore makes such a study less useful than the others that have already been completed.

 

[saizai]

My point is that the numbers have to be compared against a control group that has not received prayer. However, the fact that you have no way of determining if any or all persons in the control group were prayed for by someone outside of the experiment, you cannot actually say the group is an effective control. Any results you get, positive, or negative will be meaningless because you plain don't know who was really prayed for, and who wasn't.

Wrong. I am not testing the diff between (people who get no remote prayer) and (people who do).

I am testing the diff between (people who are normal, and may be getting prayer in their usual manner) and (people who ALSO get extra prayer through me).

You are making a believer's argument that the results would be a false negative... which given the forum means you're probably confused.

 

[saizai]

My suggestion to saizai:

1. Take an existing study
2. Identify any flaws saizai exist in the methodology
3. Propose the specific changes made to address said flaws

Introducing new elements (online participation, self-selection, etc) without providing a full analysis of their effect on the study only increases the opportunity for flawed methodology, and therefore makes such a study less useful than the others that have already been completed.

I'm not interested in replicating previous studies because I haven't the funds or the manpower to do them. But I do have the skill and ability to do one that's online.

Self-selection cannot result in a false positive. Neither can online participation.

Please, if you have a criticism, explain how it will result in a false positive.

 

[digithead]

I translated your suggestion into Matlab and ran 50,000 simulations. The simulated false positive rate was 4.908 percent, just as one would expect. So for this little piece of the puzzle, Saizai would appear to have a perfectly valid procedure...which I'm sure will therefore fail to win the million dollars.

My mistake, you are still assuming equal variances which is not a valid assumption when your sampling from a heterogeneous population. In a prior life I could suggest to you how to correctly simulate this but I've given up on the sweet life of statistics for the hard life of criminology where there are no answers...

But to give an example of my concerns, say you've recruited 50 people into the study, 4 of them have localized skin cancer, 25 have non-Hodgkins lymphoma, 19 have leukemia, and 1 has beginning pancreatic cancer and 1 has end-stage pancreatic cancer.

You randomize into groups of 25 with each person having an equally likely chance to be picked for the treatment group and you get the following sample:

Treatment: 3 skin cancer, 12 non-Hodgkins lymphoma, 10 leukemia
Control: 1 skin cancer, 13 non-Hodgkins lymphoma, 9 leukemia, 1 beginning pancreatic cancer, 1 end-stage pancreatic cancer

Note that this sampling scheme is as likely as any other sampling under your randomization technique. Can you see why you need to control for disease type, disease severity, and medical treatment? Can you see why a quality of life measure might be higher in the treatment group solely by the type of disease proportion it has? It need not be intentional for bias to creep in because you have to compare apples to apples or in this case, disease to disease...

Look at the cardiac study that Randi talked about, they only looked at those who had CABG procedure yet they still controlled for complications and other confounders. You have to account for possible confounders both in your sampling scheme and in your statistical analysis...

And as I've said, performing a prayer study is an exercise in futility given the 15 or so studies that have been performed over the years and the fact that none that could stand up to rigorous scientific scrutiny showed an effect. It would be a waste of time, effort and money and from an ethical standpoint might make someone abandon treatments that work in favor of divine intervention...

 

[Cuddles]

I'd say the most significant problem is that you have assumed 400 people. Is there any realistic chance of this number of people actually being studied here? I would expect maybe 10 or 20 people for a preliminary study like this. I don't think anyone here will claim that randomisation doesn't work with large numbers, but with the small number likely to be used it must be proven that the two groups have similar makeups. I notice saizai didn't answer this question last time I asked it.

 

[petre]

I'm not interested in replicating previous studies because I haven't the funds or the manpower to do them. But I do have the skill and ability to do one that's online.

Self-selection cannot result in a false positive. Neither can online participation.

Please, if you have a criticism, explain how it will result in a false positive.

Even if you're not, perhaps people that participate in the study (actual participants, observers, etc) are concerned about false negatives. To ignore their concerns ailenates them. Such a study would self-select for persons that are not concerned with false negatives.

I simply see no reason to perform yet another test with obvious flaws and complete disdain for making adjustments to improve sampling and analysis.

If you wish an even simpler test of prayer with no chance of creating a false positive, simply pray that every person on earth will just suddenly believe in prayer.

Finally, the JREF tends to be a bit of a stickler on allowing confounders in protocols (by that I mean, the potential for a false negative). They'd likely insist on a claim along the lines of, "I believe prayer has an effect, (describe the effect), (describe a test that will demonstrate the effect if it exists)". If you admit that your test may not conclusively demonstrate said effect even if it exists, your protocol will likely be rejected.

 

[saizai]

Cuddles - As I've said, the number of participants is purely limited to what can be done practically. I have no other reason to put a limit on it.

Also I should point out (again...) that excessively small values of N would result in it being impossible to get p<.05, so that point is somewhat moot.

petre - That's interesting; I didn't realize you were advocating for the believer's perspective. Please be more explicit about this in the future.

I do not have "disdain" for improvements - but on this forum, I have to assume at the outset that everyone will be advocating the skeptic's perspective, and require that they be logical (i.e. prove a false positive result) to justify a critique.

Could you please specifically list what you believe the "obvious flaws" to be, how to correct them, whether they would calse a false positive or negative, and how they would do so?

If you wish an even simpler test of prayer with no chance of creating a false positive, simply pray that every person on earth will just suddenly believe in prayer.

Finally, the JREF tends to be a bit of a stickler on allowing confounders in protocols (by that I mean, the potential for a false negative). They'd likely insist on a claim along the lines of, "I believe prayer has an effect, (describe the effect), (describe a test that will demonstrate the effect if it exists)". If you admit that your test may not conclusively demonstrate said effect even if it exists, your protocol will likely be rejected.

I am not, and will never be, claiming that prayer works (if it does) on quite such a dramatic scale.

JREF shouldn't be concerned with false negatives except out of the goodness of their hearts (ha.) and limiting the amount of backpedalling the testee can do - or more relevantly, to ensure that the testee is willing to agree on camera that the protocol was fair, mutually agreed, and gave an adequate chance to test their claim.

I admit that my test will not demonstrate an effect of prayer if prayer only works for people you know personally, or people you are in contact with. I don't see why this is a problem.

ETA: P.S. Nowhere in the challenge does it say that I'm required to be a Believer (though that is indeed the usual case; as I've said, I am agnostic on this). It just requires that I be able to state clearly what a positive and negative result would be, and devise a protocol that ensures there will be no false positives that is mutually acceptable.

P.P.S. Your use of "confounders" to mean "things which could result in a false negative" is rather unusual; the usual use is to mean "things which could result in a false positive if not accounted for".

 

[saizai]

Also, an enhancement from the believer's perspective (not relevant to the MDC, but relevant to a more general study):

I'll have the study be run asynchronously; that is, people can join at any time and immediately be active without having to wait for the required number N to be ready. Each round will conclude once N(R) Recipients have completed a year each. This makes the site more user-friendly and immediate-gratification-y (heh), and should not in any way detract from the validity of the protocol so long as N(R) is not determined on the basis of "live" results (which can be assured by simply making the assignments database inaccessible to me).

Whenever a Recipient is done with one year, no matter which their previous status was, they'll be shifted to "known recipient" mode - i.e. they will definitely be receiving prayer, told so, and put in more immediate contact with their Healer(s). We'll still gather info about their progress per usual. Recipients who are in their first year and in the active group would have a higher priority in the assignment algorithm though.

This should allow a secondary, correlative study about the effects of uncertain prayer (i.e. the 50-50 unknown of each Recipient's first year) vs certain prayer (i.e. thereafter). The effects of certain prayer of course can be dismissed as placebo effect, so are not intended to demonstrate any paranormal claim, but would nevertheless be of definite perceived (and actual) utility to the people involved.

(Frankly, this is a point that's always puzzled me about the practice of medicine: why not try to enhance the placebo effects as much as possible if your goal is to heal the patient? Obviously, one must control for them as a confounder [note that I'm using this to mean "thing which could result in a false positive"] when doing research for the effectiveness of treatments, but that doesn't mean that the placebo is ineffective; quite the opposite has been proven true repeatedly.)

I can't think of any way to implement this as a causal-effects study; to do so you'd need to cause some of the recipients to be shifted and some not, or assign them differently at the outset, which would probably not be very user-friendly / good PR (which is a consideration in a study run via website).

 

[Cuddles]

Cuddles - As I've said, the number of participants is purely limited to what can be done practically. I have no other reason to put a limit on it.

Also I should point out (again...) that excessively small values of N would result in it being impossible to get p<.05, so that point is somewhat moot.

No it isn't. Unless you can demonstrate that the prayer and control groups have similar composition, especially in regards to serious vs. mild disease, no-one will accept it as valid.

What sort of numbers do you expect? You may not be limiting the participants, but do you think you will have 1000 people, 10 people or somewhere in between?

P.P.S. Your use of "confounders" to mean "things which could result in a false negative" is rather unusual; the usual use is to mean "things which could result in a false positive if not accounted for".

"Confounder" always means anything that will cause the test to give a false reult, positive or negative. False positives can be dangerous if they suggest that a treatment is effective when it really isn't, so they tend to have more attention focussed on them. In this case a lot of attention is paid to false negatives because believers often make excuses for a study not finding anything, so much care must be taken to show that there is nothing that could provide false results in either direction.

 

[saizai]

No it isn't. Unless you can demonstrate that the prayer and control groups have similar composition, especially in regards to serious vs. mild disease, no-one will accept it as valid.

FWIW I have said that I will be tracking this data upfront... so it should be quite easy to demonstrate.

What sort of numbers do you expect? You may not be limiting the participants, but do you think you will have 1000 people, 10 people or somewhere in between?

Somewhere in between. It'll depend in large part on how much traffic / word of mouth is generated, and how many participants can be gotten through more normal means (eg asking cancer doctors, church groups, cancer support groups, etc to disseminate info).

Of course, I'd like to have as many as possible.

"Confounder" always means anything that will cause the test to give a false reult, positive or negative. False positives can be dangerous if they suggest that a treatment is effective when it really isn't, so they tend to have more attention focussed on them. In this case a lot of attention is paid to false negatives because believers often make excuses for a study not finding anything, so much care must be taken to show that there is nothing that could provide false results in either direction.

I see. In that case, could you please respond to my request above?

"Could you please specifically list what you believe the "obvious flaws" to be, how to correct them, whether they would calse a false positive or negative, and how they would do so?"

Of course, to claim anything as a false negative, I expect you will be taking the stance of a Believer from some particular theological framework for the sake of argument; it would be helpful if you specified which one.

 

[petre]

ETA: P.S. Nowhere in the challenge does it say that I'm required to be a Believer (though that is indeed the usual case; as I've said, I am agnostic on this). It just requires that I be able to state clearly what a positive and negative result would be, and devise a protocol that ensures there will be no false positives that is mutually acceptable.

While technically true, the problem is that even if you don't believe in it you have to find people that do. Namely, when (if) you are asked to provide 3 affidavits from people stating they've observed what you intend to demonstrate. A general affidavit would be fairly easy to get I'd imagine (i.e. "I believe the power of prayer works", etc) but to get someone to claim that they believe your protocol, specifically, will produce results may be more difficult.

 

[saizai]

While technically true, the problem is that even if you don't believe in it you have to find people that do. Namely, when (if) you are asked to provide 3 affidavits from people stating they've observed what you intend to demonstrate. A general affidavit would be fairly easy to get I'd imagine (i.e. "I believe the power of prayer works", etc) but to get someone to claim that they believe your protocol, specifically, will produce results may be more difficult.

I don't think I can do so, because this study has never been performed before in the manner I want to do it. Plus, the affidavit you suggest wouldn't meet the standard anyway.

However, I don't think it comes under the clause for which that "3 affidavit" requirement was made (namely, really extreme claims of personal power), so hopefully it should not be relevant. Also, you're conflating two different things: affidavits confirming that it's something worth JREF's bother (i.e. to filter out excessively extreme claims, like creating lights around oneself spontaneously), and people to participate. The latter should not be difficult.

You still haven't answered my request though.

 

[Startz]

I'm sitting here in a jury assembly room waiting to be called for a jury panel. I offer this as an excuse in case the following seems incoherent to readers.

There's a shaggy dog story that goes something like this. A magician (let's call him JR for short) claims that he can make an elephant materialize onstage out of thin air using an ordinary pack of playing cards.

Various scientifically-minded skeptics have a rousing argument about the statistics of such a demonstration. Questions are raised about when the deck is shuffled whether hearts will be randomly distributed or not. And whether the test results depend on if the jokers have been removed from the deck. Then there is the question about whether JR can be trusted to shuffle the deck randomly or whether an observer from the audience should cut the cards.

So while all skeptical eyes are focused on the card deck in JR's right hand and all minds are working on the statistics of card decks, JR holds out a handful of peanuts in his left and an elephant lumbers onto stage sight unseen.

The moral (besides that jury duty can be mind-numbing) is that perhaps the attention being paid to statistics is displacing attention that might be better spent looking for loopholes and tricks.

But since about all I have to contribute is on statistics, I re-ran the simulation using two different variances and reducing the sample size to 20. In 50,000 simulations there were 5.36 percent false positives, about as one would expect.