Perhaps this is, if everyone will forgive the cliche, a teachable moment. (I hope not to be boring everyone with statistical detail.) saizai believes that his idea is subject to straightforward statistical test. Others claim you can't ignore "confounders." First, the general principle. Then, an illustration.
In general, "controlling for confounders" is a critically important step. Imagine your outcome was whether patients died in a hospital and you were trying to see if being given last rites made a difference. If you tested for differences in means without controlling for patient health (a confounder) you'd get a silly answer. That's because those given the last rites are probably a lot sicker.
BUT! It isn't absolutely necessary to control for confounders that are uncorrelated with your observed explanatory variables.
It's okay to omit confounders if they are uncorrelated with selection into control vs treatment group and uncorrelated with the data gathering process.
To illustrate this I made up a computer example in which 400 observations are randomly divided into a "prayed for" and "not prayed for" group. An outcome y is determined by a strong confounder, a modest random effect, and no effect at all of prayer. Then I do a standard 2-tailed t-test for differences in means at the 0.05 level and see how often "prayer has no effect" is rejected.
I did this 10,000 times and a found a significant effect of prayer 4.48 percent of the time. Just about what you'd expect.
This seems to illustrate that saizai 's claim that confounders don't matter in his application is basically right. Or it may illustrate that people are talking at cross-purposes, in that my "model" of saizai's experiment misses an important element that would make a difference. I hope that showing this toy model will make it easier for others to point out the specifics of what's missing.
For those interested, here's the code (in Matlab).
Code:
%{
confoundedPrayer.m
Monte Carlo to illustrate testing for prayer efficacy
in presence of confounder
Dick Startz
August 2006
%}
rand('state',0); %% reset random number generators
randn('state',0);
nMonte = 10000
n = 400
rejectSum = 0;
for iMonte = 1:nMonte
prayFor = rand(n,1)>0.5;
notPrayFor = ~prayFor;
y = 2 + 3*rand(n,1) + 0*prayFor + randn(n,1);
meanDif = mean(y(prayFor)) - mean(y(notPrayFor));
stdErr = sqrt(var(y(prayFor))/sum(prayFor) + var(y(notPrayFor))/sum(notPrayFor));
rejectSum = rejectSum + (abs(meanDif/stdErr)>1.96);
end
disp(['False rejections occurred ',num2str(100*rejectSum/nMonte),' percent of the time']);
