Check my methodology - prayer study

[saizai]

Startz: I misinterpreted your #1.

However, it is still quite unnecessary for a study run by an automated website capable of handling relational databases (as mine is). I consent to allowing up to three computer specialists of JREF's choosing familiar with Ruby on Rails to examine my code to ensure that its mechanism is correct. I do not grant any right to the code, and it is not a part of the "data" to which JREF has a right as specified in the rules.

 

[Yoink]

But that is exactly what you are doing by not accounting for the confounders of disease level, demographics, frailty, etc...

And it does not have to be volitional but mere unintentional if you are not aware that you are doing it. For instance, if I load fruit onto a plane for distribution and unknowingly, an insect happens to be in some of the fruit, I am introducing something that potentionally might ruin the fruit or harm the consumer...

However, If I am aware of these insects and inspect the fruit before loading (placing) them on the plane, I stand a better chance of preventing the harm from occurring...

But if I've been made aware of this problem and dismiss it as inconsequential (which it seems you are doing by all your rhetoric) then it's willful blindness...

As for your practical signficance=statistical significance=0.001. That's just silly, because then your results are solely based on sample size and not on a real scientific hypothesis placed within a theoretical construct. Irrespective of your hypothesis, If I were a reviewer and you were looking for funding to do this type study design, I'd place you on the denied pile because you're not even close to accounting for all of the sampling problems and confounders that will occur...

Seriously, seek the assistance of Ph.D. biostatistician with clinical trial experience, they can help you if you really want a decent study design...

Digithead: I know that experimental design is incredibly tricky and that all sorts of apparently robust studies prove upon subsequent analysis to have hidden statistical flaws, so I'm not doubting that you may be putting your finger on a real problem here, but I'm struggling to imagine the sequence of events that will lead to a false positive if there is a reasonable large sample size.

Is the kind of scenario you are imagining something like this? Let us say that the sample is 100 people, and 10% are at Death's Door, 50% are Very Sick and 40% are Not So Sick: if this group is randomly divided it is pretty possible that we might get 8-2 division of the "Death's Door" patients into groups A and B, right (just under 5%, I think). Of course, that means there is a correspondingly greater likelihood of a few more "Very Sicks" in the B sample. I think we're getting into statistically insignificant probabilities to have one side have markedly fewer DD's AND VS's.

Is it your view that the higher number of DD's might so warp the experiment as to give a significant false positive?

So then is the solution to this a matter of "grading" the participants into cohorts and then modifying the randomly distributed groups to ensure rough parity within cohorts? How do you do this without either disrupting blindness or breaking randomness?

 

[saizai]

20. Any Recipient who can be clearly demonstrated to have communicated with any other Participant, or who refuses to sign a statement saying that they have not, will have their data removed from the analysis for all purposes above. The final signoff rule will be excepted for Recipients who die in the course of the study, and who have previously signed their commitment not to contact other Participants.

WEBSITE

21. The entire study shall be run using an automated website programmed in Ruby on Rails. All assignments shall be done by this code, as shall the display of Healer's current assignment, the online collection of survey results, and the presentation of copies for the Participants to sign and mail in. The code to site may be examined by up to three computer science specialists of JREF's choosing to ensure that there are no errors, back doors, security flaws, etc. The databases containing Participant contact information, Healer-Recipient assignment pairs, and group assignment will be kept secure from reading by any parties other than the website code itself until the completion of the round(s) in which those Participants are involved. *

22. Claimant, and only Claimant, shall have continued read/write access to general pages of the site, e.g. the front page, FAQ, layout, CSS, etc.

 

[saizai]

Also: A stratified random distribution would be acceptable to me, so long as I alone define the criteria for the subgroups. From each subgroup, assignment would proceed in the usual 50-50 random way.

And Yoink: Even if the sample size were small, it would still be unable to result in a false positive, because of the bound on significance for what constitutes a positive result.

 

[saizai]

7. Upon commencement of a round, all participating Recipients will first be divided into subgroups based on specific criteria of the Claimant's sole discretion. Half of each subgroup wil be randomly assigned by computer to be in the Active group; the other half will be in the Control group. The last one, if there are an odd number of Recipients in a subgroup, will be assigned at random. No Participant, nor any Experimenter who has potential contact with any Participant, will be allowed to have access to this data until after the Round is complete. This ensures that the experiment is a double blind. *

 

[Startz]

>snip

2. Of course. I elect as the form of my definition a chunk of code to be run on the server.

I don't think I understand whether JREF gets to see the code before signing the agreement.

3. It is a pre-test; you can consider it part of the negotiation. I consider the terms that I reserve for myself to define unilaterally before commencement to be completely acceptable to a skeptic's needs for the protocol no matter what I define them to be.

Does JREF get to see what statistical test is going to be used before they sign on?

 

[digithead]

digithead: I claim no a priori theory about the mechanism or effect of prayer. The first round is inteded to find out (or tune) the latter; I will not attempt to look at the former. We are not going to discuss theory, just as you would not discuss it with a dowser.

I'm not asking you for a theory, I'm asking you what is the theoretical difference that matters just as I would as a dowser what accuracy would he or she consider "proof" that dowsing works...

Your insects analogy is inaccurate.

If you expect others to back up their claims, then you should extend the same courtesy. Declarative statements are meaningless without proof...

 

[saizai]

Startz:

I don't think I understand whether JREF gets to see the code before signing the agreement.

They get to see the code before the commencement of round two, and verify that it in no way breaches double blind, installs a back door, accesses a recipient's assignment(s) status, or accesses Healers' data. They do not get to reject a proposed code based on any other reason.

Does JREF get to see what statistical test is going to be used before they sign on?

Could you elaborate on what you mean by this, i.e. in what way the proposed protocol is underspecified?

Yes, they get to see all tests I decide on before the beginning of the second round.

digithead:

I'm not asking you for a theory, I'm asking you what is the theoretical difference that matters just as I would as a dowser what accuracy would he or she consider "proof" that dowsing works...

To quote Startz:

I completely agree that "scientifically important" is what matters. But I think that in this case 1e-308 is "scientifically important." [Of course, detecting an effect of the size would take a *real* large sample. ]

Your insects analogy is inaccurate because it makes this analogy:
there is a pool of subjects, S
a subset of the subjects are flawed ("insects"), Sf
the pool is divided randomly into two subsets, Sa and Sb. By random distribution, Sf should be equally distributed between Sa and Sb on a bell curve likelihood

Your analogy claims that the presence of Sf in S corrupts the entire pool S (by insects breeding and infecting other fruit). That has no analogue, and does not affect diff(f(Sa), f(Sb)).

 

[digithead]

Startz:

Your insects analogy is inaccurate because it makes this analogy:
there is a pool of subjects, S
a subset of the subjects are flawed ("insects"), Sf
the pool is divided randomly into two subsets, Sa and Sb. By random distribution, Sf should be equally distributed between Sa and Sb on a bell curve likelihood

Your analogy claims that the presence of Sf in S corrupts the entire pool S (by insects breeding and infecting other fruit). That has no analogue, and does not affect diff(f(Sa), f(Sb)).

No, my analogy is not wrong. I was demonstrating that by being aware of possible confounders, one has to adjust for them or they are willfully ignoring the possible contamination...

And your methodology will corrupt the pool if there ends up being a higher proportion of Sf in one group than another. Your simple random sample methodology does not prevent this from happening...

And relying solely on the Gaussian distribution is risky sometimes. What if your outcome and independent variables are distributed binomial (or multinomial if there are different levels of confounding)? You have to stratify to make sure that these are accounted for. What if the underlying distribution is Cauchy or Gamma or Beta? Then you're going to need even more samples if you're going to rely on the central limit theorem...

Hence why most clinical trials now use random and mixed effect general linear (and occasionally nonlinear) models with non-Gaussian link functions because these allow you risk-adjust both for heterogeneity within a group and across groups while using the correct underlying distribution...

Again, see a Ph.D. biostatistician with clinical trial experience, they can help you design a better study. And you will get better suggestions than relying on criticism derived solely on skeptics forum...

 

[jskowron]

Saizai says

"They get to see the code before the commencement of round two, and verify that it in no way breaches double blind, installs a back door, accesses a recipient's assignment(s) status, or accesses Healers' data. They do not get to reject a proposed code based on any other reason."

This seems to be a potential sticking point to me, at least as far as the MDC. If it were my million bucks, I would not agree to automatically accept any formula. There are certainly condition's other than those listed in the above quote that would make the formula invalid and/or unreliable. For example (admittedly extreme) the code could be as follows:

if condition=control group, then score= 0
if condition=prayer group, then score= 100

such a code does not breach double blind, install a back door, ivolve accessing a recipient's assignment(s) status, or involve accessing a Healers' data. However, such a code would lead to invalid findings of a significant difference between the control and prayer group. Based on your conditions set forth above, JREF would not be able to reject the code.

Also-

Stats Point 1- how many statistical comparisons do you plan to make per round (e.g. how many p values will you be calculating)? If there is more than one planned comparison, you should to adjust your alpha to account for the number of comparisons. For example, if you agree the p<.05 will be your significance level (alpha), and are doing three statistical comparison, each the significance level of each individual test would be p<.05/3, or p<.017. The more tests you do, the more likely you are to encounter type 1 errors (false positives). This procedure corrects for this.

Stats Point 2- Depending on what statistics you use, testing the hypothesis "the prayed for group will have a higher (or lower) mean score than than the control group" is different that testing the hypothesis "the prayed for group will have a different mean score than the control group." For statistical reasons, it may be advantageous to declare the expected direction of the difference.

Good luck. I can appreciate the process you're going through. There are many a Masters or Ph.D. candidate who think they have covered everything, only to find out, hopefully at the thesis/dissertation proposal meeting, rather than defence, that they missed something.

 

[saizai]

jsk - I specifically said that the code would not have access to the person's group assignment status; your code does so. Please come up with another example of an unacceptable code.

Pt 1: There is only one comparison in the 2nd and 3rd rounds, that of the Score. The type 1 error potential is a necessary part of round 1.

Pt 2: Please elaborate. I specifically count a worsening in health among the active vs control group as a positive result, and will not cede that point. (Though I certainly don't want or expect it, it counts.)

Pt 3: Indeed.

 

[Yoink]

Pt 2: Please elaborate. I specifically count a worsening in health among the active vs control group as a positive result, and will not cede that point. (Though I certainly don't want or expect it, it counts.)

One fairly obvious problem with accepting either "improvement" or "deterioration" as "positive" results is that it requires a larger sample size to protect against non-random distribution. If you are concerned only with, say, Group A doing better than Group B, then you have only one kind of error to worry about--that you don't get an unrepresentative number of super-ill people into Group B. But with your current set up, you also have to worry about the opposite (a non-random distribution of super-well people getting into Group B). As I see it, that would mean that you'll need something like double the sample size to meet the same confidence level. (A real statistician will no doubt correct me).

 

[petre]

I should add that JREF is likely to suggest that you do indeed simply perform your first trial on your own, and after you obtain the data you require then submit an actual application with specific criteria for success.

 

[GzuzKryzt]

(Saizai) has been invited to apply. The protocol will need a lot of work, but I'll not be discussing that until after an application is received.

This will indeed eat up lots of energy.

Saizai, I will grant your "study" its deserved time - after your application got accepted. I just see questions marks popping up all over the place due to experiences with wanna-be-claimants on this very forum. No offense. Really.

 

[Gr8wight]

Draft JREF M$C protocol for PrayerMatch experiment

Protocol:
SETUP
1. Two pools of people will be recruited by Claimant to participate in the study. These people will be unr...

saizai,

Do you believe that prayer is efficacious in the healing of disease?

 

[saizai]

petre - That's what the first round is.

GK - What sort of questions that relate to your experiences with other "wanna-be-claimants"? I feel I've been quite forthright, rational, comprehensible, and sane...

Gr8wight - As I said on the About page (did you read it?), I am agnostic on the subject, as I have not seen sufficiently convincing evidence either way. I am conducting this study purely out of a desire to investigate it. My religious/spiritual beliefs in no way require that prayer work, or that it work in this manner.

 

[Cuddles]

I think a very important question is how many people do you plan to have? If there is a small sample size (only 10-20 patients) then Digithead is entirely right that the random selection will not give an equal distribution and your results will be meaningless unless you account for this. If you have a very large sample size (1000 people) then your method should be OK. It seems very unlikely that you will get this many people for a preliminary trial. Unless you tell us how may people you will actually be using we have to assume the worst.

 

[jskowron]

Saizai-

Sorry- I misunderstood your condiditions when giving my example. You're correct, my formulas would require the experimenter to not be blinded to condition, which you say is valid reason for rejection of the formula. How about this extreme example-

If round 1 score > overall mean, then new score = 100
If round 1 score <or= overall mean, then new score= 0

This type of formula would potentially magnify any small differences that may exist between groups and magnify them to the point of being statistically significant.

(Please be clear that I am not accusing you of being a scammer when I present these extreme examples. However, your inclusion of the phrase "They do not get to reject a proposed code based on any other reason" is a little strange [and unscientific] and hints, to me at least, at mistrust or even duplicity.Surely you recognize that scammers are out there, and the MDC makes JREF a target. I cannot imagine JREF will agree to a formula for calculating a score without knowing what it is.)

As for the directionality of difference between groups- You seem to be proposing a test of the difference between 2 group means. The most basic statistical procedure used to do so is a t-test. The t statistic is normally distributed (e.g. when all possible values of t are graphed, the graph takes on a bell-curve shape). To simplify things- if you hypothesize that the prayer group score will be higher than the control, p=.05, you are statistically saying that the t score will fall in the right 5% end (tail)of the bell curve (this is known as a 1 tailed test). If you hypothesize simply that there will be a difference between the means, p=.05, you are statistically saying that the t score will fall in either the right 2.5% or the left 2.5% (a two-tailed test). As can be seen, a lower difference in the means (with prayer>control) will be consiedered significant in the one tailed test, but not in the two tailed. If there is any theoretical basis to do so, a one tailed-test should be employed. While you might not care about type-2 error (not noticing a positive result when there is one), it is proper statistical procedure to minimize both type 1 and type 2. Regardless, any statistical program (or t-test distribution chart, if you're calculating the statistic by hand!) will require you to specify ahead of time whether you test is 1-tailed or 2-tailed.

sorry for all the stats geek stuff, but once you go down the road of using inferential statistics, it is important to follow all the road signs.

 

[Gr8wight]

Gr8wight - As I said on the About page (did you read it?), I am agnostic on the subject, as I have not seen sufficiently convincing evidence either way. I am conducting this study purely out of a desire to investigate it. My religious/spiritual beliefs in no way require that prayer work, or that it work in this manner.

In that case, I think you are in the wrong place. The JREF is not a research foundation, and the challenge is not a vehicle for scientific inquiry. The challenge is aimed at those who make a claim, not those who just want to find out. You should be pursuing this at a university somewhere.

 

[Ladewig]

Prayer is one of those things that is probably impossible to test in a rigorous double-blind experiment without doing things that no ethics review board would ever allow (lying to patients about whether or not they were being prayed for, and lying to "pray teams" about whether or not the people they're praying for are really sick etc.

Why would no ethics board approve that? How is it different from giving placebos to sick people?