Check my methodology - prayer study

[Skeptic Ginger]

skeptigirl, gr8wight:

I've said repeatedly.... I am testing for the ADDITIVE effect of prayer.

I am not going to try to make sure that the control group is not prayed for at all (that would be impossible as well as unethical). What I can say with certainty is that they aren't getting any extra prayer from their participation, and with statistical certainty that they (as a group) are getting less than the active group.

That diff is what I am testing.....

Perhaps I can add to the point others are making about a larger sample size.

First, if you are going to test the effect of additional prayer, as opposed to 'any' prayer you will need to quantify baseline prayer amounts and define what you mean by additional. You have to address exactly what you are testing in terms of quantity over baseline prayer rates. "Additional prayer" is too vague in this case.

Then you will either need a very large group so that you can assure an even distribution of baseline prayer volumes for the individuals in the study, or you will have to collect data on the baseline prayer rates of those individuals and include that in your analysis. This is similar to the discussion about severity of disease in both the control group and the study group.

Say you want to test a group of people who are self selected so they are likely to be being prayed for. Those people might have any volume of prayers being said for them perhaps depending upon the size of the congregation they attend. At some cut off point you may be assured an equal distribution randomizing the group into two. Depending on the range of prayer volume within the group, that cut off point may be higher or lower.

Say the baseline prayer rate is 1 prayer or no prayer. If you divide a group of 2 subjects they will be 100% different. If you pray for the guy with no prayer, both the test group and the control group will have one prayer each and will not differ.

Say the number of baseline prayers in a group of 20 subjects is between 0 and 1,000. Ten of your group of 20 attend the same church and the congregation of 1,000 regularly pray for all members of the church with cancer, (they told each other about the study). Toss a coin in the air 20 times. How evenly was your division of heads and tails? Do that several times to get a sense of how a random division of a very diverse group of 20 ends up being distributed in each group.

The more diverse the group, whether it be disease severity, disease type which we haven't even gotten to here, or baseline prayer rate, the larger of a sample you need to see an even distribution in your test group and your control group.

The only other way to address dissimilarities in both groups is to measure everything and show the groups are evenly distributed. Severity of disease, age, gender, economic status, type of cancer, quality of medical care, baseline prayer rate, religious conviction and probably more things I am not thinking of have to be accounted for as equal in both groups or your results might be reflecting the difference in the group rather than the difference in prayer volume.

The way researchers generally address small sample size in a test study before undertaking a larger more expensive study is to select a very homogeneous group for the initial smaller study. You won't have this with you current method of recruiting subjects.

 

[Startz]

>snip

I don't think I understand your example. Could you elaborate?

How about something like this? An unscrupulous challenger signs up a few confederates with a unusual first names or recognizable pictures. Then the challenger also signs up some friends to do the praying. (They get to see pictures.) All the praying people need to do is identify 1 subject to the challenger. Then it's easy to arrange a statistically significant result no matter how large the sample is. In other words, a minute amount of cheating is all that's needed to rig the results.

 

[Skeptic Ginger]

I stand by my original statement.

[Original statement: "Placebos generally aren't given for serious diseases. New treatments are compared with accepted treatment, but not with a placebo. Deliberately withholding treatment from a potentially fatal disease like cancer would be considered serious misconduct, and possibly murder, if there is any treatment available that gives a better survival rate than placebo."]

Obviously I agree that a double-blind, placebo controled trial is the ideal, but this does not always happen. Much of the debate over evidence-based medicine (within the medical community, not in the media) has come from the realisation that there are many treatments, especially in surgery, that have never been tested properly.

If a treatment is accepted for a life-threatening condition then any new conditions are usually "only tested against the previous treatment", since if someone dies while on placebo the people conducting the trial will be guilty of witholding treatment. This is not the best situation for science, but sadly this is how it works in the courts. Signing a waiver that states you are aware you may only get a placebo does not affect this.

Surgery is even worse, since there are risks associated with any surgery. This makes it almost impossible to do placebo controls, although it has been done a few times for minor operations. Much surgery is accepted because if something is wrong, cutting it out seems to be an obvious way to solve it. Unfortunately this may not always be true, and recently some procedures have been brought in to question precisely because they have only been compared with other procedures and never with a placebo.

All the links you posted in response to Yahzi seem to refer to non-fatal diseases and as such your arguments are entirely true. This is not the case that I was arguing, where there are serious ethical problems involved with placebos when a lack of treatment can be fatal.

You have addressed 4 separate issues.

First, the issue I addressed (above in bold) is patently false. Is that the statement you are standing by?

The second issue is whether a known treatment should be withheld in order to study a potentially better treatment. That is done, though rarely and under specific circumstances. Typically it is when the known treatment isn't tolerated by the patient or the known treatment offers a very poor prognosis and the new treatment has very high promise. Patients with terminal diseases and very poor prognoses sometimes volunteer to forgo standard treatments in order to further research even if they may die but future victims might be helped.

And many times there is no treatment yet established or the experimental treatment addresses a different aspect of the disease than the standard treatment addressed. For example a number of women took Tamoxifen after their breast cancer was treated to see if it lowered the rate of recurrence.

The third issue you have raised is how do you control for placebo effect in surgical procedures. That's a whole different ball of wax. Recently a study was done involving sham surgery that actually involved an incision in the control group. It was extremely controversial but the results indicated the surgical procedure did indeed have a placebo benefit and not a physical benefit.

The forth issue is probably more one of communication than substance. You have the right idea. Known treatments which could be used are generally not withheld in order to test new treatments. But when you state, "only tested against the previous treatment" that is technically incorrect. Certainly we want to know if a new treatment should be given in conjunction with and old treatment or instead of. That would be dealt with in the nature of the treatment and the study design. But the actual study would still use a placebo control if possible.

Your statement, "if someone dies while on placebo the people conducting the trial will be guilty of witholding treatment", implies giving a placebo means you must also withdraw treatment which is of course, silly. In addition to that, it wouldn't be the placebo that was at issue, it would be the failure to treat. The person could die on either the experimental treatment or the placebo if the known treatment was withheld.

What you are suggesting, and I don't think you mean, is that a study of a new treatment would involve withholding the known treatment from a group that was receiving the experimental treatment and comparing them to the group receiving the known treatment. Until there is evidence the new treatment is effective, it isn't going to be compared to a known treatment by giving one group the old and one group the new treatment. Once it is determined that both treatments work, then you might see a study comparing the two treatments to each other.

 

[saizai]

I haven't the time to respond to replies since my last comment at this point, but:
1. http://www.prayermatch.org is now up finally, though running slowly because fcgi still isn't working (grr)
2. I'll probably use a supplemented SF36 for monthly HRQOL surveys
3. I'll probably use a TNM / Ann Arbor staging for quarterly doctor surveys (plus a few other questions)

The site is almost done; what's left to do are the monthly/quarterly reports, matching algorithm, and doctor logins. That should be relatively easy and not take too much more time.

 

[Gr8wight]

I haven't the time to respond to replies since my last comment at this point, but:
1. http://www.prayermatch.org is now up finally, though running slowly because fcgi still isn't working (grr)
2. I'll probably use a supplemented SF36 for monthly HRQOL surveys
3. I'll probably use a TNM / Ann Arbor staging for quarterly doctor surveys (plus a few other questions)

The site is almost done; what's left to do are the monthly/quarterly reports, matching algorithm, and doctor logins. That should be relatively easy and not take too much more time.

Let us know when you have your positive result to report, so we can insult you over your refusal to release your raw data to be examined.

Oh, sorry, that was someone else.

 

[saizai]

Gr8wight - That was just rude.

 

[Gr8wight]

Gr8wight - That was just rude.

What, comparing you to Gary Schwartz was rude? Does Gary know you feel that way about him?

Prove me wrong.

Yes, I'll wait.

 

[saizai]

Gr8wight - You didn't mention a name. You implied that I wouldn't release the data, and that therefore you would insult me. That's a coward's insult.

Given that you're evidently not willing to carry on a polite conversation, I won't be responding you from now on. G'day.

 

[Gr8wight]

Gr8wight - You didn't mention a name. You implied that I wouldn't release the data, and that therefore you would insult me. That's a coward's insult.

Given that you're evidently not willing to carry on a polite conversation, I won't be responding you from now on. G'day.

That's OK. I'll still wait.

 

[saizai]

This is one of our major concerns. It doesn't matter how good your method is if you don't have enough people. 25 is not large enough unless you can absolutely guarantee that the two groups are of similar composition. You have said that you will be able to observe this, but you have not said how you will observe it, or what you will do about it if there is a problem.

1. I'm having >25.
2. I observe by the very simple expedient of asking.

No study has control and active groups that are exactly the same. They just draw from the same pool and make sure the study itself doesn't create differences. It's statistically unlikely (by definition, in fact) for a difference to exist - there'll be one p*100% of the time, for whatever p level you choose.

As digithead has said, Startz's model is not complete. It uses only one variable where many need to be considered, and was set up as a quick example that has not been shown to accurately model what you are proposing. It is not up to us to show that this model is wrong, it is up to you to show that there is nothing else that could affect it.

Proving a negative is impossible. You claim there is a real problem, therefore you have the much simpler task of proving a positive.

Just handwaving a claim that it's insufficiently controlled isn't good enough, especially when the monte carlo sim shows otherwise.

You must explain exactly how you obtain the scores or the whole trial is meaningless. The method must be decided in advance and cannot be changed depending on the results, since this is exactly how bad conclusions can be made from otherwise good trials. The classic statistical error is to take a set of data and analyse it until you find a correlation with something, which is almost always possible. This may not be the case here, but you must show that it is not.

Quite so. But did you completely ignore the places where I said that I would be determining the score equation in advance of obtaining data?

Not all people are above fraud. Signing a statement does not mean they mean it. How will you prove that they are telling the truth?

They don't know what group they are in, therefore they have no way to lie in a way that would influence the results.

As Gr8wight and I have said, this is simply not true. You need very large samples before you can rely on randomisation. If you expect a sample of around 25, as in the study you refered to, this is nowhere near large enough. Also, you must show that randomisation achieves this, whatever your sample size, not just assume it does.

Not so, sorry. I've seen plenty of robust studies with N ~= 25 that still manage p<.05 or <.01 or <.001 even. Depends on the distribution of the measure in the pool. In any case it's self-correcting: you can't obtain a p<.05 with a too-small pool. Simple enough.

False negative or false positive, the important word is "false".

Only if you're concerned with defending a believer's perspective. If you're just concerned with protecting the challenge against fraud, then false positives are it.

This is the worst kind of analysis possible. If you gather data from people and then try to find a correlation with anything, you will find one. This is why studies only ever focus on one cause and try to control for all others. Occasionally a strong trend may be noticed that is commented upon and recommended for further study, but a trial that is set up to examine one possible correlation cannot reliably comment on any other.

Again you don't seem to have read what I wrote very carefully.

Correlational aspects are only going to be used to inform the design parameters of the next round(s). That is consistent with standard scientific method. What is being tested is the causal.

Point 1 asked for your measure for the outcome, which you explicitly stated you would not provide and said you would change for different trials. This is not acceptable for a medical trial.

I said I would provide it before each trial in question. This is perfectly acceptable.

In any event thanks for the material, I will be using it next time I teach research methods, I'm pretty sure any undergrad can figure out the same issues that some of us have raised against it...

Glad you enjoy reading.

However, I should point out that everything I have written is my copyright and I explicitly do not grant you any rights to use it in any manner whatsoever.

It's time to abandon arguing about the protocol and try to discover why you feel it is so important to test the power of intercessory prayer on disease outcome. Especially when numerous published studies have shown its ineffectiveness. You claim you're agnostic and a skeptic but what is your motivation for doing this study? Because a true skeptic would take the already overwhelming evidence in failing to reject the null to conclude that intercessory prayer has no effect on disease outcome and move on to new matters. You've given us the how now give us the why...

I'm not interested in discussing my motivation beyond what I have already stated: curiosity as a true (weak) agnostic. I decline to get dragged into an argument about theology, philosophy, and the like.

Am I the only one bothered by the that last sentence? It seems rather confrontational (and smacks of distrust) for what is being presented as a plea to "check my methodology." When doing research, one should accept the fact that there are many ways to be wrong- probably more than there are ways to be right. Setting the conditions for which you can be judged as being wrong, a priori, is just not ok. You need to be open to errors being discovered, and willing to correct those errors. Through that statement, and his approach to those trying to help, saizai has indicated that he is not open to corrections or suggestions.

I'm quite open to correcting real errors.

I'm not open to "correcting" things that aren't really errors, or that are merely whims. I am only interested in your input insofar as it ensures that my methodology is tight. No score equation I can possibly choose, within the parameters I gave, would be a methodological flaw - and therefore I make it explicit that I can choose whatever I want.

This is for the simple reason of ensuring that everything in the application is totally explicit so there is no arguing later about what the terms are.

How about:
- prayer only works if a specific diety is addressed (FSM maybe?)
- prayer only works if done by a priest
- prayer only works if done on Sunday
- prayer only works if done by 100 or more people

Tracked correlatively. If it's the case then this info will be used to filter later rounds' participants.

- prayer only works if no one tracks the results

Inherent flaw in the design and indeed in all possible designs I can think of. Acceptable.

- prayer only works if you donate heavily to a church
- prayer only works if spoken in Latin
- prayer only works if done while standing on your head

Not tracked. If so, oh well, my miss.

This is where actually believing that it works a certain way comes in handy. You can then narrow down what you think actually matters and find out if you're right. With no belief, there are a great many number of factors you need to control for to make it a really worthwhile test. The "likely" result of "no effect" (given that the only proposed improvement to existing studies is sample size) will at least have some meaning if you had some belief, in that it will encourage you to re-examine it.

You cannot, in principle, explicitly control for all possible factors. It's simply impossible by definition. That's what randomization is for.

So I suppose if someone were to present an existing study that used 100 or more people, you would identify an error in its methodology that your study will avoid somehow, or you will agree that even the sample size of that study was insufficient, and increase your definition of "too small of a sample size" to include the new study?

I haven't seen this hypothetical study, therefore I cannot comment.



One further improvement I have thought of:

I'll set an arbitrary score equation for the first round - essentially a random guess. This allows JREF to participate in the first round as well as the second and third.

If the first round is positive, then we go directly to the third as the 'final test'; if not, then we go to the second round as the new 'preliminary test' with a score equation based on the actual data gathered in the first round.

 

[Cuddles]

1. I'm having >25.
2. I observe by the very simple expedient of asking.

No study has control and active groups that are exactly the same. They just draw from the same pool and make sure the study itself doesn't create differences. It's statistically unlikely (by definition, in fact) for a difference to exist - there'll be one p*100% of the time, for whatever p level you choose.

If you ask them, then how is this a blinded trial?

No-one ever said the groups have to be identical, but they must be similar, and must be provably so. 25 is simply not big enough, to rely on randomisation alone you need hundreds at the least, preferably thousands. A related issue is that, since there are so many types of cancer, no two people in the study will be similar. This would mean that no matter how randomly they are grouped, you will never get a meaningful result.

Proving a negative is impossible. You claim there is a real problem, therefore you have the much simpler task of proving a positive.

Just handwaving a claim that it's insufficiently controlled isn't good enough, especially when the monte carlo sim shows otherwise.

Where is the handwaving? Your sample is simply not large enough. The one simulation done was very basic and did not take into account any of the variables we have said could affect the results, such as age, social status, etc. Since you have not defined your sample group any more specifically than "anyone with cancer", we cannot give any more specific problems, we can just point out the areas you have not covered that could, and in many cases are likely to, affect the result.

Quite so. But did you completely ignore the places where I said that I would be determining the score equation in advance of obtaining data?

You said "Extant data will be analyzed after the first round and used to create a Score Equation.". Is this some new meaning of the words "in advance" that I wasn't previously aware of?

They don't know what group they are in, therefore they have no way to lie in a way that would influence the results.

So why is a statement even needed? If you are not concered about the possibility of cheating, why would you require them to sign something saying they are not? And if you are concerned, how can you prove that doing this would actually prevent them lying?

Not so, sorry. I've seen plenty of robust studies with N ~= 25 that still manage p<.05 or <.01 or <.001 even. Depends on the distribution of the measure in the pool. In any case it's self-correcting: you can't obtain a p<.05 with a too-small pool. Simple enough.

There may be studies that can achieve good preliminary results with small samples, but this is not one of them. You are looking for a (presumably) small effect with numerous possible confounders, most of which you haven't even considered.

Also, this is nothing to do with randomisation. If you take some ill people and some healthy people and asign all the healthy ones to a treatment group and all the ill ones to a control you will get an extremely significant result saying the treatment worked because the statistical analysis assume that the original groups were similar.

Only if you're concerned with defending a believer's perspective. If you're just concerned with protecting the challenge against fraud, then false positives are it.

Not true. With virtually every test of the paranormal, a negative result causes the claimants to say that the conditions were wrong or the test was unfair. In order to maintain any credibilty the JREF must ensure that any test will actually detect what it is looking for.

Again you don't seem to have read what I wrote very carefully.

Correlational aspects are only going to be used to inform the design parameters of the next round(s). That is consistent with standard scientific method. What is being tested is the causal.

Not really. As I said, it is well known that if you have a dataset with many variables, you are almost guaranteed to find correlations between some of them. If you use these to alter subsequent tests you are doing bad science, plain and simple. They could be used as the basis for an entriely different set of tests, but this does not seem to be what you are planning.

Also, you said "No difference in the Score Equation, participation criteria, or significance test will be permitted between second and third rounds.". The posts to which I was repying did not say that you would only be altering the equation after the first round, so I assumed you meant you would alter it after every test. I apologise if this is not the case.

I said I would provide it before each trial in question. This is perfectly acceptable.

I thought you had said that it would not be available, but it appears I misread one of your posts, so I apologise. I am still concerned that you will provide it, but will not allow any discussion of whether it is acceptable to anyone else.

Glad you enjoy reading.

However, I should point out that everything I have written is my copyright and I explicitly do not grant you any rights to use it in any manner whatsoever.

Is content posted on a public forum covered by copyright? And if so, is it copyright to the poster or the forum's owner?

One further improvement I have thought of:

I'll set an arbitrary score equation for the first round - essentially a random guess. This allows JREF to participate in the first round as well as the second and third.

Why does it have to be a guess? If you want to measure if people get better from prayer then your equation should measure this. It may not be perfect, and I would consider consulting qualified people who run similar tests, but there should be nothing random about it at all.

If the first round is positive, then we go directly to the third as the 'final test'; if not, then we go to the second round as the new 'preliminary test' with a score equation based on the actual data gathered in the first round.

This is just silly. Either a test is the JREF preliminary or not. If it is not, then it cannot be counted as one retroactively, if it is, then a negative result will count as a failiure. If all applicants were allowed to cherry pick their positive results the prize would have been won long ago.

 

[digithead]

Glad you enjoy reading.

However, I should point out that everything I have written is my copyright and I explicitly do not grant you any rights to use it in any manner whatsoever.

You appear to know as much about copyright law as you do about designing clinical trials...

Exactly how is my educational use of your study a violation of copyright? You've placed your ideas in the public marketplace. The title of the thread you started is "Check my methodology - prayer study". Seems to me that you've put your study out there for others to discuss, debate, and critique...

Sorry, but everything you've discussed in this forum is now out there for fair use, regardless of your copyright or frustration of how others might use your study...

 

[saizai]

I disagree. Fair use is only for limited extracts of a work for reasonable purposes (e.g. education). It just so happens that I have taught a college course myself, and for it had reason to obtain copyright permission - and have previously researched it in some depth. You can't quote an entire work without permission. You can refer people to a public discussion, but the fact that I have posted it publically doesn't mean that you're granted any rights to it other than to read it where I posted it.

Copyright law, btw, is not primarily about "ideas" but about content, i.e. my words as written. Ideas are what patents are for.

Note that fair use has four required tests, one of which is: "amount and substantiality of the portion used in relation to the copyrighted work as a whole". (http://www.copyright.gov/title17/92chap1.html#107) E.g. you can't (without permission) copy a short story that I write and publish online in its entirety, even for a classroom discussion.

I suggest you read the law more before accusing me of not knowing it.


P.S. You evidently haven't read the registration agreement for the JREF forum, which states in part:
Copyright

Any post or article published on the JREF forum by a Member is the copyright of the Member and may not be reproduced, copied or otherwise re-published without the express permission of the Member. By posting on the Forum a Member grants the JREF a non-exclusive licence to publish, republish or reproduce their work, in its entirety or as the JREF sees fit, in perpetuity. The James Randi Educational Foundation is the copyright holder of the JREF Forum.


That means that in addition to violating my copyright you would be violating the registration agreement...

 

[saizai]

If you ask them, then how is this a blinded trial?

What does asking them questions have to do with the blinding? You seem to be confused as to what 'blinding' means...

No-one ever said the groups have to be identical, but they must be similar, and must be provably so. 25 is simply not big enough, to rely on randomisation alone you need hundreds at the least, preferably thousands. A related issue is that, since there are so many types of cancer, no two people in the study will be similar. This would mean that no matter how randomly they are grouped, you will never get a meaningful result.

Wrong, sorry. They are statistically identical by definition. Perhaps you haven't read as many actual research studies as I (the ones I've seen are mainly in the areas of cognitive science and neurology, fwiw) but there are studies conducted and published routinely with n<50 which nevertheless produce perfectly sound results.

Where is the handwaving? Your sample is simply not large enough. The one simulation done was very basic and did not take into account any of the variables we have said could affect the results, such as age, social status, etc. Since you have not defined your sample group any more specifically than "anyone with cancer", we cannot give any more specific problems, we can just point out the areas you have not covered that could, and in many cases are likely to, affect the result.

I disagree. Please provide a monte carlo sim that does take into account your supposed other factors and explain how they would affect the result even after randomized double-blinding. (Note the and; I want actual numbers rather than just your handwaving about "this will affect it" without anything to back that statement up.)

You said "Extant data will be analyzed after the first round and used to create a Score Equation.". Is this some new meaning of the words "in advance" that I wasn't previously aware of?

You haven't read the rest of it evidently.

The score equation created is used in the next round. Which has new data collected. Thus for that round the SE was made in advance.

So why is a statement even needed? If you are not concered about the possibility of cheating, why would you require them to sign something saying they are not? And if you are concerned, how can you prove that doing this would actually prevent them lying?

It's simply one more measure, to address this concern even though as I explained it's not really valid.

Also, this is nothing to do with randomisation. If you take some ill people and some healthy people and asign all the healthy ones to a treatment group and all the ill ones to a control you will get an extremely significant result saying the treatment worked because the statistical analysis assume that the original groups were similar.

What part of "randomized double blind" are you not getting? I'm not "assigning" people to treatment vs control on the basis of their condition.

Not true. With virtually every test of the paranormal, a negative result causes the claimants to say that the conditions were wrong or the test was unfair. In order to maintain any credibilty the JREF must ensure that any test will actually detect what it is looking for.

That's fine. I assert that the test I have proposed is sufficient for me. The end.

Not really. As I said, it is well known that if you have a dataset with many variables, you are almost guaranteed to find correlations between some of them. If you use these to alter subsequent tests you are doing bad science, plain and simple. They could be used as the basis for an entriely different set of tests, but this does not seem to be what you are planning.

Agreed to the first sentence only.

They can be used to inform later tests - where the same standard of "decide in advance, randomized double blind" is used. And it's the normal practice in fact, whether explicitly labeled as such or not.

Also, you said "No difference in the Score Equation, participation criteria, or significance test will be permitted between second and third rounds.". The posts to which I was repying did not say that you would only be altering the equation after the first round, so I assumed you meant you would alter it after every test. I apologise if this is not the case.

I changed my mind on the first two (SE & PC): so long as they are fixed before the start of a given round, there is no reason to make them identical between rounds. Whereas there is reason to change them, to make the test more sensitive and based on actual data gathered last time so that you're not including a bunch of extraneous variables that turn out not to be affected.

I thought you had said that it would not be available, but it appears I misread one of your posts, so I apologise. I am still concerned that you will provide it, but will not allow any discussion of whether it is acceptable to anyone else.

I'm not interested in allowing discussion except for the purposes of avoiding a false positive. IMO there is no possible way for this to happen given the constraints I already have placed. Can you think of one?

Is content posted on a public forum covered by copyright?

Absolutely yes. See above post.

And if so, is it copyright to the poster or the forum's owner?

The copyright is always to the originator of the work unless they have signed a contract that says otherwise (i.e. the registration agreement, quoted above). Per the Berne convention (IIRC) there is no need to explicitly say this; it's automatic.

Why does it have to be a guess? If you want to measure if people get better from prayer then your equation should measure this. It may not be perfect, and I would consider consulting qualified people who run similar tests, but there should be nothing random about it at all.

Because I don't know in advance how they will get better. Maybe just HRQOL? Maybe a submeasure? Maybe their cancer staging? $ spent in treatment? There are a lot of variables, and fixing which ones you're testing for in advance without data (i.e. for round 1) is entirely arbitrary.

This is just silly. Either a test is the JREF preliminary or not. If it is not, then it cannot be counted as one retroactively, if it is, then a negative result will count as a failiure. If all applicants were allowed to cherry pick their positive results the prize would have been won long ago.

It's not "retroactive"; again you seem to have significantly misunderstood what I wrote.

Round 1 positive, preliminary -> Round 3 final (skip round 2)
Round 1 negative, prelim -> Round 2 ?, prelim

How is this "retroactive"? (Suggestion: look up the definition before you answer.)

 

[digithead]

I disagree. Fair use is only for limited extracts of a work for reasonable purposes (e.g. education). It just so happens that I have taught a college course myself, and for it had reason to obtain copyright permission - and have previously researched it in some depth. You can't quote an entire work without permission. You can refer people to a public discussion, but the fact that I have posted it publically doesn't mean that you're granted any rights to it other than to read it where I posted it.

Copyright law, btw, is not primarily about "ideas" but about content, i.e. my words as written. Ideas are what patents are for.

Note that fair use has four required tests, one of which is: "amount and substantiality of the portion used in relation to the copyrighted work as a whole". (http://www.copyright.gov/title17/92chap1.html#107) E.g. you can't (without permission) copy a short story that I write and publish online in its entirety, even for a classroom discussion.

I suggest you read the law more before accusing me of not knowing it.


P.S. You evidently haven't read the registration agreement for the JREF forum, which states in part:
Copyright

Any post or article published on the JREF forum by a Member is the copyright of the Member and may not be reproduced, copied or otherwise re-published without the express permission of the Member. By posting on the Forum a Member grants the JREF a non-exclusive licence to publish, republish or reproduce their work, in its entirety or as the JREF sees fit, in perpetuity. The James Randi Educational Foundation is the copyright holder of the JREF Forum.


That means that in addition to violating my copyright you would be violating the registration agreement...

You're also making assumptions that I will be copying your exact words, postings, etc. which is not necessary at all to describe your study...

In fact, I can sum up exactly what I'd ask my class on a test:

"Someone has proposed a study to determine if intercessory prayer creates better outcomes for sick people. They want to randomize about 50 participants into a control (no prayer) and treatment group (prayer) and then measure if a person got better or not. They argue that their simple randomization scheme will be sufficient to overcome possible confounders such as demographics, type of disease, disease severity, etc. Do you agree? Explain your position."

I know I haven't violated my membership agreement. Have I violated your copyright? Not in the least...

And anyone that wants to reproduce my test question, feel free...

So how can you prevent me from using your study as an example?

 

[digithead]

It just so happens that I have taught a college course myself

I sure hope it had nothing to do with research methods or statistics, because you obviously don't understand confounding or heterogeneity that your study fails to account for...

 

[ChaosEngineer]

I think there's a fundamental problem with the nature of the prayers.

Normally, prayers for healing don't have any malicious intent. They're just: "Please let Grandma get better."

But in the case of a prayer study, there's implied malice: "Please heal the strangers in the experimental group while withholding healing from the strangers in the control group." No matter how positive the person praying tries to be, there's always going to be something in the back of their mind that's hoping for negative results in the control group. No good deity would answer such a wicked prayer.

The solution is to not use sick people as test subjects. Use something like coin flips instead.

Here's the appropriate prayer: "Dear God, I've done my budget for the month and I have exactly $X free after I pay household expenses. I'm going to flip a coin 100 times and count the number of times that heads comes up. I solemnly vow to donate that percentage of the money to charity, and to spend the remainder (if any) on liquor and pornography."

If you consistently throw heads 90-100% of the time, then that'll prove that your prayer was effective. That should qualify you for the million dollar challenge, but you might need to pledge the money to charity.

 

[saizai]

You're also making assumptions that I will be copying your exact words, postings, etc. which is not necessary at all to describe your study...

In fact, I can sum up exactly what I'd ask my class on a test:

"Someone has proposed a study to determine if intercessory prayer creates better outcomes for sick people. They want to randomize about 50 participants into a control (no prayer) and treatment group (prayer) and then measure if a person got better or not. They argue that their simple randomization scheme will be sufficient to overcome possible confounders such as demographics, type of disease, disease severity, etc. Do you agree? Explain your position."

I know I haven't violated my membership agreement. Have I violated your copyright? Not in the least...

And anyone that wants to reproduce my test question, feel free...

So how can you prevent me from using your study as an example?

Doing that is not a violation of copyright.

However, don't forget to include the fact that it is a double blinded randomized control-group trial, and that there is a requirement for a certain amount of statistical significance.

And I'd like to see what responses you get.

 

[saizai]

I think there's a fundamental problem with the nature of the prayers.

Normally, prayers for healing don't have any malicious intent. They're just: "Please let Grandma get better."

But in the case of a prayer study, there's implied malice: "Please heal the strangers in the experimental group while withholding healing from the strangers in the control group."

You misunderstand the setup. They are praying for one particular individual. They don't know anything at all about the rest.

Your argument is only valid if you're willing to also argue that every prayer for one person is a prayer against everyone else.

The solution is to not use sick people as test subjects. Use something like coin flips instead.

No thanks; I am interested in whether intercessory prayer works for realworld medical usage in the way I've set up, not in the way you propose. Of course it's a perfectly valid test; it's just not the one I'm doing.

 

[Cuddles]

What does asking them questions have to do with the blinding? You seem to be confused as to what 'blinding' means...

Blinding means you anyone involved in the trial cannot know which group any participants are assigned to. If you determine that the groups are equivalent by asking participants questions you must know which groups htey are in, and therefore you are not blinded, by definition.

Wrong, sorry. They are statistically identical by definition. Perhaps you haven't read as many actual research studies as I (the ones I've seen are mainly in the areas of cognitive science and neurology, fwiw) but there are studies conducted and published routinely with n<50 which nevertheless produce perfectly sound results.

Strange, the words are all English, but somewho it makes no sense. Are you seriously claiming that just because you have a random selection, any groups you choose will turn out to be identical?

I disagree. Please provide a monte carlo sim that does take into account your supposed other factors and explain how they would affect the result even after randomized double-blinding. (Note the and; I want actual numbers rather than just your handwaving about "this will affect it" without anything to back that statement up.)

No, it's your test, you do it. You asked for us to point out any problems, and we have. It's your responsibility to either show that they are not actually problems, or fix them, not ours.

Also, when you say "supposed other factors", are you seriously claiming that you believe there are no other factors involved other than prayer? We have already mentioned numerous confouding factors such as type and severity of disease, wealth, social status, age, sex, race, etc. So far you haven't even tried to address these, and seem to be pretending that they don't exist, or are magicked away by randomising an inadequtely sized sample.

You haven't read the rest of it evidently.

The score equation created is used in the next round. Which has new data collected. Thus for that round the SE was made in advance.

In the post I quoted you claimed that you were creating the equation before collecting the data. Clearly this is not true for all your tests, even if it is for some. Therefore your equation is extremely likely to be picked specifically to give a positive result. This is not good.

What part of "randomized double blind" are you not getting? I'm not "assigning" people to treatment vs control on the basis of their condition.

As I said, the statistical analysis has nothing to do with the randomisation. If you cannot prove that your two groups are essentially similar then your analysis will be meaningless no matter how small your p value.

That's fine. I assert that the test I have proposed is sufficient for me. The end.

I agree, the end. If you refuse to make the test sufficient for anyone apart from yourself you have no chance of ever being tested.

Agreed to the first sentence only.

They can be used to inform later tests - where the same standard of "decide in advance, randomized double blind" is used. And it's the normal practice in fact, whether explicitly labeled as such or not.

Yes, they can be used to inform tests for the new correlations you have found. That is exactly what I said. They can't be used to conduct further tests for the same thing, since if you are looking for something else it is no longer the same test.

I changed my mind on the first two (SE & PC): so long as they are fixed before the start of a given round, there is no reason to make them identical between rounds. Whereas there is reason to change them, to make the test more sensitive and based on actual data gathered last time so that you're not including a bunch of extraneous variables that turn out not to be affected.

The JREF specifically says that the preliminary and final tests have the same protocol. If you change your measure this would not be the case.

I'm not interested in allowing discussion except for the purposes of avoiding a false positive. IMO there is no possible way for this to happen given the constraints I already have placed. Can you think of one?

You may not be interested, but it is likely that the testers will be. They may accept the equation as given, but if you refuse to allow even the possibility of changing it I really doubt you will be tested. Consider that you could come up with an equation that guaranteed a positive result. If they can't alter this then you would win whatever the result. I am not suggesting that you will actually do this, but with a million dollars at stake you can bet that the JREF will not allow this option to be available.

Absolutely yes. See above post.



The copyright is always to the originator of the work unless they have signed a contract that says otherwise (i.e. the registration agreement, quoted above). Per the Berne convention (IIRC) there is no need to explicitly say this; it's automatic.

Fair enough. I know very little about this sort of thing.

Because I don't know in advance how they will get better. Maybe just HRQOL? Maybe a submeasure? Maybe their cancer staging? $ spent in treatment? There are a lot of variables, and fixing which ones you're testing for in advance without data (i.e. for round 1) is entirely arbitrary.

Do you really not see the problem with this? The whole point of a trial is htat you test for one thing and control for all the other things you can think of. As I have said before, and you agreed with, if you look for a change somewhere you will find one. This is the entire reason the measure is always specified in advance.

It's not "retroactive"; again you seem to have significantly misunderstood what I wrote.

Round 1 positive, preliminary -> Round 3 final (skip round 2)
Round 1 negative, prelim -> Round 2 ?, prelim

How is this "retroactive"? (Suggestion: look up the definition before you answer.)

ret·ro·ac·tive [ rèttrō áktiv ]


adjective

Definition:

applying to past: relating or applying to things that have happened in the past as well as the present

You are attempting to apply the status of "JREF preliminary test" to the test which occured in the past. QED. As I said, if everyone was allowed to choose which test was the official preliminary after they knew the result, everyone who took the test would pass.

In addition this would cause severe problems with the rest of your protocol, since you seem so keen on altering your measure of improvement. Since, as I said above, the JREF says that the preliminary and final tests must use the same protocol you would not be allowed to change anything (except the sample of course), and would therefore be left with your "random guess" at what your measure should be.