Check my methodology - prayer study

[Thinktoomuch]

I'm sorry. I can't tell how I'm supposed to take this. Perhaps it's just my pre-coffee state.

Linda

My apologies, I was too curt and possibly ambiguous. Perhaps it is my post-senility state.
I'll try to explain what prompted me to butt in and blurt out as I did. I am getting progressively impatient with trying to scrape little nuggets of wisdom in an ever increasing pile of trivialities. After scanning six pages of running around in circles, I saw that you finally put your foot down in the name of professionality, therefore, I tried to address you from finance professional to academic professional. Gulliver's intervention provided what appeared to be a viable alternative to my hazy plan, but the flak that he is copping now (I had not realised he was a theist, by the way, this is not going to make things easy for him in this context) suggests that this is another dead end. If Saizai wants to go that way and you are happy to vouch for the protocol, however, I am quite happy with it. This will require a lot of running around, though.

So, here is my plan to cut the c*ap.

- You do what you professionally do and give me a quote of what your honorary would be for providing a JREF-proof protocol and an estimate of how much it would cost to conduct the study properly. If you decline to do so, please also refrain from flogging again a dead horse on these pages.

- If your honorary is, as I expect, reasonable (I would like to think that I am generous but not stupid, and rely on the fact that you deemed appropriate to spend a lot of time on this for nothing), I will offer Saizai to pay for it if he commits to conducting the study at the required standard and following your protocol. If he is not prepared to do so, there is no more point for anybody here to continue with the trivialities, as he is obviously not in good faith as regards his presence in this forum.

Again, I do not believe that my bank account is seriously threatened.

 

[GzuzKryzt]

I love it when the plot thickens.

 

[Jekyll]

I suggest that cheaters could create bad data points, assuming that a fair God wouldn't intercede for a cheater, even when receiving prayer support. This problem would reduce the test's ability to detect legitimate effects on those who don't cheat. So cheaters, under this assumption, hurt, not help saizai's case.

I'll keep pondering the point, but I leaning toward no cash incentives.

Better educated now,
Gulliver

Well that's an interesting idea, but even if god did disown cheaters I'm not sure that He would disown people who deliberately did badly in IQ tests so that their church receives additional donations.

Anyway, baring the big G. taking personal offence, cheaters will only serve to increase the noise of the data and not skew it
in any particular direction which should make it harder for S. to win if there is an effect, and not make any difference otherwise. It's just something to keep in mind when you're planning.

 

[fls]

My apologies, I was too curt and possibly ambiguous. Perhaps it is my post-senility state.
I'll try to explain what prompted me to butt in and blurt out as I did. I am getting progressively impatient with trying to scrape little nuggets of wisdom in an ever increasing pile of trivialities. After scanning six pages of running around in circles, I saw that you finally put your foot down in the name of professionality, therefore, I tried to address you from finance professional to academic professional. Gulliver's intervention provided what appeared to be a viable alternative to my hazy plan, but the flak that he is copping now (I had not realised he was a theist, by the way, this is not going to make things easy for him in this context) suggests that this is another dead end. If Saizai wants to go that way and you are happy to vouch for the protocol, however, I am quite happy with it. This will require a lot of running around, though.

Gulliver's protocol is also inadequate.

So, here is my plan to cut the c*ap.

- You do what you professionally do and give me a quote of what your honorary would be for providing a JREF-proof protocol and an estimate of how much it would cost to conduct the study properly.

I have been willing to volunteer my services, as do other professionals who participate in this forum and the JREF. Saizai has made it clear that he is not interested in the study that I would recommend, making issues of payment irrelevant.

If you decline to do so, please also refrain from flogging again a dead horse on these pages.

When dictating to me what behaviour of mine is unacceptable, it would help if you were specific. At this point, all I can tell is that I am not allowed to provide criticism. Specific standards will allow me to assess the feasibility of implementation. Alternate means of assisting you to deal with your disturbance over my posts may be recommended.

- If your honorary is, as I expect, reasonable (I would like to think that I am generous but not stupid, and rely on the fact that you deemed appropriate to spend a lot of time on this for nothing), I will offer Saizai to pay for it if he commits to conducting the study at the required standard and following your protocol. If he is not prepared to do so, there is no more point for anybody here to continue with the trivialities, as he is obviously not in good faith as regards his presence in this forum.

Again, I do not believe that my bank account is seriously threatened.

I think you've made your point.

Linda

 

[Thinktoomuch]

Quote:
If you decline to do so, please also refrain from flogging again a dead horse on these pages.

When dictating to me what behaviour of mine is unacceptable, it would help if you were specific. At this point, all I can tell is that I am not allowed to provide criticism. Specific standards will allow me to assess the feasibility of implementation. Alternate means of assisting you to deal with your disturbance over my posts may be recommended.

By interpreting my pleading as "dictating" to you and an indication of my "disturbance" over your posts you suggest a personal antagonism that was not intended. I leave to the other readers to see for themselves in what proportion this misunderstanding is due to my expression or to your excessive sensitivity. My apologies regardless. You became the subject of my request because of your claim that writing protocols is what you do for a living. It would have been directed to any other who had said so.

It was obvious to me that if you did not want to recommend a protocol it would be because you are convinced Saizai "is not interested in the study that you would recommend", therefore, my conclusion was that there would be no point in continuing with useless criticism, hence the vernacular flogging of the dead horse. If you disagree, I would be interested in your reasoning: I see it only as demeaning.

In any case, I am sincerely grateful for your response. Just by the act of responding you have forced Saizai to either contradict you and show interest in a scientifically valid protocol if it is provided to him at no cost, or stop lying about being in this forum to learn, in which case I would expect all sensible people to also stop giving useless advice. Either way my objective of attempting to raise a little the usefulness of the forum has been achieved. Whether my attempt is actually effective and the desired result is actually achieved, time will tell. As you say, I have made my point.

 

[fls]

If you decline to do so, please also refrain from flogging again a dead horse on these pages.

When dictating to me what behaviour of mine is unacceptable, it would help if you were specific. At this point, all I can tell is that I am not allowed to provide criticism. Specific standards will allow me to assess the feasibility of implementation. Alternate means of assisting you to deal with your disturbance over my posts may be recommended.

By interpreting my pleading as "dictating" to you and an indication of my "disturbance" over your posts you suggest a personal antagonism that was not intended. I leave to the other readers to see for themselves in what proportion this misunderstanding is due to my expression or to your excessive sensitivity.

Excessive sensitivity?!!! Let's look at this objectively. If we define the "dead horse" in this case as "driving home the point that posters are not seriously considering what others are saying" (specifically of the form "Saizai you are not listening/sincere" and "[everyone else] is not offering useful suggestions" (from Saizai)), then it really got going on page 6. Since then there have been 21 posts that could reasonably be characterized as "dead horse beating". Out of that total, 3 posters have contributed 2 DHBP (Dead Horse Beating Posts), 2 have contributed 3 DHBP, 1 contributed 4, and 1 contributed 5, making the average contribution 3 DHBP. I contributed 2 DHBP, which is not only out of the top three, it's also below average. Therefore, for you to single me out as the only recipient of your chastizement demonstrates a directed attack that was quite undeserved.



Seriously though, even though I know better, it's hard for me to resist taking advantage when the way something is expressed allows for two different interpretations. I think it's out of my system now.

My apologies regardless. You became the subject of my request because of your claim that writing protocols is what you do for a living. It would have been directed to any other who had said so.

It was obvious to me that if you did not want to recommend a protocol it would be because you are convinced Saizai "is not interested in the study that you would recommend", therefore, my conclusion was that there would be no point in continuing with useless criticism, hence the vernacular flogging of the dead horse. If you disagree, I would be interested in your reasoning: I see it only as demeaning.

I agree, but I think you also are beating the dead horse, even though you are presenting it as an attempt to move the discussion forward. You got your dig in by stating that you didn't think Saizai would accept your offer. And I suspect that you didn't really think I would take you up on your offer, either.

In any case, I am sincerely grateful for your response. Just by the act of responding you have forced Saizai to either contradict you and show interest in a scientifically valid protocol if it is provided to him at no cost, or stop lying about being in this forum to learn, in which case I would expect all sensible people to also stop giving useless advice. Either way my objective of attempting to raise a little the usefulness of the forum has been achieved. Whether my attempt is actually effective and the desired result is actually achieved, time will tell. As you say, I have made my point.

Maybe there is something to salvage. It is often hard to tell just exactly when someone has made it clear that they are insincere. Most of the time I think that it is best to give people the benefit of the doubt. It would probably lose a cost-benefit analysis, though.

Linda

 

[Gulliver]

Theism and Impact

Gulliver's intervention provided what appeared to be a viable alternative to my hazy plan, but the flak that he is copping now (I had not realised he was a theist, by the way, this is not going to make things easy for him in this context) suggests that this is another dead end.

While I thank you for your concern, I ask that you not allow my theism to worry you. I have no difficulty compartmentalizing (is that a word?) my non-scientific beliefs from the real world. I do, however, make every effort to disclose fully my weaknesses but other rely on me.

To clarify, I am not religious, do not attend any religious gatherings, do not pray, and never expect any intervention for any supernatural beings. I just conclude that there is not that is truth than we can prove from the observations of nature. I believe, but do not know, that this includes the existence of another intelligence.

I commit to you to behave with the utmost integrity in all scientific or rational efforts.

Earnestly,
Gulliver

 

[Gulliver]

Gulliver Proposal Update

Gulliver's protocol is also inadequate.
...
Linda

I, of course, appreciate your taking time to review my protocol.

Not wasting any time, I had asked Friday for independent reviews by a full professor, Ph. D., M.D. of neuroscience and an associate professor, M. D. of family medicine (to complement my Ph. D. in operations research). I'm happy to report that both were excited about the test. The neuroscientist is eager to modify the Boston test to this protocol. Both gave their preliminary approval on the design, calling it ingenious. (*blush*)

(I'm having a tough time convincing them that we have to wait on saizai. I suspect that we're going to run this protocol without him unless he gets moving. The family doctor sits on a subcommitte of the University's Human Subject's Review Board and has already sent me the release text (for the website to display during registration) they'd require in order to get the University's support.)

I believe that this won't be the first time that an applicant has seen someone else take over his idea.

(Oh and just to clarify, we all three believe that odds that the test will show a positive is about the same as p.)

Eagerly,
Gulliver

 

[fls]

I, of course, appreciate your taking time to review my protocol.

Not wasting any time, I had asked Friday for independent reviews by a full professor, Ph. D., M.D. of neuroscience and an associate professor, M. D. of family medicine (to complement my Ph. D. in operations research). I'm happy to report that both were excited about the test. The neuroscientist is eager to modify the Boston test to this protocol. Both gave their preliminary approval on the design, calling it ingenious. (*blush*)

The concerns I had with your protocol included the use of outcome tests of unknown reliability, validity or relevance, the possibility of a carryover effect (which you will likely be unable to exclude) biasing your results because of the crossover design, a highly selected population making generalizing or drawing conclusions difficult, and the high probability that the study will contribute nothing to the advancement of knowledge in this area. It would be inadequate to lay the question to rest if negative. And if positive it suffers from the same issue as Saizai's study - false positives are far more likely than true positives, making it difficult to know whether there is anything worth pursuing further. And without measuring relevant outcomes, it would be hard to know what a true-positive even means (e.g. if it's even a good thing).

It's not my time and money (at least, I hope it's not my tax dollars ), though. And I'm starting to feel very much like a third-wheel on this thread since this is clearly up to you and your colleagues now.

I'm having a tough time convincing them that we have to wait on saizai. I suspect that we're going to run this protocol without him unless he gets moving. The family doctor sits on a subcommitte of the University's Human Subject's Review Board and has already sent me the release text (for the website to display during registration) they'd require in order to get the University's support.)

I believe that this won't be the first time that an applicant has seen someone else take over his idea.

(Oh and just to clarify, we all three believe that odds that the test will show a positive is about the same as p.)

Eagerly,
Gulliver

Good luck.

Linda

 

[Gulliver]

Answering Concerns

fls,

I can't help but notice that you're using the past tense. I'll assume that you meant to use the past perfect (happening now and in the past.)

You'd didn't ask me to reply regarding your concerns. I'm not sure that we can repair our relationship enough to make a discourse here useful, but I'm willing to try. I've dealt with many experts over my tenure in the academia who seem to own only a sniper's rifle. I hope that you're willing to use another tool of a more cooperative nature. Please consider, for example, your original unexplained "is inadequate" sniping.

Allow me to respond neutrally to your most recent comments.

The concerns I had with your protocol included:

the use of outcome tests of unknown reliability, validity or relevance,
I haven't specified the outcome tests, so I believe that you attack a straw man. I suggest we can very much rely about the Boston short-term memory recall test that is well-documented and widely accepted even in a web-based environment. If you attack the Boston test in this protocol, I ask respectfully that you provide your reasons.

the possibility of a carryover effect (which you will likely be unable to exclude) biasing your results because of the crossover design,

I don't understand this comment. Please tell me your reasoning that the carryover effect can't be blocked by blocking its variable in the ANOVA (or, if we chose, by a linear regression).

a highly selected population making generalizing or drawing conclusions difficult, and

Here you're correct. We could only conclude that the results apply to churchgoers and that only if assume that prayer effects individuals near me the same as those far from me. I suggest that many accepted and useful studies ,including even the Harvard study referenced earlier in the thread (only bypass patients in several hospitals), have a more selective sample population.

the high probability that the study will contribute nothing to the advancement of knowledge in this area.

I believe this is a conclusion that you've not adequately supported.

It would be inadequate to lay the question to rest if negative.

I'm sure that it cannot lay all questions about the effect of prayer to rest. But I'm sure you're aware that science progresses in small steps. This study if negative would add to the body of evidence. If positive at very small p value, we could only conclude the need for further review of the question.

And if positive it suffers from the same issue as Saizai's study - false positives are far more likely than true positives, making it difficult to know whether there is anything worth pursuing further.

You fail to provide any evidence or even reasoning to back up your claim, "false positive are far more likely than true positive". I kindly ask you to provide such reasoning or some evidence.

And without measuring relevant outcomes, it would be hard to know what a true-positive even means (e.g. if it's even a good thing).

I disagree, of course, with your comments about the relevance, as I've stated above. I don't believe that we need to concern ourselves whether improvement in short-term recall is "a good thing". The study can clearly be neutral on this.

It's not my time and money (at least, I hope it's not my tax dollars ), though.

Since you included the emoticon, I infer that you're not serious on this issue.

And I'm starting to feel very much like a third-wheel on this thread since this is clearly up to you and your colleagues now.

I must respectfully disagree. It is not clear to me at all. I remain, for one, at saizai's pleasure in pursuing this study. For two, I have been quite clear, and not just in this thread, that I value constructive criticism from JREF Forum members.

Good luck.

Linda

May I also solicit from any Forum member their thoughts on the Gulliver protocol, especially as it relates for fls's concerns?

Gratefully,
Gulliver

 

[Thinktoomuch]

While I thank you for your concern, I ask that you not allow my theism to worry you.

No worry at all, as I already said to Linda. Anybody who commits "to behave with the utmost integrity" has my full support. And my money is after tax disposable income .

 

[Cuddles]

I must respectfully disagree. Intelligence tests are well established as not varying under many conditions. Caffeine intake is probably the most notorious, but please reference http://www.springerlink.com/content/t25366v1554q33m0/ for an interesting summary of a recent article in the peer-reviewed _Cellular and Molecular Life Sciences (CMLS)_. Even given your claim, won't the A-B nature of the experimental design eliminate any influence? Let's say 10 out of 100 subjects cheat to improve their performance, by whatever means. Since they don't know in which trial they're receiving prayer support, they can't bias the result.

Ah, sorry, it seems I missed the bit about swapping the groups over. I agree that that should eliminate any bias between the two groups. However, I am still concerned about the sample size. Do you have a minimum number you would consider acceptable? With a very small sample, such as the 50 that Saizai proposes, if just a few people perform differently in the A and B rounds there could be a significant difference between the two groups. Sample size is of course one of the biggest concerns in health studies, but as far as I can see neither you nor Saizai have adequately addressed it. You say that you will visit churches handing out fliers, but what will you do if not enough people register? Or if too many drop out during the test?

Now, Cuddles, I really think you need to sit in a cozy armchair and ponder your statements above. You're much too smart to make this claim. We aren't interested in how it works. We are concerned that after eliminating other factors can we show with statistical accuracy that a paranormal force effected the outcome under controlled situations. If so then the claimant deserves our further consideration. I would, of course, defer to JREF for the next test if this one shows a positive result. I would not claim to have proven anything.

If you insist on maintaining your position, would you please provide a quote from the JREF FAQs or instructions that eliminates "medical studies" as "just not challenges"?

Unfortunately the FAQs are all gone at the moment, so I can't find an exact quote. I believe it says something along the lines of "Results will be immediately obvious to any observer". The point being that either you have an ability or not. If you can levitate, anyone watching can tell that either you are floating off the ground or not. If you say you can find water, then anyone can see that finding even just 19/20 hidden containers is something very special, regardless of how or why you claim you do it. Studying hundreds of people for a couple of years and then performing a detailed statistical analysis to find any small difference between two groups is just not the same. I don't know if there is anything in the challenge rules that forbids it, and in fact I suspect there is not, but it is certainly very much against the spirit of the JREF and its challengs and I very much doubt they will ever accept a test along these lines.

As I, and others, have said before, the JREF is not a research funding organistion. The challenge is there mainly as a PR tool to expose frauds and also to find evidence for paranormal abilities if any actually exist. If you want to conduct lengthy research into healing effects, this is not the place to do so. In fact, you mention that you have talked to a professor and may do the test regardless of what Saizai does. This is exactly what should be done. This sort of research belongs in universities and possibly hospitals, not in paranormal challenges.

On the issue of financial rewards, this simply cannot be acceptable. It is standard procedure to pay everyone involved in a test a small amount (in university this was normally a beer or some sweets ), but you cannot reward people based on the outcome of the test. Paying people who give you the result you are looking for is a very clear conflict of interest. Regardless of whether you believe they couldn't influence the result, such a test could never be taken seriously and would certainly never be published.

Finally, I believe what Linda is saying is that neither you or Saizai have said how you will actually measure any positive effect, or any effect at all. In order for anyone to know if the results are valid, we have to know exactly what you plan on measuring, how you plan on measuring it and how you will analyse the results. The problem is not that the test is not valid, it is that we don't know what the test is, so we cannot tell if it is valid or not.

hope that we can disagree here and maintain our professional friendship. You have my respect, and I ask your indulgence to allow me to disagree so tersely.

With real gratitude,
Gulliver

No problem. Life would be so boring if everyone agreed all the time.

 

[fls]

fls,

I can't help but notice that you're using the past tense. I'll assume that you meant to use the past perfect (happening now and in the past.)

What happened in the past was the idea that it would be useful to involve myself. (make that a rueful smile)

You'd didn't ask me to reply regarding your concerns.

No. I felt obliged to state them since you picked up on my comment to Thinktoomuch. But I left it up to you to decide whether you were interested in looking at any of it. It is really the people you are working with (and yourself) that you need to satisfy, not me.

I'm not sure that we can repair our relationship enough to make a discourse here useful, but I'm willing to try. I've dealt with many experts over my tenure in the academia who seem to own only a sniper's rifle. I hope that you're willing to use another tool of a more cooperative nature. Please consider, for example, your original unexplained "is inadequate" sniping.

I'm sorry that I've alienated you. My comment was thoughtless. I did not realize that you were seriously considering implementing your protocol. I thought you were just treating this as an interesting exercise, since it seemed pretty obvious (to me) Saizai was not going to go along.

I haven't specified the outcome tests, so I believe that you attack a straw man. I suggest we can very much rely about the Boston short-term memory recall test that is well-documented and widely accepted even in a web-based environment. If you attack the Boston test in this protocol, I ask respectfully that you provide your reasons.

I thought you were looking at whether prayer influences health and that you were suggesting measuring blood pressure and mental acuity. I did not realize that you had decided to test the effect of prayer on normal short-term memory.

I don't understand this comment. Please tell me your reasoning that the carryover effect can't be blocked by blocking its variable in the ANOVA (or, if we chose, by a linear regression).

It can be blocked (if detected), which effectively negates the crossover part of the trial. And usually crossover trials do not have enough power (the reason for using the design in the first place) to adequately exclude the effect making the "if detected" part unreliable.

Here you're correct. We could only conclude that the results apply to churchgoers and that only if assume that prayer effects individuals near me the same as those far from me. I suggest that many accepted and useful studies ,including even the Harvard study referenced earlier in the thread (only bypass patients in several hospitals), have a more selective sample population.

I was thinking also of the generalizability of negative results.

I'm not sure how you are using the terms "selection", "sample" and "population", though.

I believe this is a conclusion that you've not adequately supported.

The statements that followed were meant to support that conclusion (that may not have been clear).

I'm sure that it cannot lay all questions about the effect of prayer to rest. But I'm sure you're aware that science progresses in small steps. This study if negative would add to the body of evidence. If positive at very small p value, we could only conclude the need for further review of the question.

We already have a lot of research on the effects of prayer. Those studies already effectively include the small step you are thinking of adding. If anyone is not yet convinced that prayer is ineffective, then the addition of a small study to the overall power is unlikely to be convincing. And as I mentioned to Saizai, a positive result (for the reasons given below) will serve to provide an out for those who wish to suggest there may be something to study (justifying further expenditure of time and money) without providing useful guidance as to what that might be.

You fail to provide any evidence or even reasoning to back up your claim, "false positive are far more likely than true positive". I kindly ask you to provide such reasoning or some evidence.

I brought up this issue in this post. This is the relevant paper that explains this issue in greater detail.

I disagree, of course, with your comments about the relevance, as I've stated above. I don't believe that we need to concern ourselves whether improvement in short-term recall is "a good thing". The study can clearly be neutral on this.

These are the kinds of questions I would ask when assessing relevance. Why would change in short-term memory in someone without impairment be a relevant outcome (i.e. why would you hypothesize this is something that would be affected by prayer)? Is this test capable of registering improvement in the absence of impairment? Does it provide any ability to discriminate in a normal population?

Since you included the emoticon, I infer that you're not serious on this issue.

It hardly seemed fair to single you out on this issue.

I must respectfully disagree. It is not clear to me at all. I remain, for one, at saizai's pleasure in pursuing this study. For two, I have been quite clear, and not just in this thread, that I value constructive criticism from JREF Forum members.

May I also solicit from any Forum member their thoughts on the Gulliver protocol, especially as it relates for fls's concerns?

Gratefully,
Gulliver

Ah, I thought you were talking about the possibility of proceeding without him.

Linda

 

[Gulliver]

Gulliver Proposal Update #2

All,

I'm most grateful for your support here. I see three wonderfully insightful and kind emails in a row. I don't have the words to express how precious I find your feedback and encouragement. Please consider us all friends working to educate others, and each other.

For now I beg your indulgence. I've received enough feedback to generate many days of work. I liked how the family doctor responded with a long silence to Cuddles's and fls's comments and then stated she had some thinking to do. Please consider the Gulliver Proposal "up on blocks" while we see whether we can resolve the issues you've raised. I may need longer than some of you may consider ordinate. My cancer is confining to bed rest most days now, so please allow some extra time. I promise to report back on each item as soon as we have progress to report.

On a side (and somewhat selfish note), has anyone else noticed how quiet saizai has been of late?

Pondering and Planning,
Gulliver