Constructing a new age "energy" test, suggestions and help appreciated.

How did you calculate your figure LL?


Well, um, I took 2 and raised it to the eighth power and then multiplied that by itself.

I suck at math and am probably wrong but 1 in 300 seems terribly low to me. There are two choices. If 1000 people take the first trial, only 500 are going to be right. Of those, only 250 will be right for the second trial, 125 in the third, 62.5 in the fourth, 31.25 in the fifth, 15.625 in the sixth, 7.8 in the seventh, and 3.9 in the eighth.

Then, you take that 0.39% of the first thousand people and run them through the test all over again. Only 0.39% of them are going to pass. That means that only 0.0015% of the first group of 1,000 is going to pass the second series. That's 15 people out of 1,000,000 or about 1 person in 65,746.

I freely admit that I may have figured this all wrong.

There
 
JREF accepts the odds of 1 : 1000 with Pavel in his pre-test, I guess we should aim for the same.
 
Well, um, I took 2 and raised it to the eighth power and then multiplied that by itself.

Ah, I'm afraid you've oversimplified a bit.

I suck at math and am probably wrong but 1 in 300 seems terribly low to me. There are two choices. If 1000 people take the first trial, only 500 are going to be right. Of those, only 250 will be right for the second trial, 125 in the third, 62.5 in the fourth, 31.25 in the fifth, 15.625 in the sixth, 7.8 in the seventh, and 3.9 in the eighth.

Then, you take that 0.39% of the first thousand people and run them through the test all over again. Only 0.39% of them are going to pass. That means that only 0.0015% of the first group of 1,000 is going to pass the second series. That's 15 people out of 1,000,000 or about 1 person in 65,746.

The trouble with your analogy is that the 500 people who fail your first trial are eliminated from all subsequent trials. Compare this to one person tossing a coin 10 times. By your analogy if they fail the first toss they fail the whole test but they might fail the first toss and then get all of the subsequent tosses correct.

What you have calculated are the odds of correctly predicting 8 coin tosses in a row twice in two trials each with only 8 tosses. There is only one way to achieve that and that is to be correct every time, hence the very low odds.

What i was calculating were the odds of correctly guessing at least 8 out of 10 tosses twice. There are lots of ways to do this. The binomial formula tells us how many ways there are to get k guesses right out of n. There is only 1 way to get 10 out of 10 but there are 10 ways to get 9 out of 10 and 45 ways to get 8 out of 10.

For example,

MMHHHHHHHH
MHMHHHHHHH
MHHMHHHHHH

etc, where M represents a Miss or incorrect guess and H represents a Hit or correct guess and we consider the odds of each to be 0.5. So, there are 56 ways to score at least 8 out of 10 by chance . There are 2^10 = 1024 possible outcomes for 10 coin tosses so the odds of guessing at least 8 out of 10 correctly in one trial are 56/1024 = 0.0547 or about 1 in 18.

The odds of achieving this result twice in a row are 0.0547^2 = 0.00299 or about one in 333.

I freely admit that I may have figured this all wrong.

And that is what separates you from the woo contingent ;)

I, too, am open to the possibility of my own calculation being incorrect as it has been a fair while since my last course in statistics.
 
Last edited:
JREF accepts the odds of 1 : 1000 with Pavel in his pre-test, I guess we should aim for the same.

The trouble is, with your test as it is currently described you will be setting a pass mark as either 8 out of 10 or 9 out of 10.

The odds of guessing at least 8 out of 10 in both trials are about 1 in 300
The odds of guessing at least 9 out of 10 in both trails are about 1 in 9000

I'd suggest going with the 1 in 300 odds as that would be good enough for an unofficial test and would leave less room for claims of stress to perform on the part of the subject.

In the end it's up to you though.
 
I'd recommend not complicating the evaluation with two trials, which could have conflicting outcomes and then it'd be grounds to say that the results were ambiguous.

A run of 10 should be sufficient. If you get a positive result, you can talk about doing another run later as a replication with other witnesses.

OR... use a run of 20. 14/20 is a p value below .05, which is a common threshold for social sciences, and 16/20 is a p value below .01, which is more strict threshold in the social sciences and an entry-level threshold in medical research.

If they're interested in demonstrating something that is allegedly a natural phenomenon more like a physical force, it should be in an experiment with p value below 10^-6. They'd need to obtain at least 19/20.
 
What you have calculated are the odds of correctly predicting 8 coin tosses in a row twice in two trials each with only 8 tosses.


I actually considered that problem and then decided that I didn't have the ability to figure out what to do about the two extra trials. So, I ignored them. Thanks for the great explanation.
 
I'd recommend not complicating the evaluation with two trials, which could have conflicting outcomes and then it'd be grounds to say that the results were ambiguous.

A run of 10 should be sufficient. If you get a positive result, you can talk about doing another run later as a replication with other witnesses.

OR... use a run of 20. 14/20 is a p value below .05, which is a common threshold for social sciences, and 16/20 is a p value below .01, which is more strict threshold in the social sciences and an entry-level threshold in medical research.

If they're interested in demonstrating something that is allegedly a natural phenomenon more like a physical force, it should be in an experiment with p value below 10^-6. They'd need to obtain at least 19/20.


Thanks for your contribution blutoski.

Here are the current suggestions:

A: 1 trial with 10 runs.
B: 1 trial with 20 runs.
C: 2 separate trials with 10 runs each.

THE CLAIMANT feels and believes that she's dealing with a natural phenomenon, in practise, a stream of "energy" that can heal people. I think we should go through extra effort to make sure that she is comfortable with all of the test conditions, before and after each trial. The open test plays a big part in this. How big? Who knows. I guess it depends on how much she is willing or able to fool herself IF the results are not good enough for a pass.

A: this would be the simplest and fastest, what would you suggest for a pass to replication? I know she claimed a 100% success rate, but I'd also like to give her some room for error.

Replication: 9 or more hits out of 10.

B: for me, the biggest question mark here is the length, we have to reserve 2 minutes for every healing session, that's 40 minutes already. Then the communication and transition. That would be at least another 40 minutes. 1 hour 20 minutes is a long time wearing all the equipment and keeping the "channels open". Afterwards she could try to convince herself that she had to regulate her "sensitivity" or something like that. One of our main objectives is to give her a good and fair chance to prove her abilities to herself.

Pass: 16 or more hits out of 20.

C: the way I see it, the benefit of this over option B is that she can rest in between the trials and we can use the open test both times to make sure she is feeling good about everything.

Pass: 8 or more hits out of 10 in both trials.


NOTE: Tapio just informed me that THE HEALER might need more time to "get in the mood" than 2 minutes. Well, well, our mistake for accepting THE CLAIMANTs word for that. We should have our second meeting thursday or friday. I will bring all these suggestions to the table. Thanks everyone, you've been very helpful.
 
Last edited:
STATUS UPDATE:

THE HEALER does not want to go through with the test. No specific reason was offered during the phone call, but THE CLAIMANT speculated that it's because THE HEALER is a busy person. She is now looking for another healer to help us proceed with the test.

We agreed that we are going to use the option A or the option C, it was agreed that the option B is too long for a test of this nature. We are leaning towards option A, and require 9 hits out of 10 to proceed to the second test. If she gets 8 or less hits out of 10, it is agreed that THE CLAIMANT does not have this ability and we will proceed to testing her other abilities, which include recognizing the "energy" of a (presumably) healing stone.
 
which include recognizing the "energy" of a (presumably) healing stone.


Not surprised about the healer.

That stone thing looks pretty easy to test, assuming your friend can differentiate whether it's under a coffee can or in the next room.
 
I'd suggest 8 out of 10 being the pass mark for replication.

I can't help but wonder if the subject, on scoring 8 out of 10 and being failed, would continue to believe that she really has the power she claims. By having a follow up test you can demonstrate either consistency or fluke.

One other thought occurs. Do not tell the subject how she has done until the end of the trial. If possible, have the healer leave the room when the subject indicates whether she thought she felt the healer or not. If you do this you may wish to set up a video camera to record the subject's answer so that fair play is insured.

My reasoning behind the recommendation above is as follows. If either the subject or the healer sees that the trial is not going well whey may wish to bring it to a close, or, if they continue, they may cite the "bad vibes" of the fist wrong guess or two as the reason for a subsequent poor performance.

The danger here is that they may wish to try the test again at a later date and, if they consistently end the trails which are not going well, then their list of complete trials will be biased towards occasions when they guess well by chance. This concern perhaps goes a bit beyond what you are hoping to achieve with your test but it is sound methodology nonetheless.

I'm sorry to hear that your first healer fell through. Better luck with the next one.
 
Not surprised about the healer.

That stone thing looks pretty easy to test, assuming your friend can differentiate whether it's under a coffee can or in the next room.


Yes, it should be way simpler to test. I'll get back with that later :)
 
One other thought occurs. Do not tell the subject how she has done until the end of the trial. If possible, have the healer leave the room when the subject indicates whether she thought she felt the healer or not.


Yes, this is what we've had in mind from the beginning, and just for the reasons you mentioned :) She signals us when she is ready to make her decision - we call the healer out - she signals us whether she felt the "energies" or not. We also decided that we will not use any vibrating phones to signal her, because they might emit "energies" that disturb the process. Instead we could use a long string tied to her toe and pull that whenever the healer is in place and ready to perform.

And thanks, what was I thinking? If we were happy with the option C, 8 out of 10 should be enough to pass for the replication, of course. I think Tapio even touched on this point during our meeting but somehow we didn't think it through at the time, I guess it was because our time was ending and we still needed to contact THE HEALER for the latest info.
 
Status update:

The original HEALER is back in business. We are aiming to perform the test this friday. Nice :)
 
Excellent news.

I wish you the best of luck in running your experiment and look forward to reading your results and conclusions. I'd also be very keen to hear about your subject's conclusions.

'beeees
 
From what you write I doubt she would be cheating - but there are still ways to signal who is who.

e.g. if I wanted to cheat, I would put my hands on the table next to the claimant - if I did that just about hard enough she claimant could feel it, and know it was me. With a bit of training I could ensure that the hand left to hear head hits the table (bed, or whatever) slightly before the one to the right, making the sign even more secure.

That might not even be visible on video, and you would have to look for it specifically even if it was.
 
I don't know if this is do-able (hey, I'm just the ideas guy), but is it possible to make the test more "double blind". Would it be possible to have two claimants (i.e. the claimant and a placebo claimant) in addition to the healer and placebo healer? One healer would work with one claimant at the same time as the other healer works with the second claimant and neither healer knows whether they are working on the "real" claimant or the placebo claimant. I know this adds more complexity to the test and I don't know how the healers would be shielded from knowing which claimant they are working with, but I believe this would go a long way to remove any possible communication between the (real) healer and claimant.
 
From what you write I doubt she would be cheating - but there are still ways to signal who is who.

Agreed. Collusion is the biggest risk at this point, especially since you are dealing with a single "known" healer. Since it seems that the claimant didn't want to do it without that one person, it gives me pause. There's also the fact that she insists on having the hands near the only part of the body where all five senses are located. I wouldn't *expect* cheating, but I'd at least plan for it.

Just off the top of my head I can think of a couple of ways to cheat. The easiest one to conceal is breathing through the nostrils in such a way as to move air on to her face in a pattern (one long, two short for example). Most in-camera microphones would not pick it up over background noise. It would barely be perceptible visually.

So, perhaps one additional idea would be to have the claimant lay face down on a massage table (the ones with the gap for your nose). A cardboard box cut properly could cover her head thus providing making other senses a bit more difficult to use.

As for the timing issues, I'd be happy to splice together 20 minutes worth of music that announces (over white noise, of course) when it's time to switch to a new person. Stick it on an MP3 player and Bob's your uncle.

Also, I might have missed this, but how is she signaling which is which? We can't expect her to remember all 10, can we?
 
Hi everyone. Thanks for your ideas.

I've known the CLAIMANT personally for years and trust her. She's not the kind of person that would even think about cheating, and she would've gone through the test with somebody else (as long it's a person she knows to have "strong energy").

This, of course, doesn't mean we won't be taking the possibility of cheating seriously. The HEALER will probably wear some kind of mask preventing him from giving signals by breathing forcefully. We also thought of hanging delicate feathers right above the CLAIMANTS face to reveal this.

Heh, we'll definetely have to take a picture of the set up when all is in place. I bet it's going to look freaky enough :covereyes.

Also, I might have missed this, but how is she signaling which is which? We can't expect her to remember all 10, can we?

I think KUKO mentioned something about this. The test will be held in an apartment with three rooms. One for the actual process, one for monitoring (a door with a window to see through to the test room) and one for the HEALER and PLACEBO HEALER to wait in (no windows to see through). When the decided time for each run is up the CLAIMANT will be notified by pulling a string attached to her finger or toe. At this point she's not to give any kind of signal. A second pull of the string is applied AFTER the HEALER/PLACEBO HEALER has left the room and gone back to wait. Then the CLAIMANT will give a signal by lifting her left/right hand. If she comes to a conclusion on who's "healing" her before the time's up, she will interrupt the run by first raising a fist. Then, after the HEALER/PLACEBO HEALER has left the room she'll be informed by pulling the string to signal the OBSERVER (me) on who she thinks it was.

Due to several reasons (concerning Kuko and myself) we've had to postpone the test until next week. But not to worry...it WILL happen!
 
From what you write I doubt she would be cheating - but there are still ways to signal who is who.

e.g. if I wanted to cheat, I would put my hands on the table next to the claimant - if I did that just about hard enough she claimant could feel it, and know it was me. With a bit of training I could ensure that the hand left to hear head hits the table (bed, or whatever) slightly before the one to the right, making the sign even more secure.

That might not even be visible on video, and you would have to look for it specifically even if it was.


We don't think THE CLAIMANT is going to cheat, but we will do our best to eliminate any kind of cheating (conscious or unconscious) nevertheless. It was agreed from the start, that the HEALER and the PLACEBO HEALER are not allowed to touch anything, or even move their hands while in the experiment room. There will be markers in the bed (as well as in the waiting room for the healers to prepare their hands in the right position before entering the experiment room), both healers will have to keep their hands within the limits of the markers at all times when inside the experiment room. We will practise enough to make sure that both healers know how to act inside the experiment room. If THE OBSERVER spots something outside of the agreed protocol, he will interfere.

UncaYimmy said:
Since it seems that the claimant didn't want to do it without that one person, it gives me pause. There's also the fact that she insists on having the hands near the only part of the body where all five senses are located. I wouldn't *expect* cheating, but I'd at least plan for it.


She was going to look for another healer, but somehow the original healer changed his mind and agreed to participate. It was actually our suggestion that she would try to get the best healer she possibly could, and also one that she was already familiar with in practise. In case of random results, this would at least make it more difficult for her to explain that we should've used a better healer. The head is supposedly the most sensitive part regarding the healing "energies", again, we thought this would be the ideal starting point for the test.

I'm quite confident that we can block cheating from this test. If the results are positive, the next step will be with the Finnish Skeptics who are offering a handsome prize for something like this, I'm sure they will take the precautions even further than us. This is still only a home test, although we will try our best to take everything we can into consideration, otherwise it would not serve the purpose.

Coveredinbeeees said:
I wish you the best of luck in running your experiment and look forward to reading your results and conclusions. I'd also be very keen to hear about your subject's conclusions.


Thanks CB, you've been a HUGE help in designing this test. Much appreciated. We will report fully after the test :)

Unfortunately, as Tapio mentioned, I couldn't arrange anyone to replace me @ work this friday, so we had to postpone the test to the near future. 1 - 1. The positive thing is that the original HEALER is back in business :)
 

ISF - Join now!

Every member here is approved by hand. No bots, no spam, just people who care about evidence and honest debate.

Membership is free!

Create your free account

Back
Top Bottom