Meta-analysis of JREF tests

[Paul C. Anagnostopoulos]

T'ai, I think evidence determines theories, too. But not "evidence" produced by meta-analyses of experiments that had no theories to begin with.

If people think there is enough evidence for psi or dowsing or whatever, then it's time they come up with some theories, devise experiments with hypotheses based on those theories, and go for it.

~~Paul

 

[tedly]

I think this quote is germane.

"Parapsychology is motivated by belief in search of data, rather than data in search of explanation, says James Alcock, in an article (PDF) in the Journal of Consciousness Studies ..."

It's from
SciTechDaily
about halfway down the right side of the page. I'd supply the direct link but I'm trying to plug one of my favourite web pages.

 

[T'ai Chi]

T'ai, I think evidence determines theories, too. But not "evidence" produced by meta-analyses of experiments that had no theories to begin with.

I'm not trying to employ meta analysis in order to come up with a theory. I'm trying to use meta analysis in order to accumulate experimental and correlational results across independent studies.

 

[billydkid]

Data?

I'm not trying to employ meta analysis in order to come up with a theory. I'm trying to use meta analysis in order to accumulate experimental and correlational results across independent studies. [/B]

There is no data. Nitwits come claiming to be able to dowse or remote view or perform telekinesis and can't - how does that constitute data?

 

[T'ai Chi]

Theories don't determine evidence. It is the other way around. Although, there is obviously cooperation between evidence and theories.

If there is no or incomplete evidence to warrant a theory, that is fine with me. However, just because something doesn't have a or an acceptable theory, doesn't mean that one shouldn't examine the evidence. Perhaps examining the evidence could lead to a theory and extend our understand about the supposed phenomenon in many ways.

As far as I know, there in fact have been theories proposed: quantum mechanics, the holographic universe theory, teleological model, thermal fluctuation model, behavior model, various electromagnetic theories, decision augmentation theory, and probably several others.

I don't know much about any of these theories, other than that they exist, and people are, in fact, working on them. (I just did a Google search).

I believe this is different from things like 'no theories of psi exist' and 'no experiments are being done to investigate these theories'.

 

[T'ai Chi]

Re: Data?

There is no data. Nitwits come claiming to be able to dowse or remote view or perform telekinesis and can't - how does that constitute data?

There certainly is data from some of the studies.

Say a dowser attempts to find a film cannister with water in it. There are 2 cannisters, and only one has water in it.

He does 20 trials, and gets 7 hits.

If you standardize that score, call it Z, it will be negative (in this case).

If we have another dowser who, out of 30 trials with 2 cannisters, got 17, this Z will be positive.

If we do something similar for each dowser and calculate their Z-scores, we will have a bunch of Z-scores.

If there was absolutely nothing going on but chance, we'd expect some Z-scores to be negative and some to be positive, so that the combined Z-score would wash out and be about 0.

OR, when combined, the combined Z-score could be very negative, or the combined Z-score could be very positive, meaning that the studies, when examined as a group, showed preference against chance in some way.

 

[thaiboxerken]

Ok. Here is the meta-analysis.

100% of all claimants failed.

People don't have superpowers.

 

[thaiboxerken]

If there is no or incomplete evidence to warrant a theory, that is fine with me. However, just because something doesn't have a or an acceptable theory, doesn't mean that one shouldn't examine the evidence. Perhaps examining the evidence could lead to a theory and extend our understand about the supposed phenomenon in many ways.


I believe this is different from things like 'no theories of psi exist' and 'no experiments are being done to investigate these theories'.

There is no evidence of "psi". There are TONS of theories on how it might work though. Although, they are not really theories, but speculations based on personal beliefs and nothing more. So.. no evidence + no theory = no psi.

 

[Paul C. Anagnostopoulos]

T'ai said:

OR, when combined, the combined Z-score could be very negative, or the combined Z-score could be very positive, meaning that the studies, when examined as a group, showed preference against chance in some way.

Yes, but what does that mean? You don't know, because you have no theory. It could be the same flaw in the all the experiments. It could be a bunch of different flaws. It could be psi. It could be a miracle from the lord. You don't know!

Anyway, if you like this sort of thing, there are meta-analyses up the yin-yang from "real scientists."

~~ Paul

 

[Jeff Corey]

Or, as pointed out by D.M.Stokes in the May/June Skeptical Inquirer, it could very well be the File Drawer effect.
Great article.

 

[T'ai Chi]

T'ai said:
Yes, but what does that mean? You don't know, because you have no theory. It could be the same flaw in the all the experiments. It could be a bunch of different flaws. It could be psi. It could be a miracle from the lord. You don't know!

Anyway, if you like this sort of thing, there are meta-analyses up the yin-yang from "real scientists."

~~ Paul

The 'theory' is that if the applicants do no better than chance as a group, as we expect, we'd expect the combined Z-score to be around 0 and insignificant.

Right?

 

[Paul C. Anagnostopoulos]

T'ai said:

The 'theory' is that if the applicants do no better than chance as a group, as we expect, we'd expect the combined Z-score to be around 0 and insignificant.

That's a hypothesis about the results of the experiment itself, not about any underlying theory. A zeroish Z confirms the null hypothesis. A significant Z refutes the null hypothesis. In either case, it's a big so what because the hypothesis is not interesting.

As far as dowsing is concerned, there never was anything "unexplained" that needed experimental investigation. I wander around with a stick, dig a well, and find water. I wander around without a stick, dig a well, and find water.

~~ Paul

 

[T'ai Chi]

T'ai said:
That's a hypothesis about the results of the experiment itself, not about any underlying theory. A zeroish Z confirms the null hypothesis. A significant Z refutes the null hypothesis. In either case, it's a big so what because the hypothesis is not interesting.

I disagree. It certainly is interesting to many people.

If we obtain results that are not expected by chance, then we should investigate that further, because obviously something non-chance could be going on. (effects or errors)

 

[thaiboxerken]

I disagree. It certainly is interesting to many people.

Do you count yourself and the frog in your pocket as "many" people?

So far, you're the only one I know of that wants a "meta"analysis.


If we obtain results that are not expected by chance, then we should investigate that further, because obviously something non-chance could be going on. (effects or errors)

Only if the tests were all similar. They are not.

 

[Paul C. Anagnostopoulos]

T'ai said:

If we obtain results that are not expected by chance, then we should investigate that further, because obviously something non-chance could be going on. (effects or errors)

I agree: Assuming the meta-analysis was meaningful, we should investigate further. But in the case of the JREF, further investigation simply means new claimants for the prize, who will come along regardless of any meta-analyses.

In the real world, psi researchers take the meta-analyses as evidence for psi. They are not.

~~ Paul

 

[Prester John]

A meta analysis could be done on all tests which produce some sort of score, correct guesses etc. There would have to be a degree of similarity in what is being measured.

If you assume no one has psychic ability then most results should show a score of 0 (no different from chance) with a few showing slightly positive or slightly negative results - normally expected variance. Some sort of graphical representation would be easiest.

It all depends on how compatable the data is really.

 

[T'ai Chi]

In the real world, psi researchers take the meta-analyses as evidence for psi. They are not.

~~ Paul

I think that some people could be doing that. But yes, that is wrong.

However, if nothing is going on, we expect the results to be compatible with chance. If the results are not compatibible with chance, then something is going on.

That 'something' could be real effects or errors.

 

[Dub]

IMO meta-analysis would be pointless. In the tests which would create data that could be analysed statistically - such as dowsing etc, the claimant fails if they cannot perform significantly better than by chance alone. So far everyone has failed. Thus, meta-analysis of this data would simply come to the same conclusion that each individual test came to - that the claimant can perform no better than by chance alone.

 

[T'ai Chi]

In the tests which would create data that could be analysed statistically - such as dowsing etc, the claimant fails if they cannot perform significantly better than by chance alone. So far everyone has failed. Thus, meta-analysis of this data would simply come to the same conclusion that each individual test came to - that the claimant can perform no better than by chance alone.

It certainly could happen that way, you're correct.

However, since meta-analysis examines things as a group, it could turn out that, as a group, dowsers, while still failing to acheive significance in their individual tests, still have more positive scores than what is expected by chance.

(or of course, more negative scores than what is expected by chance).

 

[Charlie in Dayton]

I find this attempt to square the circle extremely confusing.

If the individuals as individuals couldn't do squat, why would the individiduals as a group produce kumquat salad?

QUESTION: Did the individuals who claimed they could do X actually do X?
Yes = 1, No = 0.

Nobody has ever been able to do whatever they said their personal paranormal X was.

0 = 0. 0 to the Yth power is still 0.

And (he said as he put his hand to his furrowed brow in a decent Karnak The Magnificient impersonation) don't give me this bunk about I don't understand meta-analytical techniques. I don't have to understand something that doesn't apply here.

I repeat -- 0 = 0. 0 to the Yth power is still 0.

Why is it necessary to add flour, baking soda, sugar and milk here? Them little brown things are rabbit pellets, and all the additives in the world ain't gonna convince me that the end result is chocolate chip cookies...