Meta-analysis of JREF tests

[Flo]

Analyzing the data and subjecting it to conditions that were not in existence at the time of the test is BAD SCIENCE!!! It's rewriting the rules halfway through the game. That's not the way things are done, and you know it.

The individuals either have the paranormal ability, or they do not. What third option might a meta-analysis of the data show? Do you want to quantize it? Fine -- zero still equals zero.

What is it specifically that you think might be shown by this analysis? How do you hope to show some statistically significant result when there is NO DATA to support it?

You keep trying to put a number to all this. You can't, because YES and NO are not numbers!

You're trying to make something out of nothing here (literally). That's bad science. Stop it.

Actually, I've seen the same line of argument on a French-speaking forum ( www.sceptiques.qc.ca ) . The demand for a statistical analysis doesn't relate to the actual results of experiments but on the conditions surrounding them, i.e. what factors could be taken into account to explain the consistently negative results and find some sort of a hint of a beginning of positive results ("negative psy", lack of accounting for the psychological and emotional conditions of the testees, ratio of believers:unbelievers among people attending or controlling the tests, age of the captain, influence of the phases of the moon, bad feng-shui, ...).

 

[pgwenthold]

A quick note. This is clearly a troll. If you have some number of tests accross some number of subjects and the results for each subject are not significant there is not,, short of woo-woo statistics, that you can suddenly get significance.

Nonsense.

See the previous example. Suppose we run a test where the outcome 1/4 by chance. We run 10 trials, and the person gets 3 right. Is that a significant result? Not at all.

Now, suppose we test 1000 people, and they all get 3/10 right, and fail. Significant? You bet it is.

3000/10000 is far from random chance, despite the fact that not a single individual hit a significant rate.

No woo-woo statistics involved.

 

[Ed]

Nonsense.

See the previous example. Suppose we run a test where the outcome 1/4 by chance. We run 10 trials, and the person gets 3 right. Is that a significant result? Not at all.

Now, suppose we test 1000 people, and they all get 3/10 right, and fail. Significant? You bet it is.

3000/10000 is far from random chance, despite the fact that not a single individual hit a significant rate.

No woo-woo statistics involved.

The stats are not woo-woo (maybe) the interpretation is. Clearly any series of random numbers will vary from chance and sometimes they might even vary significantly. This is as it should be. The difficulty arises from the source of the data. If the experiment is not well designed and if there is not a falseifyable hypothesis, you are dealing with a fishing expedition, not science. I note the elaboration of purported intervening factors (with very impressive terminology, I might add) to explicate PK and other phenomena. Fine and good except that it is not clear that there are any phenomena at all.

What it all comes down to is that there is no clear evidence of anything paranormal to begin with.

What, precisely, is the difference between the current state of paranormal research and a bunch of labs generating random numbers and trying to assign a paranormal explination to them? If you hung your professional hat on being a researcher into the paranormal, and basically all you got for your efforts is bupkus, would you not analyze the hell out of it?

As an old friend likes to say "You can torture data to say anything".

 

[Ed]

Actually, I've seen the same line of argument on a French-speaking forum ( www.sceptiques.qc.ca ) . The demand for a statistical analysis doesn't relate to the actual results of experiments but on the conditions surrounding them, i.e. what factors could be taken into account to explain the consistently negative results and find some sort of a hint of a beginning of positive results ("negative psy", lack of accounting for the psychological and emotional conditions of the testees, ratio of believers:unbelievers among people attending or controlling the tests, age of the captain, influence of the phases of the moon, bad feng-shui, ...).

This is data torture. Now, if there were some consistant effect, looking at what influences it would make sense. This is more on the order of "let's manipulate things until we get something and then we can declare success". So what they are saying, in essence, is that there is an effect, we don't need research we know it exists now we need to find out why we don't see it. This is non-falsification, not science, and religion.



Back to basics: What is the hypothesis that is being tested?

 

[Flo]

This is data torture. Now, if there were some consistant effect, looking at what influences it would make sense. This is more on the order of "let's manipulate things until we get something and then we can declare success". So what they are saying, in essence, is that there is an effect, we don't need research we know it exists now we need to find out why we don't see it. This is non-falsification, not science, and religion.

Back to basics: What is the hypothesis that is being tested?

This is exactly what I answered the guy. As could be expected, his answer was a mix of ad hominem and protests of good faith.

 

[T'ai Chi]

A quick note. This is clearly a troll.

Spare us your dismissive ad hominems, and please stick to the topic.

If you have some number of tests accross some number of subjects and the results for each subject are not significant there is not,, short of woo-woo statistics, that you can suddenly get significance.

Really? If most of the standardized scores are positive (or negative), that wouldn't be something?

No one is saying that if the combined standardized score was significant that that would prove the paranormal or whatever. What they are saying is that if the combined standardized score was significant, that is, not near 0 and non-significant as we'd expect by chance, then something is going on, either an effect or problems with the design of the tests, or something else.

 

[Ladewig]

Why do you think that there are enough quantifiable tests in the JREF preliminary test history to see trends rather than noise?

If you did meta-analyze the data and came up with some positive result, what would you do with that information? editted to add: Never mind - I just saw your answer in the previous post.

----------------
Maybe we should meta-analyze the JREF data and then combine those results with the results of other meta analysis to produce an epi-meta-analysis?

 

[T'ai Chi]

The individual can either do what they claim, or they cannot. Yes or no. 1 or 0. Black or white. Chevy or Ford. Pepsi® or Coke®. Either/or.

That is true.. But in many dowsing experiments they are dowsing, say, 5 film cannisters for 1 of them that has gold in it. The dowsers might do 30 trials. We'd expect them to get 30*(1/5) = 6 hits by chance alone.

Statistics are kept other than 0 or 1 outcomes, that can be analyzed.

Analyzing the data and subjecting it to conditions that were not in existence at the time of the test is BAD SCIENCE!!!

I disagee. It is good science to analyze the data, and to test the hypothesis of the combined standardized score being what we'd expect by chance.

By your rules, no one could ever do a meta analysis in any discipline.

The individuals either have the paranormal ability, or they do not.

Oh, I agree 100% with that. They either do or don't. I agree. What I am wondering is is the combined standardized score near 0 and non significant as we'd expect it would be.

How do you hope to show some statistically significant result ..

Whoah, back up. I don't hope to show anything. I'm an impartial investigator simply investigating an interesting question.

 

[Ed]

Spare us your dismissive ad hominems, and please stick to the topic.

[/b]

Really? If most of the standardized scores are positive (or negative), that wouldn't be something?

No one is saying that if the combined standardized score was significant that that would prove the paranormal or whatever. What they are saying is that if the combined standardized score was significant, that is, not near 0 and non-significant as we'd expect by chance, then something is going on, either an effect or problems with the design of the tests, or something else. [/B]

Please, if they were positive there would have been results. Or, are you suggesting pooling all scores and standardizeing them? You surely don't mean that, do you? That would likely be junk statistics.

I will also ad hominum as I see fit, thank you. Your topic is moribund.

 

[Ed]

Whoah, back up. I don't hope to show anything. I'm an impartial investigator simply investigating an interesting question. [/B]

Your contention regarding this type of analysis is meaningless, I'm afraid. You surely see why, do you not?

 

[jj]

That is true.. But in many dowsing experiments they are dowsing, say, 5 film cannisters for 1 of them that has gold in it. The dowsers might do 30 trials. We'd expect them to get 30*(1/5) = 6 hits by chance alone.

Statistics are kept other than 0 or 1 outcomes, that can be analyzed.


And how do you analyze the results from tests with different statistics, different expected outcomes, etc?

Note, I'm not saying you can't, strictly speaking, but I am asking you how you would make meaning from such results.

By your rules, no one could ever do a meta analysis in any discipline.


Close, but not quite. The whole problem is that it is really quite a rare case where a meta-analysis is appropriate.

Even tests of different people using the exact same methodology and such need to be tested for consistancy.

Testing the relevance of two different tests on different people at different times is just not meaningful.

Whoah, back up. I don't hope to show anything. I'm an impartial investigator simply investigating an interesting question.

You're "impartial", yet you insist in other threads, for instance, that there is positive evidence for homeopathy?

Uh huh. And my name is Dilbert. Really, it is.

 

[T'ai Chi]

Please, if they were positive there would have been results. Or, are you suggesting pooling all scores and standardizeing them? You surely don't mean that, do you? That would likely be junk statistics.

I am interested in testing if the combined standardized score is near 0 and non-significant as expected by chance. Everything else said about my 'agenda' is what people invent out of thin, but amazingly stuffy, air.

Dismissing meta analysis as 'junk statistics' or 'data mining', is not correct.

I will also ad hominum as I see fit, thank you. Your topic is moribund.

Of people continuing ad hominems, I have no doubt.

Moribund. I love word of the day.

 

[Ed]

You're "impartial", yet you insist in other threads, for instance, that there is positive evidence for homeopathy?

Dilbert, you forget the first rule of politics when your hand is caught in the cookie jar ... Deny, deny, deny. It is similar with paranormal research ..Analyze, analyze, analyze.

In reputable research, they pretty much figure out how something is to be analyzed up front, this going back to the well stuff stinks to high heaven.

 

[T'ai Chi]

It is similar with paranormal research ..Analyze, analyze, analyze.

Analyzing. I guess some scientists are definitely guilty of that. However, they also have models that can be dismissed or kept based on the data. They also have hypotheses that can be falsified.

In reputable research, they pretty much figure out how something is to be analyzed up front, this going back to the well stuff stinks to high heaven.

There are tests. They have statistics. They are probably similar enough to be combined, to see if the combined stadardized score is near 0 and non-significant as we'd expect by chance.

What goes on in other threads has no bearing whatsoever for this thread. If I said in another thread that God exists and I can prove it, etc., that has no bearing on the topic of a meta analysis for the JREF tests.

I'm amazed I even have to say that to a group of skeptical thinkers.

 

[pgwenthold]

The stats are not woo-woo (maybe) the interpretation is.

How so? Are you denying that 3000/10000 is more likely significant than 3/10?

Clearly any series of random numbers will vary from chance and sometimes they might even vary significantly. This is as it should be. The difficulty arises from the source of the data. If the experiment is not well designed and if there is not a falseifyable hypothesis, you are dealing with a fishing expedition, not science.

He is talking about meta analysis of JREF challenges. Are you suggesting that JREF is a fishing expedition?

Moreover, you changed the point. The original claim, to which I responded, is that a lot of non-significant results cannot lead to a significant result. My example shows this to be clearly wrong. You can get significance in a large data set without ever having to have significance in any of the small sets.

What it all comes down to is that there is no clear evidence of anything paranormal to begin with.

This misses the point. Perhaps you can't see the evidence because all the data sets that have been tried are too small?

A psi effect that increases the probably correctly answering from 25% to 26% is a real effect, but you are going to have a hard time seeing it if you only carry out 1000 repititions. On the other hand, if you have 1000 people carry out a thousand reps, you will have a better chance of seeing it.

I'm not saying that there is such a thing, but you can't dismiss the possibility that it is there.

 

[Ed]

Analyzing. I guess some scientists are definitely guilty of that. However, they also have models that can be dismissed or kept based on the data. They also have hypotheses that can be falsified.



There are tests. They have statistics. They are probably similar enough to be combined, to see if the combined stadardized score is near 0 and non-significant as we'd expect by chance.

What goes on in other threads has no bearing whatsoever for this thread. If I said in another thread that God exists and I can prove it, etc., that has no bearing on the topic of a meta analysis for the JREF tests.

I'm amazed I even have to say that to a group of skeptical thinkers. [/B]

Luci, your knowledge of stats is too poor to carry on. What is the probability of drawing 911 on a specific date again?

 

[Skeptical Greg]

......

I'm not saying that there is such a thing, but you can't dismiss the possibility that it is there.

Ahhh.. But you can dismiss the possibility that it is significant.. Like an atom of arsenic in a gallon of water..

The question becomes; " To whom, is it worthwhile to determine if it is ' there ', no matter how ' insignificant '? "

 

[Lurker]

Tai:

I don't see how you could a meta analysis. The claims are all so different, with such different output. Even if you grouped dowsers together, for example, even their claims and subsequent tests would be quite different.

Some dowsers dowse for water, others? Some probably say they can get within "X" meters of the water, where "X" varies for each claimant. What other differences are there that result in them getting different output for the Randi Challenge? The possibilities are endless.

Sorry, I don't see how a meta analysis could be done. If you come up with a method, feel free to try and see what happens.

Lurker

 

[Ed]

How so? Are you denying that 3000/10000 is more likely significant than 3/10?

You mean 3/10 with a sample size of 10000 as opposed to 10. For a given difference n enhances significance. You miss the point. If you mix outliers in with other data enough to change the results, you have junk. If one dowser gets 10/10 and another gets 0/10 the net result is not 5/10 or 10/20 unless the experiment was designed as a multi subject effort. One outlier would (should) raise red flags all over the place.


He is talking about meta analysis of JREF challenges. Are you suggesting that JREF is a fishing expedition?

I am saying that mushing results together is done at one's peril. Again, if no challenger passed the test, why would you expect that glomming the results together would yield any different. I'd really like to understand your thinking. To put it another way [stevie wonder] Nothin' from nothin' leeeeves nothin'[/stevie wonder].

Moreover, you changed the point. The original claim, to which I responded, is that a lot of non-significant results cannot lead to a significant result. My example shows this to be clearly wrong. You can get significance in a large data set without ever having to have significance in any of the small sets.

A single subject design that calls for 100 trials might not be significant whereas the same design that produces the same scores over 10,000 trials may be. If that is what you meant I grant you that. However that is not to say that 100 subjects with 100 trials each with the same score would yield significance UNLESS the experiment was designed that way. You can aggregate to your heart's content but if it is not apples and apples you have a curiosity, nothing more (that and a note to self to design a better experiment before you go thru 100 uncontrolled replications). My beef with woo-woo's is that they might lump stuff together and declare it proof without doing the necessary homework.


This misses the point. Perhaps you can't see the evidence because all the data sets that have been tried are too small?

A psi effect that increases the probably correctly answering from 25% to 26% is a real effect, but you are going to have a hard time seeing it if you only carry out 1000 repititions. On the other hand, if you have 1000 people carry out a thousand reps, you will have a better chance of seeing it.

True. Now, why oh why over the last 50 years or so has no one done just that, or, if they have where are the results? Dosen't the absolute lack of evidence create a doubt in your mind? Forget about fancy stats, just the fact that support for ALL paranormal claims requires a dance is not the least bit troubling?

I'm not saying that there is such a thing, but you can't dismiss the possibility that it is there.


No, I cannot on an intellectual level any more than I can dismiss the possibility of the existance of God. I am an agnostic on God but, given the evidence or lack thereof at hand I am an atheist. So it is with woo-woo stuff. I recognize that there is a possibility for it's existance however, given the evidence and fraud (Schwartz for example) the childlike credulity of some, I believe it is ◊◊◊◊◊◊◊◊. This does not mean that I cannot maintain an open mind. It just means that proponents need to be rather compelling.

 

[jj]

In reputable research, they pretty much figure out how something is to be analyzed up front, this going back to the well stuff stinks to high heaven.

Man, oh man, Ed Save Us I hope they didn't get any of THAT in the well, for sure!