Meta-analysis of JREF tests

[T'ai Chi]

Meta analysis is a sound statistical method to examine scores from similarly conducted experiments as a group.

A meta analysis is not interested in if the applicants could do the experiment or not. A MA uses a method and combines all the standardized scores of the applications into one combined standardized score.

0 = 0. 0 to the Yth power is still 0.

We aren't examining 0's (although some standardized scores could be 0's). We are looking at their standardized scores and combining them into one standardized score.

Let's just see if the combined standardized score will be around 0 (and non-significant) as expected by chance.

But... what if most of the standardized scores are positive (or negative) ? What if the results are off from what we expect by chance?

don't give me this bunk about I don't understand meta-analytical techniques. I don't have to understand something that doesn't apply here.

Why exactly doesn't a meta analysis apply to similarly conducted tests where we can obtain standardized scores? I'm trying to understand why you think it doesn't apply. I'd like something a little more applicable than the '0 * 0! + 0^500 = 0 forever!!!' type of stuff.

Them little brown things are rabbit pellets, and all the additives in the world ain't gonna convince me that the end result is chocolate chip cookies...

It is a good thing I could care less about convincing you. I'm simply interested in scientifically examining the scores and seeing where that leads.

 

[CFLarsen]

T'ai Chi,

You need to do a little homework first, e.g. read about how the tests are designed.

Each challenge is different from any other, simply because it has to be tailored to the specific claim of the claimant.

It would be highly unfair of Randi if he insisted that Dowser A should take the same test as Dowser B, if Dowser A didn't claim the same as Dowser B.

So, there are not "standardized scores" to do a meta-analysis on.

The only thing we can look at is the results. They are all the same: "Failed". No need to look for anything there.

 

[T'ai Chi]

The challenges are different, of course, but the tests may be very or somewhat similar, enough to do a meta analysis on, in dowsing, and several other areas.

So, there are not "standardized scores" to do a meta-analysis on.

Sure there are. There are standardized scores in any test where there is a score recorded, and an expected score, and a measure of deviation.

The only thing we can look at is the results.

That might be the only thing you want to look at, but many analyses of the data can be done.

They are all the same: "Failed". No need to look for anything there.

So you're not interested in the question of: 'Is the combined standardized score what we would expect by chance?' I am.

 

[CFLarsen]

T'ai Chi,

If you feel that the tests are similar enough to do a meta analysis on, how would you do it, then? Don't ask others, I want to hear your own reasoning.

With statistics, please. Don't worry about me, if you say something I don't understand, I'll ask.

How would you perform a meta-analysis on the JREF challenges?

 

[Charlie in Dayton]

A meta analysis is not interested in if the applicants could do the experiment or not.

This is the reason for the experiments in the first place. What is the reasoning for applying an analysis method that by admission has no bearing on the subject of the data?

You're trying to find out if paranormal ability X exists. If mathematical procedure Y doesn't assist you in determining if X exists, why waste the effort? This is like going out to buy a car with the major criterion being how good a sailboat it is when you drive it off the boat dock.

These tests are conducted on individuals. Why are you trying to lump things together and get a result for a group? Throw fifty dowsers together in a field, and all you're going to get are numerous flesh wounds as they stagger around stabbing each other with their bent coat hangers. What are you looking for here? Why cloud the picture with admittedly irrelevant procedures?

I'm of the opinion that you're looking for the mathematical anomaly that 'proves' the existence of some paranormal whatnot, yet in actuality proves nothing at all. "Sure, let's throw another bushel basket full of numbers in front of 'em -- if ya can't dazzle 'em with dexterity, baffle 'em with b*llsh*t!" After all, 2+2=5, for large values of 2...

This is not a question of standardized testing, or meta-analysis, or anything else other than a simple yes-or-no answer to the question "Can you perform what you claim under proper controlled conditions?" Attempts to throw some sort of mathematical analysis into this question are irrelevant, immaterial, and I personally believe to be a symptom of a hidden agenda.

Now, go peddle your rabbit pellets to someone else. I ain't interested...

 

[T'ai Chi]

You answer my questions, I'll answer yours.

So, there are not "standardized scores" to do a meta-analysis on.

Why do you think there are not standardized scores?

No need to look for anything there.

Are you interested in the question of: 'Is the combined standardized score what we would expect by chance?'

If you feel that the tests are similar enough to do a meta analysis on, how would you do it, then? Don't ask others, I want to hear your own reasoning.

Thanks for your concern about asking others, but I can answer your question however I deem appropriate.

There are many good references on how to carry out a meta analysis. I would consult those and proceed as recommended. In general, in those references you'd find advice like what is at the bottom of this page.

 

[T'ai Chi]

What is the reasoning for applying an analysis method that by admission has no bearing on the subject of the data?

Do you expect the combined standardized score to be what is expected by chance, or not? That certainly has bearing on the subject of the data.

You're trying to find out if paranormal ability X exists.

No, I'm not. I'm just interested in analyzing the data that others collect, and seeing where those analyses lead.

These tests are conducted on individuals. Why are you trying to lump things together and get a result for a group?

To see if we get a combined standardized score around what we expect by chance. If we get a combined standardized score that is significantly away from chance levels, then there might be something going on (effects or errors).

I'm of the opinion that you're looking for the mathematical anomaly that 'proves' the existence of some paranormal whatnot, yet in actuality proves nothing at all.

Your opinion would be incorrect then, because I'm simply interested in analyzing the data.

Attempts to throw some sort of mathematical analysis into this question are irrelevant, immaterial, and I personally believe to be a symptom of a hidden agenda.

Apparently you believe that analyzing the data in appropriate ways is innapropriate. I've already stated that I'm not out to prove anything. Your repeated insistence of hidden agendas and that I'm out to prove the paranormal is still incorrect, and will be no matter how many times you suggest it.

Now, go peddle your rabbit pellets to someone else. I ain't interested...

And as I've said, I could care less about convincing you or anybody else. I am only interested in scientifically analyzing the data and seeing where that takes us.

 

[CFLarsen]

You answer my questions, I'll answer yours.

Sure.

Why do you think there are not standardized scores?

Because I have not heard Randi describe them. On the contrary, he has written about how each test is designed individually. Why are you asking this question in the first place? Have you done any homework at all?

Are you interested in the question of: 'Is the combined standardized score what we would expect by chance?'

Nope.

Thanks for your concern about asking others, but I can answer your question however I deem appropriate.

There are many good references on how to carry out a meta analysis. I would consult those and proceed as recommended. In general, in those references you'd find advice like what is at the bottom of this page.

I didn't ask how to carry out a meta analysis. I asked how you would do it on the JREF challenges.

I've answered your questions, you answer mine.

 

[Charlie in Dayton]

Let's see if I've got the hang of this multi-level quote thing...

Charlie in Dayton
What is the reasoning for applying an analysis method that by admission has no bearing on the subject of the data?
T'ai Chi
Do you expect the combined standardized score to be what is expected by chance, or not?

These are NOT standardized tests that are given across the board. They are individual, and agreed to in advance by both parties. There may be similarities at times, and indeed two or more individuals may agree to take the same test (there's an excellent example of that in Randi's book Flim-Flam). BUT - the tests are adiministered individually, and whether or not they're similar, identical, or otherwise, the results apply to the INDIVIDUAL -- period.

T'ai Chi
...I'm just interesting in analyzing the data that others collect, and seeing where those analyses lead.

The analysis of each individual bit of data is to determine whether or not individual X can conclusively demonstrate claimed paranormal ability Y under proper and agreed-on-in-advance controlled conditions. Analysis of the data to any other conclusion is irrelevant to the point of the experiment, and serves to cloud the issue. What is the purpose of analyzing the performance of the group when it wasn't the group that was tested? We're not looking for group performance here. Now, if four guys walked up and said in advance that individually they can't do beans, but together as a group they can do all sorts of wonderful paranormal things, there would be a test of the group. At that time, I would seriously suggest that in addition, the members be tested individually and the individual test results be analyzed and compared to the group's performance. But it's not kosher to invent a group when there wasn't one being tested in the first place.

T'ai Chi
Apparently you believe that analyzing the data in appropriate ways is innapropriate.

Not at all. The bone of contention at this point is, what's an appropriate way to analyze the data? You want to create an entity that never existed and analyze it. These tests were neither designed for nor run on groups. Why are you so insistent on analyzing the data that way?

T'ai Chi
I've already stated that I'm not out to prove anything.

Horse puckey. If you're not trying to prove anything, why bother analyzing the data? What, you're going to get to the equals sign and stop there?

T'ai Chi
And as I've said, I could care less about convincing you or anybody else.

Meadow muffins. If you weren't trying to convince somebody somewhere, you wouldn't be here doing this.

T'ai Chi
I am only interested in scientifically analyzing the data and seeing where that takes us.

Rabbit pellets. That mantra of 'using the scientific method' wears very thin when it's bad science being used.
You're using bad science here. Stop trying to make peach pie out of a basket of apples.

 

[T'ai Chi]

On the contrary, he has written about how each test is designed individually.

Then an assumption of independence for doing a meta analysis is clearly met.

Randi has described the observed and expected scores (and from those you get standardized scores from something like standardized score = (observed-expected)/standard deviation) relating to dowsing, and possibly other tests in Swift.

Why are you asking this question in the first place? Have you done any homework at all?

You're asking the question of why I asked the question?? Because I want your answer.

Far from me not doing "homework", I'm asking you because you hinted that there are not standardized scores. Obviosuly in dowsing, some ESP, and several other tests, there certainly are scores. If you have observed scores, and expected scores, and a measure of spread, you have all the ingredients of a standardized score.

Why aren't you interested in the question of: 'Is the combined standardized score what we would expect by chance?' ?

I didn't ask how to carry out a meta analysis. I asked how you would do it on the JREF challenges.

And how I'd do it would follow those guidelines that I gave in the link. Simply replace "relevant literature", "independent studies", "published information", and "studies" with "JREF tests", and proceed accordingly.

 

[Charlie in Dayton]

If you have observed scores, and expected scores, and a measure of spread, you have all the ingredients of a standardized score.

The observed score is 0.

The expected score is 0.

The spread is 0.

The standardized score is 0.

These results are consistent with each individual. The results of the group score are nonexistent, because there was no group tested as a group.

This concludes our meta-analysis.

Gee, that was simple, wasn't it?

 

[T'ai Chi]

These are NOT standardized tests that are given across the board. They are individual, and agreed to in advance by both parties.

I agree, fully, 100%, absolutely. However, dowsing tests are similar, and simply differ in numbers of pipes, or numbers of cannisters, or number of participants. The basic set ups are verysimilar. And therefore a meta analysis is appropriate and can be done.

BUT - the tests are adiministered individually,

Great! That is actually a GOOD thing, because it guarantees independence between tests, something which is required for a meta analysis.

Analysis of the data to any other conclusion is irrelevant to the point of the experiment, and serves to cloud the issue.

I disagree, completely. Data can be analyzed in any way that serves to explore a question of interest. If one wants to see if the combined standardized score is near what we expect by chance, one can do that analysis.

What is the purpose of analyzing the performance of the group when it wasn't the group that was tested?

As I've stated, we are exploring whether the combined standardized score is near what we expect by chance. It should be, shouldn't it?

, but together as a group they can do all sorts of wonderful paranormal things, there would be a test of the group.

I don't know how many times I will have to say, but I am not interested in proving that any paranormal phenomena exists.

You want to create an entity that never existed and analyze it.

Oh, I disagree. The data has always existed.

Why are you so insistent on analyzing the data that way?

Let me copy and paste my answer for the 4th or 5th time: The combined standardized score should be near what we expect by chance. Is it, or not?

Horse puckey. If you're not trying to prove anything, why bother analyzing the data?

Curiousity? Exploration? Fun? Seeing if I can analyze data? Seeing if the combined standardized score is near what we expect by chance? Applying statistical methodology in new and exciting ways? Testing the hypothesis that I can still use my calculator.

Meadow muffins. If you weren't trying to convince somebody somewhere, you wouldn't be here doing this.

Very poor attempt.

I am "here" because I am interested in exploring if the combined standardized score is near what we expect by chance, and, obviously I am replying to other posters' questions, including your own.

That mantra of 'using the scientific method' wears very thin when it's bad science being used.

Could you explain your opinion of "bad science"? Are you saying meta analysis is "bad science"?

You're using bad science here. Stop trying to make peach pie out of a basket of apples.

You have utterly failed to present a coherent case for that. You must have the standard topics of science confused with the flexible methods of science.

 

[T'ai Chi]

The observed score is 0.
The expected score is 0.
The spread is 0.
The standardized score is 0.
These results are consistent with each individual. The results of the group score are nonexistent, because there was no group tested as a group.
This concludes our meta-analysis.
Gee, that was simple, wasn't it?

Well, it was simple-minded anyway.

DATA

 

[CFLarsen]

And how I'd do it would follow those guidelines that I gave in the link. Simply replace "relevant literature", "independent studies", "published information", and "studies" with "JREF tests", and proceed accordingly.

Then, you have absolutely no idea what you are doing. You want to perform a meta-analysis on something that has no common ground, based on a text you clearly do not understand, but think you can just "replace" words to make it work.

You haven't done any homework regarding how the tests are being done.
You haven't exhibited any knowledge whatsoever regarding how to do a meta-analysis.
You haven't exhibited any knowledge whatsoever regarding what a meta-analysis is.
You haven't understood why you can't do a meta-analysis on the tests.

I think you just like to throw fancy terms around to appear smart. Well, you are not. Your own posts prove that.

Let's try again: In your own words, please explain how you would do a meta-analysis.

In your own words. That's where you get in trouble, isn't it?

 

[EdipisReks]

 

[T'ai Chi]

Then, you have absolutely no idea what you are doing.

You statement of dismissal does nothing to rationally debate your case.

You haven't done any homework regarding how the tests are being done.

What are you talking about? Where is your evidence for that claim? What do you mean by homework? Have you read any of the Randi commentaries that talk about specific dowsing tests? Do you agree or disagree that said commentaries talk about observed and expected scores? Do you agree or disagree that, with an easily calculated measure of spread, one can obtain standardized scores? I have read these, and understand statistics and how it applies to these tests, so how does that equate with my not doing my homework?

You haven't exhibited any knowledge whatsoever regarding how to do a meta-analysis.

Interesting. Are we in the same thread? Note: I don't expect you to answer rhetorical questions.

I've discussed meta analysis plenty in this thread, and gave pointers to a fairly complete link on meta analysis. The disconnect comes from you assuming I don't understand what is on the link, or how it applies to the JREF tests-something which you assume.

You haven't exhibited any knowledge whatsoever regarding what a meta-analysis is.

You are clearly incorrect. I've stated that it is to combine results from similar experiments, and I've also provided a link that explain what a meta analysis is. Do you dispute this?

You haven't understood why you can't do a meta-analysis on the tests.

No person has given any good reason why one can't. So tell us Claus, why can't one do a meta analysis? In order to say that you can't do a meta analysis, I assume you know all the specifics about meta analysis theory and application. Is it safe to say that you do know a lot about meta analysis, Claus?

I think you just like to throw fancy terms around to appear smart. Well, you are not. Your own posts prove that.

Let's not discuss what your posts show.

I want to see if the combined standardized score is near what we expect by chance. Everything else you psychics read into my words is your invention, conjured out of thin, very thin but amazingly stuffy, air.

Let's try again:

Let's not try again, Claus.

I am not required in any way to answer your questions in my own words ad nauseum if there are already sources out there explaining exactly how to carry out a meta analysis.

I'm not interesting in debating your online personality, I'm interested in analyzing some data, and discussing statistical issues. If you irrationally believe I am trying to appear smart by throwing around terms, then you have nothing to contribute to the discussion, and your further postings will not regretably be ignored.

If, on the other hand, you wish to discuss actual science, statistical, or testing issues, that would be most welcome.

 

[dissonance]

T'ai Chi, what's your statistics background? Graduate level? Undergrad? Interested layperson? We (OK, I) might have an easier time explaining why what your asking is inappropriate if we knew where you were coming from on this.

 

[tedly]

T'ai chi

If you take the meta-analysis maybe nothing is still going on.
We have a frighteningly common belief that there is something out there. Suppose we get a thousand by thousand matrix of possible interactions. If an interaction is so unlikely that it only has a one in 10000 chance of occurring, we find 100 such artifacts in our dataset. Similarly a belief so wierd that only one in a million could bite on it, has 6000 people out their that hold it as a basic tenet of their faith.

If you torture the data long enough it will confess to anything. The members seem to be sufficiently humane that they don't want to torture the data. Let go.

 

[CFLarsen]

T'ai Chi,

You are very right - this is not about me, it's about you.

Why are you unable to explain, in your own words, how you are going to do a meta-analysis on the JREF challenges?

It should be easy for you: You claim to have done your homework, by reading what Randi has said about the challenges. You also claim to know how to do a meta-analysis. You seem reasonably able to express yourself verbally. So, what's stopping you?

If you don't want to explain it, then just go ahead and do this meta-analysis of yours.

What's stopping you?

 

[69dodge]

Originally posted by T'ai Chi
I want to see if the combined standardized score is near what we expect by chance.

Suppose it's not. What do you think we could conclude from that? How would it change your beliefs or your actions? In short, what difference would it make?

If the results would make no difference, there's no point in doing the meta-analysis to begin with.