• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

Meta-analysis of JREF tests

This thread has gone on for long enough.

T'ai Chi, if you want to analyze the data, then go to the JREF headquarters and analyze the data. Performing that type of meta-analysis is outside the stated purpose of the The Foundation. Furthermore, even it were something within the Foundation's charter, the funds are too limited to devote to a task that might give a result other than zero.

Lastly, I would suggest that before you book an airline ticket, ask yourself, "why do I think there are enough quantifiable tests in the JREF history that trends will be visible above the noise."
 
Ladewig said:
This thread has gone on for long enough.
Click to expand...


Perhaps because people reply? ... ;)


T'ai Chi, if you want to analyze the data, then go to the JREF headquarters and analyze the data.
Click to expand...


The fact that I am merely suggesting the idea, has nothing to do with me going to Florida, and the fact that I am not personally doing the analysis has nothing to do with discussing the idea of doing one.

It would be nice if the data was publicly available (without specific names of people involved, of course) though, then we could all analyze and compare our results.


Performing that type of meta-analysis is outside the stated purpose of the The Foundation.
Click to expand...


Sure, I agree with that. That is not the purpose of the JREF.

Question: Is it outside of the stated purpose of "The Foundation" to allow outsiders/independent investigators access to their dataset?
 
FutileJester said:

The axioms of the scientific method don't depend on statistics.
Click to expand...


It would have been more proper of me to say that the 'scientific method' and the disciplines of Statistics have many overlapping areas: inference, hypothesis testing, quantitative, experimental design, issues about bias, randomization, models, etc.


Standardized score of what measure?
Click to expand...


Something like: number of times correct idendtifying the film cannister that has gold in it, or identifying the pipe that has water flowing through it.


If we did it's possible some interesting areas for study could be revealed. For example, it could be found that out of all categories of claimants, dowsers lived perceptibly longer. Or that clarivoyants have the strongest religious convictions. But without this additional data, we're basically stuck with hits. And I don't yet see any way that meta-analyzing the hits would be suggestive of anything concrete enough to focus a new study on.
Click to expand...

Perhaps we could look at score by gender, or score by age, or score by time spent learning dowsing, for starters. I'm personally not too interested in relating their scores to any other variables (although it is a very interesting area to explore). I'm just interested in seeing if the combined standardized score is near 0 and non-significant as we'd expect it to be.
 
T'ai Chi said:
Something like: number of times correct idendtifying the film cannister that has gold in it, or identifying the pipe that has water flowing through it.
Click to expand...


Okay, so basically we're analyzing hits, however they are defined for an individual test. I still don't see what useful results we can derive from an analysis of hits. To re-ask an unanswered question from before, can you give an example along the lines of "the score for measure X was much greater than Y, which suggests Z"? Or specifically for this case, "the standardized score for dowsing hits was X, therefore Y".

The purpose of a meta-analysis can't be just to wave our hand in the air afterwards and say "look we found something". It should suggest correlations not originally seen which can then be separately tested in their own controlled studies. What studies, in principle, could be suggested by the results of analyzing dowsing hits?

Perhaps we could look at score by gender, or score by age, or score by time spent learning dowsing, for starters. I'm personally not too interested in relating their scores to any other variables (although it is a very interesting area to explore).
Click to expand...

I agree, probably not much interesting to be found there. My point is that these types of studies, looking for correlations between measures that are available but which were not the principle concern of the study, seem much more likely to provide usable results. I doubt enough data of that sort is consistently available in the trial data in any case.
 
Ed said:
Hmmmm..... Talking to yourself, Luci?
Click to expand...

Wow, quite abnormal behavior for a skeptic...

You might want to rethink your strategy, as I can assure you, that Lucianarchy and I are quite different people entirely.

It seems like you are simply acting based on your deep-set beliefs, so carry on.
 
T'ai Chi said:


Wow, quite abnormal behavior for a skeptic...

You might want to rethink your strategy, as I can assure you, that Lucianarchy and I are quite different people entirely.

It seems like you are simply acting based on your deep-set beliefs, so carry on.
Click to expand...

Address some of the issues raised on your silly thread. Luci would never do that. That will be proof.
 
Ed said:


Address some of the issues raised on your silly thread. Luci would never do that. That will be proof.
Click to expand...

So you are admitting that you don't have proof but yet you already made the claim of 'T'ai Chi = Lucianarchy'???

That is not skeptical at all I would say.

Excuse away.
 
Ta'i Chi, I know you have no obligation to jump through hoops when I ask a question, but it's frustrating to have you engage me in a discussion and then ignore my questions about your viewpoint. I think I'm giving you much more benefit of the doubt here than most others, but that's a hard attitude to maintain when you take time to answer personal slights but won't answer (repeatedly) good questions about the topic.

For the third time, and for the benefit of those of us who do not know as much about meta-analysis as you claim to, could you please answer the following questions:

  • What kind of conclusions could we, in principle, make based on the meta-analysis? Looking for statements along the lines of "the standardized score for dowsing hits was X, therefore Y".
  • What further studies could be suggested by the results of analyzing dowsing hits?

If we can't answer these ahead of time, then I can't help but agree with the conclusion others have already reached - it's just a fishing expedition.
 
FutileJester said:

For the third time, and for the benefit of those of us who do not know as much about meta-analysis as you claim to, could you please answer the following questions:
Click to expand...


Sorry FJ, I get distracted from the good questions by the people who insult, etc.

We could make the conclusion of: 'the combined standardized score was significant or nonsignificant.'

This could lead to possibly finding ways to improve the tests.
 
T'ai Chi said:
We could make the conclusion of: 'the combined standardized score was significant or nonsignificant.'
Click to expand...

I know that finding a significant score means that the score is significant. :rolleyes: But what could it mean if the score is significant?

This could lead to possibly finding ways to improve the tests.
Click to expand...

Such as? I know meta-analysis is a tool to suggest future tests, I've said so myself. The question is, which improvements or new studies could in principle be suggested by the analysis you're proposing?

My point with the earlier examples involving secondary variables was that meta-analysis using these variables makes sense, since we can easily imagine a result and what tests that result would imply. For instance, a result might be that claiming to be a dowser is correlated with strong religious conviction. We can then easily design a test that collects data about dowsing claimants and religious beliefs.

I can't think of any results from analyzing dowsing hits that would suggest a specific test or improvement. But I'm the first to admit that I don't know a lot about this, which is why I'm asking. Is there a hypothetical but specific result that would imply a specific test or improvement?
 
FutileJester said:

I know that finding a significant score means that the score is significant. :rolleyes: But what could it mean if the score is significant?
Click to expand...


It is hard to say for sure, FJ, without actually examining the data. So, I don't know. :(
 
T'ai Chi said:
It is hard to say for sure, FJ, without actually examining the data. So, I don't know. :(
Click to expand...

But don't you see that this is a fatal flaw in your proposed analysis? With other proposed (but uninteresting to us) analyses we could easily imagine ahead of time possible results.

You pointed out earlier the parallels between statistics and science; in this vein, doing a meta analysis without anticipating possible conclusions is like doing an experiment with no falsifiable hypothesis. It's fishing. Painful experience has shown that it just doesn't lead to reliable results.

In fact this is a principle I use at work - never run a test if you don't understand what the results will mean. I've seen (slightly) younger engineers go through a cycle of getting confused, running a test they don't really understand, and consequently getting more confused. Contrast this with the more experienced engineers, whose most stinging criticism of a propsed test is, "But what will that tell us?" If you can't answer that to their satisfaction, they won't run the test. Experience says it will be a waste of time.
 
Ed said:


You sure?
Click to expand...

I'm sure, this is from another thread.

http://www.randi.org/vbulletin/showthread.php?s=&postid=1870085040#post1870085040

Posted by: Thaiboxerken
I didn't like you when you posted as Who, and I still don't like you. Go meditate on that.
Click to expand...

Response from T'ai Chi:

Well I like you! *smooch* So there.

I will go meditate though. Thanks!
Click to expand...

No mention of "I'm not Who" in there anywhere.

Damn trolls. Who, you said you were done with the forum, why did you come back?
 
You must log in or register to reply here.

Back
Top Bottom