JREF Challenge Statistics

[Mercutio]

So every once in a while we see data from these tests, right?

Why not put it all in an easy to get, easy to read, format for everyone who is interested in such results from each test?

A) You and who else constitute this "everyone"? B) As I tell my kids, the question is not "why not?"; the question is "why?". These data cannot answer the questions you have wanted them to; what purpose will these data serve?

Seems reasonable.

To whom?

Is "truly interested" the same as a "true Scotsman"?

Very good! You are quite right; I really have no reason to suspect that you have any interest in the questions at all, let alone "true interest". Consider the comment retracted.

 

[CFLarsen]

Seems reasonable.

Here we go again...."reasonable". That's when we know T'ai Chi has run out of arguments....

 

[T'ai Chi]

These data cannot answer the questions you have wanted them to; what purpose will these data serve? To whom?

The data can answer questions people are interested in, descriptive ones about how many tests per year, what type, claimant characteristics, and scores.

You seem to be interested in individual results, but not a list of all the individual results. Who knows why.

; I really have no reason to suspect that you have any interest in the questions at all, let alone "true interest".

What you "suspect" doesn't really matter.

 

[Mercutio]

The data can answer questions people are interested in, descriptive ones about how many tests per year, what type, claimant characteristics, and scores.

Who are these "people"?

The data can answer these questions only about the small, self-selected sample itself. The data cannot tell us anything about the greater population, about which the questions are far more interesting.

You seem to be interested in individual results, but not a list of all the individual results. Who knows why.

Because that is all the data are good for. If I am interested in the questions you think are important, I would look for a sample which can answer them. This list cannot answer those questions.

You seem to be interested in asking questions of a database that is not designed to answer them, but not looking for a database that can. Who knows why.

What you "suspect" doesn't really matter.

Hey, I was only agreeing that my previous comment was a "No True Scotsman". Which would you prefer, that I assume you are interested, or that I assume you are not? It doesn't matter to me. If you are interested, there are better ways of answering the questions. If you are not, there is no reason to be asking them in the first place.

 

[T'ai Chi]

Who are these "people"?

Many skeptics and non-skeptics, some people who read the commentary, some people who read SI, Skeptic, Fortean Times, some people who are on science and skeptic movement bulletin boards; in short, anyone who is interested in seeing data from tests done by skeptical organizations.

You seem to be interested in them, at least when they are listed singley and sporadically.

The data can answer these questions only about the small, self-selected sample itself. The data cannot tell us anything about the greater population, about which the questions are far more interesting.

This minor point was already addressed.

 

[Mercutio]

Many skeptics and non-skeptics, some people who read the commentary, some people who read SI, Skeptic, Fortean Times, some people who are on science and skeptic movement bulletin boards; in short, anyone who is interested in seeing data from tests done by skeptical organizations.

Hmmm...perhaps I missed where these people were clamoring for this data set to be mined; if you could point me to where the request is being made by others than you, I will gladly concede the point. Might I suggest, though, that people who are interested in the sort of questions you ask might already realize that this data set cannot answer them, and are not asking the same question you are.

 

[T'ai Chi]

If one doesn't feel statistics from tests done by skeptical organizations are interesting or important, one is entitled to such an unimaginative opinion.

But there seems to be really no other way to answer questions like "What % of tests are dowsing?" other than to see actual data, for example.

Test information and results listed singlely is OK, but putting all of these single results in a table and not even combining them in a meta-analysis is taboo, according to some, for some odd reason.

 

[CFLarsen]

If one doesn't feel statistics from tests done by skeptical organizations are interesting or important, one is entitled to such an unimaginative opinion.

But there seems to be really no other way to answer questions like "What % of tests are dowsing?" other than to see actual data, for example.

Test information and results listed singlely is OK, but putting all of these single results in a table and not even combining them in a meta-analysis is taboo, according to some, for some odd reason.

Can you point out where the request is being made by others than you or not?

 

[Mercutio]

If one doesn't feel statistics from tests done by skeptical organizations are interesting or important, one is entitled to such an unimaginative opinion.

I agree. Fortunately, no one has suggested such a thing. What is suggested is that trying to make the data serve a purpose for which they were not designed is not productive.

But there seems to be really no other way to answer questions like "What % of tests are dowsing?" other than to see actual data, for example.

What purpose would the answer to that question serve? It answers only the narrowest of questions about a self-selected sample. It tells nothing about the beliefs about dowsing in the general population, nothing about belief in one's own ability to dowse in the general population...I do not understand why one would intentionally choose a biased sample to examine meaningful questions. If you agree with that, and claim only to be interested in examining the narrow, self-selected sample, I sincerely ask "why?". It is an absolutely useless question.

Test information and results listed singlely is OK, but putting all of these single results in a table and not even combining them in a meta-analysis is taboo, according to some, for some odd reason.

Exactly. It is an "odd" reason; a reason which only becomes clear with an advanced understanding of statistics. This is why combining them is such a bad idea.

Without a good understanding of statistics and methods, all sorts of investigations are seemingly "reasonable". My students have to be taught, with some care, why a self-selected sample is subject to bias; why a sample of convenience may show artifacts; why a particular alpha is used. These things are not intuitively obvious.

Taking a sample that is perfectly adequate for its original purposes but unfit for meta-analysis or inferential study and trying to do more with it, is just plain irresponsible. Anyone who cares about statistics and methodology should know that. There is enough misinformation available without intentionally adding to the pile.

 

[CFLarsen]

I do not understand why one would intentionally choose a biased sample to examine meaningful questions. If you agree with that, and claim only to be interested in examining the narrow, self-selected sample, I sincerely ask "why?".

Because it would produce something that would muddy the waters.

Which is all that T'ai Chi is here for.

 

[catbasket]

I feel it's time to quote something jj said, which I felt would be a good aide-mémoire:

Um, let's look at this T'ai Chi chap's performance. First, he's had at least 4 nics, those being T'ai Chi, Whodini, JZS, and Statisticool. Second, he's reknowned for taking things out of context, malicious misunderstanding, outright misstatement of others positions, and the like. Third, he has nothing positive to say, and appears to emerge mostly to indirectly attack or incite other posters.

T'ai - if you think you're impressing the lurkers ... well you've failed abysmally in my case.

 

[T'ai Chi]

Being that the responses I've received are 'self-selected', I can safely discard them then, according to some.

Mercutio wrote

It tells nothing about the beliefs about dowsing in the general population,

So? Where was it said that that was a goal? It is setting out to answer what the statistics are of those who took the test, you know, that whole descriptive vs. inferential thing that I've repeatedly said.

It is an absolutely useless question.

Frequently it is said that the plural of "anecdote" and is not "data". I'm starting to think that some around here believe that the plural of "complaint" is "rebuttal".

This is why combining them is such a bad idea.

Check out the bold in what I wrote

Test information and results listed singlely is OK, but putting all of these single results in a table and not even combining them in a meta-analysis is taboo, according to some, for some odd reason.

That is, the possibility of a meta analysis was raised. The main issue is getting all the statistics for the tests from skeptical organizations made readily available to any interested party. If an organization has a test as its main draw, the least it should do IMO is make the data from the tests more easily available.

 

[T'ai Chi]

T'ai - if you think you're impressing the lurkers ... well you've failed abysmally in my case.

Do try and stay on topic.

 

[catbasket]

Do try and stay on topic.

Ok -

Does anyone believe that after 1,000+ tests that are statistical in nature are carried out, that anyone will win by chance?

Has the JREF carried out 1,000+ tests?
Were 1,000+ statistical in nature?

No. And no. (http://www.randi.org/research/faq.html#1.3)

Will anyone win by chance?

I think it's exceptionally unikely anyone will win by chance; someone could. But -


What exactly is the point of your "what if?" scenario?

 

[Mercutio]

Being that the responses I've received are 'self-selected', I can safely discard them then, according to some.

You can, and do, so quickly here...why not when it is important?

So? Where was it said that that was a goal? It is setting out to answer what the statistics are of those who took the test, you know, that whole descriptive vs. inferential thing that I've repeatedly said.

Which does not answer my question--what is the purpose of asking the question? You get an answer--ok, what does it tell you? I can't see anything useful, but I was wrong once before, about 5 years back, so I suppose it could happen...

Check out the bold in what I wrote

Ok, so your contention is that putting them all in one table is not "combining them". I think it is, and I advised against it, even if no meta-analysis is performed.

That is, the possibility of a meta analysis was raised. The main issue is getting all the statistics for the tests from skeptical organizations made readily available to any interested party. If an organization has a test as its main draw, the least it should do IMO is make the data from the tests more easily available.

because...

You have repeated your request many times. What you have not done is justify it. Suppose you get the table of data. What purpose does it serve?

 

[T'ai Chi]

Which does not answer my question--what is the purpose of asking the question? You get an answer--ok, what does it tell you?

You're asking, for example, what the number of tests done per year tells you? It tells you the number of tests done per year. You're asking what does how well the dowsers performed tell you? It tells you how well the dowsers who were tested performed.

Anything I can do to clarify.

Ok, so your contention is that putting them all in one table is not "combining them". I think it is, and I advised against it, even if no meta-analysis is performed.

So this is where your confusion lies Mercutio. A list of data from many single experiments is not the same as combining data from experiments. The assumptions behind your "advising" are therefore flawed, and are safely dismissed.

Suppose you get the table of data. What purpose does it serve?

A question one should be asking instead, is what purpose do tests serve if one has such a hard time seeing the statistics from them.

 

[Mercutio]

You're asking, for example, what the number of tests done per year tells you? It tells you the number of tests done per year. You're asking what does how well the dowsers performed tell you? It tells you how well the dowsers who were tested performed.

Thank you.

So it tells you nothing. It has no use outside describing a self-selected sample. It is useless.

Anything I can do to clarify.

Except actually clarify. But you have answered sufficiently; if you actually did have a use for it, you'd have told it.

So this is where your confusion lies Mercutio. A list of data from many single experiments is not the same as combining data from experiments. The assumptions behind your "advising" are therefore flawed, and are safely dismissed.

Nice try, but you got it backward. My advising was based on the mere fact of listing the data. That you do not call that "combining" is utterly irrelevant, although it does speak to your understanding. You do not call this listing "combining", and therefore do not see the inherent problem. You are quite simply wrong.

A question one should be asking instead, is what purpose do tests serve if one has such a hard time seeing the statistics from them.

The challenge tests have already served their purpose. They have tested the claims of the claimants. They have no obligation to suit your purpose.

Seriously, TC, I am finding it harder and harder to believe that you are not understanding. Do you have an expert statistician that you trust, to ask to explain it to you? Failing that, could you elaborate on why you do not think that the data are not appropriate for further use? Perhaps if you do, I will be able to identify where you are mistaken. At this point, I am convinced that you are indeed mistaken--I am willing, even eager, to be proven wrong, but it seems excruciatingly clear to me that you are quite simply wrong, yet for some reason unable or unwilling to see this.

 

[T'ai Chi]

So it tells you nothing. It has no use outside describing a self-selected sample. It is useless.

What other descriptive statistics do you find "useless"?

If one is interested in skepticism, and especially the tests done by skeptical organizations, I'd think the data from their tests would be useful for similar descriptive reasons, so one can better understand claimant characteristics, experimental design, statistics in skepticism, and interpreting data, among others.

They have tested the claims of the claimants.

That is trivially true. What is also true, is that from some tests there are statistics. Can these be made more easily available?

They have no obligation to suit your purpose.

Strawman alert. No one is saying anyone has any obligation.

Seriously, TC, I am finding it harder and harder to believe that you are not understanding.

You are apparently finding a lot of things hard on this thread: arguing bias from a hypothetical non-realistic situation where optional stopping and testing the claimaint based on their expectation of performance are done, thinking the listing of single experiments is combining, and so on.

At this point, I am convinced that you are indeed mistaken--I am willing, even eager, to be proven wrong, but it seems excruciatingly clear to me that you are quite simply wrong, yet for some reason unable or unwilling to see this.

You want me to prove your opinon that my opinion is wrong, wrong?

The argument being presented is that it would be useful for skeptical organizations to make data from tests more easily available. I'm curious to understand why you believe one can be "wrong" about that opinion.

How many tests were done per year? What % of tests were on dowsing? What is the closest that someone has gotten to passing a test? How does this vary for different skeptical organizations? Broadly classified, what is the most common experimental design of these tests? Etc. I'd like to understand skepticism better, especially the results from tests done by skeptical organizations. You know what they say, if it cannot be expressed in number, it is knowledge of a meager type, or something like that.

It might work for you to know that dowsing is a claim that has been tested a lot. Others would like an actual number to really understand the issue.

 

[Mercutio]

What other descriptive statistics do you find "useless"?

Oooooh, you got me! They are not "useless"; they describe a small, self-selected sample. A sample which has already been examined, in terms of the question the sample was obtained to examine. They are only useless if we attempt to do anything else with them, like answer some of the questions your website suggests we will answer in looking at this sample.

If one is interested in skepticism, and especially the tests done by skeptical organizations, I'd think the data from their tests would be useful for similar descriptive reasons, so one can better understand claimant characteristics, experimental design, statistics in skepticism, and interpreting data, among others.

"useful for similar descriptive reasons"...are you admitting that they have no inferential use at all? And of course, you are wrong about better understanding claimant characteristics, too, at least in any practical sense. Suppose you find that there is a particular gender composition; you can make no statement about whether that composition is typical for such samples, whether it is typical for the population, about the reasons for the gender composition in this case, whether the gender composition might be an artifact of the challenge process or not...you are in possession of a trivial fact with absolutely no practical utility. You call this "useful", but other than repeating your assertion that it is, you do not explain how.

That is trivially true. What is also true, is that from some tests there are statistics. Can these be made more easily available?

It is not trivial; it was the purpose of the data. What is trivially true is that these descriptive data describe the sample.

Strawman alert. No one is saying anyone has any obligation.

Oh, good. We can safely ignore your requests.

You are apparently finding a lot of things hard on this thread: arguing bias from a hypothetical non-realistic situation where optional stopping and testing the claimaint based on their expectation of performance are done, thinking the listing of single experiments is combining, and so on.

I have explained my reasoning at every point, or at least I think I have. Please feel free to show where I am mistaken. Please feel free to show your own reasoning, and what use the descriptive statistics would serve. I have no problems admitting I am baffled by that.

You want me to prove your opinon that my opinion is wrong, wrong?

Again, I believe that I have supported my opinion. If I have not, please feel free to show where I have failed.

The argument being presented is that it would be useful for skeptical organizations to make data from tests more easily available. I'm curious to understand why you believe one can be "wrong" about that opinion.

Did you not read my earlier posts today? At 12:37 (EST), I explained this. Do you contend that making misleading data available is a good thing? Perhaps you have more trust in the innate statistical capabilities of untrained individuals to take into account the inherent problems of self-selection (among others) into account.

How many tests were done per year? What % of tests were on dowsing? What is the closest that someone has gotten to passing a test? How does this vary for different skeptical organizations? Broadly classified, what is the most common experimental design of these tests? Etc. I'd like to understand skepticism better, especially the results from tests done by skeptical organizations. You know what they say, if it cannot be expressed in number, it is knowledge of a meager type, or something like that.

Suppose you had answers to each of these. What would each tell you? Would it tell you why X many tests were done? Whether dowsers were preponderant (for argument's sake) because they had most applications, or because the tests were easier, or because they were the ones who came to a mutual understanding with the testers, or some other reason? Without experimenter manipulation, what does the correlative data about which organization tests what tell you?

If you would like to understand skepticism better, this is a terrible way to start! If you wish to answer a given question, first find out what sort of data you would need in order to do that! These data simply do not tell you anything about the questions you claim to be asking! Please....find somebody you trust who knows about methodology, to explain it to you, if you do not want to listen to me.

It might work for you to know that dowsing is a claim that has been tested a lot. Others would like an actual number to really understand the issue.

Tell me what the actual number would tell you. What would that number mean? (this is not a rhetorical question. What is it that you really think you could glean from this number?)

 

[T'ai Chi]

A sample which has already been examined, in terms of the question the sample was obtained to examine. They are only useless if we attempt to do anything else with them, like answer some of the questions your website suggests we will answer in looking at this sample.

Why do you believe they are useless? I don't find them useless. For example, the tests are done for the year 2005, but I'm interested in seeing a pie chart showing the % of type of tests for 2005. Maybe the trend in being tested for dowsing is down, up, or constant, from the years past, for example. Who knows.

You seem to want to pretend that once the test is over, one cannot get anything from the data other than 'test passed' or 'test failed', which is rather a limiting view.

"useful for similar descriptive reasons"...are you admitting that they have no inferential use at all?

Limited use for inferential statistics I'd say since the same is not randomly chosen, we don't know if they are representative. It is hard to say without seeing actual data and understanding the specific details of the tests. At this point I'm just interested in descriptive statistics of the tests done by skeptical organizations that are statistical in nature.

Suppose you find that there is a particular gender composition; you can make no statement about whether that composition is typical for such samples,

Understanding characteristics of the sample. I thought that was made clear when I talked about descriptive statistics for the sample.

Oh, good. We can safely ignore your requests.

You need to work harder at ignoring.

I have explained my reasoning at every point, or at least I think I have.

Instead of asking me to show where you are mistaken, you need to show where you are correct. Making a hypothetical example that is non-realistic (relies on option stopping and pretending that the observed data is compared to what the claimant expects) is not a good way to attempt to show you are correct.

Do you contend that making misleading data available is a good thing?

Why do you claim it is not misleading when listed singly, but is misleading when all listed together?

Suppose you had answers to each of these. What would each tell you? Would it tell you why X many tests were done?

You questiosn are akin to asking me what a person's age would tell me. Well, it would tell me how old they are. You could keep asking what that really would tell me, but it isn't a very rational way to argue.

These data simply do not tell you anything about the questions you claim to be asking!

I claim to be asking questions? Nope. I list the actual questions out, for example. But sure they do. They tell us about the sample, which is what is being asked about. How many times will that need to be repeated?

Tell me what the actual number would tell you.

Try telling me some actual numbers first. I understand you are having some trouble with that. My point is that you shouldn't be having that trouble.