JREF Challenge Statistics

[Mercutio]

Re: your undemonstrated claim that they are "biased". If they are, I'd suggest you talk to JREF since they're the ones designing biased tests according to you.

Strawman. I did not ever say that the tests were inappropriate for their intended use. It is the further use of these tests which is where a bias emerges. I have repeatedly said that; it is dishonest of you to characterise my statement as you do.

I guess you'd have to ask the people who look at such data in real life for specifics, since that data is collected in real life, and many others), and they cannot control many things either.

I asked a hypothetical question. It is intended to make a point. Your link shows totals by state for a number of categories. Of far more interest are some of the links from your linked page. The statistics there are gathered for the purpose of looking at driver safety. As such, the categories and reporting techniques are exactly what you would expect in order to achieve that aim. They do not present a situation analogous to my hypothetical.

But how about a simple example, for you- a number might hint that perhaps a change in the road, signals, or signs is in order.

"Might"? "Hint"? Yeah, that is pretty much the best you can do. The data on an unsafe intersection, if improperly collected, "might" seriously under-report the number of accidents. In comparison, then, another intersection "might" "hint" that it is worse than this first one, and get the sign or signal.

Your answer shows either that you do not understand sampling and experimental design, or that you are ignoring it for the sake of argument.

 

[Mercutio]

This

hints at it, but

(bold mine)

actually says it.

Gee...you don't link to my original post. Let's look at the words in context.

The data can answer these questions only about the small, self-selected sample itself. The data cannot tell us anything about the greater population, about which the questions are far more interesting.

You seem to be interested in individual results, but not a list of all the individual results. Who knows why.

Because that is all the data are good for. If I am interested in the questions you think are important, I would look for a sample which can answer them. This list cannot answer those questions.

You seem to be interested in asking questions of a database that is not designed to answer them, but not looking for a database that can. Who knows why.

What you "suspect" doesn't really matter.

Hey, I was only agreeing that my previous comment was a "No True Scotsman". Which would you prefer, that I assume you are interested, or that I assume you are not? It doesn't matter to me. If you are interested, there are better ways of answering the questions. If you are not, there is no reason to be asking them in the first place.

Ah...I see. You are taking my comments (or rather, you clip one sentence out of a larger comment) about this particular data set, and trying to make it look like I intended that comment to apply to all data sets. This goes back to your strawman from the previous post--I have said, and supported, that the tests are perfectly adequate for evaluating the claimants' claims, but are subject to a systematic bias when they are combined. This is what you have turned into a global statement of "Mercy seems to be under the illusion that data cannot be used for purposes other than under which they were collected." I am certain that now that your error has been pointed out, you will retract that statement.

And...in case you think, wrongly, that the combinational bias is the only problem, and wish to present the data for unspecified reasons in some large file, remember that the self-selection of subjects severely limits the utility of this sample. If you have genuine questions, there are better data sets to use. This one has done its job.

 

[T'ai Chi]

I did not ever say that the tests were inappropriate for their intended use. It is the further use of these tests which is where a bias emerges.

You have repeatedly said that, but you have not demonstrated that.

How could you? How could you possibly show that any other use of the data is biased as you are claiming?

And, again, you continue to ignore that I am suggesting the possibility of looking into a meta-analysis ("One could also consider doing some type of meta-analysis on the data if appropriate.") (underline "could", underline "consider", underline "possibility", and under "if appropriate" if that might help you understand) , among more important things like just seeing the actual data.

Why do you believe just seeing the actual data make bias magically emerge?

The statistics there are gathered for the purpose of looking at driver safety. As such, the categories and reporting techniques are exactly what you would expect in order to achieve that aim.

Um, so? So I showed a use of such data that you couldn't imagine any use for.

 

[T'ai Chi]

Let's look at the words in context.

It still doesn't change your erroneous belief that you repeat in your last post, that "..the further use of these tests which is where a bias emerges." That if the data is looked at other than intended, then it magically introduces bias. This is what you are desperately trying to argue.

This goes back to your strawman from the previous post--I have said, and supported, that the tests are perfectly adequate for evaluating the claimants' claims, but are subject to a systematic bias when they are combined.

What is your measure of this bias? Yes, numbers please.

And, yet again, you continue to ignore that I am suggesting the possibility of looking into a meta-analysis ("One could also consider doing some type of meta-analysis on the data if appropriate.") (underline "could", underline "consider", underline "possibility", and under "if appropriate" if that might help you understand) , among more important things like just seeing the actual data.

In other words, you keep droning on about combining, when I'm not even saying combining is the way to go. That is impossible to say without seeing actual data and getting a handle on specifics of the tests. Something which you are convinced you know everything about, sans data, apparently.

If you have genuine questions, there are better data sets to use. This one has done its job.

You've mentioned that already, but you still haven't shown that. If you are interested in other hypothetical data sets, then you are warmly encouraged to do some work and examine those. For those that want to examine data from tests from skeptical organizations, then examining data from tests from skeptical organizations is the way to go, Mercy.

 

[Mercutio]

You have repeatedly said that, but you have not demonstrated that.

Well, not to you, it seems. Others have understood it.

How could you? How could you possibly show that any other use of the data is biased as you are claiming?

I have not claimed that. I also have not claimed that this is the only problem. In my last post above, last paragraph, I mention the self-selection problem.

And, again, you continue to ignore that I am suggesting the possibility of looking into a meta-analysis ("One could also consider doing some type of meta-analysis on the data if appropriate.") (underline "could", underline "consider", underline "possibility", and under "if appropriate" if that might help you understand) , among more important things like just seeing the actual data.

And you are getting feedback on your suggestion. As for seeing the actual data, you have not given a reason to prefer a self-selected sample over a properly randomized one...if you can suggest a legitimate reason that this particular self-selected sample can answer a question you have, I would support it. So far, nothing.

Why do you believe just seeing the actual data make bias magically emerge?

The meta-analysis makes the first type of bias emerge. The self-selection of subjects results in the second type of bias. It does not have to emerge, it is there by the nature of the challenge. This is not a scientifically selected, random or representative sample. There is no requirement for it to be; it is the collection of people who accept the challenge. It is, necessarily, a bad sample to do descriptive statistics on.

Um, so? So I showed a use of such data that you couldn't imagine any use for.

No, you just think you did. Did you click on the links from the page you linked? The actual data were not collected by self-report; the people collecting the data had much more control over what was collected. These are not at all analogous samplings.

 

[Mercutio]

It still doesn't change your erroneous belief that you repeat in your last post, that "..the further use of these tests which is where a bias emerges." That if the data is looked at other than intended, then it magically introduces bias. This is what you are desperately trying to argue.

No desperation here; the only one who does not see it, it seems, is you.

What is your measure of this bias? Yes, numbers please.

Is this finally something you are interested in? Might I suggest that you do some research on it and get back to us?

I have presented the bias; one other here even suggested an analysis that might account for it. Everyone here but you, it seems, understands it.

And, yet again, you continue to ignore that I am suggesting the possibility of looking into a meta-analysis ("One could also consider doing some type of meta-analysis on the data if appropriate.") (underline "could", underline "consider", underline "possibility", and under "if appropriate" if that might help you understand) , among more important things like just seeing the actual data.

I believe you said this above, and I responded there.

In other words, you keep droning on about combining, when I'm not even saying combining is the way to go. That is impossible to say without seeing actual data and getting a handle on specifics of the tests. Something which you are convinced you know everything about, sans data, apparently.

No. I have mentioned two distinct reasons for not using the data. One addresses combining, but the other (the self-selection bias) does not.

You've mentioned that already, but you still haven't shown that. If you are interested in other hypothetical data sets, then you are warmly encouraged to do some work and examine those. For those that want to examine data from tests from skeptical organizations, then examining data from tests from skeptical organizations is the way to go, Mercy.

You have not mentioned any tests, any questions, for which this is the appropriate data base. As I said above, if you have a question for which this is the appropriate database, I would support it. I can't think of why you would want a self-selected sample to answer anything worthwhile, but if you can suggest something, go ahead. You have not done so yet. Oh, and "Mercy" really doesn't work, because of the different "c" sounds. I am sure if you use your imagination you can come up with something much better. Good luck.

 

[T'ai Chi]

Well, not to you, it seems. Others have understood it.

I'm only interested in your evidence (that you fail to show).

In my last post above, last paragraph, I mention the self-selection problem.

Again, that may be a problem if inference is in mind. As you've been told many times, inference is not in mind.

As for seeing the actual data, you have not given a reason to prefer a self-selected sample over a properly randomized one...if you can suggest a legitimate reason that this particular self-selected sample can answer a question you have, I would support it. So far, nothing.

In one of your early posts you ask the question of 'how many tests are statistical in nature?' That is one example of a question that could be answered. Are you now not interested in this question?

It is, necessarily, a bad sample to do descriptive statistics on.

Now that is being pretty dogmatic.

If inference were done, yes, that would be wrong. If descriptive statistics are done, that is not necessarily bad. In fact, descriptive statistics computed from such samples are done quite often.

I refer you to a book that you can check out and read (but which you probably won't and you'll ask me to read it to you): Sampling: Design and Analysis, Lohr, p. 8, second paragraph, titled What Good Are Samples with Selection Bias. She talks about the sample being valid unless the investigators wanted to generalize the estimates to a larger population - ie, they are doing descriptive and not inferential statistics (as I've been saying all along, eh?) She gives a specific example. Please, go read it.

The actual data were not collected by self-report;

Um, no kidding!, since in a fatality the people die, Mercutio. But all drivers are not assigned a number, and some selected to die and some not. That is what you are complaining about, the non-randomization, with your comment

As for seeing the actual data, you have not given a reason to prefer a self-selected sample over a properly randomized one...

Here is some more traffic fatality data:

http://www-fars.nhtsa.dot.gov/main.cfm
http://www.tsc.berkeley.edu/html/data.html
http://www.zzapp.org/rileygea/itsa/bkfat.htm
http://www.dot.wisconsin.gov/drivers/drivers/traffic/crash/fatality.htm

A lot of them look at trends even.

 

[T'ai Chi]

No desperation here; the only one who does not see it, it seems, is you.

You keep repeating your belief, but you haven't demonstrated it.

You have demonstrated you are incorrect when you say a such samples are "necessarily" bad. That is very, very, very, very, wrong.

I have presented the bias;

What is a numerical measurement of this bias?

I can't think of why you would want a self-selected sample to answer anything worthwhile,

Mercy, your lack of imagination is not a good argument.

 

[CFLarsen]

Good work, Merc.

 

[nathan]

Tai Chi, answer the question

Ah, you're back. Please answer my question. 'why have you not discovered the %of female applicants yourself from the Challenge Applications forum?' I asked you this in response #238 and response #273, yet you have not responded.

In response #218 you claimed

What % of the applicants have been female? Interesting question. Seems unnecessarily difficult to get a numeric answer.

I've told you where to get this information. Why are you incapable of determining this ratio for yourself from the challenge applications forum?

As I commented in #273, I gather you want to know the number of challenge tests per year. Why have you not determined this yourself, from the same forum?

 

[T'ai Chi]

Ah, you're back. Please answer my question. 'why have you not discovered the %of female applicants yourself from the Challenge Applications forum?' I asked you this in response #238 and response #273, yet you have not responded.

In response #218 you claimed

I've told you where to get this information. Why are you incapable of determining this ratio for yourself from the challenge applications forum?

As I commented in #273, I gather you want to know the number of challenge tests per year. Why have you not determined this yourself, from the same forum?

I'm sorry, how do you know what numbers I have and don't have?

As you've also been told, these numbers are not all the numbers we need to answer the question fully.

An obvious reason is that they don't cover multiple years, just one year.

Seeing trends from year to year could be desirable.

 

[Mercutio]

I'm only interested in your evidence (that you fail to show).

Again, that may be a problem if inference is in mind. As you've been told many times, inference is not in mind.

And as you have been told many times, that is not the only problem.

In one of your early posts you ask the question of 'how many tests are statistical in nature?' That is one example of a question that could be answered. Are you now not interested in this question?

That question applied to whether or not the data could be used for meta-analysis. If you are backing down from meta-analysis, what is your purpose for this information"

Now that is being pretty dogmatic.

If inference were done, yes, that would be wrong. If descriptive statistics are done, that is not necessarily bad. In fact, descriptive statistics computed from such samples are done quite often.

Yes, when people are interested in exploring that sample. These data have been collected with answering very specific questions in mind. We do not need to further describe these data; they have served their purpose. Unless and until you can come up with a legitimate question for descriptive analysis of these data, you are merely fishing.

I refer you to a book that you can check out and read (but which you probably won't and you'll ask me to read it to you): Sampling: Design and Analysis, Lohr, p. 8, second paragraph, titled What Good Are Samples with Selection Bias. She talks about the sample being valid unless the investigators wanted to generalize the estimates to a larger population - ie, they are doing descriptive and not inferential statistics (as I've been saying all along, eh?) She gives a specific example. Please, go read it.

I'll check if it is in our library. Meanwhile, it sounds like she is making exactly the point I have been. If you do not want to generalize the results to the larger population, you must have a question that is limited to that sample. So far, you have none. If you get numbers, you have no way at all of determining why those numbers are what they are, given that this is a self-selected sample.

Um, no kidding!, since in a fatality the people die, Mercutio. But all drivers are not assigned a number, and some selected to die and some not. That is what you are complaining about, the non-randomization, with your comment

A lot of them look at trends even.

Do people volunteer to be in car accidents?

 

[T'ai Chi]

That question applied to whether or not the data could be used for meta-analysis.

Which I said might be appropriate, might not. Without looking more at data, who knows. Just dismissing it outright is not very open-minded.

These data have been collected with answering very specific questions in mind.

How is a question of 'How many tests were done per year?' not specific for you?

Unless and until you can come up with a legitimate question for descriptive analysis of these data,

Again, your lack of imagination is of no relevance.

 

[CFLarsen]

I'm sorry, how do you know what numbers I have and don't have?

As you've also been told, these numbers are not all the numbers we need to answer the question fully.

An obvious reason is that they don't cover multiple years, just one year.

Seeing trends from year to year could be desirable.

Perhaps. But you have been told where to find the available data. You were interested in this, and you know now where to find it.

Yet, you slink away again.

 

[Mercutio]

You keep repeating your belief, but you haven't demonstrated it.

The things I keep repeating have worked, every time but one. Again, my lack of imagination may be a poor excuse, but so far I can't seem to come up with something that this last person can comprehend. I am sure it must be my fault.

You have demonstrated you are incorrect when you say a such samples are "necessarily" bad. That is very, very, very, very, wrong.

Context, dearie. You are arguing both sides of this one. I have not said they are necessarily bad as a global statement, and have already told you that this is your strawman. They are perfectly adequate for asking the questions that they were designed to ask. (remember, you told me to take it up with the skeptic's organizations? Remember, that was your strawman?) They are necessarily bad when we use them for inference. Your own source agrees with that. Until you have a research question that is limited to the sample, I must conclude you are using them for inference. Certainly, the way you talk about using the data on your website implies that you are using them for inference.

What is a numerical measurement of this bias?

Don't know. No real need to answer this question, when better samples exist to answer questions about the population. The test tube is dirty. Do we spend time looking at how dirty it is, or do we run our tests with a clean test tube?

Mercy, your lack of imagination is not a good argument.

It is not an argument. I have repeatedly asked you for examples. My ignorance of proper questions to answer with this dataset is simply that. You could cure it, and I have (as we often do when we are ignorant of something) asked someone whose actions suggest he might have some answers. So far it seems like you are as unable as I am to come up with legitimate reasons to look at those data.

 

[nathan]

I'm sorry, how do you know what numbers I have and don't have?

I'm making the assumption that you're capable of reading the challenge application forum and extracting the numbers from it. Perhaps that is beyond you.

As you've also been told, these numbers are not all the numbers we need to answer the question fully. An obvious reason is that they don't cover multiple years, just one year.

Pray tell me how finding the % of female applicants from the data available is not a desirable thing, if you want to know the % of female applicants? Obviously it will be from a subsampling of the entire cohort of applicants, but surely with your knowledge of statistics you could extrapolate that

Seeing trends from year to year could be desirable.

This is an additional question you did not pose to begin with. Methinks you just added this now to avoid doing the work of finding the answer yourself.

 

[Mercutio]

I'm sorry, how do you know what numbers I have and don't have?

As you've also been told, these numbers are not all the numbers we need to answer the question fully.

An obvious reason is that they don't cover multiple years, just one year.

Seeing trends from year to year could be desirable.

With a self-selected sample, what would the trend from year to year be able to tell you?

What would it be able to tell you that isn't better answered with a different, systematically collected, data set?

 

[Ripley Twenty-Nine]

T'ai Chi,

Would it be possible for you to post for us the following information concerning your post history? :

Date/Time of Post, Username used to Post with, JREF Board Posted in, Subject Category of Post, JREF User Replied To, and length (In characters) of post?

I think this would be very, very interesting information to have.

Thanks!

 

[CFLarsen]

I'm making the assumption that you're capable of reading the challenge application forum and extracting the numbers from it. Perhaps that is beyond you.

...

This is an additional question you did not pose to begin with. Methinks you just added this now to avoid doing the work of finding the answer yourself.

You answered it yourself: T'ai Chi wants other people to do the hard work for him.

He has done it before, when he wanted other people to find transcripts of psychic readings, only to make all sorts of tallying. Some of that tallying included gender - but no reference to year-by-year. Then, it was OK to merely count gender. Now, it isn't.

He is simply inventing a new excuse not to do any work himself. He would rather have other people do it for him, and then scold them, if they won't dance to his pipe.

Strangely enough, he has removed that "study" from his website, and has refused to explain why. Fortunately, I have a copy. Available on request.

 

[Gr8wight]

Please allow me to ask my question from several weeks ago one more time.

Because we don't have 20 dowsing experiments done simlarly. We have one dowsing experiment finding gold under a cup, one dowsing experiment finding addresses with a pendulum, one telepathy experiment sending thoughts to another person, one martial arts experiment attempting to stop an attacker without touching him... How do you combine those results?

By the way, can you tell me what element all of those tests had in common?

Justin (I won't play games by using your sock puppet name),

I don't mean to harp on this point, but it lies at the very basis of this discussion. Mercutio, CFLarsen, and others have tried to explain to you why data from many different tests cannot be combined into any meaningful meta analysis, but you ignore their knowledgeable attempts to help. My question is important because it involves the only piece of data that encompasses all of the preliminary tests. It can be stated as a simple percentage, which is easy for everyone - even you - to understand.

Have a go, won't you? If you can't come up with the answer just say so, and I'll explain it all using real easy words. I promise.