• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

JREF Challenge Statistics

T'ai Chi said:
You don't do a hypothesis test saying 'I'll set the criterion at 5, then I'll solve for alpha'.
Click to expand...

You're not supposed to (in "real" research), no.

In practice, it happens all the time, especially in parapsychology.

Just as a sample case, consider the claim of T.C. Albin, "California Weatherman," as documented by KRAMER here on this forum. Quoting briefly from the claim and acceptance:

Claim said:
I happily submit that it will snow July 27th of this year [2005] in Oakland. To be exact some time between 12:00 am and 11:59 pm on July 27th of this year it will snow in Oakland California.
Click to expand...

Acceptance said:
We accept your claim in which you state that you will "CAUSE IT TO SNOW IN OALKLAND CALIFORNIA on July 27th, 2005", providing that we agree that the Oakland, California you are citing here is the one across the bay from San Francisco, and that the snow will fall from the sky above, as if on a mid-winter Colorado Mountain ski slope, and that it be of meteorological origin, and NOT made artifically by some kind of snow-making machine. The snow must be as a result of the weather, and NOT man-made.
Click to expand...

The claim is clear, as is the ad hoc acceptance criterion. (Perhaps needless to say, it didn't snow in Oakland.) But what's the alpha cutoff of this test?
 
T'ai Chi said:
Why not 95:1? Why not 90:1? 75:1? Etc. This approach just replaces subjectively choosing alpha with subjectively choosing the prior odds. Different people choosing different priors, could lead to different results. Whereas setting alpha is pretty much standardized, therefore different people pretty much choose the same alpha of .05, .01, or .001.

If I ask a person to choose alpha, they'll typically say .05, .01, or .001. Ask a person to set prior odds, and see what range of numbers you get.
Click to expand...
A couple of points:

1) Why is it bad if different people get different results? If two people have different beliefs before seeing the results of some experiment, it makes perfect sense for them to have different beliefs after seeing the results. Why should a single experiment cause anyone to throw away entirely the lifetime of experience that led him to his prior beliefs?

2) It matters where the subjectivity is introduced. Otherwise, why do any statistical calculations at all? Why not just eyeball the experimental results, and subjectively decide what we think they indicate? We should introduce our prior beliefs where they belong: in the prior probability distribution, whose purpose is exactly to represent those beliefs. Then, from that point on, everything can be calculated objectively.

It might not always be easy to decide whether we believe that the odds are, for example, 95:1 or 75:1, but we certainly can't expect a statistics handbook to decide for us. They're our beliefs, after all. If we don't know what we believe before an experiment is done, the results of the experiment can't tell us that. They can only tell us how we ought to change whatever prior beliefs we may have had.
 
Nice post, 'dodge...can I steal some of your language for my stats classes?
 
69dodge said:
1) Why is it bad if different people get different results?
Click to expand...

Not sure anyone said it is bad.

It might not always be easy to decide whether we believe that the odds are, for example, 95:1 or 75:1, but we certainly can't expect a statistics handbook to decide for us.
Click to expand...

What analysis do you choose to accept when your priors lead to an opposite conclusion of someone who used different priors?
 
T'ai Chi said:
Not sure anyone said it is bad.
Click to expand...
I thought you mentioned it as an argument against Bayesian analysis. If you don't consider it to be an argument against Bayesian analysis, I'm glad, because I don't believe it is one. But then I don't understand why you did mention it.

T'ai Chi said:
What analysis do you choose to accept when your priors lead to an opposite conclusion of someone who used different priors?
Click to expand...
Mine, of course. And the someone should accept his.
 
69dodge said:
I thought you mentioned it as an argument against Bayesian analysis. If you don't consider it to be an argument against Bayesian analysis, I'm glad, because I don't believe it is one. But then I don't understand why you did mention it.
Click to expand...

No, I don't think it is bad if people reach different conclusions.

It is somewhat confusing if people analyze the same data, but end up reaching different conclusions because their priors were different.
 
T'ai Chi said:
No, I don't think it is bad if people reach different conclusions.

It is somewhat confusing if people analyze the same data, but end up reaching different conclusions because their priors were different.
Click to expand...
Exactly. Man, you nailed it. Cut straight to the core.

That's why we need to know why you think it is "reasonable" to set alpha to what you did.

Do you have an answer?
 
Sorry to butt in here but from someone for who maths makes my head hurt :p

Could someone help me, I've interpreted this and that other thread and from what I can gather the differences are;

1) We try to work out the best odds we can, before the experiment, taking as many factors as we can into account

2) We run the experiments, then we work out the odds. If we want to we can massage the odds due to hind sight. If peer reviewed stuff uses this method it will play it safe because peer reviewers are happy to pull it apart
 
Here again is the webpage of what I've been thinking of lately

http://www.statisticool.com/jrefchallengestats.htm

This applies not only to the JREF preliminary tests, but also similar tests by other skeptical organizations.

It would be nice if we could finally learn needed experimental tallies, p-values, & other stuff, to get an idea of if the scores are in line with what one would expect by change.
 
T'ai Chi said:
Here again is the webpage of what I've been thinking of lately

http://www.statisticool.com/jrefchallengestats.htm

This applies not only to the JREF preliminary tests, but also similar tests by other skeptical organizations.

It would be nice if we could finally learn needed experimental tallies, p-values, & other stuff, to get an idea of if the scores are in line with what one would expect by change.
Click to expand...
I see that you don't set alpha. Why not?
 
Originally Posted by T'ai Chi

Here again is the webpage of what I've been thinking of lately

http://www.statisticool.com/jrefchallengestats.htm

This applies not only to the JREF preliminary tests, but also similar tests by other skeptical organizations.

It would be nice if we could finally learn needed experimental tallies, p-values, & other stuff, to get an idea of if the scores are in line with what one would expect by change.
Click to expand...

Cluas wrote

I see that you don't set alpha. Why not?
Click to expand...

It is something that JREF and other skeptical organizations set prior to the experiment.

Oh, I just noticed, I wrote

It would be nice if we could finally learn needed experimental tallies, p-values, & other stuff, to get an idea of if the scores are in line with what one would expect by change.
Click to expand...

or "cfl next post". And then Claus replied the next post.

I must be psychic. ;)
 
T'ai Chi said:
or "cfl next post". And then Claus replied the next post.

I must be psychic. ;)
Click to expand...
Or you can count. I am in the top ten posters, and CFL has more than twice my posts. In threads like this, he is even more present. In this thread, only you and one other poster have more posts than CFL, you can safely learn to hold your own tongue, and drkitten has posted mainly in the earlier pages.

Of course, what this means is that your naive bayesian analysis told you that this was an event with a high prior probability. Once again, you agree with a more formalized mathematical approach.


:D
 
Odds are you're right, Behaviorman.
 

Attachments

  • behaviorman.jpg
    behaviorman.jpg
    63.2 KB · Views: 23
Last edited:
T'ai Chi said:
It is something that JREF and other skeptical organizations set prior to the experiment.
Click to expand...

But that's different from the alpha you set. Which leads us back to your own value. "Reasonable", yes.

So, why is that "reasonable", if JREF and others set it differently? Do you know something they don't? Something that they should know?

T'ai Chi said:
Oh, I just noticed, I wrote

or "cfl next post". And then Claus replied the next post.

I must be psychic. ;)
Click to expand...

Only you left out the ampersand, meaning "and", as well as choosing letters out of sequence. Your pathetic attempt of Bible-code post-hoc reasoning is pure woo: Leaving out what doesn't fit.

Let's see if we can use the same methods to find a universal truth:

T'ai Chi said:
It would be nice if we could finally learn needed experimental tallies, p-values, & other stuff, to get an idea of if the scores are in line with what one would expect by change.
Click to expand...

"Tai sucx."

Gee, I'm a goddamn oracle!!
 
T'ai Chi,

Why is your own alpha value "reasonable", if JREF and others set it differently? Do you know something they don't? Something that they should know?
 
You must log in or register to reply here.

Back
Top Bottom