Why has 0.05 been chosen as an acceptable p value?

[Vikram]

Maybe statisticians could answer this question for me.

A p value of 0.05 means that 1 in every 20 tests will, out of sheer randomness, yield a postitive result. This '1 in 20' principle seems to confer quite a high probability of success through chance. If one were to conduct a single test each on a 100 woo-woo beliefs, one would end up with the conclusion that 5 of them are supported by statistically significant data. Of course, they would not be replicable and all, but might it not be better to just make p values a little more stringent - perhaps 0.005 or even lower?

Any explanation as to why a relatively 'common' probability like 0.05 has been chosen as the watershed line for statistical significance?

 

[Zep]

If one were to conduct a single test each on a 100 woo-woo beliefs, one would end up with the conclusion that 5 of them are supported by statistically significant data.

Only if there was a single test for each. In reality, there would be a significant number of tests on each one to ensure that the results were consistent enough to be reliable, and THEN the p-value would be applied.

 

[Vitnir]

A compromise maybe, if you want a lower p-value you might end up with a study that is 10 times more expensive and you would have wasted that money since your study is unnecessary accurate.

Edit: Plus there are other limits. 0.01 is often used and you can sometimes see 0.1.

 

[T'ai Chi]

A p value of 0.05 means that 1 in every 20 tests will, out of sheer randomness, yield a postitive result.

You are talking about alpha here, not the p-value. In any case, we'll mistakengly reject the null hypothesis when the null hypothesis is true, 1 out of 20 times, in the long-run (it doesn't say what will happen with any specific 20 experiments), yes.

Of course, they would not be replicable and all, but might it not be better to just make p values a little more stringent - perhaps 0.005 or even lower?

Well, they are in many cases; .01, .001, and so on. It is unwise to blindly decide that p-values less than .05 are significant without making the decision of what constitutes significant p-values based on the subject matter being investigated. You are certainly right; I'd want to set alpha much lower in the case of controversial medical claims, for example.

Also keep in mind that there is a trade-off: if we decrease alpha, we increase beta (the probability of not rejecting the null hypothesis when the null hypothesis is false!).

Any explanation as to why a relatively 'common' probability like 0.05 has been chosen as the watershed line for statistical significance?

There are two main reasons that are my opinion for the p less than .05 sticking:

a) If a well-known and persuasive statistician, like Ronald Fisher, for example, introduced and/or popularized it, it could just "stick" by dogma (no matter how practical), and

b) For Z being from a standard normal distribution (which is one of the most common test statistics) one gets P(Z less than 1.96) + P(Z greater than 1.96) ~= .05, and 1.96 is so close to 2, that it makes for easy calculations in a time pre-computers, and easy for back-of-envelope calculations in any time period.

 

[BobK]

Here's an answer from a guy that knows the field.
Found here.

The provenance of the P<0.05 criterion goes back to the great pioneer of significance testing, R A Fisher, who is deemed to have given it his imprimatur. He did not in fact do this and late in his life stated that he had just used this level in his calculations as a “mathematical convenience”. Furthermore, he also stated that “without randomisation there is no significance”.

Here's a link to some other questions you might have.FAQ

I really enjoy this site. You may want to investigate the whole site.

JREF at one point had a link to his site. Don't know if they still have it tho'.

Bob

 

[SezMe]

Re: Re: Why has 0.05 been chosen as an acceptable p value?

Only if there was a single test for each. In reality, there would be a significant number of tests on each one to ensure that the results were consistent enough to be reliable, and THEN the p-value would be applied.

How do you know that "In reality" this would be the case.

Or, to quote Claus, "evidence please."

 

[Zep]

Re: Re: Re: Why has 0.05 been chosen as an acceptable p value?

How do you know that "In reality" this would be the case.

Or, to quote Claus, "evidence please."

Fair enough. Perhaps I should have phrased it more as "If the researcher is serious...".

 

[Vikram]

Zep,
True. Replicability would, of course, eliminate all problems.

Vitnir,
Hmm, I agree. It would be best to use a higher p value for 'screening' tests. If there's a positive result, one could always repeat it with a lower p value.

jzs,
Thanks for the explanation. It makes perfect sense.

BobK,
The link you posted has been officially added to my Favorites list.

*****

Looking at the 'p' issue from the point of view of the JREF challenge, since replicability is not a requirement (apart from the preliminary test and the actual test itself) and a million dollars are at stake, what p value might be suitable?

 

[Zep]

Looking at the 'p' issue from the point of view of the JREF challenge, since replicability is not a requirement (apart from the preliminary test and the actual test itself) and a million dollars are at stake, what p value might be suitable?

That would depend on the test, and the claimant, and the tester.

For example, a claim of levitation has a simple binary outcome - you levitate or you do not. There are probably simple ways to determine if this is fraud (and a stagecraft person might know some wrinkles, and thus how to detect them). So the test would probably be that the claimant actually levitates and passes the "success criteria". p-values shouldn't really enter into that, should it?

An example of where p-values WOULD be used is testing of dowsing. And this has actually been done a number of times. I suggest you look at the Australian Skeptics website www.skeptics.com.au, to see the results of same.

 

[Vitnir]

That Brignell from Number Watch seem rather biased himself when it comes to judging science. I just read his entry on smoking and fertility and I don't understand what he is ranting about. If you want to blame someone for creating panics about medical health findings point the finger to the journalists instead. Most of his points seem to have a sound basis somewhere but it seems to me that he is taking everything one step too far when he draws his final conclusions.

 

[CFLarsen]

Re: Re: Re: Why has 0.05 been chosen as an acceptable p value?

Or, to quote Claus, "evidence please."

I rarely say "please"....

 

[Mercutio]

I tell my students that .05 is the number that most people in our field use, simply because most people in our field use it. It is not magic, and as (Tukey, I think, certainly not Fischer) said, "surely god loves .06 almost as much as .05."

In theory (but the cynical part of me wants to add "but rarely in practice"), the alpha level is chosen after due consideration of the relative costs of type I and type II error (false alarms and misses, respectively). If misses are expensive (money, lives, etc.), you choose a fairly high (or more probable) alpha. If false alarms are expensive (same yadda yadda yadda), you set a more stringent (less probable) alpha level.

 

[olaf/QII]

--here we have a case where the p is less than .0001

--and the study has been reproduced many times, yet Darat the harrasser attacks me.


1999

"A total of 3674 datapoints were collected from the 4 labora-tories, of which 840 were invalid at the 0.1 % level of risk.Combining all data in the presence or absence of hista-mine dilutions, the overall effect was highly significant(p < 0.0001). However, statistical analysis demonstrated thatthe lowest anti-IgE concentration (0.04 mg/ml) was the bestfor the observation of inhibition. At this concentration, therewere 772 valid data points. Using the GLM procedure, the overall effect of all hista-mine dilutions (15th– 19thcentesimal dilutions) was signifi-cant (p = 0.0001, F = 40.07) (Table 1). The normality of thedistribution of the basophil counts was verified using Henriplots for the control but not for the histamine-dilution treatedcells (data not shown). Non-parametric analyses (Kruskal-Walis test) confirmed the inhibitory effect of histamine (p < 0.0001). "

http://216.239.39.104/search?q=cach...berfroid+M,+Sainte-Laudy+J,+Wiegant+FA.&hl=en


2004

2004 study on BASOPHILS

Full paper ----- HTML version --- 2004

http://216.239.41.104/search?q=cach...df+marotta+marini+banaudha+maharaj+ives&hl=en



Full paper ---- PDF version -- 2004

http://www.amhmg.org/Histamin2004.pdf

RESULTS ----------- P < .OOO1

 

[exarch]

It's been far too long since I took statistics, but what the heck are they doing here?

They have 3674 datapoints from 4 laboratories (how many from each laboratory?), of which they immediately dump 840 (22.86% or more than 1/5th of their results) as invalid for some reason. Then they cherrypick 772 datapoints (27.24%) from the remaining 2834 (apparently because they showed the most good results?) and conclude from that that the odds of having a good result among the selection of good results is p < 0.0001 ...



Could someone explain this here? What am I missing?

 

[Dragon]

From an earlier thread -

Originally posted by MRC_Hans

Could somebody give the olaf a push, it seems to be stuck in a groove?

 

[Number Six]

I was taught that one reason 0.05 is the magic number is that that was the level for which the tables were made. Back then people couldn't just grind out a slew of numbers in five minutes on a computer. People had to actually toil to get these numbers. And so once they had a table of numbers for particular alpha levels they tended to stick with them.

The more general problem though is that of turning a continuim into a dichotomy. The p-value can go from 0 to 1. Picking any number (be it 0.05 or something else) and using it as a black/white determintor of significance is too simplistic. There is almost no difference at all between 0.049 and 0.051 and yet one is "Significant" and the other "Non-Significant."

 

[olaf/QII]

It's been far too long since I took statistics, but what the heck are they doing here?

They have 3674 datapoints from 4 laboratories (how many from each laboratory?), of which they immediately dump 840 (22.86% or more than 1/5th of their results) as invalid for some reason. Then they cherrypick 772 datapoints (27.24%) from the remaining 2834 (apparently because they showed the most good results?) and conclude from that that the odds of having a good result among the selection of good results is p < 0.0001 ...



Could someone explain this here? What am I missing?

What you are missing is the common sense that should allow your brain to accept that there is something going on with respect to ultra-dilute solutions and bioactivity.

 

[Pantastic]

Originally posted by
What you are missing is the common sense that should allow your brain to accept that there is something going on with respect to ultra-dilute solutions and bioactivity.

If that's true, come forward and claim your million dollars.

Hello?

Anyone?

 

[T'ai Chi]

Back then people couldn't just grind out a slew of numbers in five minutes on a computer. People had to actually toil to get these numbers.

I am reading a great book (again) called The Lady Tasting Tea, by Salsburg. At one point in the book, speaking about R.A. Fisher, he mentions that Fisher used the Millionaire for his calculations.

Using the example from the book, if Fisher wanted to miltiply 3342 by 27, he put the platen on the units position, put in the number 3342, and hand cranked seven times. Then he put the platen on the tens position, put in the number 3342, and hand cranked two times. For large numbers this could obviously get very tedius! It always boggles my mind to think what an Archimedes, a Gauss, an Euler, or a Fisher could do if they had a TI-92, a copy of Mathematica, or Maple.

Salsburg estimates that Fisher needed about 185 hours of work to generate just a single table (in Fisher's "Studies in Crop Variation. I."), and estimates it took Fisher at least 8 months of 12hr days to prepare the tables for that article (which he says probably does not include the hours needed to work out the theoretical mathematics, to organize the data, to plan the analysis, and to correct any mistakes that might come up).

Yikes!

 

[Vikram]

I am reading a great book (again) called The Lady Tasting Tea, by Salsburg. At one point in the book, speaking about R.A. Fisher, he mentions that Fisher used the Millionaire for his calculations.

Using the example from the book, if Fisher wanted to miltiply 3342 by 27, he put the platen on the units position, put in the number 3342, and hand cranked seven times. Then he put the platen on the tens position, put in the number 3342, and hand cranked two times. For large numbers this could obviously get very tedius! It always boggles my mind to think what an Archimedes, a Gauss, an Euler, or a Fisher could do if they had a TI-92, a copy of Mathematica, or Maple.

Salsburg estimates that Fisher needed about 185 hours of work to generate just a single table (in Fisher's "Studies in Crop Variation. I."), and estimates it took Fisher at least 8 months of 12hr days to prepare the tables for that article (which he says probably does not include the hours needed to work out the theoretical mathematics, to organize the data, to plan the analysis, and to correct any mistakes that might come up).

Yikes!

Kinda reminds me how much I take my desktop computer for granted.

I hereby offer thanks to the Lord for this here pentium...