Statistical scandals

[amhartley]

amhartley wrote: <<[Neyman-Pearson hypothesis testing] purports to be objective (celebrating its freedom from bayesian priors) but, because its testing procedures make no statements about hypotheses, the inferences (what you interpret as statements about hypotheses) it ends with are a subjectivist human construct.>>

Yes. Which is its strength. It clearly separates the objective statements from the subjective interpretations. As opposed to Bayesian, which lumps it all together in one package.

3 questions:
1. Why is it a strength that N-P inductive conclusions are subjectively invented, rather than objectively discovered? Isn’t the main point of science to learn about the world?

2. Most bayesian analyses report separately both the relatively “subjective” prior, and the relatively “objective” likelihood and/or sufficient statistics.

3. Even the relatively “objective” elements within N-P testing have subjective elements (beyond the subjective modeling choices this testing shares with bayesianism). To wit: the critical region is determined based on the experimenter’s intentions. E.g., in a series of Bernoulli trials, the critical region depends on whether the experimenter intends to stop after attaining a prespecified number of successes (negative binomial distribution), or to stop after performing a prespecified number of trials (binomial distribution). However, this is a little nit-picky & should not detract from discussion of #1 here.

 

[amhartley]

It was originally Athon that brought up [a one-in-five chance], and it sure seemed like he was talking about alpha.

What Athon said was “I can do a statistical test and make the claim 'I have evidence that x = y'. It means little until you know how I did this, and how much confidence I have in my results. When you then learn that I have a one in five chance of being wrong, it is up to you to determine whether my results are useful.” Evidence comes from data. So, when Athon says he has evidence that such-and-such, he is making post-experimental statements. As you said, alpha is meaningful pre-experimentally only. Therefore, Athon could not have been speaking of alpha.