Odd. That doesn't match my experience doing research with actual biologists at all. Just who are you talking about here?
Thanks for a refreshing perspective & potentially helpful post. Yes, eventually we need to get around to what to do about all this. After a few years of talking with people about these issues, though, I’ve come to believe that people are not open to solutions until they are aware of the severity of the problems. Status quo is just way too comfortable.
Like I was saying, there is a temptation to minimize the weaknesses of p. In the easiest cases, at least, p is a combination of estimated effect size (delta), precision (1/sigma) and sample size
. If you know p, sigma-hat & n, you can often work backwards to estimate delta. However, this process reminds me of the 2 guys in the helicopter counting caribou:amen to all of that, except that the bayesian can in fact develop an exact Pr(Ho|data). This prob just may not apply to someone else with different priors.
My only reservation here is that I doubt our ability to assimilate all of that info together and come up with an optimal inference or decision. Even when you consider only p and delta, there’s no assurance of optimality.
If you can get people to consider subjective bayesianism, I’m behind you.
Yes, excellent paper. Or at least thought provoking. As I mentioned above, though, it’s like alchemy to try to come up with inferences about hypotheses when all one has are probabilities about data assuming hypotheses, even (or especially?) if one adds Cohen’s d or Max Like Estimators & whatnot to the mix.
Please be more specific. When I don't understand something I'm eager to know about it. And I think the sweeping statements are justified. Truly yours, The Topic Starter.
Accurate explanation. Another way to see the difference is in the case where A and B are just propositions (discarding the idea of p-values for the moment). Then, we have (if, as Art denies, it makes sense to assign probabilities to propositions like hypotheses):
That's interesting; I hadn't heard of that paradox.
Sorry to lose you. Even tho your "justification" of 0.05 'cuz, as I mentioned at other places, Pr(data given hypothesis) can differ widely from Pr(hypothesis given data), and not only in what some call "extreme cases."
I need to correct a typo:
No. According to your formulation, P(A|B)=P(A&B)/P(B). That is defined regardless of whether we assume B. The probability of getting a two when you roll a fair die is well-defined regardless of whether you actually roll a fair die.
Why are you using hypothesis testing if you don’t approve of it?
It would be even easier if you just told me.
That’s not a shortcoming of the method, but its presentation.
Nope. We’re saying what happens when we assume it.
If I say “If A then B, B is true, therefore A is definitely true”, then that’s affirming the consequent, and it’s a fallacy. If I say “If A then B, B is true, that supports A”, that’s not a fallacy.
Math “assume”: if… then
It’s not the middle word that I have a problem with. How is this a “guess”?
Confidence is about the person. Probability is about random variables.
I see no example.
A properly constructed rejection region consists of situations which would decrease our confidence. Therefore, if the rejection region contains situations that would increase our confidence, it’s not properly constructed.
Okay. The alternative hypothesis should include all situations other than the null hypothesis. If Ho says mu=0, then H1 should say mu!=0. That is, it should include the possibility that mu=.5. Am I going to be famous for pointing that out?
You don’t explain how. I’m not going to respond to your cryptic Jeb/Bush stuff.
Then explain what you meant by it.
I’ve never said it does.
Pot, kettle.
Mass is.
You said that it can be different. I’m just asking you how.
Parameters are sometimes known in statistics, I was using the term in the general mathematical sense anyway, and .05 is not a feature of hypothesis testing.
Except that you haven't shown how this is a difficulty.
Just try to say “P(A|B),” i.e., “the probability of A assuming B,” without using the word “assuming” (and it's no fair saying “given” instead). Sorry if that appears frivolous; I’m just at a loss to understand you. Applying what you say below: to calculate the probability of getting a 2 when rolling a fair die, we “see what happens when/if”, even contrafactually, a fair die is being rolled. You & I are probably saying the same thing here.
You had described a testing situation in rough terms and, to be able to discuss it, one needs more precise language. I was filling in that language but wasn’t sure exactly the scenario you had in mind.
Example: As an experiment, Oakes gave a short quiz to 70 psychology academics. Only 3 people interpreted the p-value correctly. The majority of respondents interpreted the p-value as the post-experimental probability of the tested hypothesis.
The average textbook or professor defines statistical inference as “extending sample results to a larger population,” sets up a correspondence between hypotheses & populations, and then says p-values, conf intervals, and hypothesis test results are inferential. How, then, can you hope to avoid the misunderstanding that these outcomes are statements about hypotheses? How would you change the presentation?
That seems right; I see no difference.
First, to nitpick: If A then B (A=>B), then B supports A only when B is not a tautology.
Okay.
It’s a guess in that the Neyman-Pearson H testing methodology specifies no rational way of choosing alpha. That is why people choose alpha depending on thumb-rules (such as Fisher’s 0.05) or the conventions of communities of scientists, rather than on logical/mathematical laws. In simple cases, one can reverse-engineer H testing from a bayesian standpoint to choose alpha, but that’s another story. . .
Interesting. Does confidence obey any laws, or can we assign (feel?) any degree of confidence we wish? Is it quantitative? If so, is it bounded?
I was referring to the cases, noted lower in that post, in which x supports Ho relative to H1, even though p<0.05.
Maybe it is useless to discuss this until, in the discussion a few lines back, we sort out how confidence differs from probability. In the meantime, I am just wondering how, if (as you say) confidence is “about the person,” that it is also “in a hypothesis.”
Sorry, there must be a typo. What did you mean by “mu!=0?” I hope you didn't send that to the publisher.
Not a republican, eh?
“examine” could mean “put out in the open” at least, or even better, “evaluate.”
You stated “a one in five chance of being wrong.” Someone is wrong iff s/he’s concluded a hypothesis, & the hypothesis s/he’s concluded is false, so now the hypothesis has a probability of 4 in 5. Therefore, you’re attributing a probability to a hypothesis. Unless you want to distinguish between probability and chance, in which case we’d have 3 concepts to keep distinct: probability, chance, and confidence. This is getting complicated!
I said that the inductive meanings people derive from deductive statistical outcomes are products of their minds, or of arbitrary conventions. Viz., they’re fabricated rather than discovered. You asked, “how that can be different.” If, as you seem to suggest, fabricated meaning and objective meaning are the same, then I suppose there’s no such thing as objective truth.
The focus was on alpha, which is the primary design probability of Neyman-Pearson hypothesis testing. Alpha is not just a feature, it's the pre-eminent feature.
So what am I allowed to say? Am I allowed to say “if”? “When”? “Evaluated at”?
I’ve described a simple hypothetical. Put aside your issues regarding testing, and just answer it directly. I’m not talking about testing, I’m just talking about inferences.
Well, they are statements about the hypothesis. They are not statements about the probability of the hypothesis.
I do. “A implies B” means “If we assume A, then we must assume B”. It is true whether we assume A or not.
The full statement is that if 0<P(A),P(B)<1, and P(B|A)>P(B), then P(A|B)>P(A).
That makes no sense. B supports A regardless of whether C does.
”Guess” refers to issues of fact, not choices in general.
”Confidence” is a general term. “Bayesian Confidence” is a type of confidence, and it does follow rules.
Except that that does not do what you claimed it does, nor what you are now claiming. Earlier you said it’s neutral, but now you’re saying it supports Ho.
Probability is what’s studied in probability theory. Confidence is not.
If I say that I trust someone, am I saying something about that other person? Or myself? Or both?
These assumptions are in the open. And what would evaluating them entail?
Or if he rejects the hypothesis, and the hypothesis is correct.
That doesn’t follow.
No, the focus was on the specific value of .05, and that specific value is not a feature of hypothesis testing.
Or you could have a situation where P(Ho)=1 but p=.001. Such a situation will happen only .1% of the time. If you’re willing to have it happen that frequently, then you should set alpha at .001.
It’s already impossible.
Which in turn confused me, since I expected it to mean "if I recall correctly".
Confidence intervals? Those are even worse. If 3 out of 70 don't understand what alphas is, I doubt 3 out of 140 know what "confidence interval" means. Answer: very little. A confidence really doesn't say anything about the true value, yet people think that it does.003998 said:
Any of those options indicate you’re assuming, at least provisionally, in the “mathematical” sense you identified in another post, that H is true. It doesn’t mean that you believe H, or that you’re committing yourself to act as if H. The assumption is purely for argument’s sake.
Before I can address that, we have foundational questions (raised above) to answer about confidence & its difference, if any, from probability. The only way I know to talk about statistical inference involves probabilities of hypotheses. Probability as inference obeys laws & can be discussed; confidence (in your as-yet undefined sense) as inference I don’t know how to discuss meaningfully.
Well, when no data either demonstrate H nor falsify H, we can’t say whether H. So, if we can’t even speak of Pr(H given data), then what can we say?
If, as I believe you hold, statistical inference involves assigning confidence to hypotheses (or is it to people?), then does confidence get included in this “precise mathematical formulation?”
Good point.
The degree of statistical support by B of A is only in relation to the degree B supports C. But I’m seeing now that this, too, depends on unanswered questions about “support.” Because I believe statistical “support,” or statistical evidence, in favor of H, is anything that increases Pr(H), I can talk meaningfully about support. But you, with your emphasis on “confidence,” may have something completely different in mind.
Any rational choice depends on (imperfect) knowledge of issues of fact. E.g., if I have to choose whether to invest in Stock A, I will gather & use as much information as possible about whether Stock A will appreciate or depreciate. If I have no knowledge about those issues of fact, as in the choice of alpha, I am guessing, rather than choosing rationally.
I ask again: Does your confidence in general follow rules? Or only bayesian confidence?
That was a different example. Art, we need to focus on foundational issues first. What is confidence, does it obey rules, etc.
You’ve said more about what confidence is not than what it is.
Good question; I think it leads in a potentially productive direction.
True. Either way, once some one has concluded a state (true or false) a hypothesis, they are wrong if the other state (false or true) is true about the hypothesis. So, the probability they are wrong is a probability of the other state.
Well, I guess we were not communicating, because I was talking about alpha.
You didn’t comment on the more important part of what I was saying in “For instance, in many applications, p=0.03 and Pr(Ho|data)=.20, say, could be realized given the same data. Mistaking p for Pr(Ho|data) leads here to an unjustified strength of belief that Ho is false. That strength of belief could, in turn, support a decision to bet 7 to 1 against Ho, whereas the max justified bet against Ho would be 4 to 1.” Maybe that is because you doubt probabilities apply to hypotheses at all (even tho you said here “P(Ho)=1”, but perhaps just to participate in the discussion)?
It’s impossible “to say whether someone is right for interpreting inductively a p-value (say) in such and such a way?” I’m surprised you would say that, and I wonder about the reason. Is that because you believe there are no norms for induction, or the only norm is complete freedom, or what?
We can saythat we are more willing to accept (orreject) Ho.
No.
Don't those ideas contradict each other? Whether A increases P(B) is independent of whether C increases P(B).
Which stock to pick is a choice. The claim that one will do better than the other is a guess. Picking a stock is not a guess. It's based on guesses, but it's not a guess.
Confidence in general does not follow rules. Bayesian confidence, as well as other types, do.
This is a common misconception of hypothesis testing. Alpha should be calculated before the test is conducted, before any data is collected, and before any conclusion is reached. As you say, once one has reached a conclusion, it makes little sense to talk about the probability of it being true unless you are assigning that probability to the hypothesis itself.
What do you think is the important part? What position does it establish?
It is inherently a subjective process, and any objectivity is illusury.
You stated “a one in five chance of being wrong.” If you were referring to alpha in that, you didn’t say so. I, like most scientists outside statistics, was and remain concerned about inference, i.e., “post-experimentally, what can we say about unknown states of nature, based on observed data?” Apparently, you were talking about the deductive pre-experimental probability assuming H, i.e., “pre-experimentally, assuming H, what is the chance I’ll reject H?”
Yes, that is the supreme irony in frequentist testing as “inference:” It purports to be objective (celebrating its freedom from bayesian priors) but, because its testing procedures make no statements about hypotheses, the inferences (what you interpret as statements about hypotheses) it ends with are a subjectivist human construct. Neyman-Pearson hypothesis test inferences are fabricated, rather than discovered (in N-P's defense, they explicity excluded inference from what their testing can do).
It was originally Athon that brought it up, and it sure seemed like he was talking about alpha.
It is a common misconception that alpha is based on the data, rather than the test. Once you have completed the test, the result of the test is no longer a random variable.
Yes. Which is its strength. It clearly separates the objective statements from the subjective interpretations. As opposed to Bayesian, which lumps it all together in one package.