(Another) random / coin tossing thread

[Lance Manion]

Well, it's either heads or tails, so 50-50, obviously!

Oh yeah? I meant the bag contains 2,049 pennies, smart guy.

 

[69dodge]

Sorry - I don't understand your example. What's the null hypothesis here?

Doesn't matter. Pick your favorite null hypothesis.

Yes, it's really as silly as it sounds. That's the point.

 

[69dodge]

In my example, I've already done that, and any statistican always would. If I've noticed that the bulbs seem to fail in clusters, I'll use a test based on a measure of clustering, and reject the null hypothesis if there is more clustering that plausibly seen in the Poisson distribution.

Yes, I agree. You have already considered likely alternatives, and any statistician always would. But only implicitly, and all the while insisting that he hasn't because he can't.

I don't really understand this: I'm taking the Bayesian point of view to mean choosing a prior somehow and applying Bayes Theorem. I'm not interested in what probabilities mean. To apply Bayes Theorem, you need a distribution on the degree of faultiness of the power supply. I agree it's reasonable to guess a probability that the supply if faulty, but to apply Bayes Theorem, your prior needs to be specific enough to calculate the probability of seeing a certain degree of clustering in the failures given that there is a fault in the supply. I can't imagine how you would come up with such a prior. The whole point of hypothesis testing is that you don't need to - it's good to have an idea how likely a fault is, because this affects what significance level you should choose, but there's no need to think about modelling the degree of faultiness until you've established that you have any evidence at all that anything is wrong.

Hypothesis testing lets you pretend that you don't need to. But implicit in any particular test is such a prior. If, were you explicitly shown that prior, you'd agree that it more or less reflects what you know about the power supply, then by all means accept the result of the significance test. If not, you shouldn't consider the test to have established anything, because it was based on faulty assumptions.

 

[69dodge]

Well, it's either heads or tails, so 50-50, obviously!

 

[Meridian]

Hypothesis testing lets you pretend that you don't need to. But implicit in any particular test is such a prior. If, were you explicitly shown that prior, you'd agree that it more or less reflects what you know about the power supply, then by all means accept the result of the significance test. If not, you shouldn't consider the test to have established anything, because it was based on faulty assumptions.

You don't need to if you aren't going to answer the question `what is the probability that there's a fault'. Hypothesis testing is something to do when you can't answer this question, which is almost always. I agree that something is implicit in a sensible use of hypothesis testing, but that something is much less than a prior. It's simply an idea of what kind of deviation to look for, and roughly how plausible it's presence is. That's a far cry from enough data to calculate the conditional probability of a certain deviation given a fault.

 

[sol invictus]

Hmm. Well, rotating doesn't change anything for B - it's simply off in total either way. But I agree that (if rotating is reasonable, which I don't think it always will be, but never mind) you could view A as simply predicting 4 parameters and saying nothing about the last.

Rotating changes the situation (for both A and B, although you might need to make a different rotation for each) into one which is identical to your first case.

Secondly, I think with your setup, Bayes gives the answer you don't want: viewing A as a single theory that only predicts 4 parameters, what prior probabilities are you assigning to A and B? If they are roughly equal, Bayes will tell you that given the observations, A is 99.9999% certain.

No! As we already discussed (and you agreed), with 5 sigma and w ~ 10^-10, B will be favored. Your second model is completely identical to your first in that regard.

I think you're still missing the point. The intuition scientists have (at least the ones that have thought about this) is that theories should be penalized for having new parameters, but ultimately fitting to data should win. There have been various ad hoc formulae proposed which encapsulate that intuition. Thinking about your examples made me realize that Bayes gives us one such formula, and a rather good one - it penalizes you for the extra parameter by an amount that's power law in the uncertainty (a rule which seems to fit rather nicely with my intuition for what to do in such situations), but because confidence grows exponentially with standard deviation, data can overcome that and will always win in the end.

That would certainly affect the significance level that I'd use in the test. But I don't at all see how to use it to construct a prior distribution on the degree of clustering of failures.

So you'd lower the required significance level? What if you had a third hypothesis that had nothing to do with power fluctuations (a batch of bad bulbs, say)? Why should that one suddenly become harder to rule out?

It makes no sense at all - you must use the knowledge from the newspaper article to adjust a prior, or you are ignoring an incredibly relevant piece of data. Science would never have progressed if we thought that way.

No - in fact I'd say your example supports it! If you allow that C might be correct, you presumably also allow variations of C in which the fractions of A and B vary. So this argument says that any prior distribution is ok. To obtain a conlcusion by Bayesian reasoning you need to decide on one, and it still seems to me totally clear that there is no reasonable way of doing this for two Theories of Everything.

In the examples I have in mind, the fractions of A and B do not vary.