Puzzled By Statistics

[sol invictus]

If we had a different question, maybe it would be easier to see why the sequence is less important.

It's not less important in any intrinsic or fundamental sense, and I think that's part of what's bothering the OP. Take the example I gave above - suppose you flipped 500 heads in a row and then 500 tails in a row. If you only use the mean, that looks just as probable as any other sequence with the same totals (i.e. quite probable). But no reasonable person would conclude it was random.

What's really going on is that we have certain ideas about what types of patterns are likely, or even possible, in a series of coin flips. Most of those possibilities affect the mean but not the sequence. Therefore we mainly consider theories in which the probability of the data depends only on the mean. Mathematically speaking, that's perfectly fine - we can pick any theory we want and use Bayes to evaluate its probability. But which theories we choose to consider is totally up to us, and there is no mathematical framework to guide us or tell us that the mean is in any way special - that requires some intuition about coins.

 

[GreedyAlgorithm]

Can you work through the questions about the coins and show why 990/1000 heads is strange, whereas a sequence of H and T that's 1 in 2¹⁰⁰⁰ is not strange?

But how do I know which is the right statistic to look at? I just do, it makes sense. But why? Why is that the statistic I need to look at? How do I know except for common sense?

Ultimately, physics.

In this case, physics from the get-go. We know things about how physics works, and these things must be factored into probability calculations. It's very easy to come up with mechanisms that would allow something with the initial appearance of a "fair coin" to come up heads 990/1000 times. It's very difficult to come up with mechanisms that would allow something with the initial appearance of a "fair coin" to always come up HHTHTTTHHHHTHTTHHTHHHHHHHHHTTHTHTTTTHHTTTTTHHHHHHHTTHTTHTHTHHTHHHH..."

Probability theory is not so good at justifying models. Given a model, it's easy to show you any of the math you like. But your question seems to be "how do I pick the hypotheses to test?" or possibly the subsequent "how do I justify picking this model to produce my results?" The answer to the first is "that's how genius is born". Perhaps a Bayesian super-intelligence will be able to pick out interesting hypotheses from the gigantically huge space of possible theories without a problem, but we have some trouble doing it. Occam's Razor helps (and may even be fundamental!).

The answer to the second? I'm pretty sure that a fully specified hypothesis is the model. Run the tests, see how well your model performs, see how well you did at picking that model from the set of all possible models. We've found through experience that models like "each sequence of H,T produced by many tosses of the coin is progressively less likely to occur in proportion to our ability to guess the next toss given the previous tosses compared to our ability to guess the next toss without knowing the previous tosses" (i.e. independence) work very, very well, so those are the ones we propose.

Check out Solomonoff induction and maybe Kolmogorov complexity for some ideas on how we might go about putting better priors on models.

 

[Ivor the Engineer]

1) Encode H -> +1, T -> -1.

2) Perform Haar wavelet transform.

3) Compare output against know fair coins.

 

[sphenisc]

..

 

[Elaedith]

I never did statistics past A Level, which has worked fine so far. However, I now find myself scratching my little head.

Suppose I have a hypothesis that a coin is fair. I toss it a thousand times, and 990 times it comes down heads. I can work out the probabilty of that, and say: now, this would only happen one time in (some large number I can't be bothered to calculate) if the hypothesis is correct, therefore the hypothesis only stands those odds of being correct.

Now, this seems to me to be fair and reasonable.

Thanks.

I don't think it is. The probability of getting any given outcome if a hypothesis is true is surely not the same as the probability that the hypothesis is true?

 

[Herzblut]

There must be something somewhere which explains how one's definition of "improbable" relates to the testing of a statistical hypothesis: a formal and well-founded way to tell how significant to a statistical hypothesis an improbable event really is. All events are improbable to some degree, depending on how you do the counting. What is the right way to do the counting, and how is this demonstrated?

Thanks.

To analyze the randomness of a bit stream statistically, you might use the NIST test suite:

http://csrc.nist.gov/groups/ST/toolkit/rng/stats_tests.html

 

[Ivor the Engineer]

I don't think it is. The probability of getting any given outcome if a hypothesis is true is surely not the same as the probability that the hypothesis is true?

Correct. Formally:

P(H|D) =/= P(D|H)

Frequentists calculate the RHS. Bayesians use:

P(H|D) = P(D|H).P(H)/P(D)

to attempt to calculate what we really want, which is the probability the hypothesis is true given the data. The problem is P(H), the prior probability of the hypothesis, is often (always?) subjective.

 

[Herzblut]

Correct. Formally:

P(H|D) =/= P(D|H)

Frequentists calculate the RHS. Bayesians use:

P(H|D) = P(D|H).P(H)/P(D)

to attempt to calculate what we really want, which is the probability the hypothesis is true given the data. The problem is P(H), the prior probability of the hypothesis, is often (always?) subjective.

Yeah. I like this conditional probability fallacy. Especially in legal arguments (prosecutor's fallacy), nicely described in

http://en.wikipedia.org/wiki/Prosecutor's_fallacy

Consider this case: you win the lottery jackpot. You are then charged with having cheated, for instance with having bribed lottery officials. At the trial, the prosecutor points out that winning the lottery without cheating is extremely unlikely, and that therefore your being innocent must be comparably unlikely.

Look! If you win the jackpot you must be a criminal defrauder!

 

[technoextreme]

It's not less important in any intrinsic or fundamental sense, and I think that's part of what's bothering the OP. Take the example I gave above - suppose you flipped 500 heads in a row and then 500 tails in a row. If you only use the mean, that looks just as probable as any other sequence with the same totals (i.e. quite probable). But no reasonable person would conclude it was random.

Actually thats wrong because the mean can correspond to an event that will with 100% certainty never ever ever ever ever happen in real life. Typically you only get that in discrete situations. In this case the mean does not correspond to any event. If heads is one and tails is zero and if you are assuming that they have equal chances of occurring the mean is .5 which corresponds to an event that will never happen. Though forgive me on that part I was confused about what he was asking about.

Probability theory is not so good at justifying models. Given a model, it's easy to show you any of the math you like. But your question seems to be "how do I pick the hypotheses to test?" or possibly the subsequent "how do I justify picking this model to produce my results?" The answer to the first is "that's how genius is born". Perhaps a Bayesian super-intelligence will be able to pick out interesting hypotheses from the gigantically huge space of possible theories without a problem, but we have some trouble doing it. Occam's Razor helps (and may even be fundamental!).

And this is why he picked a dumb example. He picked an example where there is a mathematical theory as to what the results will look like.

 

[sol invictus]

Actually thats wrong because the mean can correspond to an event that will with 100% certainty never ever ever ever ever happen in real life. Typically you only get that in discrete situations. In this case the mean does not correspond to any event. If heads is one and tails is zero and if you are assuming that they have equal chances of occurring the mean is .5 which corresponds to an event that will never happen. Though forgive me on that part I was confused about what he was asking about.

I don't know what you're talking about. The mean is the mean - it's not supposed to represent a single event, and no one that understood what it was would ever think otherwise.

The OP's question, at least in part, was why one should focus on the mean rather than some other statistic. There is no fundamental answer, other than that it's likely to be the sensible thing to do in the case of a series of coin flips.

And this is why he picked a dumb example. He picked an example where there is a mathematical theory as to what the results will look like.

There's nothing dumb about his example. The question he asked is perfectly valid, and the example is good.

 

[blutoski]

My impression is that Dr. Adequate is asking why, if all outcomes are equally unlikely, we can say that some particular outcome is virtually impossible.

eg: tossing a coin 10 times:

A: TTTTTTTTTT Odds: 1/1024
B: HHHHHHHHHH Odds: 1/1024
C: THTHTHTHTH Odds: 1/1024
D: HTHTHTHTHT Odds: 1/1024
E: THTTHTTHHT Odds: 1/1024

so, we test a guy's telekinetic powers, and he produces a coin series like this:

F: THTHTTHTHH Odds: 1/1024

Dr. Adequate is asking why don't we say that this guy's got supernatural powers?

It took me awhile to figure out why:

Answer: because he didn't call it in advance.

If we reverse a bit, and see how these tests play out, there is some sort of claim involved. If I claim I can force coins to flip heads, we're looking at one and only one outcome's probability. All the rest could be explained by the power not working, because they have at least one tails in them.

The claim is the one outcome: HHHHHHHHHH Odds 1/1024; "not-claim" is the set of all other outcomes Odds 1023/1024.

 

[Zeuzzz]

I wish I had chosen statistics. Theres something really deep about this question of probablities that I cant put my finger on, and is very hard to articulate. Its like the monty hall problem, and is related to statistics and Bayes various theorems. Proabably why so many of histories great mathematicians and physicists go mad, like Boltzmann, kurt godel, and many others. I wonder what they discovered. Actually, I'm glad I didn't choose statistics

 

[sol invictus]

Answer: because he didn't call it in advance.

Yes, that's what everyone else has said too.

You must specify a theory (or hypothesis, same thing). Then you can use the data to compare it to another theory (via Bayes). If your theory is that the coin is fair, all sequences are equally probable. If your theory is that the coin is weighted, some sequences are much more probable than others. Therefore, given some data, you can distinguish between the two theories.

But the math doesn't tell you which theories to consider - that's your job.

 

[balrog666]

Every sequence of 1000 tosses has exactly the same probability of 2^1000.

But any specific sequence, chosen from the space of all potential sequences of length 1000, is only related to the group (or family) of sequences that indicates a fair or unfair coin (or even uncertain, to some confidence level) through the probability computations of the number of heads/tails, which has no relation to the sequence itself.

So, even both the 2 and the 1000 are used in the probability computations, the 2^1000 is simply the answer to a different question than fair/unfair.

 

[69dodge]

There must be something somewhere which explains how one's definition of "improbable" relates to the testing of a statistical hypothesis: a formal and well-founded way to tell how significant to a statistical hypothesis an improbable event really is.

It depends on which other hypotheses you have in mind, with which you intend to replace this one (if it is rejected) as being reasonable to believe. So, the fact that everything is very improbable is not a problem, because what's relevant is not the "absolute" or "inherent" probability of an experimental result. Rather, what matters is how much more likely it is on one hypothesis than on another, it being evidence in favor of those on which it is more likely, and against those on which it is less likely.

Can you work through the questions about the coins and show why 990/1000 heads is strange, whereas a sequence of H and T that's 1 in 2¹⁰⁰⁰ is not strange?

The important question is not whether something strange happened---as you've noted, everything is strange in some sense---the important question is what you should conclude from the happening. A particular sequence of coin flips containing 500 each of H and T has the very low probability of 1/2¹⁰⁰⁰ if the coin is fair, but it has an even lower probability if the coin is biased, so naturally its occurence can't be a reason to prefer the hypothesis of bias over that of fairness. On the other hand, a particular sequence containing 990 H's and 10 T's has a higher probability if the coin is biased towards H than if the coin is fair; therefore, its occurence is evidence against fairness and for bias.

 

[lenny]

I don't think there's any rigorous way to answer that deeper question lurking behind what you asked - how we know when a sequence is really random, or instead has some underlying order. In fact I think it's formally undecidable.

yep. a finite sequence cannot be classified as random (or nonrandom).

and in practice, experimentalists have nothing else.

 

[lenny]

It depends on which other hypotheses you have in mind, with which you intend to replace this one (if it is rejected) as being reasonable to believe. So, the fact that everything is very improbable is not a problem, because what's relevant is not the "absolute" or "inherent" probability of an experimental result. Rather, what matters is how much more likely it is on one hypothesis than on another, it being evidence in favor of those on which it is more likely, and against those on which it is less likely.

so are you saying relative likelihood is "all" that matters? surely H1 may be 2^64 times more likely than H", given the data. but how is this of any interest if the probability of the data is vanishing small given H1?

in the coin case, of course, this is not an issue. in real life decision support, however...

 

[lenny]

I don't know what you're talking about. The mean is the mean - it's not supposed to represent a single event, and no one that understood what it was would ever think otherwise.

perhaps technoextreme meant the expectation (not the mean), the expectation need not correspond to any realisation of the process, and may be physically ridiculous. frightening then how often it is plotted in forecasting.

one need never expect the expectation.

 

[Dunstan]

one need never expect the expectation.

No one expects the expectation! Our chief weapon is...

no?

Well, it was worth a shot.

 

[69dodge]

so are you saying relative likelihood is "all" that matters?

No. The prior probabilities of the hypotheses matter too. I was being somewhat informal. I figured that this would get the idea across better than just saying "use Bayes's theorem", which is the more formal version.

surely H1 may be 2^64 times more likely than H", given the data. but how is this of any interest if the probability of the data is vanishing small given H1?

I don't understand what you're trying to say here.

We already have the data. It happened, however improbable we may think it was. Our job now is just to evaluate various competing hypotheses, based on it (and on our previous opinion of those hypotheses). Yes?