An Impossible Coin Flip?

[drkitten]

But back to the interpretation. Suppose the Eagles win. What does that tell us? It tells us that they are the team in the Super Bowl, but other than that, not much. For example, it doesn't tell us that the Eagles were more likely to win. Shoot, as far as we know, it could easily be that the Falcons would win 85% of the time on a neutral field. Playing in Philadelphia, it might be 65%. But events that happen 35% of the time are not uncommon (no one is surprised by a die coming up either 5 or 6, when 1 - 4 would be twice as likely to happen). Thus, if the Eagles win, we don't know if that means that they were 100% locks to win (1 - 6 on a die), or whether they were a 35% underdog on a die that happened to roll of 5 or a 6.

Although true, this doesn't help us for a variety of reasons -- basically because it answers the wrong question (and does so more or less from the opening sentence).

To start with, there's a fundamental difference between probability and statistics that you're glossing over. Probability is about the future, while statistics is about the past. Now, it's certainly true that the past is a good predictor of the future (often the best one that we have), and for this reason a good statistical model of what has happened can often be modified to a probabilistic model of what will happen. And this is essentially how the frequentists define probability. The probability now is the same number that we will give then after we run an infinite series of experiments.

And in the case of coin flips, I have no problem with seeing how that definition applies, even though I can't actually run the infinitely extended experiment.

But I'm not interested in a coin flip. I don't care about interpreting an outcome. I'm interested in the outcome itself, the outcome of an event that will happen once, and I'm interested not in figuring out why it happened (after the fact), but what will happen, beforehand. I don't want to know why McNabb won, but whether or not he will. And there's even a very simple empirical way to evaluate whether or not the assessed probability is any good -- I want to have more money in my pocket on Monday than I do today.

Which is why I'm a Bayesian. Because the frequentist interpretation doesn't tell me why I should believe someone's assessed probability. There are no meaningful statistics regarding this game, as it hasn't happened yet. On the other hand, there is "evidence" that can be interpreted in a Bayesian framework to yield a "credible-degree-of-belief" (which is one of the usual interpretations of Bayesian formalism). Now, that I understand and can work with.....

 

[Number Six]

And I think the effect of variation is underestimated by people. The NFL only has a 17 game season. But even on top of that, for any particular game a key player is injured. And schedule matters a lot. And even the way record is measured counts. Every week you get a win or a loss (discounting the small probability of a tie.) There is no gray area. If you play a tough opponent on the road and lose by 2 points it counts the same as if you play an easy opponent at home and lose by 20 points...both are 1 loss. And on and on. There are just too many factors to conclude definitely that better record = better team.

As far as Vick this weekend goes, I wasn't thinking cold per se but rather was thinking that a sloppy field negates speed and that lousy weather equates with a better chance of their being a sloppy field. So if that holds then it counts against Vick. But then again, crappy fields usually help teams with better running games and Atlanta's offense is mostly run. Maybe that means Dunn and Duckett will have big days, especially Duckett since it may be a grind it out kinda day and he's a big grunt rather than a small, quick guy.

I learned about Michael Vick's passing the hard way when I took Peerless Price early in my fantasy football draft last year. Price became a big star with the Bills. Then he signed a free agent contract with Atlanta, whose QB was Vick, and completely disappeared. He put up lousy numbers again this year too. But not coincidentally, Atlanta's tight end flourished this year. That, as well as simply watching Atlanta play now and then, led me to conclude Vick can't pass downfield very well. If you watch the game this weekend take note of how on target his downfield passes are. (I'm not talking 5 yards downfield, I mean 15 yards downfield.)

 

[pgwenthold]

Although true, this doesn't help us for a variety of reasons -- basically because it answers the wrong question (and does so more or less from the opening sentence).

it's not the wrong question. It is a different question, and it is a question of function. It's a question of using observations (what you are calling statistics) to create a predictive model. One approach to do that is to do it from a probabilistic standpoint, using correlations of past situations to assign probabilities. As others have said about weather forecasts, it's like saying that a 70% chance of rain means that "on 70% of the days where the atmospheric conditions are like this, it has rained."

To start with, there's a fundamental difference between probability and statistics that you're glossing over. Probability is about the future, while statistics is about the past. Now, it's certainly true that the past is a good predictor of the future (often the best one that we have), and for this reason a good statistical model of what has happened can often be modified to a probabilistic model of what will happen. And this is essentially how the frequentists define probability. The probability now is the same number that we will give then after we run an infinite series of experiments.

Not exactly. The probability now is the same number we _would_ get if we were to run an infinite series of identical experiments.

Note that means that we ran them all at the same time, under the same conditions. The fact that it is physically impossible is irrelevent. If we could do it, the answer we would get is the probability.

But I'm not interested in a coin flip. I don't care about interpreting an outcome.

I am interested in the outcome, to the extent that it helps me predict the outcome of the next game. The two concepts are not unrelated.

I'm interested in the outcome itself, the outcome of an event that will happen once, and I'm interested not in figuring out why it happened (after the fact), but what will happen, beforehand. I don't want to know why McNabb won, but whether or not he will. And there's even a very simple empirical way to evaluate whether or not the assessed probability is any good -- I want to have more money in my pocket on Monday than I do today.

And I can and have done this. Empirical assessment of these things is readily straightforward, and can be done all the time. There are classes in meterology dedicated to the empirical assessment of forecasts.

Another example, I have created a ranking system for NCAA women's volleyball based on this exact approach, that matches are probabilistic. I am constantly assessing the accuracy of the predictions that I make, and I can tell you how it performs. For example, when I apply a ranking to the set of data that exists, it correctly describes about 85 - 86% of the match outcomes. Thus, 85 - 86% of the time, it has the team that won ranked higher. Now, when I turn that around and look forward, the accuracy in those matches will drop to about 83%. There is definate dropoff, for a variety of reasons, but it is not large. This is why I always talk about _estimating_ probabilities for upcoming matches. We can't _know_ the exact probabilities, but we can hopefully come close. However, there is clearly uncertainties in the ratings that get assigned, and that leads to uncertainties in the probabilities.

Another thing to note, the model is not necessarily a one-to-one map. For example, if I wanted to, I could create a system that did a lot better than 85 - 86% for matches that have already been played (I can beat 90% regularly if I wanted). However, that ranking generally only gets about 78% right or so for upcoming matches. Therefore, I prefer the more predictive model. It's the difference between having a system that accurately describes the past, or is more likely to predict the future.

 

[pgwenthold]

And I think the effect of variation is underestimated by people.

This is generally true all over. Read some of the stuff by Gilovich on the concept of "hot hand" to see how people are too wont to create patterns out of randomness.

Gilovich's book "How We Know What Isn't So" is a good read in this area.

 

[LostAngeles]

Re: Re: An Impossible Coin Flip?

Is that a typo or am I missing something? Seems to me it's quite impossible to get 1,030 anythings with only 1000 flips.

I think it was supposed to come off the median and if my linear approximation is right, it's probably 32.25 rather than 30 if you want the square root of 1000. So 532.25 and 467.75. (Calculators are for wusses!)

And um, proabability isn't certainty, it's likelihood and there's no such thing as a sure bet. So yay! I contributed!

 

[69dodge]

Re: Re: Re: Re: An Impossible Coin Flip?

Originally posted by jzs
I guess I don't see a need to choose. Both ways of thinking are good at solving different problems.

I'm curious though; do you view a probability distribution as the limit of a frequency histogram?

Some problems are solved most easily by thinking about them in a way that might be described as "frequentist". But the Bayesian approach seems to me more fundamental, because one can derive frequentist-type results from it but not vice versa.

For example, a large number of independent, identically distributed random variables will probably form a frequency histogram that is close to what you expect, but I view that fact as following from a Bayesian interpretation of probability distributions and of independence, and I interpret "probably form" also in a Bayesian way.

On the other hand, I do not see how it is at all useful to define a probability distribution as the limit of a frequency histogram because, first, no finite number of past observations gives us any information about what the limit might be, and second, even if we somehow knew what the limit was, that would still tell us nothing about any finite number of future observations.

When someone says that a flipped coin has probability 1/2 of showing heads, he does not just mean that, in an infinite number of flips, half will be heads. He also means, for example, that in 100 flips, getting 90 heads is very improbable---improbable, that is, in the Bayesian sense of the word. How could one derive that from a purely frequentist approach, which does not even have the vocabulary to say it?

 

[Jeff Corey]

Pardon me, is a "frequentist" someone who counts outcomes to determine probability?
I frequently have trouble with undefined words.

 

[69dodge]

Originally posted by Jeff Corey
is a "frequentist" someone who counts outcomes to determine probability?

I'd say it's someone whose definition of probability does not permit any other way to determine a probability than by counting outcomes. (In particular, it's someone who deems too vague and imprecise the Bayesian view that probability describes the strength of one's belief about something after taking into account all available information.)

The problem with such a frequentist definition is that it doesn't permit determining a probability by counting outcomes either. How many outcomes shall we count?

I just flipped a quarter three times. It came up tails all three times. Does that mean the probability is 1 that it will come up tails on the next flip? Why not?

 

[RussDill]

I just flipped a quarter three times. It came up tails all three times. Does that mean the probability is 1 that it will come up tails on the next flip? Why not?

Your confidence interval on it hitting tails is very low, even without knowing the mechanics of a coin toss.

 

[T'ai Chi]

There's a very big one if you're dealing with non-reproducible events. I believe that McNabb and Vick are due to meet in a few days for a playoff game. They're not going to play an infinite number of games (or even two games), but I need to know what the probability of Vick winning so that I know whether to accept the 2-1 one of my friends has offered me.

If the probability of Vick winning is better than 33.3%, then it's a good bet. But there's no way we can assess that as a limit process.

Good point. But how would you rationally assign an objective probability to that event?

I'd look at the win/loss statistics of the teams when those two players faced each other.

 

[T'ai Chi]

Now, it's certainly true that the past is a good predictor of the future (often the best one that we have), and for this reason a good statistical model of what has happened can often be modified to a probabilistic model of what will happen.

Exactly. This has proven itself to be sensible in practice.

 

[T'ai Chi]

A strict definition of a frequentist is someone who limits probability to mean only a long-run frequency.

I just flipped a quarter three times. It came up tails all three times. Does that mean the probability is 1 that it will come up tails on the next flip? Why not?

It may. 3 flips isn't a long-run frequency, it is the short run. Keep going to infinity..

For that situation, say we are interested in estimating P, the probability of tails, by p. We can use a Wilson estimate to move the observed probabilty away from 1 (and, using the same reasoning if all heads came up, away from 0) to reflect a more realistic estimate.

where x is the number of tails and n the number of flips,

p = x/n

The Wilson estimate is p_w = (x+2)/(n+4)

 

[T'ai Chi]

Re: Re: Re: Re: Re: An Impossible Coin Flip?

Some problems are solved most easily by thinking about them in a way that might be described as "frequentist". But the Bayesian approach seems to me more fundamental, because one can derive frequentist-type results from it but not vice versa.

In Bayes's original work, he talked about putting a distribution on the parameter theta. He, however, talked about theta as being generated from an auxiliary physical experiment; throwing a ball on a table (ie. keeping track of the frequency!).

However, right after this, he wrote

'the same rule (Bayes theorem) is a proper one to be used in the case of an event concerning the probability of which we absolutely know nothing antecedently to any trial made concerning it'

That is basically an argument from ignorance. He was saying that although we don't know anything about theta, it is equally probable to be between 0 and 1. This non-objectively chosen theta can be a real serious issue sometimes. If, on the other hand, it is chosen as the results from previous studies, then it is coming from a frequency, just like throwing a ball on a table.

For large n, the likelihood tends to dominate the prior anyway, so the choice of prior in these cases is negligible. Also, frequently the results of Bayesian and frequentist approaches converge for large n, which is nice!

 

[Art Vandelay]

The short answer is that you can calculate the probability that you will get 2 million "successes".

That answer completely misses the point.

 

[DevilsAdvocate]

This is a quote from Leonard Susskind (theoretical physicist):

This sounds interesting but what actually is Susskind trying to say? Is he saying that it could not possibly happen (even though he can't prove it)? Surely there is some chance that it could happen?

Does anyone have any comments?

I think Susskind is playing a bit of a rascal.

He's saying that the only thing that you can know absolutely is "pure knowledge" - knowledge based on reason and logic that is independent of empirical observation. So the joke is that you can't prove anything (absolutely) that you claim to know with an empirical demonstration. Why not? Because all things depend on chance.

You can claim to know that if you drop a book it will drop to the floor. But what if it doesn't? What if the Earth's gravitation gets messed up by something unknown to you that causes the book not to drop? Obviously there is some very small chance that what you claim to absolutely know could be false. You could've only proven your claim, if you knew about the weird something that would disrupt gravity--and everything else that may affect the result. So the other little joke is that you can't really claim to know anything unless you know everything.


So Susskind is saying that "proving something" (with empirical observation) is different from "knowing something" (by logical reason). But they are related. Logical reason is based on a system where you control and know all of the conditions. Empirical observation is based on a system where you do cannot control or know (everything) about the conditions. But by having the (logical reasoned) knowledge that there is a 50% chance on a coin flip, you can predict with better than (also logical reasoned knowledge) chance of the outcome of a series of flips. So it’s a chance of a chance, or as the Student says a “probably of probablies”.

Not only that, but we can (by logical reason) determine how confident we are that different empirical results will occur -- in other words, how surprised we will be. So when the Student says, “all it means is that you'd personally be surprised if it didn't work? “ Susskind should have replied, “Yes! Absolutely!” Laws of probability (and laws of physics, and chemistry, etc.) tell us what to expect. If something unexpected happens, we are surprised. The whole purpose of science is “to reduce our surprise”, which you could inverse to say “to predict results” (but with the caveat that this is done with a measure of confidence).

The bottom line is that logical reason is pure improvable knowledge that lets us know how surprised we should be when those conditions don’t play out in empirical observation. The thing Susskind seems confused about is exactly how logical reason relates to empirical observation. That’s where it gets into philosophy.

 

[69dodge]

Originally posted by jzs
It may. 3 flips isn't a long-run frequency, it is the short run. Keep going to infinity..

But I can't keep going to infinity. No one ever can.

For that situation, say we are interested in estimating P, the probability of tails, by p. We can use a Wilson estimate to move the observed probabilty away from 1 (and, using the same reasoning if all heads came up, away from 0) to reflect a more realistic estimate.

where x is the number of tails and n the number of flips,

p = x/n

The Wilson estimate is p_w = (x+2)/(n+4)

Where does that formula come from? What are the assumptions underlying it?

For my quarter, it gives an answer of 5/7. I'd bet a lot of money that if I continue to flip that same quarter 7000 times, the number of tails will be nowhere near 5000.

 

[T'ai Chi]

But I can't keep going to infinity. No one ever can.

And you have no problem calculating areas under curves to get these probabilities?

Where does that formula come from? What are the assumptions underlying it?

I don't know. I mean, I know it was suggested by Edwin Wilson in roughly 1930, but I don't know the assumptions behind it or the reasons other than to pull p away from 0 and 1 to make it a more sensible estimate of P. I know computer simulations and some mathematical computations have been done to support this. References (which I haven't read) are

A. Agresti and B.A. Coull, "Approximate is better than 'exact' for interval estimation of binomial proportions," The American Statistician, 52 (1998), pp. 119-126

and

Lawrence D. Brown, Toni Cai, and Anirban DasGupta, "Confidence intervals for a binomial proportion and asymptotic expansions,", Annals of Statistics, 30 (2002), pp. 160-201.

For my quarter, it gives an answer of 5/7. I'd bet a lot of money that if I continue to flip that same quarter 7000 times, the number of tails will be nowhere near 5000.

I agree. There isn't a lot of information in 3 flips of a coin, so our estimate is more uncertain than if we'd flip the coin 7000 times. We'd naturally get an estimate we'd have more confidence in. The point is that 5/7 is a more sensible estimate than 1.

 

[T'ai Chi]

For my quarter, it gives an answer of 5/7. I'd bet a lot of money that if I continue to flip that same quarter 7000 times, the number of tails will be nowhere near 5000.

OOps, forgot to attach this to my previous post.

For 7000 flips, I'd expect something like this to occur

 

[69dodge]

Re: Re: Re: Re: Re: Re: An Impossible Coin Flip?

Originally posted by jzs
That is basically an argument from ignorance. He was saying that although we don't know anything about theta, it is equally probable to be between 0 and 1.

That sounds strange to you because you're thinking like a frequentist. When a Bayesian says theta is uniformly distributed on [0, 1], he means precisely that he doesn't know anything about theta except that it's between 0 and 1; he's not saying anything more than that.

(Theta is an unknown parameter here, correct? If it has some fixed value but we just don't know what that value is, frequentist notions of probability don't apply to it.)

All probability is about ignorance. If we knew for sure what was going to happen, we wouldn't need to use probability.

For large n, the likelihood tends to dominate the prior anyway, so the choice of prior in these cases is negligible. Also, frequently the results of Bayesian and frequentist approaches converge for large n, which is nice!

Sure. Large n is easy.

 

[T'ai Chi]

Re: Re: Re: Re: Re: Re: Re: An Impossible Coin Flip?

That sounds strange to you because you're thinking like a frequentist. When a Bayesian says theta is uniformly distributed on [0, 1], he means precisely that he doesn't know anything about theta except that it's between 0 and 1; he's not saying anything more than that.

That is saying that it is uniform between 0 and 1.

A uniform distribution is a specific distribution. I'm wondering how can you, objectively, go from ignorance about theta to specifying a specific distribution about it?

Personally I don't have a problem doing that in practice (ie. specifying an axiomatic uniform prior), so maybe I should ask myself that. But philosophically, it is an issue.