Probability of a probability being wrong?

[psionl0]

Really? OK. Based on your analysis, what is the probability that the true risk of an earthquake is less than 1.33%?

P(p < 1.33% | p < 1.33%) = 1

Seriously though, there is no "true risk". Any number you come up with depends on what assumptions you make and what tolerances you are prepared to live with.

If P is the probability that there are no earthquakes in 300 years and p is the probability that an earthquake will occur in a particular year then
P = (1 - p)^300
or
p = 1 - P^(1/300)

Setting a minimum value of P determines the maximum value of p so pick a number.

 

[jt512]

Seriously though, there is no "true risk".

That's a strange comment from someone who just a couple posts ago calculated a frequentist confidence interval.

Any number you come up with depends on what assumptions you make and what tolerances you are prepared to live with.

Obviously your estimate depends on your assumptions. That doesn't mean that there is no true long-run value to estimate; it only means that if you make bad assumptions, you'll calculate a bad estimate. I'm not sure what the part of your statement about tolerances has to do with the question. But a fundamental assumption of frequentist statistics is that there is a true state of nature about which we make statistical inferences.

Do you seriously believe that there is no true long-run probability that a flipped coin will land heads? If so, then what is it that we are estimating in a coin-flipping experiment? Is there no true long-run difference between Drug A and Drug B for treatment of X? What then is the effect estimate we calculate from drug trial data an estimate of?

The whole point of sampling is to calculate an estimate of a true population parameter, is it not? The point of looking at N years of data on whether or not an earthquake occurred is to estimate the true long-run probability per year of an earthquake. Surely, the more years we observe, the more our estimate converges on the true risk. If you can show that it does not, then you'll have overturned the Central Limit Theorem, the most important theorem in frequentist statistics.

If P is the probability that there are no earthquakes in 300 years and p is the probability that an earthquake will occur in a particular year then
P = (1 - p)^300
or
p = 1 - P^(1/300)

Setting a minimum value of P determines the maximum value of p so pick a number.

I asked, "Based on your analysis, what is the probability that the true risk of an earthquake is less than 1.33%?" You respond with a calculation that neither depends on your analysis nor calculates the probability I asked about! Your analysis does not lead to estimating such a probability. It can't because it was a frequentist analysis, and the concept of a probability distribution for a population parameter is nonsensical under the assumptions of frequentism.

 

[psionl0]

Do you seriously believe that there is no true long-run probability that a flipped coin will land heads?

I asked, "Based on your analysis, what is the probability that the true risk of an earthquake is less than 1.33%?" You respond with a calculation that neither depends on your analysis nor calculates the probability I asked about!

If you are going to censor the part of my post that directly (though facetiously) answers your question then it is no wonder you claim I didn't give the probability you asked for.

Post #10 made two key assumptions: 1) That we are dealing with a binomial distribution which could be approximated by a normal distribution. 2) That the probabilities calculated fall within the 95% confidence interval.

The first assumption is iffy (especially the bit about the normal approximation) but it permits a rough and ready calculation. There is no justification for the second assumption but 95% of the time, I would be correct in concluding that the probability of an earthquake in any year is less than 1.33% (1.22% if you don't use the normal approximation).

 

[Anerystos]

The two sigma rule for a 95% confidence internval is (more or less) right with the normal distribution. But this is the binomial distribution, with a small p, so this approximation is not going to work.

It is neither the normal or the binomial distributions. It is, in fact, the Poisson distribution, which describes the probability of the occurence of rare events. See, for example, the Wiki article about it. There is, no doubt, a plethora of other descriptions of it.

My memory is that there was a popular book on it called something like "Goals, Floods and Horse Kicks", but I can't find any reference to it on ABE or Bookfinder - which rather leads me to suspect my memory.

There is a good article on it in Psychology Wiki.

 

[edd]

It is neither the normal or the binomial distributions. It is, in fact, the Poisson distribution, which describes the probability of the occurence of rare events. See, for example, the Wiki article about it. There is, no doubt, a plethora of other descriptions of it.

My memory is that there was a popular book on it called something like "Goals, Floods and Horse Kicks", but I can't find any reference to it on ABE or Bookfinder - which rather leads me to suspect my memory.

The Poisson distribution is the limit of the binomial for small p and large n, so it is perfectly fine to choose to use the binomial if desired.

 

[Anerystos]

The Poisson distribution is the limit of the binomial for small p and large n, so it is perfectly fine to choose to use the binomial if desired.

As usual, it all depends. In this case it depends on what you mean by "small" and "large". Which again depends. Ad infinitum.

I agree, though, that you pays yer muney & takes yer choice.

 

[jt512]

Do you seriously believe that there is no true long-run probability that a flipped coin will land heads?

Whatever that means.

If you are going to censor the part of my post that directly (though facetiously) answers your question then it is no wonder you claim I didn't give the probability you asked for.

Post #10 made two key assumptions: 1) That we are dealing with a binomial distribution which could be approximated by a normal distribution. 2) That the probabilities calculated fall within the 95% confidence interval.

The first assumption is iffy (especially the bit about the normal approximation) but it permits a rough and ready calculation. There is no justification for the second assumption but 95% of the time, I would be correct in concluding that the probability of an earthquake in any year is less than 1.33% (1.22% if you don't use the normal approximation).

I have no problem with the binomial assumption. The normal approximation is poor in this case, but that's not a question I care about. That question I do care about is one you still have not given a clear answer to:

What is the probability, based on your analysis, that the annual risk of an earthquake is less than 1.33%?

Please just answer the question with a number, if you can calcluate one, or state that such a number cannot be determined from your analysis.

 

[byhisello99]

Wow.

History tells us that the "probability of an earthquake in a given year" is 1. The U.S. Geological Survey estimates that several million earthquakes occur in the world each year. While it is possible that no further earthquakes will occur, ever, on our planet, the probability of an earthquake in a given year is so close to 1 that it is indistinguishable from 1. As for the probability of an earthquake in a specific location of a minimum magnitude, we'll need more data.

Interestingly, events with a probability of zero occur often. The probability that on earth there will be a JREF with a post about probabilities by a user named copterchris was, until very recently, indistinguishable from zero. Yet it occurred.

Let's try the probability that there are two planets in the universe each having a JREF with a post about probabilities by a user named copterchris. Zero again, and a smaller zero than the first one (yes, there are relative sizes to zero and some infinities are larger than others). Yet, with about 4.0 x 10^22 stars in the known universe, about half of which are estimated to have planets, it may well already have occurred.

 

[edd]

As usual, it all depends. In this case it depends on what you mean by "small" and "large". Which again depends. Ad infinitum.

I agree, though, that you pays yer muney & takes yer choice.

Ok, let me make a stronger statement. The binomial distribution is exactly correct and the Poisson distribution you suggested is only an approximation to the exact solution you dismissed.

 

[psionl0]

What is the probability, based on your analysis, that the annual risk of an earthquake is less than 1.33%?

Please just answer the question with a number, if you can calcluate one, or state that such a number cannot be determined from your analysis.

I have answered this question for you in umpteen different ways.

The onus is on you to stop whingeing and read for comprehension.

 

[jt512]

I have answered this question for you in umpteen different ways.

The onus is on you to stop whingeing and read for comprehension.

Umpteen equivocations. Zero direct answers. Just give me a number. How hard can that be?

Actually, how hard it is is impossible, because I'm asking you for a probability of a population parameter, and your analysis was frequentist, and thus does not permit probabilities of parameters. That would explain why you have been unable to give me a straight answer to a simple question. What is less clear is why you can't just admit that.

 

[psionl0]

Please just answer the question with a number, if you can calcluate one, or state that such a number cannot be determined from your analysis.

OK then, "such a number cannot be determined from my analysis".

You can only calculate the maximum probability of an earthquake in any year once you have decided on your confidence interval. The larger the confidence interval, the greater the maximum probability.

Are you ready to stop this silly "give me a number" game yet?

 

[jt512]

OK then, "such a number cannot be determined from my analysis".

Thank you.

You can only calculate the maximum probability of an earthquake in any year once you have decided on your confidence interval. The larger the confidence interval, the greater the maximum probability.

*sigh*. I know the terminology is confusing, "probability of a probability," but my question isn't what is the maximum probability of an earthquake. It is what is the probability that the true annual risk, p, is less than the "maximum" you have calculated. If you happen to want to discuss it further it would be clearer to use the term earthquake risk for p. That way we can talk about the probability that the risk is less than some value of p calculated from the data.

Are you ready to stop this silly "give me a number" game yet?

The more I think about confidence intervals, the less I think I understand them. This conversation, as frustrating as it has been to both of us, has helped me to focus my thinking. If you want to continue it, I'm game, but I'll certainly understand if you're not interested.

 

[psionl0]

Thank you.




*sigh*. I know the terminology is confusing, "probability of a probability," but my question isn't what is the maximum probability of an earthquake. It is what is the probability that the true annual risk, p, is less than the "maximum" you have calculated. If you happen to want to discuss it further it would be clearer to use the term earthquake risk for p. That way we can talk about the probability that the risk is less than some value of p calculated from the data.

"Risk" is just another synonym for "probability" but I see where you are coming from.

Unfortunately, with the data being just a single statement of "no earthquakes so far" I don't see how you can get a definitive distribution - much less fill in all the blanks.

The more I think about confidence intervals, the less I think I understand them. This conversation, as frustrating as it has been to both of us, has helped me to focus my thinking. If you want to continue it, I'm game, but I'll certainly understand if you're not interested.

It is not surprising that you find yourself questioning your understanding of confidence intervals. You have the nomenclature and the mathematics down pat but if your education is similar that that of many schools and colleges that I have seen, you haven't been given a very clear picture of a random variable.

Most of the schools that I have seen just launch straight into the different types of distributions and how to calculate probabilities etc from them without any explanation of the concept of a random variable. This makes the whole exercise seem rather abstract and gives little idea of why a particular case is best described by one distribution instead of another.

 

[jt512]

It is not surprising that you find yourself questioning your understanding of confidence intervals. You have the nomenclature and the mathematics down pat but if your education is similar that that of many schools and colleges that I have seen, you haven't been given a very clear picture of a random variable.

Most of the schools that I have seen just launch straight into the different types of distributions and how to calculate probabilities etc from them without any explanation of the concept of a random variable. This makes the whole exercise seem rather abstract and gives little idea of why a particular case is best described by one distribution instead of another.

I know what a random variable is, I assure you. That's not the issue. It's the interpretation of a realized 95% (say) confidence interval that I find troublesome. Prior to sampling, the probability that a random 95% CI for a parameter µ will cover µ is .95. However, once the sampling is conducted, the realized 95% CI has no frequentist probability interpretation. It is not the case that it has a 95% probability of covering µ; it either covers µ or it does not.

I'm beginning to think that the best interpretation of a realized 95% CI is that it is a set of values of µ that are consistent with the observed data in a specific sense: it comprises precisely those values of µ that, given the observed data, would not be rejected by a hypothesis test at the .05 level of significance.

 

[psionl0]

I know what a random variable is, I assure you. That's not the issue.

Maybe you do but could still an issue if you question confidence intervals. Many a time I have had to go back to basics and question what I understood about the concept of a random variable before the latest lesson made sense to me.

Ultimately what I figured out (since nobody told me) is that a random variable is simply a trial in which all of the possible outcomes are numbers. (Rolling a dice is the classic illustration of a random variable). The X% confidence interval is simply the number range in which X% of your results will fall within each time you run the trial. It is nice if the interval is symmetric about the mean but not necessary - especially if the RV has a skewed distribution.

If you know the type of distribution of the RV and its mean and standard deviation then it is relatively easy to calculate the confidence interval. Usually there are a set of pre-digested formulas that you can use.

The question arises what to do if you don't know any of the parameters of the RV. The usual trick is to run the trial a certain number of times and generate a set of numbers which form a representative sample for the RV. The mean and standard deviation of the sample can be considered an estimate of the mean and SD of the RV (although a correction is needed to ensure that the estimate is "unbiased"). You can even construct a frequency histogram to get an idea of the type of distribution you are dealing with.

In this instance, the RV is the number of earthquakes observed in a 300 year period. The problem is that the trial was only run once and the number generated was zero. This doesn't give us a mean (zero) that we can have any confidence in and you can't get a SD at all if the sample size is only 1. Therefore, we can't really determine the confidence interval here.

As an alternative, I assumed that 1) the RV could be modeled by a binomial or normal distribution and 2) the result was within the 95% confidence interval. That is, I approached the problem backwards. You might question the validity of either assumption but I think the mathematics pans out.

I'm probably teaching you how to suck eggs here but I'm hoping that if you know where I am coming from then there might be less tension arising from any future discussions from this topic.

 

[Farsight]

Not knowing the answer to this has bugged me for ages...

If an event is hypothesized to have probability x, how many observations of the event not happening does it require to invalidate the hypothesis, or of being confident that 'x' is wrong?

e.g. I hypothesize that there is a 1 in 100 of an earthquake in a given year, how many years of no earthquakes would it take before I could be confident that 1:100 is not correct

There's some interesting answers in the thread. But the bottom line is this: you can't be too confident about this. The earthquakes might come like buses. So whenever anybody does express confidence about this sort of thing and throws out things like Bayesian and sigma and a whole load of graphs and charts and math, turn your BS detectors to max. Be sceptical.

 

[edd]

. The X% confidence interval is simply the number range in which X% of your results will fall within each time you run the trial. It is nice if the interval is symmetric about the mean but not necessary - especially if the RV has a skewed distribution.

It isn't that. If I collect a sample and calculate a 95% confidence interval of say 15 to 93, it doesn't mean that repeating the trial will give me a mean between 15 and 93 95% of the time in the future. It only means that of all such intervals I might construct, 95% of them should contain the population parameter.

A confidence interval is a range constructed for one particular sample designed to have a particular property when considered as part of a population of such constructed ranges. It doesn't have a particular useful meaning when considered alone.

 

[psionl0]

It isn't that.

It most assuredly is if you are dealing with a RV with a known distribution (the context of the quote). For example, if a pair of dice are rolled then there is a 94.44% probability that the result will be between 3 and 11 inclusive. That doesn't mean that if you roll the dice 18 times that there will only be one 2 or 12 in the mix but if you roll it 18,000,000 times, you can be pretty confident that there will be close to 1,000,000 2's or 12's in the mix.

Of course, sampling an RV of unknown distribution is more complicated as you point out.

 

[edd]

It most assuredly is if you are dealing with a RV with a known distribution (the context of the quote). For example, if a pair of dice are rolled then there is a 94.44% probability that the result will be between 3 and 11 inclusive. That doesn't mean that if you roll the dice 18 times that there will only be one 2 or 12 in the mix but if you roll it 18,000,000 times, you can be pretty confident that there will be close to 1,000,000 2's or 12's in the mix.

Of course, sampling an RV of unknown distribution is more complicated as you point out.

Well a pair of dice is a bit of a restrictive example in this case, as there are quite obvious minimum values of 2 and maximum values of 12.

What I'm trying to point out is that when you said

The X% confidence interval is simply the number range in which X% of your results will fall within each time you run the trial.

that's not normally the kind of confidence interval that gets quoted. The confidence interval is usually given as a particular instance of possible confidence intervals - a realisation from one particular observation. If we consider instead something normally distributed with μ=0 and σ=1, you might get an observed mean of -0.3, and some confidence interval around that value, not 0. You can't say that X% of the time the actual population mean is in the X% confidence interval, only that X% of the time you create such a confidence interval the population mean will be in the range you calculate then.

The confidence interval is something calculated from a particular realisation of a trial, not a property of the underlying population distribution.