Extraordinary claims require extraordinary evidence, the discussion.

[Dinwar]

Then your probability of each is .5 (given that I told you to assume that I*ate eggs).

Right--but it's a failed demonstration because it's not unknow. I would have built that data off of known information.

Thus it has no bearing on the question of how to calculate the probability of something unknown.

No.

This says it all. You don't know the system, and therefore cannot accurately calculate the probability. The reality is that if I take a nap, it's usually indicative of rather serious illness--full incapacitation, as in "can't even get to the toilet" sick. Even with the sleep deprivation that comes with an infant I've only taken six naps in the past five years, each time associated with some major illness (and once my body basically shutting off due to a migrain--the two are not dissimilar).

I say this to illustrate a contrasting situation: you don't know the system, and your guess was wrong. You can't calculate the odds, because you don't have enough data.

Yes. But I*told you that you can assume that I ate eggs, and that they were either scrambled or fried, and I*asked you whether each of those two claims was ordinary or extraordinary. Given that I had eggs, you had no evidence to support either claim over the other. Yet you definitively stated that both claims were ordinary.

First,xkcd said it best: you can't communicate poorly and pretend it's deep. If you're goinig to give me the parameters of the scenario, we can treat them as givens.

Second, I already demonstrated that I had a great deal of other data to contribute to the discussion, and provided an example of a cooking method that would yield less than 50% probability given that other information. So your analysis is flawed.

Here's another ordinary claim that you have no evidence for: a newly developed, but as yet untested, drug is more effective than an existing drug. Ordinary or extraordinary? What is your evidence for superiority. What's your prior probability of the hypothesis?

I have none. It's impossible. You've given me no information about the drug, the effects, how they're measured, what the drug is supposed to do, any side effects, how the tests were performed, etc. There's nothing you can say about it that's rational--any statement about probability oversteps what the evidence can support.

My father used to mock statistics with the following: What are the odds, if I have 50 black and 1 red ball in a paper sack, that I'll pull out a red ball? 50/50. Either it'll happen or it won't. The same type of flaw in Dad's reasoning is inherent in any probability given ini answer to your question.

 

[jt512]

You don't know the system, and therefore cannot accurately calculate the probability.

So, from experience I know you don't grasp prior probabilities, so I'm not going to go to far with you on this. In fact, I can easily come up with a prior probability, or a prior distribution of the probability of your taking a nap based on background information about nap taking.

I say this to illustrate a contrasting situation: you don't know the system, and your guess was wrong.

My "guess" wasn't wrong. Your taking a nap was not an extraordinary claim. Six naps in five years is nothing extraordinary. P(nap/day)=0.3% is not an extraordinarily low probability. 10^-10, which I*would consider the probability of any alleged paranormal phenomenon being true, is an extraordinarily low probability.

It's impossible. You've given me no information about the drug, the effects, how they're measured, what the drug is supposed to do, any side effects, how the tests were performed, etc. There's nothing you can say about it that's rational--any statement about probability oversteps what the evidence can support.

Impossible for you maybe. A good first approximation to the prior would be the percentage of new drugs that have proved to be superior to their predecessor in the clinical trial literature. A good zeroth approximation would simply be 0.5, the Bayesian representation of no knowledge.

My father used to mock statistics with the following: What are the odds, if I have 50 black and 1 red ball in a paper sack, that I'll pull out a red ball? 50/50. Either it'll happen or it won't. The same type of flaw in Dad's reasoning is inherent in any probability given ini answer to your question.

Uh, no. It isn't, but at least now I have some insight into the sources of why you think the way you do about Bayesian probability.

 

[Dinwar]

So, from experience I know you don't grasp prior probabilities, so I'm not going to go to far with you on this. In fact, I can easily come up with a prior probability, or a prior distribution of the probability of your taking a nap based on background information about nap taking.

Fair enough. What are the odds I've ftogned?

My "guess" wasn't wrong. Your taking a nap was not an extraordinary claim.

Only true if you have no actual knowledge of my nap-taking. I've been nearly killed more frequently than I've napped, and no one can claim that THAT isn't an extraordinary claim. When you can honeslty say "Someone or something has tried purposefully to murder me more frequently than X", X can be considered extraordinary for that system, can it not?

Six naps in five years is nothing extraordinary. P(nap/day)=0.3% is not an extraordinarily low probability.

It's statistically insignificant. Plus, most people nap a tad more frequently than six times in five years, putting me well outside the norm--which, if you'll recall, is my original definition of "extraordinary".

A good first approximation to the prior would be the percentage of new drugs that have proved to be superior to their predecessor in the clinical trial literature.

For a certain value of "good". By which I mean, if you define "good" as "something that provides a number". If you define "good" as "something that has any bearing on the actual system in question", however, it's a pretty sorry excuse for an estimate.

A good zeroth approximation would simply be 0.5, the Bayesian representation of no knowledge.

For a certain value of "good". By which I mean, if you define "good" as "something that provides a number". If you define "good" as "something that has any bearing on the actual system in question", however, it's a pretty sorry excuse for an estimate.

Uh, no. It isn't

Yes it is. You are saying "It'll either happen or it won't". There's no substantive difference. Feel free to dismiss me if you wish, but that doesn't change the fact that your logic is precisely the same.

 

[jt512]

I've been nearly killed more frequently than I've napped, and no one can claim that THAT isn't an extraordinary claim. When you can honeslty say "Someone or something has tried purposefully to murder me more frequently than X", X can be considered extraordinary for that system, can it not?

It's not a coherent claim, so I can't judge whether it's extraordinary or not. If you had said that, over your lifetime, there have been more serious attempts to murder than than days you have taken naps, I would call it extraordinary; and, appropriately, I would not believe it without more evidence.

Six naps in five years is nothing extraordinary. P(nap/day)=0.3% is not an extraordinarily low probability.

It's statistically insignificant.

That doesn't even make sense. How can a plain fact be statistically insignificant? It is no surprise that the scientist who repeatedly claims that you don't have to understand statistics to be scientifically literate has such a poor understanding of both frequentist and Bayesian statistics.

Plus, most people nap a tad more frequently than six times in five years, putting me well outside the norm--which, if you'll recall, is my original definition of "extraordinary".

I have no idea why you would think that being somewhat in the left tail of a probability distribution is extraordinary. On the contrary, somebody has to be there.

For a certain value of "good". By which I mean, if you define "good" as "something that provides a number". If you define "good" as "something that has any bearing on the actual system in question", however, it's a pretty sorry excuse for an estimate.

No, I define a prior probability the way every Bayesian statistician does: the probability based on whatever background information I have. At the extreme, no background information implies a uniform prior probability distribution, at least when the hypothesis space is discrete and finite (and arguably when it is not).

Yes it is. You are saying "It'll either happen or it won't". There's no substantive difference. Feel free to dismiss me if you wish, but that doesn't change the fact that your logic is precisely the same.

Dinwar, it's not that hard a concept. If you have exactly two competing hypotheses, and no information on which to base a belief about which is more likely to be true, then your prior probability for each hypothesis is .5. This is nothing at all like your dad's dumb example, where he has all the information to compute the correct probability, but he completely ignores it and assigns .5 to each hypothesis because, duh, there's two of them.

 

[Kevin_Lowe]

That doesn't even make sense. How can a plain fact be statistically insignificant?

If I had to take a guess, I'd say that Dinwar remembers from uni that p<0.05 is "statistically insignificant", has forgotten or never knew in the first place that this concept only refers to the p value of a false positive and not to p values in general, and hence now believes that anything which happens less than one time in twenty is "statistically insignificant".

That's the best I can do at reconstructing the possible errors which could lead to such a post.

 

[Dinwar]

Dinwar, it's not that hard a concept. If you have exactly two competing hypotheses, and no information on which to base a belief about which is more likely to be true, then your prior probability for each hypothesis is .5.

jt512, this is not a hard concpet: if you have no other information than the two possible hypotheses, you don't have enough data to draw any conclusions about probability.

Your statement about probability isn't invalid, in that it's a good aproximation if one is forced to provide an answer to "Which is more probable?" However, if one is not forced to do so, one should avoid giving probabilities to hypotheses one knows nothing about. It is incumbant upon rational folks to acknowledge the limits of their knowledge, and if all you know is that a set of hypotheses exist your knowledge does not permit you to asign probabilities for any of them to be correct. Something I learned very early on as a scientist is that it's okay to not know something--and sometimes, it's manditory to admit that you don't know it. All of your arguments sound like nothing more than a refusal to admit this foundational concept of science.

This refusal is not without its dangers. The issue is, once humans have an answer we tend to stick with it. There are innumerable studies demonstrating this. And 50% is an answer. You may think you're going to hold it tentatively, but the reality is that most people won't. They'll tend to stick with it, and argue that it's the correct one well past the point where it's been shown to be nothing more than a placeholder. This has had a detrimental impact on my field, numerous times in its past, and I'm pretty hesitant to perpetuate an error merely because it makes you feel better to have a number.

In science, ALL interpretations MUST be supported by data. If your data is nothing more than the number of hypotheses you can generate that fit the data, I do not believe there is sufficient data to assign any probability to them. Doing so even tentatively overstates the case.

 

[jt512]

jt512, this is not a hard concpet: if you have no other information than the two possible hypotheses, you don't have enough data to draw any conclusions about probability.

That assertion only confirms what I already knew: when it comes to Bayesian statistics you're ignorant and unaware of it.

Your statement about probability isn't invalid, in that it's a good aproximation if one is forced to provide an answer to "Which is more probable?"

And that sentence directly contradicts your previous one.

However, if one is not forced to do so, one should avoid giving probabilities to hypotheses one knows nothing about.

Says the guy who doesn't understand probability.

It is incumbant upon rational folks to acknowledge the limits of their knowledge...

Which is exactly what giving probabilities of 50% to two competing hypotheses is doing. If you had more information about them—information that makes one of them more likely than the other—you'd give them different probabilities.

...and if all you know is that a set of hypotheses exist your knowledge does not permit you to asign probabilities for any of them to be correct.

Except that up-post you admitted that it is valid to assign probabilities equal probabilities to them in exactly that scenario.

Something I learned very early on as a scientist is that it's okay to not know something--and sometimes, it's manditory to admit that you don't know it.

It's too bad that you didn't learn Bayesian inference early on as a scientist. Oh, but wait, you're the one who says education in statistics is not necessary for scientific literacy.

All of your arguments sound like nothing more than a refusal to admit this foundational concept of science.

Oddly enough, all of your arguments show that refuse to admit a foundational concept of Bayesian statistics—and I'm not just being snarky. The necessity of equal probabilities for elements of a discrete, finite space is foundational in Jaynes' development of the subject.

This refusal is not without its dangers. The issue is, once humans have an answer we tend to stick with it. There are innumerable studies demonstrating this.

Bayesian inference is the process of updating prior probabilities in light of new data. It is one of the most important tools a scientist can have. Scientists do this every day—even you do, I suspect, although perhaps not formally. There are innumerable studies, in which it is done formally, that lead to successful conclusions.

And 50% is an answer. You may think you're going to hold it tentatively, but the reality is that most people won't. They'll tend to stick with it, and argue that it's the correct one well past the point where it's been shown to be nothing more than a placeholder. This has had a detrimental impact on my field...

The very field you claim doesn't require knowledge of statistics. How interesting.

 

[Dinwar]

At this point, we're just insulting one another and not getting anywhere. If you want to pretend to have knowledge without data, enjoy. I'm out.

 

[jt512]

At this point, we're just insulting one another and not getting anywhere. If you want to pretend to have knowledge without data, enjoy. I'm out.

Well, for the last time then, giving two hypotheses 50% probabilities means that you have no knowledge that would lead you to favor one over the other.

 

[eight bits]

Dinwar

Actually, it doesn't. The term "extraordinary" is never defined; ...

Why should it be? Sagan wasn't dictating legislation, he was explaining, informally and in ordinary language, why he disbleieved personal reports of abduction by space aliens, despite having a well-earned reputation for thinking that life might exist and evolve off-Earth.

The parallels to Hume's famous comments on miracles are unmistakable. Like Hume, Sagan focuses on the relative ease and plausibility of mistakenly (or fraudulently) reporting a miracle compared with witnessing an actual miracle. In context, then, with respect to claims, "extraordinary" refers to what might be attention-getting about reporting that you have been abducted by aliens, and with respect to evidence, the term refers to having a form which is difficult to fabricate if the claim it accompanies is false.

Sagan's proposal is, on its face, a heuristic for managing one aspect of evidentiary interpretation. Compared with the expression of other heuristics, it is adequately phrased "When you hear hoofbeats, think horses, not zebras" is lousy advice in some parts of Africa. Yes, it is, but the phrasing is chosen for its mnemonic virtue, not as a literal prescription. Humans who might benefit from the advice would also plausibly know that what the figure of speech refers to is not zoology, but rather to what is usual and expected versus what is unusual and far-fetched.

Sagan's formulation does, however, invite misterpretation in terms sometimes described as "Bayesian." Hume obviously wasn't a Bayesian, since he was dead before Laplace decided to forego the credit for an innovative application of probability to uncertainty management. If Sagan had "Bayesian" sympathies, he kept them well-hidden.

...it's simply placed in with the implication that it's somehow greater than regular evidence.

In context, "somehow" is different in kind, specifically not personal testimony and not supporting exhibits that are easily fabricated. Sagan knows that testimony and uncompelling props might and do resolve many routine uncertainties with satisfactory confidence.

On some points arising,

If priors bother you, then fine, there are other likelihoodist (and parallel but weaker) approaches to evidentiary interpretation which don't rely on priors. In the limit of copious and easily understood evidence ("interocular trauma"), all admissible methods get pretty much the same answer. Duh. In the absence of such evidence, there is only poorer or better supported personal opinion, often diverse. If you have the luxury of never needing to choose a course of action despite your uncertainties, then by all means, avoid poorly supported personal opinion. Know however, that many people do not always have that luxury. Try to be patient with them.

In any case, Bayesianism has little to do with what Sagan meant. This thread, however, seems to have been about what Wiseman meant. I wouldn't know Wiseman's thoughts about Bayes. "Extraordinary claims require extraordinary evidence" is a well-turned catchy phrase, and plainly survives its maker. It is unsurprising, then, that it might come to be quoted with different intentions than Sagan's original purposes. Professor Sagan is not in a position to object.