On the nature of evidence

[athon]

You seem to be putting the conclusion before the evidence the opposite way with your, "If fairies did exist". I would argue that the fairy sighting would be evidence against the conclusion that they do not exist, rather than evidence for the conclusion of their existence, if this makes any sense.

Haha. I think I burst a blood vessel.

Let me think about this; the way I see it, we start from a position which gives no weight to fairies either existing or not existing. As such, 'fairies exist' is a useless position to adopt or make decisions based on. i.e., a null hypothesis.

Yet science is based on accumulating evidence. Imagine it slightly different - a new particle in physics where one should not exist. To start with the assumption it cannot exist, therefore it can't be evidence for one risks missing out on a novel discovery. If such a particle did exist, observing it would be evidence of it. The inference of 'novel particle' is only dismissed as a result of applying Occham's Razor, and not because the observation is not evidence for it.

I guess a simpler way of putting it is that we can't state definitively that fairies do not exist (you can't prove a negative), so you also can't definitively say that seeing one cannot be evidence of one existing.

Athon

 

[Darat]

But that's putting the conclusion before the evidence. It's starting with the assumption that fairies don't exist and therefore the evidence can't be in support of it. If fairies did exist, seeing one would be evidence for it, surely?

...snip...

Now, it is incredibly unlikely that fairies exist, however if science is about accumulation of relative evidence then claiming something is not evidence simply because we have already concluded it isn't real is putting the cart before the horse.

I don't think so since your evidence would merely be additional evidence to what we currently have in regards to whether fairies could exist and the evidence we already have is against the existence of fairies.

Often when a claim is made for say fairies a counter argument is "you can't prove a negative", and whilst that truism is useful for some claims it isn't for many others.

If I knew nothing about the history of fairy claims, if I did not know what I do about how the world (appears ) to work and you said you'd seen a fairy I'd say that then it would be evidence that I need to consider that you'd seen a fairy. However since I already have access to a mass of evidence that rules out fairies existing, I have to evaluate the new evidence of you seeing a fairy in light of all that evidence as well.

This is where Sagan's truism "extraordinary claims require extraordinary evidence" comes into play, to overturn the current "extraordinary evidence" we have for the non-existence of fairies we need evidence that is in itself extraordinary. I'd suggest in the case of fairies this probably would require the capture of a specimen. (Watch out Tinkerbell!)

 

[Darat]

....snip...

I guess a simpler way of putting it is that we can't state definitively that fairies do not exist (you can't prove a negative), so you also can't definitively say that seeing one cannot be evidence of one existing.

Athon

This is something I disagree strongly about i.e. the idea that we can't prove a negative, we certainly can and we often do. For instance "There is no elephant in my room", that negative is very easy to prove, a simple observation proves it and the only way to say it doesn't is to go down the route of "elephant of the gaps".

If something is claimed to be real we can often prove that it can't be. (Not to go down the philosophical route just take "real" to mean anything we can interact with either directly or indirectly.)

 

[Francesca R]

This is something I disagree strongly about i.e. the idea that we can't prove a negative, we certainly can and we often do.

I think that technically we "can't prove" something does or doesn't exist in most cases. There may be an elephant that you are missing. And that might not be a side table despite a lot of evidence that it is.

Reasonable doubt, and the man on the Clapham Omnibus, come to mind.

 

[Francesca R]

Let me think about this; the way I see it, we start from a position which gives no weight to fairies either existing or not existing. As such, 'fairies exist' is a useless position to adopt or make decisions based on. i.e., a null hypothesis.

Null hypotheses are assumptions, or straw men. At some point you need to invoke assumptions and then you test them. Practically I don't think it matters too much which is the null. Statisticians (especially econometricians, which I am supposed to be able to do) prefer to reject a null hypothesis as the orthodox way to derive "proof", but I am not convinced that their techniques would not be equally valid if they were to set up tests to accept them. It's concerned with "general to specific" but I forget why (Even though I am at work and could look it up)

 

[Darat]

I think that technically we "can't prove" something does or doesn't exist in most cases. There may be an elephant that you are missing. And that might not be a side table despite a lot of evidence that it is.

Reasonable doubt, and the man on the Clapham Omnibus, come to mind.

That's the "elephant of the gaps argument" I was referring to and the semantic shuffle we often use.

We know (again lets not getting into philosophical meanderings of what "know" means) that a side table is not an elephant. To state that the side table might be an elephant in someway requires you to provide proof of your extraordinary claim before I need to consider it as even a possibility, given the evidence I have that a side table is not an elephant. My proof by observation can also be refuted by changing the definition so if you say that an elephant doesn't just mean a "large grey coloured quadruped mammal".

However I still maintain that there ain't an elephant in my room and that I can prove that (as long as we use our usual definitions for words).

 

[athon]

I don't think so since your evidence would merely be additional evidence to what we currently have in regards to whether fairies could exist and the evidence we already have is against the existence of fairies.

Can something only ever be defined as evidence in association with other observations, then? It still seems to be assuming a conclusion and then deciding something is evidence in favour of it, rather than being led to a conclusion through the observations.

It's a bit like stating a murder weapon is evidence of a particular murder only in retrospect after the crime has been solved. If I have a bloody knife and a man's fingerprints on it, it is evidence of that man having used it to kill somebody up until it has been determined convincingly otherwise.

Often when a claim is made for say fairies a counter argument is "you can't prove a negative", and whilst that truism is useful for some claims it isn't for many others.

I fully agree, and think it needs to be put into context. As you go on to say, there are parameters you can put on a field which means a negative can be proven. 'In this limited field of X items, Y does not exist'. However, science is not full of a known number of items. The field, while perhaps not infinite, is unknown. We can conclude that stating fairies exist is a useless statement, but we cannot state definitively that they do not exist. It's just not a statement that can be made outside of a specific set of boundaries.

Obviously we don't entertain the notion that fairies exist, particularly on flimsy evidence such as somebody saw it. However we could still argue non-parsimonous situations where fairies could exist.

If I knew nothing about the history of fairy claims, if I did not know what I do about how the world (appears ) to work and you said you'd seen a fairy I'd say that then it would be evidence that I need to consider that you'd seen a fairy.

This idea disturbs me, however, as the practice is to assume that a conclusion is concrete and final. I don't think we need to go through history's examples of why such an approach is is dangerous to scientific progress.

However since I already have access to a mass of evidence that rules out fairies existing, I have to evaluate the new evidence of you seeing a fairy in light of all that evidence as well.

Why 'rules out'? I'll further dig my grave here by going even more ridiculous - there is no 'ruling out' a Matrix-style universe, other than it being against parsimony. As much as I despise metaphysical notions of laws external to what we can observe, they cannot be ruled out. We don't use them simply because without reason to acknowledge they exist, they are superfluous. However parsimony isn't a universal law - it is a useful assumption about how the universe works.

This is where Sagan's truism "extraordinary claims require extraordinary evidence" comes into play, to overturn the current "extraordinary evidence" we have for the non-existence of fairies we need evidence that is in itself extraordinary. I'd suggest in the case of fairies this probably would require the capture of a specimen. (Watch out Tinkerbell!)

I agree that in order to outweigh contrary evidence and support fairydom (is that a word? is there a fairysub? ), there needs to be extraordinary evidence. However, that still not mean observing a fairy is not evidence for one - it just means it on its own will not lead to a conclusion in support of the inference.

Null hypotheses are assumptions, or straw men. At some point you need to invoke assumptions and then you test them. Practically I don't think it matters too much which is the null.

I agree with all of this except for the last part. Maybe it's just how I've been taught, but I find it difficult to see the merit in starting with a hypothesis which starts with the assumption that something is true. Starting with 'there are no fairies', this is a position which can be falsified. Until that happens, you cannot state either way with 100% certainty. The thing a lot of people trip up on is equating this with the statement 'there could be fairies', which isn't quite the case. Any notion of fairy existence is useless information which cannot be used to make a worthwhile prediction.

Statisticians (especially econometricians, which I am supposed to be able to do) prefer to reject a null hypothesis as the orthodox way to derive "proof", but I am not convinced that their techniques would not be equally valid if they were to set up tests to accept them. It's concerned with "general to specific" but I forget why (Even though I am at work and could look it up)

If anything, you've given me food for thought. I'm more than happy to be swayed on this point. Thanks.

Athon

 

[Francesca R]

I agree with all of this except for the last part. Maybe it's just how I've been taught, but I find it difficult to see the merit in starting with a hypothesis which starts with the assumption that something is true. Starting with 'there are no fairies', this is a position which can be falsified. Until that happens, you cannot state either way with 100% certainty. The thing a lot of people trip up on is equating this with the statement 'there could be fairies', which isn't quite the case. Any notion of fairy existence is useless information which cannot be used to make a worthwhile prediction.

When dealing with statistical significance, 100% certainty is no good to you. It's all about confidence intervals for rejecting claims, which in turn are confidence intervals for accepting the opposite. It is more usual to reject than accept, but the way I see it, it makes little difference. In orthodox statistics you reject a specific null hypothesis (such as "the real equilibrium oil price is stationary through time") but that leaves you with something non-specific ("OK it's not stationary . . . so what is it?"). So what people do is set up a general null hypothesis ("the real equilibrium oil price is not stationary") and try to reject that instead, which then allows them to say "With X% confidence the price of oil, adjusted for inflation, reverts to a constant value over time". Really this involves discarding a range of other hypotheses without testing them at all, but it's what happens.

Similarly, I think, with fairies. Our own heuristics and biases from observational experience, give us very useful rules to simplify our own assumptions and tests. But true critical thinker would try to step out of those. Then again--a true free-thinker might struggle to reach many conclusions.

 

[Darat]

Can something only ever be defined as evidence in association with other observations, then? It still seems to be assuming a conclusion and then deciding something is evidence in favour of it, rather than being led to a conclusion through the observations.

I'm struggling to think of any evidence that could exist in total isolation, the idea that it could seems to me to be a Platonic ideal. We can only make an observation because of the interactions everything has with everything else.

It's a bit like stating a murder weapon is evidence of a particular murder only in retrospect after the crime has been solved. If I have a bloody knife and a man's fingerprints on it, it is evidence of that man having used it to kill somebody up until it has been determined convincingly otherwise.

Isn't that how science does in fact work with evidence? The evidence for plate tectonics was around prior to the theory of plate tectonics, the evidence led to the theory but now what will happen (because that theoretical model seems to match reality) is any new evidence is tested against our current model to see if it supports that model, only if it doesn't do we reconsider our model.

...snip...

Why 'rules out'?

...snip...

Because the current model we have based on the current evidence we have not only makes the "prediction" that fairies (as magical little people with wings that fly) do not exist but also offers an explanation as to why you might claim to have seen a fairy. In other words the evidence you bring to the table regarding fairies is already incorporated into what we currently know.

...snip...


I'll further dig my grave here by going even more ridiculous - there is no 'ruling out' a Matrix-style universe, other than it being against parsimony. As much as I despise metaphysical notions of laws external to what we can observe, they cannot be ruled out. We don't use them simply because without reason to acknowledge they exist, they are superfluous. However parsimony isn't a universal law - it is a useful assumption about how the universe works.

...snip...

This is more philosophy and I think it would derail the thread from your intended purpose?

...snip...


I agree that in order to outweigh contrary evidence and support fairydom (is that a word? is there a fairysub? ), there needs to be extraordinary evidence. However, that still not mean observing a fairy is not evidence for one - it just means it on its own will not lead to a conclusion in support of the inference.

...snip...

I think part of the problem here is that the statement "I've seen a fairy" is not in fact evidence it's a conclusion; the evidence would be more along the lines of "I saw a little man with wings flying and he turned someone into a toad".

 

[Darat]

...snip... Then again--a true free-thinker might struggle to reach many conclusions.

Many? I'd say any!

 

[Francesca R]

Can something only ever be defined as evidence in association with other observations, then? It still seems to be assuming a conclusion and then deciding something is evidence in favour of it, rather than being led to a conclusion through the observations.

Both routes happen, don't they? Theory precedes discovery in may cases, and discovery leads to theory formation in others. Then the two are worked in synergy to tighten up to conclusions. I think both routes are responsible for the current "knowledge base".

 

[athon]

Isn't that how science does in fact work with evidence? The evidence for plate tectonics was around prior to the theory of plate tectonics, the evidence led to the theory but now what will happen (because that theoretical model seems to match reality) is any new evidence is tested against our current model to see if it supports that model, only if it doesn't do we reconsider our model.

Of course. Yet new observations are considered as evidence for a variety of inferences - only then are they whittled down using Occham's Razor and compared with what is known. If a new observation contradicts what is known, overturning all previous theories should be considered, even if only briefly enough to dismiss it. Hence that observation is evidence for a contradictory theory (if other things as well).

I know fundamentally this is mostly a semantics discussion. We don't disagree on the process. However in communicating skepticism and science, I think understanding the terms and how they should be applied is important.

Because the current model we have based on the current evidence we have not only makes the "prediction" that fairies (as magical little people with wings that fly) do not exist but also offers an explanation as to why you might claim to have seen a fairy. In other words the evidence you bring to the table regarding fairies is already incorporated into what we currently know.

Ah, I see the problem. Excellent point - I've been lazy with using the word 'fairy' instead of 'little flying humanoid', which in itself presumes a conclusion. Cool.

This is more philosophy and I think it would derail the thread from your intended purpose?

Not necessarily. The term 'evidence' lies at the core of the philosophy of science - hell, people use the term 'evidence' for all sorts of non-materialistic notions. I'm not suggesting that they are practically correct, however we can't afford to simply assume a materialistic philosophy is correct for the sake of randomly choosing a side. We need to consider an observation as evidence for X - regardless of its philosophical position - and then follow this to a conclusion which either supports it or dismisses it.

I think part of the problem here is that the statement "I've seen a fairy" is not in fact evidence it's a conclusion; the evidence would be more along the lines of "I saw a little man with wings flying and he turned someone into a toad".

Yup. I just got that.

Athon

 

[fls]

...(you can't prove a negative)...

Athon

You can't prove a negative universal statement.* But you can't prove a positive universal statement either. Rather than saying you can't prove a negative, a statement which only applies to some negative statements (but people will misconstrue as applying to their particular negative statement, thereby rendering it safe), it makes more sense to say that you can't prove a universal statement (excluding those that represent definitions/tautologies). Otherwise you are forced into a corner where you ignore the normal process of science (which is generally not about universal statements) or you tie yourself into philosophical knots trying to fit a large, mammalian quadruped in the space behind the bookcase.

Linda

*Even this statement is qualified (as has already been pointed out) as merely impractical, rather than impossible, since it reflects our practical knowledge of the field.

 

[Beth]

But that's putting the conclusion before the evidence. It's starting with the assumption that fairies don't exist and therefore the evidence can't be in support of it. If fairies did exist, seeing one would be evidence for it, surely?

Now, it is incredibly unlikely that fairies exist, however if science is about accumulation of relative evidence then claiming something is not evidence simply because we have already concluded it isn't real is putting the cart before the horse.

I guess it comes down to agreeing whether an observation can only be evidence for one thing or for many other things.

Athon

I quite agree. You've phrased things far better than I could though. As usual!

 

[Beth]

When dealing with statistical significance, 100% certainty is no good to you. It's all about confidence intervals for rejecting claims, which in turn are confidence intervals for accepting the opposite. It is more usual to reject than accept, but the way I see it, it makes little difference. In orthodox statistics you reject a specific null hypothesis (such as "the real equilibrium oil price is stationary through time") but that leaves you with something non-specific ("OK it's not stationary . . . so what is it?"). So what people do is set up a general null hypothesis ("the real equilibrium oil price is not stationary") and try to reject that instead, which then allows them to say "With X% confidence the price of oil, adjusted for inflation, reverts to a constant value over time". Really this involves discarding a range of other hypotheses without testing them at all, but it's what happens.

[derail to lecture about statistical hypothesis setting]
Statistically the way it's set up makes a great deal of difference because the probability of making a correct decision when rejecting the null is precisely defined, easily computed, and can be set by the experimenter while the probability of making a correct decision when failing to reject the null (called the power of the test) is not so easily computed and cannot be set by the experimenter. Instead it is dependent on the true but unknown state of reality and the sample size used to compute the test statistic.

The way statistical hypothesis testing works is that you define the null as the hypothosis you assume true and wish to reject in favor of the alternative hypothesis. In addition, the mathematics behind it require that the null hypothesis contain the equality while the alternative hypothesis is a strict inequality. (It has to do with computing probabilities of closed sets versus open sets.)

You decide on a test statistic for your data and compute the probability of observing as test statistic of that size under the assumption that the null is true. Then you can reject the null if the probability of the observed values of the test statistic (p) is smaller than alpha. You can accept the alternative as being true with a confidence of 1 - p.

The reason that we do it this way (which seems backwards to many people) is that it allows us to specify the level of confidence we want to have before accepting the alternative. When we do, we can be very confident we are making a correct decision. However, when we fail to reject the null, the probability that we are making the correct decision can be much much lower. [/derail]

Ah, I see the problem. Excellent point - I've been lazy with using the word 'fairy' instead of 'little flying humanoid', which in itself presumes a conclusion. Cool.

Depends on how you define fairy I guess. I would have thought that 'little flying humanoid' = 'fairy'. But I guess is could have been an alien, eh?

 

[Francesca R]

Statistically the way it's set up makes a great deal of difference because the probability of making a correct decision when rejecting the null is precisely defined, easily computed, and can be set by the experimenter while the probability of making a correct decision when failing to reject the null (called the power of the test) is not so easily computed and cannot be set by the experimenter. Instead it is dependent on the true but unknown state of reality and the sample size used to compute the test statistic.

The way statistical hypothesis testing works is that you define the null as the hypothosis you assume true and wish to reject in favor of the alternative hypothesis. In addition, the mathematics behind it require that the null hypothesis contain the equality while the alternative hypothesis is a strict inequality. (It has to do with computing probabilities of closed sets versus open sets.)

You decide on a test statistic for your data and compute the probability of observing as test statistic of that size under the assumption that the null is true. Then you can reject the null if the probability of the observed values of the test statistic (p) is smaller than alpha. You can accept the alternative as being true with a confidence of 1 - p.

The reason that we do it this way (which seems backwards to many people) is that it allows us to specify the level of confidence we want to have before accepting the alternative. When we do, we can be very confident we are making a correct decision. However, when we fail to reject the null, the probability that we are making the correct decision can be much much lower.

I agree about that, but the alternative hypothesis is general, only excluding the null but not being a specific alternative to it. So it seems to me that the conclusion this gives you is that something is not [whatever the null was], yet I often see it emphasised as if it was that something is [some specific alternative to the null] (but not the only conceivable one) with high probability.

 

[Hokulele]

Depends on how you define fairy I guess. I would have thought that 'little flying humanoid' = 'fairy'. But I guess is could have been an alien, eh?

I think you are somewhat missing the point Darat was making. By defining fairy as a little flying humanoid, you have already assumed a conclusion. For example, if I claim to see a fairy, I have already concluded something about what I saw that is unsupported (the existence of fairies). If, however, I claim to see something that was flying, about 6 inches tall, and wearing a small, green dress, I am not making any conclusions about the identity of the object, and I have a starting point for figuring out what it really was.

The trouble with the first line of thinking is that it can lead to false conclusions, or worse, lead to an end to inquiry. If I define "fairy" to mean "what I saw", it is pretty much the end of the road for critical thinking.

 

[Ivor the Engineer]

[derail to lecture about statistical hypothesis setting]
Statistically the way it's set up makes a great deal of difference because the probability of making a correct decision when rejecting the null is precisely defined, easily computed, and can be set by the experimenter while the probability of making a correct decision when failing to reject the null (called the power of the test) is not so easily computed and cannot be set by the experimenter. Instead it is dependent on the true but unknown state of reality and the sample size used to compute the test statistic.

The way statistical hypothesis testing works is that you define the null as the hypothosis you assume true and wish to reject in favor of the alternative hypothesis. In addition, the mathematics behind it require that the null hypothesis contain the equality while the alternative hypothesis is a strict inequality. (It has to do with computing probabilities of closed sets versus open sets.)

You decide on a test statistic for your data and compute the probability of observing as test statistic of that size under the assumption that the null is true. Then you can reject the null if the probability of the observed values of the test statistic (p) is smaller than alpha. You can accept the alternative as being true with a confidence of 1 - p.

The reason that we do it this way (which seems backwards to many people) is that it allows us to specify the level of confidence we want to have before accepting the alternative. When we do, we can be very confident we are making a correct decision. However, when we fail to reject the null, the probability that we are making the correct decision can be much much lower. [/derail]

One problem with hypothesis testing is with a large enough sample size practically any null hypothesis can be rejected.

I suggest the proponents of acupuncture perform their studies with thousands of volunteers rather than just a few dozen, then they could quote results with p-values much less than 0.05 and shut the sceptics up*.

*Yes, I know it wouldn't shut you (or me) up.

 

[Beth]

I think you are somewhat missing the point Darat was making. By defining fairy as a little flying humanoid, you have already assumed a conclusion. For example, if I claim to see a fairy, I have already concluded something about what I saw that is unsupported (the existence of fairies). If, however, I claim to see something that was flying, about 6 inches tall, and wearing a small, green dress, I am not making any conclusions about the identity of the object, and I have a starting point for figuring out what it really was.

The trouble with the first line of thinking is that it can lead to false conclusions, or worse, lead to an end to inquiry. If I define "fairy" to mean "what I saw", it is pretty much the end of the road for critical thinking.

It was a just a joke Sorry it didn't fly.

 

[Beth]

One problem with hypothesis testing is with a large enough sample size practically any null hypothesis can be rejected.

No. A null hypothesis that's true has only an alpha probability of being rejected regardless of the sample size because the sample size is taken into account when making that determination.

You may be confusing hypothesis testing with a regression analysis which tests for signicant correlations between variables. With a large enough sample size, any non-zero correlation can become statistically significant.