Brain Evolved Preference for Anecdotal Evidence

[Skeptic Ginger]

SkeptiGirl: The cited study does not say that nobody got flu-like symptoms; it simply says that patients getting a placebo shot instead ALSO got flu-like symptoms at about the same rate. So I think Pax's post was correct, but incomplete. Something along the lines of, "...a flu shot can cause [symptoms] -- of course, so can getting a saline injection. " might be clearer.

But that is also false according to the evidence.

The flu vaccine is not what's causing the symptoms, but they are occurring. Psycho-somatic effects aren't imaginary! They're just caused by some still-unknown mechanism(s).

You are forgetting that we know what is causing the symptoms. It is simply the coincidental timing of flu shots which are given during the peak in other mild infectious diseases.

Some subjects get actual rashes from placebos! And some get symptom relief. There's a world of interesting stuff to be discovered in the realm of mind-body effect--that science has not yet figured out the mechanism doesn't make it mythical. And recognizing that such effects occur isn't surrendering to woo explanations of them, either.

I know, it's a minor detail, but I am fascinated by placebo effect and have had to change how I think and talk about it to be accurate. The real world is so much more amazing than any fantasy ever written!

Where is your evidence that the occurrence of symptoms after flu shots are psychosomatic? I can show you other evidence that is not the case.

The way to sort out the effect of something such as a flu vaccine is to subtract the background rate. If you get zero when you subtract the background rate then the variable you are testing caused zero % of the effect.

Now there is the issue of a no test arm in a study. That would be the true background rate. And many studies don't include a no treatment arm. But if you had an ideal study you would start with a large group, you would randomly assign members of the group to a no treatment arm, a placebo arm and a flu shot arm. If the rates of mild systemic symptoms was the same in all three groups then the flu shots caused ZERO % of the systemic symptoms. If the placebo and flu shot groups had more systemic symptoms than the no treatment group, then you would be seeing the effect of getting a shot.

Only if the flu shot group had more systemic symptoms than the other two groups, then that would be evidence the effect was due to the flu vaccine.

You seem to be making a common mistake here. You are attributing cause to a variable, (getting the shot), when the evidence suggests some other variable, (more likely frequency of upper respiratory infections at the same time we give flu vaccine) is responsible for the effect.

There is no evidence the effects are psychosomatic. While one needs that no treatment arm to confirm that, the assumption the symptoms are coming from the shot are unsupported. Such a conclusion is short sighted. It fails to consider the obvious variable, the frequency of common respiratory infections in Oct and Nov.

You are correct to consider placebo effect as one possible variable. And I have an advantage of knowing that study is but one of several pieces of very good evidence that confirm side effects from flu vaccine, psychological or not, are 99% myth. But I'm pretty sure you'd have a hard time finding any evidence that the placebo effect contributes to systemic symptoms after flu shots. What you will find, however, is people pay more attention to their symptoms after a shot and in a study. So that headache which would have been forgotten a week later is instead remembered if the patient associated it with the flu shot.

ETA: On the larger issue of convincing versus just showing how a method is faulty, a counter-example can be helpful. On the issue of Correlation is not Causation, I like to point out that there is a strong correlation between espresso drinks consumed by parents and kids' test scores.

After a moment, I explain that this is because AFFLUENCE is strongly correlated to both! Wealth enough to make $3 coffee drinks a common expenditure means one or both parents are making good money; that is strongly linked to literacy and level of education. Parental literacy and level of education are linked to kids' scholastic achievement. It would be erroneous to credit some caffeine-induced Life Energy that jumps from the parent's aura to the kid's as the source of the initial correlation. (Editor's note: I assume in the UK this would not work, since the percentage of coffee junkies is so much lower.) Usually, this example works because people can see how there IS a real correlation, and yet no causation.

I've tried these kinds of examples. But there is something about the convincing personal experience that blocks the person seeing the correlation.

 

[Skeptic Ginger]

Maybe it's an odd twist but since mankind's ability to reproduce is no longer controlled by natural selection it is appropriate that our rapid anecdotal response behavior be replaced by analytic thinking. IOW, natural selection has favored those who were able to overthrow natural selection!

That natural selection forces are no longer operating in the human species is an unsupportable conclusion. But maybe you could take this to another thread.

 

[Skeptic Ginger]

Someone upthread mentioned cognitive dissonance, and I suspect that's a big part of what happens. People will nod and smile and agree, oh, of course, that's bad reasoning, but can't connect it to themselves. We have big ol' blind spots when it comes to our own thought processes, and unless we can accept that even smart people make mistakes in their reasoning sometimes, our minds will do whatever it takes to save us from the cognitive dissonance that would come from even noticing anything that would make us look irrational or stupid.

I have no idea whether it actually makes a difference or not, but whenever I can, I like to use an anecdote about myself to make my point. I tell about my own mistakes in reasoning without the slightest bit of embarrassment, because, after all, everyone makes those kinds of mistakes at some point or another. They don't make me any less smart, and, by implication, they don't make anyone else less smart, either. (In fact, I can get a bit smug at being smart enough to have caught myself, but that's a whole other issue. )

We live in a mistake-averse society. It's only a thought, but I suspect that if we can find ways to make people less afraid of being wrong, their ability to spot their own errors and correct them will improve, whether those errors are due to overreliance on anecdotes, correlation-causation confusion, false memories, confirmation bias, or anything else.

I think people's information filters do play a big part here. That personal experience is heavily favored by one's brain. And there is something that keeps people supporting their original conclusions in spite of contradictory evidence.

Teaching critical thinking skills early on in childhood education is probably the most effective option.

 

[cj.23]

Actually, a rather obvious variable on the flu thing springs to mind. Given that most flu vaccines are administered i'm guessing at doctors surgeries, where in the UK at least you will wait with other patients, many of whom are in for mild seasonal disorders, might one simply not contract the cold or whatever while waiting for the shot in a disease rich environment?

I recall the appalling legends which prevented many people seeking vaccination against flu in the uS on a couple of occasions historically- Gina Kolata's excellent book Flu deals with them well, and the public health implications as I recall.

Also, flu vaccinations only protect against the most virulent strains predicted for that season, so quite likely many people who are vaccinated will go on ot get flu perfectly naturally?

cj x

 

[Skeptic Ginger]

Right, yeah I was thinking legally. Problem is that I have seen all observations and personal experience classed as anecdotal, and as such then statements like "my father is called Gunnar" become ancedotal and considered less valuable. Now as a historian, I would note that the most valuable evidence availsble to us is primary witness reports, written often, and without direct observation history would go nowhere as a discipline. Ditto Darwin. If he had ignored his personal observations in the Galapagos, and not then formulated theories based upon his own observations, no theory of evolution. And so on and so forth. Alll of modern science was developed with observations as a key criteria. So anecdotal clearly means something far more specific than "observation based" - though increasingly I see it misued that way - Sagan did it for instance - it means hearsay or unrelaible second hand testimony, the stuff of rumour. If Soapy saw the locjh Ness monster eating petunias in his living room tonight,and reported his experience, that would not be anecdoatal evidence. If I said Soapy had had that experience and i knew cos Larsen mentioned it to me while we were buying laptops, that would.

So anecdotal evidence must be defined very strictly - and as the term is taken from law, I have used that definition. Here is a modern American definition of the type I regard as nonsensical

I think I have addressed this in previous posts here. I am used to using the term, anecdotal evidence, because it has a specific meaning in the medical sciences. But to avoid confusion, I think 'personal experience' fairly well describes what I am referring to for those who are not used to the term, anecdotal evidence, as it is used in medical science.

...However we have a relationship one presumes between eyewitness testimony and the structural engineers reports. While much eyewitness testimony is doubtless flawed, as observations are, I would assume the structural engineers reports simply explain what eyewitnesses experienced on the day? If there was a large disagreement between the eyewitness reports and the official theory, as in some early models of the Hindenburg disaster, we might well want to revise till we find agreement, or a clue to the oddities of the eyewitness reports?...

Take the classic case, the assasination of JFK. Eyewitness reports are confusing and contradictory, and the physical evidence is clearly vital, but we might expect some degree of agreement in a correct theory?

Eye witness evidence and scientific observations are not really what I am referring to here. But since you've brought this up, it is worth pointing out that a trained observer is different from a random observer. And some eyewitness accounts are going to be more reliable than other accounts. For example you can identify a person you witnessed if you know them while you may not make an accurate identification if it was someone you didn't know.

Surely one might also argue that in some people a hypersensitive immune reaction gives minor flu like symptoms (but no flu)? I'm assuming flu is a dead vaccine, and sure the correlation of dates is not necessarily significant - for exactly the reasons you say. It would be simple to conduct a trial where the vaccine was administered at a different time of year, or patients were isolated from possible disease vectors? ...

Nice study, but the possibility of a hidden variable remains. Saline solution is just as likely to provoke an immune response as the actual vaccine - it's not the virus making people ill, but there bodies response to the process of vaccination. If you include the no treatment arm however you have a perfectly sound study, and I would fully accept the results.

My flu shot example is so classic that even people in the thread are arguing the anecdotes over the evidence. These conclusions are even less supportable than Miss Kitt's. I think it's safe to say you won't find any evidence that saline provokes an immune response. And I already explained in an above post that there is no evidence the vast majority of people get any reaction from flu shots other than a mild sore arm in a few people and extremely rare reactions such as an allergic reaction.

People do have vasovagal syncope after shots and blood draws. That is generally unrelated to what is in the syringe. But people do not in any study I have ever seen, get placebo illness symptoms from vaccines. I think the range of placebo effect is a lot narrower than is being suggested here. That symptoms are paid attention to is known to cause a perceived increase in symptoms following a flu shot. But again, that doesn't completely explain the widespread myth that flu shots cause illness symptoms. Those supposed after shot symptoms are 99% myth. They simply don't happen. Many people attribute any illness they get within a month or even longer of a flu shot as being caused by the shot. And, they selectively remember so that they will tell you they got sick "after every shot" when in reality they have just forgotten the shots that were not followed by symptoms.

Yes, but we are now back to my original point - neither of the thinsg cited in the article were actually anecdotal. In both cases what happened was simply a false correlation, as in the flu vaccine example. And this happens just as much in real science as in pseudo-science.

Imagine I perform a study of the population of sticklebacks in a river, and measure the variable pollution froma sauce factory. As pollution increases, the stickleback population drops drastically. Applying careful statistics i find the correlation is significant.

If however I fail to note the increase in the population of stickleback-in-Worcester-sauce-loving herons, I may well have made a false corellation. Sure my brain is programmed to accept this - I have noted a correlation between say snogging my girlfriends sister and being dumped, putting my hand in a fore and it burning, and getting drunk at office parties and humping off the roof and physical injury. In almost all cases correlation does indicate causality - in a few it soes not.

There may be some areas where correlation more often than not indicates causality. But it is not so common when it comes to things that tend to have lots of potential variables such as in medical research.

Many years ago I became ill three times after visiting a burger joint near my house. I had eaten there for ears on and off with no ill effects, but now i seemd to be getting sick every time. Why? Well I thought maybe the new management had let hygeine slip. It semed a bit unlikely as the staff were mainly the same though. I thought it through, and kept going. And I was ill again, a fourth time. I switched to chicken burgers, and was fine. It took weeks of patient experimentation and a couple more queasy nights before I discovered it was nothing to do with beefburgers (the false correlation), or hygeine. It was that some items were served with a blue cheese sauce, and for reasons I have still not sicovered eating blue cheese sauce makes me really ill. I tested this by buying aj jar, and waiting till my girlfriend used some in a meal - and siure enough I was ill, even though I did not know it was in there.

If your assumption about cause was correct, then the public health would likely have been alerted to outbreaks of illness associated with that restaurant. I think this shows one of the problems I am talking about. Your study was flawed. Yet you consider it was a valid study.

So my point here is the problem is nothing to do with anecdotal evidence: it's actually false correlation, where the apparent correlation is actually explicable by another variable. The problem is in our lives and in science the overwhelming majority of correlations are significant, which renders us prone to pay attention. Correlation does not require Causality: but in most cases it implies it. When it doesn't, then we have a problem, and make mistakes. ... cj x

In other words, you can't see the flawed thinking going on here. I don't mean to be rude about it. But these are exactly the kind of false conclusions I am talking about.

 

[cj.23]

Hey skeptigirl, no offence at all. OK, so we are dealing here actually simply with the arguments for what Guyatt & Sackett called evidence based medicine, is that right? I read Archibald Cochrane's book from the mid 70's when I worked at the College of Health (1991-1994), and lived through many of the controversies over these issues. The article strikes me as claiming a far wider application than just medicine however, and still evidence based medicine is simply that supported by widespread statistical randomized/double bllind trials as far as I recall - the name is misleading, as all diagnosis are evidence based after all. I have no doubt of the efficacy of EBM, or the excellent medical statisticians out their, like FLS who posts on this forum, but I still do not think the article makes sense. I just wanted you to know a) you are not at all rude, I probably need to clarify and b) that I intend to reply, just been busy and had not seen your latest till just now.

cj x

 

[Skeptic Ginger]

Hey skeptigirl, no offence at all. OK, so we are dealing here actually simply with the arguments for what Guyatt & Sackett called evidence based medicine, is that right? I read Archibald Cochrane's book from the mid 70's when I worked at the College of Health (1991-1994), and lived through many of the controversies over these issues. The article strikes me as claiming a far wider application than just medicine however, and still evidence based medicine is simply that supported by widespread statistical randomized/double bllind trials as far as I recall - the name is misleading, as all diagnosis are evidence based after all. I have no doubt of the efficacy of EBM, or the excellent medical statisticians out their, like FLS who posts on this forum, but I still do not think the article makes sense. I just wanted you to know a) you are not at all rude, I probably need to clarify and b) that I intend to reply, just been busy and had not seen your latest till just now.

cj x

The thread is about addressing the underlying causes of why people fail to think critically rather than just addressing the facts people have wrong. The thread discusses one example, brain functions. We know that some characteristics of the human brain frequently lead to false conclusions about causal relationships when only coincidental relationships actually have occurred. In other words people naturally look for explanations for the things they observe. In order to think critically and therefore draw more valid conclusions, we need to recognize the underlying mechanisms which cause people to go wrong.

I used the term, anecdotal evidence. It confused some people. My reference to the term, 'anecdotal evidence' in medical research was merely to clarify the definition for the discussion. Trained observers and the nature of eye-witness accounts are not what the discussion is about. The discussion is about the strength of personal experience (more than just witnessing something) and how that impacts people's beliefs to the degree they don't believe the scientific evidence even when it is more reliable.

I used the 'flu shot false conclusions' as an example. But it caused some problem in the discussion because many people in the forum also are affected by experiences, or people they trust's experiences when it comes to false beliefs about flu vaccine. It would seem that to discuss how personal experience can lead to false conclusions, one must remember, no one is immune. Our brains are designed to draw these conclusions.

That is why the subject of underlying mechanisms for barriers to critical thinking skills needs to be discussed in the first place. And, no, this is not strictly about medical research. Medical beliefs happen to be particularly rife with personal experiences. However, so are psychic experiences and ghost encounter experiences.

 

[Big Les]

Now as a historian, I would note that the most valuable evidence availsble to us is primary witness reports, written often, and without direct observation history would go nowhere as a discipline.

I've wrestled with this (because as a fellow historian I DO think that anecdote is the same thing or on the same order as the "anecdotal evidence" that skeptigirl and Shermer discount), and I think you simply have to accept that accurate history and archaeology just aren't possible. The overwhelming majority of the past is essentially lost to us. Just think about your own life, and how much of a true sense of it some future historian could get from only a handful of documents (for example).

This doesn't make it futile. Imperfect, even fundamentally flawed, understanding, is still worth seeking. And history is always more about those of in the present than those dead and gone. But it does make it (anecdotal evidence) the best we have in terms of the past, and the worst we have in terms of establishing present day events. Still not worthless, but the lowest form of evidence.

Then there's the issue of accepting anecdote regardless - for some things, this risk is worth taking. If someone tells you your house is on fire, a) there's precedent and plausibility. and b) you have a lot to lose by ignoring the evidence and little to lose by following it.

When it comes to what we collectively call "woo", these are things that have never been proven to occur by science, and so anecdotes surrounding such events can (sceptics agree) be rejected out of hand.

Far from crippling it, scepticism is a boon to history, because it means you lower your expectations of what's achievable, you get closer to the truth, and you can still speculate based on the evidence, as long as you make clear that's what you're doing.

 

[Skeptic Ginger]

I've wrestled with this (because as a fellow historian I DO think that anecdote is the same thing or on the same order as the "anecdotal evidence" that skeptigirl and Shermer discount), ...

It is interesting that however clear one describes one's view, people ignore what you say and churn out their own versions.

I do not discount anecdotal evidence. Half the research in medicine relies on anecdotal evidence. What you fail to understand here is the difference between systematically collected anecdotal evidence, and using un-systematically collected stories without controlling for other possibilities meaning you take everyone's word for it that they know something caused something.

It's important that people for example, report beliefs about symptoms they think their flu shot caused. Then it is up to the researchers to test the hypothesis that the shot and not something else caused the symptoms. You don't just say, "OK, that must be a correct conclusion and therefore it is a fact."

When it comes to what we collectively call "woo", these are things that have never been proven to occur by science, and so anecdotes surrounding such events can (sceptics agree) be rejected out of hand.

Far from crippling it, scepticism is a boon to history, because it means you lower your expectations of what's achievable, you get closer to the truth, and you can still speculate based on the evidence, as long as you make clear that's what you're doing.

You have made a false statement here that these anecdotes are dismissed out of hand. To the contrary, they are followed up on with research at least in medicine. If the research shows the conclusion to be wrong, then the individual's conclusion about the relationship between the two things is a false conclusions.

If you wear your hat backward and your ball team wins the game, did your hat make that happen? Was your backward hat required in order for that win? If it happened a couple times, lots of people believe the hat actually had an influence. That doesn't mean it did.


So it would help if you got your facts straight here about who is dismissing what evidence and which conclusions.