Accuracy of Test Interpretation - The Facts

[Wrath of the Swarm]

The following link contains a great many links to brief papers arguing over the matter.

I think it's clear that there are all kinds of arguments out there.

http://psycprints.ecs.soton.ac.uk/archive/00000357/#html

 

[steve74]

I note that you have failed to support your assertion that you haven't changed your position when evidence was provided to the contrary.

People aren't *completely* unaware of the base rate - but some researchers mistakenly thought they are precisely because performance on such questions is so terrible.

It's most certainly not an illusion.

These would be the same mistaken researchers you were citing to support your point that doctors can't accurately interpret tests, yes?

The following link contains a great many links to brief papers arguing over the matter.

I think it's clear that there are all kinds of arguments out there.
http://psycprints.ecs.soton.ac.uk/a.../00000357/#html

There are indeed all sorts of arguments out there. None of them are arguing your point that because doctors can't answer the sort of question you posed they are unable to interpret test results.

 

[Wrath of the Swarm]

Unable to interpret? No, obviously doctors reach conclusions about test results constantly.

They can't interpret them rationally, which is why they do so poorly when asked to answer a simple question.

 

[steve74]

Unable to interpret? No, obviously doctors reach conclusions about test results constantly.
They can't interpret them rationally, which is why they do so poorly when asked to answer a simple question.

That would be the simple question posed by the 'mistaken researchers' you mentioned earlier? Would you please make your mind up which research you support and which you reject, it's rather tedious to have to keep up with your twists and turns. I have presented evidence as to why the question you posed was flawed. You even seemed to agree for a moment when you mentioned the researchers mistakes and now you seem to be saying that the question had validity. Extraordinary.

 

[Wrath of the Swarm]

Well, I'm sorry you're having such trouble following. I'll try to be more direct.

Point 1: People, even doctors, are extraordinarily bad at answering certain kinds of statistical questions.

Point 2: If the presentation is changed to give more explicit information in a different format, people become more accurate, but not necessarily much more so.

Even when the question is posed in a frequentist manner, and the frequencies of each subgroup are explicitly presented, people do quite badly. I acknowledge that, in one study, people could reach the right answer about 92% of the time, but this study is an exception to the general pattern.

The more thought people are required to put into the problem (as in working out the frequencies of the subgroups by themselves), the worse they do.

As some of the research I linked to points out, doctors when discussing how they analyze test results rarely consider base rate; they use certain heuristics instead.

 

[steve74]

Point 1: People, even doctors, are extraordinarily bad at answering certain kinds of statistical questions.

Statistical questions which have nothing to do with how tests are interpreted in real life.

Point 2: If the presentation is changed to give more explicit information in a different format, people become more accurate, but not necessarily much more so.

Even when the question is posed in a frequentist manner, and the frequencies of each subgroup are explicitly presented, people do quite badly. I acknowledge that, in one study, people could reach the right answer about 92% of the time, but this study is an exception to the general pattern.

Once again you're starting with the 'one experiment' nonsense. As I've already pointed out to you I have referenced five studies (and there are many more) which show that your point is incorrect. This study is certainly not an exception to the general pattern but you seem to ignore all the other studies as they don't sit well with your dogma.

As some of the research I linked to points out, doctors when discussing how they analyze test results rarely consider base rate; they use certain heuristics instead.

Yet even non-medical students take base rates into account when base rates are presented in a realistic way.
Once again though you ignore this rather obvious point because it doesn't fit in with your dogma. Let's return to the original point; you claimed that because medical students & doctors did poorly on the sort of question you posed that they couldn't interpret test rationally. This is manifestly not the case. Merely by changing the phrasing of the question even non-medical undergraduates performed extremely well on such questions. I can conclude only one of three things:
1. You realise you were incorrect but continue to assert that you are correct purely because you do not wish to admit your error.
2. You realise you were incorrect but continue to assert that you are correct because you can't let go of your 'doctors can't do stats, doctors can't interpret tests rationally' dogma.
3. You honestly believe that the question demonstrates what you think it does because you have willfully ignored the evidence to the contrary.
None of them shows your arguments in a remotely flattering light.

 

[Wrath of the Swarm]

The degree of certainty of medical tests is extremely important. I agree that doctors do not interpret tests this way in real life - that's a problem.

You're taking the results of a single study and ignoring all the rest of the data which contradicts your point. This strongly suggests your arguments are ideologically motivated.

I think your point is quite clear, steve74.

 

[steve74]

You're taking the results of a single study and ignoring all the rest of the data which contradicts your point. This strongly suggests your arguments are ideologically motivated.

Once again you're willfully ignore the other studies I have referenced. I am not relying on one study and you know it. As for you asserting that my position is ideologically motivated well that's just f*cking hilarious. You claim there is no critical thinking on these boards but you just trot out received opinions from outdated studies and when new evidence challenges your dogma, you stick your head in the sand.

Edited - spelling

 

[yersinia29]

WOTS has a bias/agenda against doctors. In 90% of the threads he posts in, he reveals this bias.

Its perfectly clear now that contradictory evidence has been presented, he is so fervently anti-doctor that he is blinded.

WOTS apparently has a complex regarding doctors. He probably applied for medical school and got rejected; as a result he is now on a crusade to try and prove that doctors are wrong or dumb or stupid or whatever.

WOTS criticizes others on here for siding with Rolfe and says they are not free thinkers. He then reveals his hypocrisy by refusing to change his arguments in teh light of evidence which clearly contradicts his position. I've been calling WOTS a hypocrite and a non-rational crusader since I've started posting. This is just further confirmation.

 

[yersinia29]

The degree of certainty of medical tests is extremely important. I agree that doctors do not interpret tests this way in real life - that's a problem.

Real life is the only thing thats relevant. If doctors screw up on theoretical statistical examples but get them right when the same data is presented in a practical, real-life context, then your argument that doctors cant interpret tests correctly is bunk.

 

[T'ai Chi]

This question was given to a group of faculty, staff and fourth-year students at Harvard Medical School.

From the original study:
...

Was this a take-home question, or one they had to answer on the spot?

The basic argument is that human reasoning does not approximate Bayesian reasoning.

Heck, I'd go as far as to say human reasoning isn't even good with frequentism (think: birthday problem, monty hall problem, and others) in many situations.

 

[steve74]

Was this a take-home question, or one they had to answer on the spot?

They answered on the spot.

Heck, I'd go as far as to say human reasoning isn't even good with frequentism (think: birthday problem, monty hall problem, and others) in many situations.

Maybe so, but those are different problems to test interpretation. Obviously with more complex problems people will struggle.

 

[davefoc]

I agree with WOTS here, I think.

WOTS point was that doctors performed poorly when tested on their understanding of a concept which is a critical aspect of determining the reliability of a diagnosis which, would seem to be a critical aspect of the job of many if not most doctors.

The fact that rewording the question in a way that improves the likelyhood that a doctor is likely to get the question right seems not to be relevant to WOTS's point unless WOTS worded his question in such a way that masked the fact that the testee did actually understand the concept.

I don't think WOTS did, but I understand that there might be a disagreement there.

It also seems like the kind of question that a doctor should routinely see in his training and should not have difficulty with even if the wording wasn't perfect.

I sympathize to some degree with where WOTS is coming from in general with regard to doctors, but I lack the facts to back up my thoughts with regard to this. I see old people inundated with medications and I really am skeptical about how much science is behind that. It has been stated many times about how a few medical deveolopments such as vaccines plus improved nutrition and sanitation are responsible for the preponderance of the improvement in life spans. So why all of a sudden do old people need masses of medications if things aren't all that much better than before masses of medications for old people was the norm?

 

[drkitten]

I agree with WOTS here, I think.

WOTS point was that doctors performed poorly when tested on their understanding of a concept which is a critical aspect of determining the reliability of a diagnosis which, would seem to be a critical aspect of the job of many if not most doctors.

The fact that rewording the question in a way that improves the likelyhood that a doctor is likely to get the question right seems not to be relevant to WOTS's point unless WOTS worded his question in such a way that masked the fact that the testee did actually understand the concept.

I don't think WOTS did, but I understand that there might be a disagreement there.

It also seems like the kind of question that a doctor should routinely see in his training and should not have difficulty with even if the wording wasn't perfect.

I sympathize to some degree with where WOTS is coming from in general with regard to doctors, but I lack the facts to back up my thoughts with regard to this. I see old people inundated with medications and I really am skeptical about how much science is behind that. It has been stated many times about how a few medical deveolopments such as vaccines plus improved nutrition and sanitation are responsible for the preponderance of the improvement in life spans. So why all of a sudden do old people need masses of medications if things aren't all that much better than before masses of medications for old people was the norm?

Um, because masses of old people wasn't the norm? No old people -> no medication for old people. If you want to go back to a median life expectancy of 35 years, we can probably reduce geriatric drug expenses by a considerable margin. A gerontologist friend of mine explained it to me once as "all deaths before 65 are preventable." Once you get past 65, it becomes more difficult (and very expensive) to keep people alive and functional.

The problem is that the definition of "understanding" as used in this thread is deeply, deeply flawed. It's no surprise that humans are bad at "formal" reasoning; this has been known since at least Wason and Johnson-Laird's 1970 card experiment. People are much better at situated cognition. Doctors, being people, are not immune to this effect.

So what?

Let me be absolutely clear about the effect I refer to here. People have difficulty interpreting statements like "If there is a vowel on one side of the card, there must be an even number on the back" in an experimental context. People have no difficulty interpreting "If you are drinking beer, you must be over twenty-one." The statements are formally identical. Even
graduate students in logic screw this task up. People are better at situated cognition.

Doctors lack an "understanding" of Bayesian statistics? So what? They're not statisticians, they're doctors. The question is whether or not they can perform the tasks to which they have been trained and that society expects of them (producing proper diagnosis and treatment in a clinical setting).

 

[Wrath of the Swarm]

So go get a screening test for a serious disease and see whether it's still a problem.

Especially if the doctor wants to do a biopsy to confirm the diagnosis.

 

[drkitten]

So go get a screening test for a serious disease and see whether it's still a problem.

Especially if the doctor wants to do a biopsy to confirm the diagnosis.

I still fail to see the problem.

Let me spin it around for you. You go to a doctor to get screened for medicobabbleosis, and the test comes back positive.

What do you want the doctor to do, ignore the test results? Why did you want to be screened in the first place, then?

 

[Wrath of the Swarm]

Of course not. Nor do I want the doctor to conclude that I have the condition and begin giving me a treatment with health prices.

The example you used, of the rule "if you're drinking beer, you must be at least 21" - what would you say if bartenders couldn't answer it properly?

Now if we change the subject to diagnostic testing, and doctors can't answer properly, what does that tell you? Isn't that question already in a context they should understand?

 

[steve74]

Let me be absolutely clear about the effect I refer to here. People have difficulty interpreting statements like "If there is a vowel on one side of the card, there must be an even number on the back" in an experimental context. People have no difficulty interpreting "If you are drinking beer, you must be over twenty-one." The statements are formally identical. Even
graduate students in logic screw this task up. People are better at situated cognition.

Doctors lack an "understanding" of Bayesian statistics? So what? They're not statisticians, they're doctors. The question is whether or not they can perform the tasks to which they have been trained and that society expects of them (producing proper diagnosis and treatment in a clinical setting).

What he said.

Interesting paper on Wason's selection task for anyone interested.

 

[steve74]

The example you used, of the rule "if you're drinking beer, you must be at least 21" - what would you say if bartenders couldn't answer it properly?

The point is that people can answer this sort of question well. They only struggle when questions are phrased in ways that don't relate to human experience. Most people don't understand the language of formal logic but that doesn't mean they are incapable of logical thought. The paper I linked to in my previous post covers this topic in great depth.

 

[drkitten]

Of course not. Nor do I want the doctor to conclude that I have the condition and begin giving me a treatment with health prices.

So you're saying that you want the doctor to order the biopsy (or other more definitive test) to confirm the diagnosis, then? But in this case, it doesn't matter whether or not he can guess the probability of a false positive, since the biopsy will clear that up.

Fortunately, this is standard medical practice.

The example you used, of the rule "if you're drinking beer, you must be at least 21" - what would you say if bartenders couldn't answer it properly?

Now if we change the subject to diagnostic testing, and doctors can't answer properly, what does that tell you? Isn't that question already in a context they should understand?

No, it's not, as has been pointed out upthread. Doctors don't usually deal with abstract incidence percentages (when presented with incidence frequencies, they do a lot better), nor do they usually deal with test results absent a clinical evaluation. In fact, doctors are specifically trained to disbelieve test results if they disagree with the clinical presentation.

Furthermore, doctors almost never need to make a definitive diagnosis on the basis of a screening test. Standard practice is to order another, better test when a screening test is failed, and to produce a diagnosis and treatment plan only with the results of more extensive testing are in. Jumping to a diagnosis on the basis of a single preliminary test is strongly frowned upon.

So you're presenting doctors with data in a format they don't usually use, and asking them to make a judgement they would ordinarily be disciplined for making. It's not surprising that an extraordinary situation would give poorer performance. Wason could have told you that thirty-five years ago.