Poll: Accuracy of Test Interpretation

[Wrath of the Swarm]

Now tell me how, if you are treating every individual the same, that is as members of this "population" with 0.1% incidence, you can still simultaneously declare that the chance the test has given out the correct result is only 9.02%.

Strawman. The conclusion from a positive result that a given person has the disease has the 9.02% chance of being correct.

The chance of the test being wrong for any person is 1%. The chance of the test being wrong for a particular subset of people isn't necessarily the same.

Don't you understand any statistics at all?!

 

[Paul C. Anagnostopoulos]

Mentioning one's own book is an appeal to authority?

~~ Paul

 

[Rolfe]

Mentioning one's own book is an appeal to authority?

Probably. One has to counter the ad-homs somehow.

Rolfe.

 

[geni]

For my argument to be valid, I would have to use the same question as was used in the studies. That's a basic point of experimental design - which Rolfe clearly knows nothing about. You can't test the validity of an experiment without recreating its structure.

The test is already invalid as a repeat due to the change in context. Thinking that you have equiverlence suggests that you havn't though th through properly

There are no unstated assumptions

Even your supporter dissagrees with you here. You assumption is that p₁=p₂. You can see this in you own calculations back on page one

 

[Wrath of the Swarm]

Mentioning one's own book is an appeal to authority?

The implication is that she's an expert on the subject, as she had cause to write a book.

Since she's demonstrating a complete lack of comprehension in this thread, I'd hate to read her book.

geni: Nothing about the context was changed. May I ask what you think changed between my question and the one presented in the studies?

 

[steve74]

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

Point 3: The question, as it stands, is perfectly comprehensible.

Point 4: It doesn't matter why the doctor ordered the test. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

Point 5: If you want to link together multiple tests, fine. The analysis of the results becomes much, much more complicated. We have to determine the error rating(s) of the first test, the degree to which the first and second tests are independent, determine whether the initial tests are actually uniform (doctors can plausibly use many different symptoms to develop suspicious, and the probabilities for each might not be the same) and so forth.

Point 6: You're only making yourself look more like a fool the more you continue this, Rolfe. Admit you were wrong and get it over with.

With regard to point 1: I'd like to know how you know the question you asked was the same question asked in the research, when you fully admit that you don't can't remember which research you read?

I'd guess you are referring to Casscells et al. (1978) (certainly the most famous example of a base rate neglect study in med students) where a similar example was given to a group of faculty, staff and fourth-year students at Harvard Medical School. Only 18% got anywhere near the correct answer. The question, in their study, was phrased rather more exactly than in your question, specifically:

“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?_ ____%”

As you can see this study was quite specific in mentioning false positives rather than your vague talk of 'accuracy'. Your question is underspecified as has been pointed out to you many times.


Casscells, W., Schoenberger, A. and Grayboys, T. (1978). Interpretation by physicians of clinical laboratory results. New England Journal of Medicine, 299, 999-1000.

Of course if you were referring to another study I'm sure you'll cite it.

 

[Interesting Ian]

Too easy, but what's the chances of you having the disease if you get three positives in a row?

Adam

Just looked at it now and I thought . .ummm . .surely it's very obviously 10%. But when I voted I scracely thought that 13 out of the previous 14 would have voted the same as me!

I'm agreeing with everyone. This is seriously worrying

 

[Rolfe]

Strawman. The conclusion from a positive result that a given person has the disease has the 9.02% chance of being correct.

The chance of the test being wrong for any person is 1%. The chance of the test being wrong for a particular subset of people isn't necessarily the same.

Don't you understand any statistics at all?!

Wrath, it seems I understand them better than you.

You are simply continuing to assert that under the very restricted conditions you impose, this figure is correct. What I am trying to get through to you is that in the real world, these conditions do not apply.

Just tell me one actual diagnostic test which has been through proper sensitivity and specificity testing and has come out with identical results for both. And I don't want vague manufacturers' claims, I want your actual studies, with actual patients and actual reference testing for comparison. I can find you plenty that aren't identical. (Unfortunately they're in a box in my office, not on the Net, because the Veterinary Record has only dragged itself into the IT age this year.)

[Digression. If you go through the entire thousand cats for 0.1% incidence, 99% sensitivity and 99% specificity, you do indeed get 99% of results correct for that particular combination. Though I doubt if you knew that. I'd still like to see you do the working.]

No, it's easier than that JUST TELL ME THE CALCULATION YOU USE TO DERIVE THAT 99% ACCURACY/ERROR RATE FIGURE. Can you do it? Given that even you can't possibly assert that EVERY test has equal sensitivity and specificity.

Or simply admit that you meant to say "99% specificity" in the first place (because that was all we needed to know), but were a bit loose in your terminology.

Now, back to the more interesting question.

If you test everybody in the world, and pool their results, then your figure is correct. 9.02% of the positive results are correct. (And 99.99898% of the negative results are correct, to save you a job.)

Guess what. We don't care. We don't test everybody in the world, and even if we did, we wouldn't be testing them as unidentified zombies, but as individuals with their own characteristics.

If the individual in question is in a group less likely than the whole to be affected (that is clinically normal) then we reduce the probability of the positive result being correct accordingly, by plugging in the correct incidence in the group to which this patient belongs. As the prevalence to be considered has to be the prevalence in the population to which the patient belongs.

But if he is in a group more likely to be affected, we increase the probability of the positive result being correct.

The bare probabilities applicable to the population of "everybody in the world" may be of interest to statisticians, but they are only a starting point when making a clinical decision.

If you had worded the question in a totally abstract way, asking for the percentage of positive results which would be right assuming that the incidence in the population being tested was 0.1% (and we'd managed to agree that it was the specificity which was 99%), then fine.

But you can't ask a question about an individual patient, with vital information which you simply leave out, and continue to assert that this validity still maintains.

Rolfe.

 

[Paul C. Anagnostopoulos]

Ian said:
I'm agreeing with everyone. This is seriously worrying

That is because this thread is causing a rampant, free-floating vortex in the space/time continuum. Consider the players. Consider the opinions. Consider the personalities. Nothing like this has ever occurred in this universe before.

~~ Paul

 

[Wrath of the Swarm]

There is no vital information being left out. You've given everything you need to know about the conclusion and the factors affecting the test.

Symptoms are irrelevant. They matter only to presorting, which is a form of test. Combining two tests makes everything much more complex, and it's not the situation asked about in the research (I mentioned several posts after I complained about not being able to find it that I was looking in the wrong place for references).

What if the disease we were discussing was HIV, or a similiar infection with few (if any) obvious symptoms?

The test has objective and universal error rates that are sometimes the same for alpha and beta and sometimes different. In this example, they are the same. We do not want to consider extraneous details that would make answering the question harder - you can barely manage this one as it is.

I am pointing out that, when presented with a simple question involving the use of a diagnostic test with known accuracy in a particular generic situation, the vast majority got it wrong.

I suspect that if you had asked them, most of them would have predicted they'd get it right.

This is the problem. Rolfe, as the resident mindless-defender-of-the medical-status-quo, denies that there is a problem and attacks that which makes the existence of the problem clear. When dealing with people whose positions are grossly incorrect, her mindless rancor actually aids her. But she can't tell the difference - she'll attack anything and everything.

 

[Rolfe]

The question, in their study, was phrased rather more exactly than in your question, specifically:

“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?_ ____%”

Now that is valid. State clearly that the figure being given is the "false positive rate", which is simply 100 - specificity (or to be absolutely exact, the other way around, that is specificity is defined as 100 - the false positive rate). So, specificity is 95%. Perfectly clear.

"ASSUMING YOU KNOW NOTHING ABOUT THE PERSON'S SYMPTOMS OR SIGNS."

Exactly. State that clearly, and we understand the question. There is only one possible answer. In this case, the probability of the result being correct is only 1.87%. (We can say nothing about the probability of a negative result being right because we have not been told the sensitivity.)

I can see how only 18% were anywhere near. The combination of quite poor specificity (95% isn't very good) and very low disease incidence pushes the number very low, and if you're guessing and not working it out then it's not very intuitive.

Wrath's basic premise isn't so wrong. Left to themselves without being led through the arithmetic, a lot of medics and vets do make a wrong guess. However, that is one of the things clinical pathologists are for. You ring me up and you say, do I trust this, or do I have to do further testing.

Or in the real world, where knowing nothing about the patient's clincial signs or the reason the test has been requested isn't on the agenda, the intuition jumps to

• a negative test in a patient who probably doesn't have the disease is probably right
• a positive test in a patient who probably dosn't have the disease is probably wrong
• a positive test in a patient who probably does have the disease is probably right
• a negative test in a patient who probably has the disease (least firm situation as getting right over to the right-hand side of the graph isn't that common clinically, but certainly should not be taken at face value, and needs follow-up)

Let the experienced clinician do it by true instinct, and they will be right most of the time without working it out, so long as you allow them all the information they usually have.

Ask the question in a deliberately statistical manner, without allowing a full calculation, and the intuitive answer may well be wrong.

The danger lies in drumming the bare statistics into people (as Jacobson did) without qualifying it. The result is often that the simplified position of "negative results are always right, positive results are always wrong" might be the take-home message.

Which is where the NegTest came in.

More common is the (also false) position that negative results are always right, and positive results always have to go to the reference method. Not so bad, but it leads to unnecessary doubt being raised about the obvious true-positives in the obvious clinical cases, and underappreciation that even your negative results can be wrong, again especially if the patient is showing clinical signs.

So without fully understanding the complete range of possibilities, as Wrath clearly doesn't, you end up with people who are making worse seat-of-the-pants assumptions if you approach the problem this way, than you'd get if you left them alone.

Just to recap. Wrath got the question wrong. That's quite obvious. He didn't state the terms as clearly as the original did, if this is the original which is being quoted. (And if he has an original which asks his exact question, first show me the reference and second, it was a bad study.)

Yes, we could see what you meant. But please don't continue to assert that this was the only possible way the question could be taken.

Rolfe.

 

[Rolfe]

What if the disease we were discussing was HIV, or a similiar infection with few (if any) obvious symptoms?

More ad-homs. Wrath, this is a substitute for rational argument, not part of it.

Now you're doing the "what-if" game. You didn't tell us anything, originally. So you've been shown the wide range of possibilities which might arise, depending on the precise nature of the "what-if" you want to plug in.

It comes to the same thing, anyway.

If the doctor had no reason for testing other then ticking the box for the insurance company, your figures are in the right ball-park. Subject to subtracting the percentage of HIV-infected people with recognisable clinical AIDS from the "incidence in the population being tested" figure.

If on the other hand the doctor noticed a lesion suspicious of (at an extreme example) Kaposi's sarcoma that the patient had overlooked, that bet is completely off.

The original study correctly and honestly stated that (however improbably) the people being asked were not to know anything about the clinical signs. Wrath didn't appreciate the necessity for this part, and so we have all these futile attempts at self-justification.

Rolfe.

 

[Wrath of the Swarm]

If we're talking about a given patient's symptoms, we're not talking about the overall accuracy of the test any more. The entire point has been lost. We're also no longer talking about a single test, but a combination of two tests that probably aren't independently distributed.

You're completely incapable of admitting that you could be wrong, much less actually so.

Medical professionals simply aren't very good at thinking explicitly and rationally through their job. They rely on instinct and acquired knowledge.

That is a major weakness. It means that they will be very poor at detecting flaws in established medical techniques - and they certainly are.

In some cases, it also means that they will deny reality as long as they don't have to admit they don't know what they're talking about.

I'm done. Peace out.

 

[steve74]

If we're talking about a given patient's symptoms, we're not talking about the overall accuracy of the test any more. The entire point has been lost. We're also no longer talking about a single test, but a combination of two tests that probably aren't independently distributed.

You're completely incapable of admitting that you could be wrong, much less actually so.

Medical professionals simply aren't very good at thinking explicitly and rationally through their job. They rely on instinct and acquired knowledge.

That is a major weakness. It means that they will be very poor at detecting flaws in established medical techniques - and they certainly are.

In some cases, it also means that they will deny reality as long as they don't have to admit they don't know what they're talking about.

I'm done. Peace out.

Your're going Wrath? So soon?
That's a shame because I thought you might stick around and answer my question as to why you claim to be posing the exact same question posed in a study that you, er, can't even remember the name of.

Your basic point stands that some doctors need better training in probability theory but your example was badly phrased and your unwillingness to admit that fact makes you look very foolish indeed.

 

[Interesting Ian]

Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

I agree.

 

[Rolfe]

If we're talking about a given patient's symptoms, we're not talking about the overall accuracy of the test any more. The entire point has been lost.

YOU were talking about raw probabilities, your problem is you never made that clear. You assumed the doctor was not allowed to have information he clearly did have, and demanded without even specifying that this information (which in the real world would be a vital part of the decision-making process) should be disallowed. And got badly tangled as a result. (Once again, we now know the original study did not make that mistake.)

Yet again you've used the term "overall accuracy" without specifying what the heck you mean.

• Probability that any random positive result is right, with these given parameters? (positive predictive value)
• Probability that any random negative result is right, with these given parameters? (negative predictive value)
• Percentage of affected patients who will test positive? (specificity)
• Percentage of unaffected patients who will test negative? (sensitivity)

These are all clearly defined terms. Please quit with this "99% accuracy", "error rate", "overall accuracy" without saying which of these, or which function of these you refer to. Your persistent refusal to get your terminology correct has not only been the cause of all this from the start, it demonstrates clearly that you only have the most superficial grasp of the subject.

Your refusal to explain how you would calculate this "accuracy" figure you initially specified as 99%, reveals your lack of understanding of what you are talking about.

Your off-the-cuff formulation of the problem with two crucial errors in it with respect to the original was perhaps understandable, in someone with no practical experience of the field, but your repeated refusal to admit to these errors, or to examine the effect of failing to define these terms on the range of answers that can be admitted, is juvenile and ignorant.

The problem is that most of us saw where you were at the very beginning, and what your unstated assumptions were, and your agenda in putting the question the way you did. It was a piece of cake to solve the arithmetic, for your assumptions.

The interesting bit has been to look at how limiting these assumptions were, and how different the answers might be if you allow equally or indeed even more probable assumptions. Then when we see that an experienced clinician might be instinctively operating from the second set of assumptions, while the statistician wants to chain him down to the unrealistic first set of assumptions, we get a better overall appreciation of the implications of the exercise as a whole.

Wrath, however, will never understand this. For someone who seems to pride himself on his thinking, he's distressingly "in the box", and unable to accommodate his understanding of the subject to take account of assumptions other than those he made himself.

I'm off to bed, goodnight.

Rolfe.

 

[Rolfe]

I agree. (with Wrath, that is.)

I rest my case.

Rolfe.

 

[Interesting Ian]

I rest my case.

Rolfe.

So Rolfe, what have I done to annoy you?

 

[Dancing David]

All three. You were told the base rate of the disease in the population, and the accuracy of the test, which includes with the alpha and beta rate.

It's nice to see that most of the people who bothered responding did indeed choose the correct answer, although now we'll never know how many formuites would have chosen differently. A shame.

Some of us vote before we read the thred, that is why I am the sole vote at 70%, which was just my guess.

I think that it is important in testing to know which way he test runs, Does it run to false positives or to false negatives.

Most people who use the OTC pregnancy tests are encouraed to find out that they generaly test positive in error and not negative in error.

And here in the US it depends on your test results and your insurance. I have known people who were sent home when they had cardiac enzymes indicating they were having a severe MI, because they didn't have insurance.(Happened in Decatur Illinois, two months ago to an african american, self employed and uninsured, 55 years old.)

 

[Rolfe]

So Rolfe, what have I done to annoy you?

Sorry Ian, nothing personal.

But I know who was considered to have lost the "does 0.99999(infinity) equal 1?" argument, even though as someone who hadn't explored that particular problem in any depth, I initially and instinctively thought your position was correct.

Rolfe.