Poll: Accuracy of Test Interpretation

[Wrath of the Swarm]

I have read them. The question is whether you ever do - or if you think about what you read.

The quote you've presented from this person you argued with is utterly correct. "A negative test result .... is reliable in predicting that a cat does not have the infection/disease. .... negative test results are good prognosticators of non-infected cats even if the sensitivity .... of the test is not good."

That's the simple truth. It may not be a desirable test in a wider sense because it misses infected cats... but that has nothing to do with the points you quoted.

Obviously, the population tested will affect the results, in the sense that if everyone has the condition, there's no such thing as a false positive (or if everyone lacks it, there's no false negative and so forth). What's important is the accuracy of the test (whether in general or in distinguishing between alpha and beta error), because that is an objective, universal property of the test that doesn't change across populations.

Your hysterical rantings don't echo, Rolfe.

 

[Wrath of the Swarm]

Why perform a test that misses 10% of the infected patients?

Maybe because not performing the test misses 100% of the infected patients by definition.

Because tests with low false negative rates usually also have high false positive rates? Because overall accuracy in diagnostic testing is extremely difficult to accomplish?

No, obviously he's just performing the test to make money off of his victimized clients, and you're swooping in to save the day!

 

[ceptimus]

If we have a test for a disease that doesn't exist in the population (say we have a test for smallpox - and smallpox has been eradicated) then no matter how 'good' the test is, any positives that it reports will be wrong.

I think when Wrath said the test was 99% accurate, and gave no further information, then you have to assume that out of every 100 people tested, whether they have the disease or not, 1 person will be told the incorrect result.

I realize this is not a likely scenario in the real world - I was just treating it as a puzzle. I like puzzles.

 

[Rolfe]

Your hysterical rantings don't echo, Rolfe.

Anyone else care to say if I'm ranting hysterically?

Wrath, you made two classic errors of presentation when you posted that question. With, I note, the not-very-well-hidden agenda of showing how clever you are and how stupid medical professionals are.

The first was the one which was obvious to everyone, where you quoted an "accuracy" figure which was meaningless as it stood, without stating that you were implying that sensitivity and specificity were both 99%.

I assumed this was just a sloppy way of saying that specificity was 99%, because sensitivity wasn't relevant to the question anyway. You have however dug yourself a deeper and deeper hole by declaring that this "99% accuracy" is some sort of combined sensitivity and specificity figure. This is a meaningless concept. You can't simply take an arithmetical mean of the sensitivity and specificity and call it "accuracy", and to assume (and to assume that we would assume) that they were equal is ludicrous.

100% sensitivity, 50% specificity. 75% accuracy?
100% specificity, 50% sensitivity. 75% accuracy?
75% sensitivity, 75% specificity. 75% accuracy?

These are three very different products, and nobody in their right mind would consider them all under the same banner, as "75% accuracy". By the way, if these were all that was available, which would you choose to stock, and why?

We can go round the houses on this relatively minor point all night.

However, the more important point is, why did the doctor decide to do the test? He knows that, and he will take it into account in deciding whether to believe a positive result or not.

Scenario 1. Wrath goes to the doctor for an insurance medical, feeling fine. Doctor checks him over carefully, and finds nothing wrong. But the insurance form requires that he tick the box to test for Galloping Varicella, as a routine.

In that situation, a 9.02% probability that the test Wrath described is correct is actually an overestimate unless all people with Galloping Varicella are clinically normal. What you need is the incidence of Galloping Varicella in the clinically normal population - which will undoubtedly be less than the incidence in the population as a whole, which of course includes those who are in the last stages of terminal disease from the condition.

Scenario 2. Wrath goes to the doctor for an insurance medical, feeling fine. Doctor checks him over carefully, and notes a couple of worrying things. He has a mole in the middle of his back, where he can't see it, and there is a faint but just perceptible cast to his left eye. The doctor knows that these two features, found together, are very suggestive indeed of Galloping Varicella, in fact he is about 80% sure Wrath has the condition. Although nothing is said about it on the insurance form, he decides to perform the test. Of course, knowing that there is still a 20% chance he is wrong, he just tells Wrath that the test is "routine", to save possibly unnecessary worry.

In that situation, there is a 99.75% probability that a positive result is right.

Do we criticise the doctor if he decides to break the bad news at that point?

This is a classic error. Look at a problem from outside, ignoring the inbuilt assumptions with regard to way of working that people build up over the years. Assume one scenario, and only one, because you don't have the experience to imagine any other. Then ambush some professionals with your assumed scenario, and completely fail to realise that they may be (consciously or unconsciously) answering the question from the point of view of a different scenario, the scenario they are familiar with.

The fact is, there is always a reason for testing, and that reason is part of the interpretation. Insurance requirement, although no suspicion? Strong clinical suspicion? Wrath believes that doctors do the test without thinking about this. They don't. But it's often so instinctive that you can make them look unreasonably stupid by pulling a Wee Kirkcudbright Centipede on them. (Note, that text is all that is available, but it is the result of a bad transcription from a sung version by an American who didn't understand the lingo. For a start, the dance is the "Palais Glide", not the "parlor glide". What were they thinking of!)

And just to note this again, five out of five posters got the "right" answer before I posted a syllable, so Wrath's stated purpose of showing how bad the people on this forum are at this sort of reasoning was doomed from the start.

But we're having a much more entertaining discussion now, aren't we?

Rolfe.

Late edits only for spelling typos.

 

[Dragon]

This site has a nice discussion of the issue in simple terms. More importantly, it references research studies and asserts that the problem has been replicated many times.

Okay, so it's not a stellar reference... but I think it proves my point. My problem is what medical resources don't discuss the issue much - you'll find a lot more if you do a general Google on "do doctors have problems with Bayesian reasoning?"

Good link, Wrath - pity you didn't read it before your OP.

From the link (using breast cancer as an example) -

Figuring out the final answer always requires all three pieces of information - the percentage of women with breast cancer, the percentage of women without breast cancer who receive false positives, and the percentage of women with breast cancer who receive (correct) positives.

Oh dear - how many pieces of information in your OP?

 

[Dragon]

On the subject of Bayes' Theorem - this thread that ca3799 has just started in GS&P got me thinking - surely it applies to polygraph testing? Can't find anything on http://antipolygraph.org/ about it yet - hmmm....

 

[pgwenthold]

And just to note this again, five out of five posters got the "right" answer before I posted a syllable, so Wrath's stated purpose of showing how bad the people on this forum are at this sort of reasoning was doomed from the start.

But we're having a much more entertaining discussion now, aren't we?

Oh, but it is fun. The clown came in as pompous as could be intent on showing us how much smarter he was than everyone else, and then after fooling no one, got schooled big time.

After having it blow up in his face, then he runs to the literature (where, as demonstrated by Dragon, blows it again), apparently abandoning his attempted exercise.

Entertaining it has been, hmmmm?

 

[Wrath of the Swarm]

From the link (using breast cancer as an example) - Oh dear - how many pieces of information in your OP?

All three. You were told the base rate of the disease in the population, and the accuracy of the test, which includes with the alpha and beta rate.

It's nice to see that most of the people who bothered responding did indeed choose the correct answer, although now we'll never know how many formuites would have chosen differently. A shame.

 

[Dragon]

All three. You were told the base rate of the disease in the population, and the accuracy of the test, which includes with the alpha and beta rate.

...

Nope - we had to assume what you meant by "accuracy". Rolfe has explained this to you already.

 

[Rolfe]

Just a comment.

The first (permanent) part of my sig line wasn't chosen by accident.

Rolfe.

 

[Wrath of the Swarm]

Wrath, you made two classic errors of presentation when you posted that question. With, I note, the not-very-well-hidden agenda of showing how clever you are and how stupid medical professionals are.

Pointless character assassination. The problem certainly doesn't make me look any smarter - I failed it the first time I saw it, many years ago.

It does make you look dumber, but since you're not a medical professional that doesn't quite count, does it?

The first was the one which was obvious to everyone, where you quoted an "accuracy" figure which was meaningless as it stood, without stating that you were implying that sensitivity and specificity were both 99%.

I wasn't "implying" it - it's a consequence of what I said. Sloppy interpretation.

You have however dug yourself a deeper and deeper hole by declaring that this "99% accuracy" is some sort of combined sensitivity and specificity figure. This is a meaningless concept. You can't simply take an arithmetical mean of the sensitivity and specificity and call it "accuracy", and to assume (and to assume that we would assume) that they were equal is ludicrous.

No one's claimed anything about an arithmatic mean - except you.

And it's certainly not ludicrous for the alpha and beta rates to be equal. It's somewhat improbable in the same sense that it's improbable for any two values to be the same, but there's no reason it can't happen.

That's two 'misinterpretations' made by you.

These are three very different products, and nobody in their right mind would consider them all under the same banner, as "75% accuracy".

Except anyone who uses English in the standard manner. Of course, we weren't considering such a situation - the accuracy of the test was established without reference to alpha and beta rates.

We can go round the houses on this relativly minor point all night.

But that would involve discussing how wrong your statements are and how pointless your objections have been, and we don't want that.

In that situation, a 9.02% probability that the test Wrath described is correct is actually an overestimate unless all people with Galloping Varicella are clinically normal. What you need is the incidence of Galloping Varicella in the clinically normal population - which will undoubtedly be less than the incidence in the population as a whole, which of course includes those who are in the last stages of terminal disease from the condition.

That's two tests. The first test involves an examination of the obvious symptoms - it has an accuracy all its own. Considering the results of test B in the light of test A is perfectly reasonable medical practice - but it's not an effective way to determine the accuracy of test B.

You have continually ignored this point, I suspect because you know you've made a major error and want desperately to direct attention away from it.

In the question, there were no other tests specified other than the one I gave you all information on. The potential ability to perform other tests has no bearing on the question I asked.

In that situation, there is a 99.75% probability that a positive result is right.

True. When we discuss that situation, we'll drop you a line.

Wrath believes that doctors do the test without thinking about this.

Liar. You have no idea what I believe, so you make up a position for me that you know you can successfully attack.

The point is that not this is how medical testing is performed. The point is that doctors fail to answer the question correctly. Being the self-appointed forum apologist for the field of medicine in general, you leap to explain how doctors base their judgments on additional clinical data, blah blah blah... ignoring the point that THEY CAN'T CARRY OUT A SIMPLE MATH PROBLEM.

 

[Wrath of the Swarm]

Nope - we had to assume what you meant by "accuracy". Rolfe has explained this to you already.

If I say that I can identify a randomly-chosen card while blindfolded with 80% accuracy, would you have problems understanding that as well? Would you demand I offer accuracy ratings for each type of card?

The test has 99% accuracy; without further specificiation, that means that any response it gives has a 99% chance of being correct and a 1% chance of being wrong. There's your alpha and beta rates right there.

No further categorization is given; none is needed. You had all the information needed to answer the provided question.

 

[geni]

I didn't. I don't know the standard equations so was trying to work it out from scratch and ran into an equation with two unknows which is of course unsolverble. You card analogy is false since the way you have stated it the question being asked is different from t he one in this thread.

 

[Wrath of the Swarm]

Two unknowns? Then you certainly weren't working the problem properly.

It is perfectly reasonable to talk about a test that has an accuracy rating. Not all tests have different rates of false positives and false negatives - and even if they did, we don't always care.

When given an accuracy and the population prevalence, you had sufficient information to solve the problem. I could have made it more complex and somewhat more realistically probable, but that not only wasn't necessary, it would have invalidated my point that doctors were unable to answer the question correctly. If I changed the question, why would I bring up those studies?

Admit it - your objection is groundless.

 

[Rolfe]

Sob.

There are an infinite number of answers to this question, based on the information Wrath didn't clarify. However, Wrath chooses to declare only "his" answer to be correct.

Once again, Wrath, the scenario you post cannot exist in the abstract. You didn't tell us why the doctor did the test.

If it was for no other reason than because it was a box that had to be ticked (for example in an insurance medical), your 10% probability of the positive result being correct is in fact an overestimate, because you didn't tell us the incidence of the condition in the population with no suspicious clinical signs (less that 0.1% obviously, but we don't know how much less).

If it was because you came in with clear clinical symptoms suggestive of the condition, then the probability of the positive result being correct is pretty high (depending on a number of clinical factors).

Forgive me if I'm inclined to assume that if the doctor decided the result was correct, it might have been because he knew he was in the latter scenario.

You cannot put forward a hypothetical situation like this, then get miffed when people point out that your "correct" answer is only correct if a number of details which you haven't specified are exactly the way you have tacitly assumed them to be.

You did imply that the doctor's appointment was for a routine checkup, without any particular presenting signs. That's fine. But you didn't say why the doctor wanted to check for the condition. Now you impose more conditions than you originally stated, that he was just doing it for fun, or the greater enrichment or the laboratory, or (more likely) the test was a condition of an insurance policy or an employment contract. It could easily have been because the doctor's clinical acumen smelled a very aromatous rat.

You, however, want to assume the scenario that makes the doctor look a fool.

Now, for God's sake put me out of my misery and tell me what the bloody blue blazes that "99% accuracy" figure is supposed to mean. Stop assuming that sensitivity and specificity are equal, I know and you know and the entire medical laboratory profession knows that only hypothetical tests come like that.

So, for a real-life test, like the ones I deal with every day, which have unequal sensitivity and specificity, how are you calculating what you call "accuracy"?

the accuracy of the test was established without reference to alpha and beta rates

I assume that by "alpha and beta rates" you mean sensitivity and specificity - that's OK, we obviously come from areas with a different vocabulary. But I'd like to get this clear. So, if not like that, how in all that's holy was it established?

(Note, this part of the argument is not of my making. I originally assumed that Wrath meant 99% specificity, since specificity was the only figure relevant to the sum he had set. It's Wrath himself who keeps saying now that the 99% somehow incorporates both sensitivity and specificity. Not my problem if he can't then explain how.)

Wrath. Who can't carry out a simple maths problem? People here got it right. And if you still think I used a crib, I'll repeat that I wrote the spreadsheet I used myself, years ago, and only mentioned it to explain why I could do multiple scenarios of the problem relatively quickly.

We made the assumptions you wanted us to make. We got the "right" answer by your lights. However, we also realised where you were mistaken, which was in assuming that the reason for carrying out the test was irrelevant to the doctor's decision as to whether to go with the result or not.

Deal with it.

Rolfe.

 

[yersinia29]

Wrath is a troll guys. Dont waste your time. He wont address your points, and will just continue to spill his bile. He gets a philosophical thrill out of trying to obfuscate the arguments involved.

Wrath, I'm still waiting for your Bayesian analysis of MRI and x-rays.

 

[Wrath of the Swarm]

Why the doctor performed the test has no bearing on the correct answer! Does it matter why Farmer Brown took away three apples from the box that held seven? No!

And there aren't an infinite number of answers. Without specifying different values for alpha and beta, we consider only error. Alpha and beta values follow from the overall accuracy.

Again: the hypothetical test was 99% accurate, so there was a 99% chance that any result it came up with would be correct. That tells you what the alpha and beta rates are - they're equal in this particular case.

Even you aren't stupid enough not to recognize this, so I'm forced to conclude you're being intentionally deceptive in order to support your 'point'.

 

[yersinia29]

Why the doctor performed the test has no bearing on the correct answer!

Of course it does you fool.

The incidence of lupus in the overall population of women is x%

The incidence of lupus in women with a butterfly rash, photosensitivity, and Raynaud's phenomenon is x+y%

doctors dont just run lupus tests on random women. Therefore, the incidence thats used in calculating false positives and other parameters depends on x+y, NOT x.

 

[ceptimus]

Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

 

[geni]

Two unknowns? Then you certainly weren't working the problem properly.

We we need to know what 99% accurcy means. It means that some of the tests are giving false posertive or negatives. Therfore the inacurcies are due to either false posertives or negatives. What is is the ratio of these inacrucies ah can't work that one out on the data given problem is unsolverble.

It is perfectly reasonable to talk about a test that has an accuracy rating. Not all tests have different rates of false positives and false negatives - and even if they did, we don't always care.

But in this case we do case because it can have a big effect on the answer.

Admit it - your objection is groundless.

You used to work for edexcel didn't you? The problem as stated is unsolverble.