Poll: Accuracy of Test Interpretation

[Badly Shaved Monkey]

From a post of mine on page 4:



Now: let's say that the chance of false positives isn't the same as false negatives. Oh, the first is .40 and the second is .30, just to pick two random numbers.

Now, let's say we use a group of 10 sick patients for the test. We get no false positives (since everyone is sick) and 3 false negatives. So the accuracy is 3/10 or .30.

Now let's use a group of 10 healthy patients. We get no false negatives and 4 false positives. Accuracy is .40.

Now let's use 5 healthy and 5 sick patients. We get 5 * .4 false positives and 5 * .3 false negatives, for a total of (on average) 3.5
wrong answers out of ten.

See? The accuracy changes depending on the population.

Now let's pretend the false positive rate is .40 and the false negative rate is also .40.

In the first case, we get an accuracy of .40, just like the second case.

In the third case, we get 5 * .4 false positives and 5 * .4 false negatives, for an overall accuracy of .40.

No matter what sample population I give the test, it will always have an accuracy of .40.

Thanks for finding that.

I see the problem now. You think that by specifiying accuracy to be 99% and giving no other information means that it must mean sens and spec are both 99% or else you would have given us those parameters separately. I'm not sure how we wwere to know that, but I can see that is probably what you thought.

Yes? At last?

 

[Badly Shaved Monkey]

As you seem more interested in arguing than providing the formula that answers my puzzle, here it is:

A = (1 - P) * F_p + P * F_n

Where:

A = accuracy
P = proportion of population infected by the disease
F_p = proportion of false positives when only uninfected persons are tested.
F_n = proportion of false negatives when only infected persons are tested.

Note that all these are proportions (in the range 0 to 1) and not percentages. Multiply by 100 to express them as percentages.

But if anyone has learned anything from this thread, it should be that blindly plugging numbers into a formula you don't understand is a bad idea. That and the fact that you shouldn't let emotion enter into your argument, especially if mathematics or science are the the subject being argued over.

Forgive me for asking, but what is the provenance of this definition given the part of the problem seems to be a variety of uses of the term 'accuracy'?

 

[Badly Shaved Monkey]

Anyway, I must leave this for the moment. Patients waiting and blind clinical guesses to be made.

 

[yersinia29]

BSM,

Wrath makes the mistake of assuming that the only relevant issue here is strict mathematics.

From a strict mathematical sense, "accuracy" can be a substitute for both sensitivity and specificity values.

That does not change the fact that its a poor way to describe the problem.

 

[Hellbound]

I know I said I was done, but I couldn't pass up this little gem

Now: let's say that the chance of false positives isn't the same as false negatives. Oh, the first is .40 and the second is .30, just to pick two random numbers.

Now, let's say we use a group of 10 sick patients for the test. We get no false positives (since everyone is sick) and 3 false negatives. So the accuracy is 3/10 or .30.

Wait a minute. 7 of the responses were accurate, so the accuracy is .30?

Now let's use a group of 10 healthy patients. We get no false negatives and 4 false positives. Accuracy is .40.

Again, 6 results were accurate, but the accuracy is .40?

Now let's use 5 healthy and 5 sick patients. We get 5 * .4 false positives and 5 * .3 false negatives, for a total of (on average) 3.5 wrong answers out of ten.

So is the accuracy here .65 or .35? Educate me, oh great statistics expert.

See? The accuracy changes depending on the population.

Now let's pretend the false positive rate is .40 and the false negative rate is also .40.

In the first case, we get an accuracy of .40(psst...60%), just like the second case.

In the third case, we get 5 * .4 false positives and 5 * .4 false negatives, for an overall accuracy of .40(psst...60%).

No matter what sample population I give the test, it will always have an accuracy of .40.

Psst...60%.

I think you have displayed perfectly your expertise in the field, Wrath. Especially after calling Rolfe to task several times for mistakes she made. That plank must be heavy, no?

 

[ceptimus]

Forgive me for asking, but what is the provenance of this definition given the part of the problem seems to be a variety of uses of the term 'accuracy'?

You are mistaken in your belief that accuracy has a variety of meanings. Of course as an English word, it does have many meanings, as any dictionary will show, but when it is applied to the outcome of tests that give a binary (yes/no) answer, then the meaning is (or should be) totally clear to any mathematician, scientist or engineer.

As I said many pages ago it simply tells you how good the test is - if a test is 99% accurate, it means that (in the long run) for every 100 tests you do, one of the results will be wrong. How could it possibly mean anything else? Can you give an alternative definition that could apply in a mathematical context?

 

[Prester John]

As I said many pages ago it simply tells you how good the test is - if a test is 99% accurate, it means that (in the long run) for every 100 tests you do, one of the results will be wrong. How could it possibly mean anything else? Can you give an alternative definition that could apply in a mathematical context?

and as such, it has virtually no usefullness for evaluating the utililty of medical binary test results. It does not convey sufficient information. Which is why sensitivity and specificty are used.

 

[yersinia29]

ceptimus,

I'll say it again. Whats true in a strict mathematical sense, and whats good language to use in a study are 2 very different things.

Like I said, there is no medical test known to man that has equal sensitivity and specificity values. Thats why "accuracy" is never used in regards to describing medical tests.

That doenst change the fact that in a strict mathematical sense accuracy = sensitivity and specificity. But whats relevant in a pure mathematical sense is not the only issue involved here.

 

[ceptimus]

and as such, it has virtually no usefullness for evaluating the utililty of medical binary test results. It does not convey sufficient information. Which is why sensitivity and specificty are used.

But it does give all the information necessary when it is unchanging, as I demonstrated above.

'A particular test is always 99% accurate.'

This tells you everything you need to know. Of course, you couldn't say this except in the unlikely circumstance that alpha = beta, but when you can say it, it conveys ALL the information.

To say, as you just did, that it doesn't convey sufficient information is simply wrong.

 

[yersinia29]

Of course, you couldn't say this except in the unlikely circumstance that alpha = beta, but when you can say it, it conveys ALL the information.

Change "unlikely" to "unprecedented" or "unheard of"

 

[Wrath of the Swarm]

Oops. How silly of me.

I gave the error instead of the accuracy (which was one minus the value I gave).

My mistake. Of course, I've never claimed to be an expert in anything.

I believe that when the correct terms are subsituted, you'll find that my argument is correct.

 

[ceptimus]

ceptimus,

I'll say it again. Whats true in a strict mathematical sense, and whats good language to use in a study are 2 very different things.

I agree with you of course. The point is that it was Rolfe and her supporters who began the quibbling. She and they were wrong, and have accepted that. Anyone who brings up the same argument, hasn't read and understood the thread, so we have to keep pointing out the same error over and over.

Now, lest it be misunderstood, I agree that two figures should aways be given for any real world diagnostic test. Even in the unlikely circumstance that both were the same, they should still both be given.

Wrath's question was not a real world case - it was a hypothetical example designed to show up people's misunderstanding of simple statistics, and it has succeeded remarkably.

 

[Prester John]

But it does give all the information necessary when it is unchanging, as I demonstrated above.

'A particular test is always 99% accurate.'

This tells you everything you need to know. Of course, you couldn't say this except in the unlikely circumstance that alpha = beta, but when you can say it, it conveys ALL the information.

To say, as you just did, that it doesn't convey sufficient information is simply wrong.

You need to know the false positive and false negative rate. Accuracy as defined :

if a test is 99% accurate, it means that (in the long run) for every 100 tests you do, one of the results will be wrong

does not do that except in exceptional circumstances. As these circumstances do not occur in the medical field, the term accuracy as defined by you is not used.

As i said many pages ago, this is a pure v applied argument. Yes under specified circumstances you can define accuracy to encompass sensitivity and specificity. This is not practical in medical statistics and is not done

 

[geni]

Wrath's question was not a real world case - it was a hypothetical example designed to show up people's misunderstanding of simple statistics, and it has succeeded remarkably.

Well of course you'd say that your the only person who hasn't been wrong yet.

 

[Wrath of the Swarm]

As these circumstances do not occur in the medical field, the term accuracy as defined by you is not used.

WRONG!

Those circumstances are rare in medicine, but they can apply. They apply even more often outside of medicine.

 

[Prester John]

WRONG!

Those circumstances are rare in medicine, but they can apply. They apply even more often outside of medicine. [/B]

The first part of my sentence may have been a very slight exageration, but the second is correct. Accuracy is not used to describe binary tests. Lets put is another way, in the 10+ years i have worked in laboratory doing tests, i have not seen accuracy used to describe a binary test result. this is my practical experience, i am not a statistical expert, but neither am i statitically challenged.

I have no comment on the use outside medicine (of the term accuracy), but as the original question concerned a medical test, it was an innapropriate term to use.

 

[Wrath of the Swarm]

The question is not whether that value is shown on medical test result forms.

In order to understand the concepts of false positive, false negative, true positive, and true negative, the concept of accuracy must be learned first.

If you know those four, you're presumed to know the one.

So... you can't actually find any mathematical errors in the question, can you?

 

[Paul C. Anagnostopoulos]

Ceptimus said:
Wrath's question was not a real world case - it was a hypothetical example designed to show up people's misunderstanding of simple statistics, and it has succeeded remarkably.

Hmm, sounded like it was trying to be a real-world case to me. A hypothetical example would have been worded without reference to medicine.

Do people really misunderstand statistics all that badly, or was this mostly a giant rat-hole having to do with whether a specific sort of accuracy might or might not be relevant under the ambiguous conditions of the question?

~~ Paul

 

[Wrath of the Swarm]

The scenario I provided would work just fine in the real world. It would simply be less likely than scenarios where tests don't have universal accuracies.

That doesn't make a bit of difference to the question - if anything, it makes the question easier to answer than it would have been otherwise.

 

[jj]

Do people really misunderstand statistics all that badly, or was this mostly a giant rat-hole having to do with whether a specific sort of accuracy might or might not be relevant under the ambiguous conditions of the question?

~~ Paul

And Paul sinks the 3-point shot. Paul shoots and scores 3, WOTS pulls a technical and Paul gets a free shot at goal!