Poll: Accuracy of Test Interpretation

Knowing only the specificity is not enough to solve the problem, despite what Rolfe would have you believe.

If the rates of false positives and false negatives are the same, then we can give the test an overal accuracy.

If they're not the same, though, the accuracy becomes dependent on the population being tested. In other words, instead of the formula you just presented giving the same value in all situations, it will change according to the situation.

"Assumptions" indeed.
 
Badly Shaved Monkey said:


WotS

Does this look like a reasonably authoritative use of the term 'accuracy'?

If 'accuracy' is said to be 99% does this not allow sensitivity and specificity to vary from 99%? If 'accuracy' = 99% permits specificity to vary from 99% doesn't your original question require us to assume that you meant specificity=99% while paying little attention to what sensitivity may be in order to give the ball-park answer of "10%" in the poll?

Is this correct?
Click to expand...
I know you asked WotS, but I would like to comment.

The biostat professor's definition of the term is the same as the one WotS has been using throughout. Further, if the same accuracy applies to any population, with any frequency of occurence of the disease, it then (mathematically) follows that the specificity and sensitivity must be the same as the accuracy.

If the accuracy only applies to a specific population, then the specificity and selectivity are related to each other (and to the accuracy, and actual incidence of the disease) by a mathematical function. Puzzle: What is the function?

Edit to add: The wording of 'what the professor said' is a bit vague - it's the number of people tested that matters, not the total population.
 
Hey here's an interesting paragraph that can keep everyone happy for a while.

http://my.execpc.com/~mjstouff/articles/specsen.html

"Bottom line: when someone tells you that a diagnostic test, done properly, is 99% accurate (meaning that both the specificity and the sensitivity = 99%), the actual 'accuracy' of the test will in fact really depend on how common the disease you are testing for is in the population you are testing.

This example clearly points out why diagnostic tests are designed to confirm a diagnosis for rare conditions. They should not be used to go 'fishing' for a diagnosis (which unfortunately happens as an inappropriate use of many diagnostic tests).

In real life, we don't nab 10,000 people randomly and run them through AIDS tests. People who tend to get AIDS tests are people who are at high risk for a variety of factors (lifestyle, occupation, medical condition such as hemophilia, transfusion recipient and the like), so they are in a statistical sense a different 'population' than the general population.

Even so, keep in mind that the test is most certainly NOT 99% 'accurate' in the way all of us think about accuracy, even though both its specificity and sensitivity are 99%. You always must take into account how common the disease you are testing for is in the population you are testing. Many, MANY tests have much lower specificity and sensitivity than the 99% I've used in this example. However, if the condition these tests are trying to diagnose is much more common in the population, then what we think of as 'accuracy' becomes better for a given level of specificity and sensitivity as the prevalence of the disease increases."

The writer goes on to define 'accuracy' as TP/(TP+FP) and gives and example with 'accuracy' of only 92%, yet sensitivity and specificity are both 99%.

It's also inteesrting to note that she puts 'accuracy' in quote marks most of the time to indicate the lability of the concept in contrast to WotS's assertion tht it is a uniquely defined parameter.

This handy definition also contradicts the one that appeared earlier in this thread of (TP + TN)/(TP + TN + FP + FN), which serves to emphasise the arbitrariness of te term.

(Edited to amplify)
 
I think this thread has shown why people need to be careful about terminology, and also why for pos/neg tests sensitivity and specificity values are always given, even if they are the same.

The sheer amount of ambiguity over what WoS meant by the use of accuracy (as evidenced by the length of discussion) can be used as an argument as to why it shouldn't be used.

Incidentally, i think that the vast majoritory of the populace is very ignorant about statistics, and i would be unsuprised to find that doctors are included i this group. However it should also be remembered that in diagnosis of patients statistics is not the only factor :)
 
I will admit, when I first read Rolfe's post I was so angry about her intentionally spoiling the poll that I didn't even notice her glaring mathematical errors.

Go back and look at what she wrote. She claimed specificity is the percentage of positive results that are correct.

This is wrong. http://www.fpnotebook.com/PRE18.htm

She's also wrong about the definition of sensitivity.

Then she claimed that all we needed to know was the specificity (which is wrong no matter what definition you use, the correct one or the "Rolfe special").

Reality check: this "expert" who supposedly wrote a book on the subject doesn't know the definitions of basic terms. Everything she's said in this thread has been grossly wrong.

And most of you mindlessly repeated her statements! I can go back, read the thread, and see you people repeating statements she'd made as if they were fact!
 
Wrath of the Swarm said:
Knowing only the specificity is not enough to solve the problem, despite what Rolfe would have you believe.

If the rates of false positives and false negatives are the same, then we can give the test an overal accuracy.
Click to expand...
1. I acknowledged that was an error. Please see above, I'm not doing it all again.

2. If the rates of false positives and false negatives are the same, then the rates of false positives and false negatives are the same. That's all. It doesn't justify the introduction of a completely foreign term (foreign to binary test discussion, that is) to describe this unusual occurrence. There is a reason why this term isn't used in this context, please see above again, because trying to derive any universally-applicable definition is confusing at best, please see (less far) above.

Yes, assumptions.

Rolfe.
 
ceptimus said:
The biostat professor's definition of the term is the same as the one WotS has been using throughout.
I don't think that's right, because the prof's definition allowed for sens and spec to vary whereas WotS insists they are both uniquely specified by a single figure for 'accuracy'

It also seems to be the case that there is not, as I have already mooted, any agreed definition of 'accuracy' because the second example I have now posted gives a different, yet reasonable version, and it looks exactly like WotS made up his/her/its version 'on the fly' when setting the question. If that was not the case why do we still lack WotS producing a citation to back up his definition and why do we have two others that differ from it?

Further, if (I think this 'if' is one of Rolfe's criticisms and is illustrated by the example I have just posted from here http://my.execpc.com/~mjstouff/articles/specsen.html ) the same accuracy applies to any population, with any frequency of occurence of the disease, it then (mathematically) follows that the specificity and sensitivity must be the same as the accuracy.

If the accuracy only applies to a specific population, then the specificity and selectivity are related to each other (and to the accuracy, and actual incidence of the disease) by a mathematical function. Puzzle: What is the function?

Edit to add: The wording of 'what the professor said' is a bit vague - it's the number of people tested that matters, not the total population.
Click to expand...
 
Wrath of the Swarm said:
She claimed specificity is the percentage of positive results that are correct.

This is wrong.
Click to expand...
Yes, it is wrong. That describes the PPV, in fact. And if I said that, I made a unintentional mistake. There are places in here where I was typing too fast, I have noticed. Including a total typo of "precision" when I meant something else.

So go on, I'm off on holiday, go through the thread to find the typos, feel free. I've acknowledged the genuine error.

Wrath acknowledged his too, earlier.

Rolfe.
 
WotS

Will you please address the question of differing definitions of 'accuracy' that I have covered in my last two posts? Thanks.
 
It's already been explained to you, BSM.

I explained what 'accuracy' meant several pages back. It's the same one as you got from the professor.

That value can only be defined ahead of time (independently of the sample population) if a certain set of conditions hold. They applied in this case. End of discussion.
 
http://www.rapid-diagnostics.org/accuracy.htm

"Accuracy can be expressed through sensitivity and specificity, positive and negative predictive values, or positive and negative diagnostic likelihood ratios. Each measure of accuracy should be used in combination with its complementary measure"

i.e. 'Accuracy' is the general term for the whole area not a specific parameter within it.


This would represent a third reasonable viewpoint, moving us even further from the idea of 'accuracy' as a uniquely defined numerical parameter that completely specifies specificity and sensitivity.
 
They weren't just typos, Rolfe! You repeated them many times, and the people who repeated you did so many times as well!

You presented several arguments with those concepts. Didn't you notice you'd screwed up completely?

You don't understand the concept of accuracy, you don't know how to analyze statistical aspects of the test to reach conclusions, you thought consequences that follow inevitably from the given information were "assumptions"...

You're not an expert at all. You don't know a damn thing about what we've been discussing.
 
Some time ago someone asked why people were bothering with nitpicking over page after page of this thread, isn't it simply that it is an interesting exercise to try to resolve this puzzle of what is really meant by the superficially simple idea of 'accuracy'?
 
http://www.imtech.res.in/raghava/mhcbench/parameter.html

" The term accuracy was defined to provide a single measure of performance. It is defined as the proportion of correctly predicted peptides." Peptide assays is what this page was discussing.

"Accuracy
The proportion of correctly predicted peptides (both binders and non-binders)
((a + d)/(a + b + c + d))*100"

This is the same as the prof's definition.

WotS

Please explain how this definition uniquely specifies sens and spec for a given 'accuracy'
 
No, you don't get it.

That definition applies when looking at a particular sample for the particular test. For that sample, the test's accuracy was given by that formula.

Without referencing a specific set of results, that definition is meaningless. The concept of 'accuracy' doesn't apply - except when the chance of error does not depend on the nature of the test population.

In such a case, alpha must equal beta by definition. That is the only time the accuracy of the test can be considered without mentioning a particular sample population.
 
Wrath of the Swarm said:

I explained what 'accuracy' meant several pages back.
Not very well it would seem since you claim that your definition uniquely defines sens and spec, whereas, notwithstanding your claim that
It's the same one as you got from the professor. , the prof's one does not, so yours and his can't be the same. Could you please lay them out in parallel as simple equations so you can show us what you mean and how it is the same definition as the professor's even if it differs from the other one I have just cited.
Click to expand...
 
[edit] Fixed a repeated error where I gave the error instead of the accuracy. Many thanks to those kind enough to point this out.

From a post of mine on page 4:

"Accuracy" is the proportion of correct test responses to total test responses. (Just what the English definition of the word would imply.)
Click to expand...

Now: let's say that the chance of false positives isn't the same as false negatives. Oh, the first is .40 and the second is .30, just to pick two random numbers.

Now, let's say we use a group of 10 sick patients for the test. We get no false positives (since everyone is sick) and 3 false negatives. So the error is 3/10 or .30, and the accuracy is .70.

Now let's use a group of 10 healthy patients. We get no false negatives and 4 false positives. Error is .40; accuracy is .60.

Now let's use 5 healthy and 5 sick patients. We get 5 * .4 false positives and 5 * .3 false negatives, for a total of (on average) 3.5
wrong answers out of ten, which means accuracy is .65.

See? The accuracy changes depending on the population.

Now let's pretend the false positive rate is .40 and the false negative rate is also .40.

In the first case, we get an error of .40 and an accuracy of .60, just like the second case.

In the third case, we get 5 * .4 false positives and 5 * .4 false negatives, for an overall error of .40 and accuracy of .60.

No matter what sample population I give the test, it will always have an accuracy of .60.
 
Wrath of the Swarm said:

Without referencing a specific set of results, that definition is meaningless. The concept of 'accuracy' doesn't apply - except when the chance of error does not depend on the nature of the test population. As I have been pointing out, no one else seems to share your idea that 'accuracy' itself is uniquely definable, never mind whether it uniquely defines the parameters sensitivity and specificity, and in the prof's example it does not.
Click to expand...
 
As you seem more interested in arguing than providing the formula that answers my puzzle, here it is:

A = (1 - P) * F<sub>p</sub> + P * F<sub>n</sub>

Where:

A = accuracy
P = proportion of population infected by the disease
F<sub>p</sub> = proportion of false positives when only uninfected persons are tested.
F<sub>n</sub> = proportion of false negatives when only infected persons are tested.

Note that all these are proportions (in the range 0 to 1) and not percentages. Multiply by 100 to express them as percentages.

But if anyone has learned anything from this thread, it should be that blindly plugging numbers into a formula you don't understand is a bad idea. That and the fact that you shouldn't let emotion enter into your argument, especially if mathematics or science is the the subject being argued over.
 
You must log in or register to reply here.

Back
Top Bottom