ceptimus said:
It doesn't need to, as IN THIS CASE they are both the same. Of course, IF they were not both the same, then both would have to be given.
It does need to. A descriptive parameter for the test which is only applicable in the unlikely chance of the two values being the same is useless.
You can see the artificiality of the situation from the way you posted your calculation (which was clear, thank you).
Try the test on 100,000 people known to have the disease, the test says that 99,000 have it, and 1,000 don't.
Estimate of probability of a false negative is 1,000 / 100,000 = 1%
Now try the test on 100,000 people known to be free of the disease, the test says 99,000 are clear and 1,000 of them have it.
Estimate of probability of a false positive is 1,000 / 100,000 = 1%
You had to force the two groups to be the same, or your scenario simply wouldn't have worked. But in real life there is no reason at all why they should be the same. They are not independent variables, because (as Wrath pointed out) there is a tendency for improvements on one side to cause a deterioration on the other - efforts to eliminate false negatives tend to increase false positives and vice versa. They tend to be
opposite variables, which actually makes it
less likely that they will happen to be identical in any given example.
Wrath has been asserting that this "accuracy" definition is a "more powerful" way to describe the test. However, you have just shown quite clearly that the term simply cannot be used at all except in the very unlikely chance that false positive rate and false negative rate are identical. This is precisely why it is not used. There is in fact no meaningful way to combine the two sides for the general case, and give an overall "accuracy" figure which does not vary with prevalence. This is why it is standard practice to quote sensitivity and specificity separately.
ceptimus said:
There are plenty of real world tests where the chance of a false positive IS the same as the chance of a false negative, even though this may not be true of medical tests.
This may be so. But we are not talking about other situations here, this discussion is specifically about medical tests. For the purpose of going on to discuss the competency or otherwise of doctors, as Wrath has told us several times. Therefore the deliberate use of a defining term which is simply not applicable to the medical testing situation is perverse, to put it mildly.
ceptimus said:
The fact remains that Wrath's question, as posted has a definite answer.
No, Ceptimus, it doesn't.
You cannot calculate a positive predictive value (which was in effect the question) unless you know the SPECIFICITY. That is the percentage of unaffected patients who test positive. (We don't care, for this question, how many affected patients test negative, we don't need to know.)
Wrath did not provide this information in a way that could be understood without making assumptions.
My assumption was, OK, I know I need the specificity figure. I've been given something called "accuracy", which I know is not a meaningful concept in the context of this type of testing. I will assume that this is just sloppy terminology, and that what I have actually been given is in fact the specificity. I did this, and got the expected result.
However, Wrath explicitly denies that this is what happened. He states that he deliberately used this "accuracy" term because it is "more powerful", because it incorporates both sensitivity and specificity. I ask again, how can a term be "more powerful" for the purpose of (real-world, in which we actually live and in which the terms we choose to use have to have general utility) test description, when it cannot be used at all in most (real-life) instances?
("The very fact that I used the term at all should have told you that I was referring to the rare and unrealistic situation of equal specificity and sensitivity!" Oh, God give me strength!)
Others reading the thread realised that "accuracy" must somehow incorporate sensitivity and specificity - and started speculating how. Arithmetical mean? This loses information, as Geni pointed out, but not only that. As Wrath pointed out, you can't just give equal weight to sensitivity and specificity when in reality more unaffected patients will be tested than affected, and so a higher rate of false positives will be disproportionately reflected in the "wrong" results. This notion is actually dragging the discussion back towards the concept of the predictive value, which is in fact the
answer, not the question.
(In fact, this is the reasoning that led to the adoption of the predictive value calculation as another way to characterise test performance, with all its advantages and disadvantages which we will no doubt get to some time next week.)
So, we're left scratching our heads. Wrath has told us the "accuracy", specifically denying that he just meant to say "specificity". We need the specificity figure. How to get it? We don't know what Wrath means by "accuracy", because as you've just demonstrated, it's not a term which has a meaningful definition which can be used to characterise real-life tests.
Now we begin to suspect that the only way to get any sense out of this is to assume that Wrath must mean that both sensitivity and specificity are 99%. This is such an unlikely situation in real life (for medical tests, but remember, we are specifically dealing with medical tests, and we are trying to test those used to handling medical tests), that it hadn't really occurred to us that he could possibly mean that. But maybe he does.
Yes, he does. Since the question cannot be answered without the specificity value, Wrath must have given us the specificity value, we're told. OK, that's what I thought, but no, "accuracy" isn't specificity. But I still have to be able to deduce the specificity from this "accuracy" value. Arithmetical means run into the sands of the predictive value. So there is only one other possible explanation. He means that for this remarkable test, sensitivity and specificity are identical, therefore he can produce this all-in-one "accuracy" figure (which we've never heard used in the real world, for reasons already gone into).
Can you see that this inevitably involves an assumption? Either that Wrath meant specificity when he said accuracy, or that he meant that specificity must equal sensitivity for this very singular assay, and therefore accuracy means both sensitivity and specificity, which brings us back to where we were, accuracy here means specificity.
However you slice it, it is an assumption. Necessitated by Wrath's choice to parachute-in this "accuracy" figure, rather than simply use the standard terminology for the discipline which is applicable to all tests, not just a (tiny) subset.
Recall, Wrath kept declaring that he was posing the question in exactly the way it had been put to the medical personnel in the studies he was replicating (from memory). But when sources were finally produced, none of them quoted an "accuracy" value. In fact the popular choice was the "false positive rate" (100 - specificity), which is in effect the same information, but the term is more intuitive in its meaning than specificity itself. No problem with this, it's the right way to do it.
Why did Wrath choose to do it differently?
I think, because he knows pure statistics, but not applied statistics. He was trying to do it from memory, and didn't understand what any medical person would automatically know - that tests are usually not of equal sensitivity and specificity, and that specificity is the term you need to know. He therefore got into a huge tangle which was entirely unnecessary.
I'm sorry if I've still failed to explain it to you. Doesn't the fact that none of the source studies introduced an "accuracy" figure reveal anything, even if my explanations are inadequate?
Yes, I threw an early stone, overtly. But it was as a response to an unprovoked covert stone, Wrath's OP. I knew
exactly where he was going. I'd quite like to go there and explore that place. But to set out to trap/criticise the medical community and yet not to employ the unambiguous terms that are provided for the purpose of this question, and which it turns out
were employed by his sources, got my back up from the first moment. If Wrath is going to set himself up as the oracle in the medical statistics department, then for goodness sake use the correct
medical statistics in the OP!
However, that wasn't my main reason for pulling the trigger on that sloppily-put and simplistic problem. My main reason was that Wrath had shown us a scenario
deliberately designed to make the doctor look stupid. First he had built into the scenario a clinical examination, which led to the test request, but he did not tell us the reason for the request. This is hugely dishonest, because it is this reason that would influence the doctor's decision to accept the test result as correct. Which we're told he did.
Wrath tells us that the doctor examined the patient. Then he decided to order the test. Then he received a positive result. Which he decided was correct.
He wants us to assume that there was no special reason for the ordering of the test. That the figure to be assumed for "prevelance" in the case of this particular patient is the 0.1% figure for the population as a whole. And that the doctor therefore jumped to a wrong conclusion for no good reason.
But these are all assumptions. We can make equally (if not more) valid assumptions fron the same data.
The doctor ordered the test because of something he observed while examining the patient. When he got the positive result he was perfectly well aware that the condition had only a 0.1% incidence in the general population, but he knew that the probability of this patient having the disease was much greater than that, so the predictive value of this result
in this patient was much higher than the baseline 9.02%, indeed high enough to make it a racing certainty.
The reason I say this is the more likely scenario is that Wrath couldn't just ask us the probability that the positive result was correct, oh no. He chose to tell us that the doctor decided it was correct, and ask us the probability that the
doctor was wrong. Now Wrath has a very low opinion of doctors. But I don't think they're as stupid as he assumes. If part of the information I'm explicitly given is that "the doctor chose to believe that the result was correct" I feel I am entitled to use this information to reflect on the entire problem. If he made this decision, might not this be an indicator that he'd requested the test because of clinical suspicion, not as a routine?
Ceptimus, wording of these problems is of crucial importance. You mustn't tell the audience too little, or too much. Here, we were told too little, in that we had to guess at the specificity figure we needed, and we're kept in the dark as to the reason for requesting the test. And we're told too much, in that we're told what the doctor concluded. We didn't need that to be able to work out the basic maths, but once we've been told it, it introduces other possible assumptions which may affect the interpretation of how the problem should be viewed.
Again, when we look at the source material, we find that the question isn't "how likely is it that the doctor was wrong?" It's "how likely is the test to be wrong?" Much more neutral, but not Wrath's style.
And (certainly in the example quoted by Steve74, which looked like the original 1978 study) there has to be some way of indicating that we are not allowed to take signs and symptoms into consideration (explicitly stated in the 1978 question), or in fact that the patient is low-probability and we should use a low-probability in calculating predictive value.
Wrath missed that part completely.
Now is we ignore Wrath's justifications for his wording, how could it have been worded to pose the same question, but to rein in all those assumptions and keep the reader on the desired train of thought? Easy.
Last month I had to go for an insurance medical. The doctor could find nothing wrong with me, but the insurance company required that I have a blood test for a particular disease, which actually has a 0.1% incidence in the clinically heathy population. The test, which has a false positive rate of only 1%, came back positive. What is the probability that I really have the disease?
That's how you do it (and believe me, I've set this one for clinicians often enough, I know how careful you have to be to stop them jumping all over the question - but this one is bomb-proof).
Note that I explicitly identified the patient in question with the population I gave the incidence figure for. No weaselling that the prevalence isn't necessarily valid for that individual. It's the right prevalence. And by the way, no silly "accuracy", you have a number there from which you can directly derive the specificity. And I didn't tell you what the doctor thought. Why should I? You're the doctor! Come on, what do you think? I've been very careful to give you NO reason to go for a high probability that the result is right. Will you still fall for it?
And quite often, they do.
And that, actually, is the
start of the class, not the end.
So, I went for Wrath. This is no more than he dishes out - he frequently tells posters they can expect no mercy from him. The reason I did it is that he was setting out to have a go at medical comprehension of statistics, but he himself had been extremely sloppy in his wording of the question. A question which is regularly used in the sort of classes I teach, and the parameters of which are well-known.
I wish he'd worded the question better. We might have had a much more constructive discussion by now. But Wrath takes no prisoners. When he himself is less than perfect, why should we refrain from retaliation?
Rolfe. Hoping to start the real discussion soon.
