Wrath of the Swarm said:
If you're such an expert, why did you make such a basic mistake in the first place?
This may be the best question in the latter part of the thread, and I hope I can now answer it truthfully.
Well we all know what Wrath thinks. He's told us so many times. I'm brain-dead. Like all other medical professionals (except I forgot, I'm not a medical professional). He's never going to change that opinion, no matter how explicable the mistake.
However, it is explicable. I was telling the truth about the published papers, the book, the book reviews and the lectures on the subject. And by the way, there weren't any mistakes in these. So why did I make a mistake here? And how come I got the right answer in spite of it?
Remember, I said I used a spreadsheet to derive the answer. I wrote that spreadsheet in about 1996, when I became involved in this particular subject at a detailed level. I had to do so many of these predictive value calculations that it was a lot easier to do it like that, also I wanted to display that graph I showed in my first post, as many of the concepts are a lot easier to get across from a graph. So I became used to doing the calculations using the spreadsheet I was familiar with.
This displayed the difficulty posed by Wrath's incorrect formulation of the question rather differently from the back of the envelope method. Whichever way you look at it, Wrath gave only one parameter (without specifying which), when two were required. Does the number fit one slot only, or is there some way to derive the two different parameters from the single value?
Wrath's constant assertions that his unique use of the word "accuracy" was intended to force the two values to be equal are completely illegitimate. Even if the two values are
allowed to be equal to make the problem simpler, they have to be
free to be non-equal, because there is nothing constraining them from being non-equal. This is why the term accuracy isn't defined and isn't used in this context - because it is meaningless. There is no way to derive an "accuracy" value for a real-life assay except by a fluke, and the terms employed have to be able to be used with real-life assays or the whole exercise is futile.
When you look at the difficulty from the point of view of the spreadsheet, it is intuitively fairly clear that one of the numbers is much more necessary than the other. Small differences in specificity make a huge difference to the resulting predictive value for this particular problem, while larger differences in sensitivity make relatively little difference. Like this.
<TABLE BORDER="1"><TR><TD ALIGN=CENTER VALIGN=BOTTOM>
Variable value</TD><TD ALIGN=CENTER VALIGN=TOP>
PPV with
specificity constant</TD><TD ALIGN=CENTER VALIGN=TOP>
PPV with
sensitivity constant</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>100%</TD><TD ALIGN=CENTER VALIGN=TOP>9.10%</TD><TD ALIGN=CENTER VALIGN=TOP>100%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>99.5%</TD><TD ALIGN=CENTER VALIGN=TOP>9.06%</TD><TD ALIGN=CENTER VALIGN=TOP>16.54%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>99%</TD><TD ALIGN=CENTER VALIGN=TOP>9.02%</TD><TD ALIGN=CENTER VALIGN=TOP>9.02%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>98%</TD><TD ALIGN=CENTER VALIGN=TOP>8.93%</TD><TD ALIGN=CENTER VALIGN=TOP>4.72%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>97%</TD><TD ALIGN=CENTER VALIGN=TOP>8.85%</TD><TD ALIGN=CENTER VALIGN=TOP>3.20%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>96%</TD><TD ALIGN=CENTER VALIGN=TOP>8.77%</TD><TD ALIGN=CENTER VALIGN=TOP>2.42%</TD></TR><TR><TD ALIGN=CENTER VALIGN=TOP>95%</TD><TD ALIGN=CENTER VALIGN=TOP>8.68%</TD><TD ALIGN=CENTER VALIGN=TOP>1.94%</TD></TR></TABLE>
I certainly didn't think this through at the time. If I had, I wouldn't have made any mistake.
I simply saw, quickly, that if specificity was pegged at 99%, then not defining the exact sensitivity within the range of expected values for a good assay didn't really make much difference. In effect, the answer was close-enough to 9 for any value between 95% and 100%. So click the "10%" button and let's get on to the interesting part. (My real mistake was in giving that 9.02% figure so precisely - I didn't stop to check
what sensitivity value was in the damn sensitivity box when I increased the number of significant figures visible in the cell to see just how close to 10% we were, and it happened to be 99%. In fact to that number of significant figures the influence of sensitivity is indeed appreciable, if small.)
No doubt Wrath is going to jump all over this. Assumptions all over the place. But remember, assumptions
he was generating in his inexplicable desire to force his own assumption, that sensitivity and specificity are equal. As this was such a false assumption, considered as a legitimate method of handling the calculation in general, I went without sufficient thought for the other assumption that would give a meaningful answer, that the figure we had been given was the one we
really needed, the specificity.
Wrath will no doubt assert that this proves I'm incapable of coherent thought, and can only plug numbers into my spreadsheet without thinking. However, Wrath, consider. This is the risk you run if you don't use the correct terms to describe the problem you're setting. You may think that the assumption you're trying to force is inevitable, but it may be that someone who sees the problem from a different perspective (in this case, seeing the necessity for using only terms that can be used for all assay conditions, not just a flukey subset seldom encountered in real life) may reject that assumption, and find some other more or less workable one.
So why was I so careless? Because it was the mistake of Wrath's that didn't really matter. It was (or should have been) easy to clarify, and it wasn't a mistake that carried any agenda or affected the result of the problem in any meaningful way. It's still ~10%, whatever. We're not arguing about that. We'd just have liked an unambiguous question.
At the time I made this error, I was already furious. Furious and fed up with Wrath's constant anti-medicine agenda, one plank of which I now understood for the first time, and with the fact that the problem set was clearly worded in such a way as to facilitate the argument that doctors are stupid. Which Wrath indeed lost no time in setting out.
To elaborate. Wrath has been asserting for months now that it is a known fact that doctors can't think. And much of the time he supports this by saying that "doctors can't do Bayesian analysis". Didn't pay much attention. Yes, I understand the statistics I need to understand, but I don't have all the statistician's language. I didn't realise that this particular problem was what he meant. Then I did.
This problem has a number of dangerous angles. For one thing, it can be used as Wrath is trying to use it, to ambush medical personnel into giving the "wrong" answer, even though the same people might in fact make the right assumption if the situation was presented to them in a more familiar context. For another, it is the start of a very dangerous and counterintuitive logic trail, which is often mis-presented in such a way as to undermine the clinician's confidence in an intuitive approach which usually leads to the right course of action, and substitute assumptions which are only valid for the well-patient screening situation, forcing them to be inappropriately applied to the clinical testing situation.
This is the aspect which particularly interests me, and examination of the false logic trails leading from it is the reason I have become especially familiar with the statistics. To the point where I can normally handle them well enough to write books and articles and give lectures about them.
However, in my haste to advance to the interesting part, I made a mistake in the way I described my assumptions stemming from Wrath's minor terminology error. Instead of saying, for this particular problem the exact value of sensitivity (within reason) makes no material difference, therefore we assume you've just given the specificity value, I wrongly, without thinking, said the sensitivity was completely irrelevant. And then I was called sixteen sorts of idiot, a whore and God knows what else. And I don't believe Wrath even realised exactly what I'd done. He was so intent on denying that his wording required any sort of assumption at all, and on finding new and creative insults to throw at me, that he never even looked to see if there was
any rationale to what I was saying. Because, you see, I'm a pretend medical type who can't possibly understand anything about all this mathematical stuff, so the correct response is obviously abuse.
By the time the subject was dragged back to the "accuracy" point, I was seeing so much red that I woefully failed to go back and make sure that, however right I might be about Wrath's wording requiring an assumption, my explanation of the assumption I'd hastily made was actually correct. As Wrath was only concerned to defend his position that no assumption was required, I was only concerned to demonstrate that it was. Which it was, as we've finally seen.
All right, that should be a lesson to me. When handling a very familiar problem which you haven't worked through
from first principles for several years, don't let anger prevent you from checking your working.
Whether it will be any sort of lesson to Wrath on the danger of making implicit assumptions when formulating from memory a well-known problem designed to annoy people, I very much doubt.
Rolfe.
