Poll: Accuracy of Test Interpretation

No, you can't, because then the accuracy of the test cannot be said to be 99% any more.

Now the equation for accuracy is no longer a constant, but a variable that depends on the particular sample population you put into it. Give the test to a population composed entirely of healthy people, and a certain percentage of the answers will be wrong. Give the test to a population of sick people, and a different percentage of the answers will be wrong. Give the test to a mix, and the percentage will change depending on the relative proportion of sick and healthy people.
 
Wrath of the Swarm said:
No, you can't, because then the accuracy of the test cannot be said to be 99% any more.

Now the equation for accuracy is no longer a constant, but a variable that depends on the particular sample population you put into it. Give the test to a population composed entirely of healthy people, and a certain percentage of the answers will be wrong. Give the test to a population of sick people, and a different percentage of the answers will be wrong. Give the test to a mix, and the percentage will change depending on the relative proportion of sick and healthy people.
Click to expand...

For what it's worth, I just checked with the biostats professor here (the guy who brings in most of the department funding through consulting with the local hospitals), and he offered an entirely new definition of "accuracy" in this context :

The accuracy of the test is the number of trials for which the test got the correct answer divided by the total number of trials. (More formally, the true positives plus the true negatives, the sum divided by the total population).

He specifically rejected the idea that "accuracy" could be twisted to mean "both the specificity and the sensitivity."

Rolfe? Ever seen this usage?
 
Ok someone work out the accuracy for the case where for every 100 people there are 1.000101 false negatives per hundred people and no false posertives and for when there are 1.000101 false posertives per hundred and no false negatives.
 
If the chance of a false positive isn't the same as the chance of a false negative, though, that value will change depending on the incidence within the test population.

Ignoring statistical variation and looking only at the averages for the moment, we'll find different values for the accuracy if we test populations with different rates of the disease.

Imagine alpha is .2 and beta is .3. Then if we test a population where everyone is negative, the test will have a 20% accuracy. If we test a population where everyone is positive, the test will have a 30% accuracy.

Without making reference to the specific test population, we can't determine the accuracy. It's a variable that depends on an unknown factor.

Again: the only way we can speak about objective, universal accuracies for the test is if alpha equals beta. To simplify everything, I gave such an case in the original question. No assumptions are needed.
 
Wrath of the Swarm said:
Again: the only way we can speak about objective, universal accuracies for the test is if alpha equals beta. To simplify everything, I gave such an case in the original question. No assumptions are needed.
Click to expand...
Apart from the assumption that the 'accuracy' you gave was objective and universal, and not simply correct for the test population you had already identified.
 
Wrath of the Swarm said:
If the chance of a false positive isn't the same as the chance of a false negative, though, that value will change depending on the incidence within the test population.
Click to expand...

But you told us the population so we don't need to worry about that.
 
But I gave it for the general population.

Each individual tested is a population of one; either the incidence is zero, or the incidence is one. Either the person doesn't have it, or they do.

The accuracy of the test, in any particular case, then becomes undefined unless we know whether the person has the disease or not - and since the point of the question was determining whether or not the conclusion the doctor reached was correct, I couldn't tell you that.
 
"Doctor, how accurate is this test you're giving me?"

"Well, Timmy, that depends. If you actually have the disease, there's an 80% chance the test will detect it and a 20% chance it won't. If you don't have the disease, though, there's a 5% chance the test will detect the disease anyway and a 95% chance it won't."

Do you see why I used a test with a universal accuracy? I wanted to make everything as simple as possible.
 
Wrath of the Swarm said:
"Doctor, how accurate is this test you're giving me?"

"Well, Timmy, that depends. If you actually have the disease, there's an 80% chance the test will detect it and a 20% chance it won't. If you don't have the disease, though, there's a 5% chance the test will detect the disease anyway and a 95% chance it won't."

Do you see why I used a test with a universal accuracy? I wanted to make everything as simple as possible.
Click to expand...

No the above is simpler.
 
Wrath of the Swarm said:
Do you see why I used a test with a universal accuracy? I wanted to make everything as simple as possible.
Click to expand...
But you didn't STATE that that's what you were doing. You left it to be assumed. Which was not making things as simple as possible.

Now I've admitted to where I caused unnecessary contortions in the discussion, by erroneously offering you two "assumptions" you might have wanted us to make.

But you wouldn't say, no, there is only one assumption which will get you an answer, so that is the assumption I intended you to make. Because yu won't concede that that was an assumption.

Rolfe.
 
Martin said:
Apart from the assumption that the 'accuracy' you gave was objective and universal, and not simply correct for the test population you had already identified.
Click to expand...
Yes. That is what I referred to earlier when I said there was some (slight) room for quibbling. A better wording would have been:

A particular test for a disease always gives results with 99% accuracy.

In the country where I live, one in every thousand people are known to suffer from the disease. Before I took the test, there was no reason to assume I was specially at risk - that is to say I had the usual 1 in 1000 chance that applies in my country.

Yesterday, I took the test, and the test says I have the disease. What are the chances now that I have the disease?
 
Earlier in the thread, Wrath, you wrote this:

.
Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.
Click to expand...

I've already shown point 1 to be a falsehood, so lets move onto point 2. In the study I quoted and the studies that you linked to, the questions were not phrased using the term accuracy, which you used in your question. Wrath, are you now claiming that that the researchers and statisticians who wrote these questions screwed up? Should they have used the term accuracy? No, they should not have done and neither should you have done. In the context of such a study the term is meaningless, the subjects would need to make an assumption as to what it means. You phrased the question wrongly. You have had plenty of opportunities to admit this and correct your error but you have failed to do so. This is the point you have consistenly failed to get.

By the way, Wrath, you have now called me a liar several times in this thread. Twice, I have asked you how you defend these assertions and twice you have failed to respond. It seems to me likely that this is because you know I haven't lied and only made the assertions in order to draw attention from your own lies
 
geni said:
Ok someone work out the accuracy for the case where for every 100 people there are 1.000101 false negatives per hundred people and no false posertives and for when there are 1.000101 false posertives per hundred and no false negatives.
Click to expand...
That situation can't apply to a general population geni. If everyone has the disease, then it is impossible for false positives to occur. Similarly, if no one in a particular population suffers from the disease, then false negatives are impossible.

Note that Wrath's accuracy figure is much more universal - it simply means that the result for one in every 100 tests will be wrong. This figure can work on a population where everyone has the disease, or no one, or any mixture of the two.
 
ceptimus said:
Yes. That is what I referred to earlier when I said there was some (slight) room for quibbling. A better wording would have been:

A particular test for a disease always gives results with 99% accuracy.

In the country where I live, one in every thousand people are known to suffer from the disease. Before I took the test, there was no reason to assume I was specially at risk - that is to say I had the usual 1 in 1000 chance that appies in my country.

Yesterday, I took the test, and the test says I have the disease. What are the chances now that I have the disease?
Click to expand...
It's better, but you're still making the assumption that specificity and sensitivity are equal in this particular test. And it's still an unstated assumption.

Also, while you have pinned down the prevelance figure by stating explicitly that you want the 1 in 1000 to be seen as applying to that patient, it's not entirely realistic.

If the overall incidence in the country is 1 in 1000, then the prevalence in people with no observable risk factors is going to be less than 1 in 1000. Because some of these people are going to be excluded from the group because they are showing risk factors or clinical signs.

So if you're going to tell us anything in particular about this patient, such as the fact that he has no observable risk factors, the prevalence figure you really want isn't the prevalence in the entire population of the country but the prevalence in the subsection who are showing no risk factors.

It may seem a slight quibble, but to be absolutley correct here you would be better to give the incidence of disease in clinically unremarkable individulas, since you have told us that we are dealing with a clinically unremarkable individual, or to do what Steve74's test did, and simply say nothing about clinical presentation and state explicitly that you want the problem worked without reference to any signs or symptoms.

I prefer the former approach, because it's more realistic.

Rolfe.
 
Rolfe said:
It's better, but you're still making the assumption that specificity and sensitivity are equal in this particular test. And it's still an unstated assumption.
Click to expand...
No. You are wrong Rolfe. I am beginning to despair of you ever understanding this.

If I state that a test ALWAYS gives results that are 99% accurate, then it FOLLOWS that the specificity and sensitivity MUST both be 99%. There is NO assumption. It is simply mathematically impossible for it to be any other way.
 
It's not an assumption, Rolfe. It is a stated property of the test.

It is remarkable how unable you are to grasp even this, the simplest of points. It is not an assumption.

ceptimus: I see your point, but I really don't think that wording was necessary. If I say the test has an accuracy, it can only be a universal one if I'm expressing any meaning at all. Whether I spoke about the general population immediately before doesn't matter, because the value would change for each different population.

So I agree that your wording would have been easier to understand for some, but I still don't think there's a problem with the way I worded things. (I have been wrong before, though, so take this with a grain of salt.)
 
Rolfe said:
It's better, but you're still making the assumption that specificity and sensitivity are equal in this particular test. And it's still an unstated assumption
Click to expand...
Not this time, no. Ceptimus' test is always 99% accurate, regardless of test population. That means that you'll get 99% accuracy for any subset of the population you choose. Choose the disease-free and disease carrying populations as your subsets, and you'll see that it must be true that specificity = sensitivity = 99% in this case. Wrath didn't specify that his accuracy was independent of the test population he'd set up, so there's some wiggle room there. The version Ceptimus gave seems solid to me.
 
ceptimus said:
Note that Wrath's accuracy figure is much more universal - it simply means that the result for one in every 100 tests will be wrong. This figure can work on a population where everyone has the disease, or no one, or any mixture of the two.
Click to expand...
No, it's not universal. It only applies to the very unlikely situation of a test with equal sensitivity and specificity. The fact is that you need to know both figures (sensitivity and specificity), every time that the test you're using doesn't fit this idealised scenario.

Rather than make the reader assume that you must be implying the idealised scenario because you only gave one figure where two were needed, it's still clearer just to say "sensitivity and specificity are both 99%".

You can also dress it all up like they did with the breast cancer question, giving raw data for true positives and false positives, which contains the necessary information within it.

Rolfe.
 
If the accuracy hadn't been independent of the population, it would have a different value for different populations, and the value I gave would not only have been utterly useless but technically incorrect as well.

Without a specific testing population, the accuracy of such a test is not defined. (I suppose we could do some calculus and sum up the accuracies across different populations, but that's a lot of work to get a statistic that has little to no value in performing calculations.)

So since I gave it, it must be independent of population.

In hindsight, the question would have been less open to misunderstanding if it had been worded differently - but then we wouldn't have given Rolfe this chance to humiliate herself utterly, so in the end I'm quite pleased.
 
Wrath of the Swarm said:
In hindsight, the question would have been less open to misunderstanding if it had been worded differently - but then we wouldn't have given Rolfe this chance to humiliate herself utterly, so in the end I'm quite pleased.
Click to expand...
I'm glad you're pleased, Wrath. But it's still an assumption, certainly the way you worded the question.

And if you were really smart enough to spot the mistake I was making, you could have pointed it out about ten pages ago. But that would have involved opening yourself to the possibility of conceding that your wording was open to misunderstanding, as you have just done.

Edited to add: Since Wrath seems to be taking this post to imply that I believe I'm "humiliated utterly", I'd just like to clarify that I've admitted making a silly error. However, Wrath nevertheless worded his example so that an assumption had to be made, which was the essence of the disagreement in the first place. He has, grudgingly, admitted this, finally.

Yes, I feel silly - because rushing at Wrath the moment I saw the hole he was about to walk into led me to throw away the advantage by not considering what I was doing with the minor ambiguity carefully enough. This is not clever tactics, I agree.

However, the hole is still there. It's just unfortunate that there has been such acrimony that we may never get to see the bottom of it.

Rolfe.
 
You must log in or register to reply here.

Back
Top Bottom