Poll: Accuracy of Test Interpretation

[Brian the Snail]

But I know who was considered to have lost the "does 0.99999(infinity) equal 1?" argument, even though as someone who hadn't explored that particular problem in any depth, I initially and instinctively thought your position was correct.

No, I think Ian's was the probabilty of finding random sequence in infinite series type-thing argument.

But I think the two questions are pretty much equivalent anyway. Or at least equally as silly.

 

[ceptimus]

I suppose we are really arguing about semantics.

If you make the assumption that the test has different accuracies for false positives and false negatives, then the question cannot be answered.

As the question was proposed in a way that suggested that there was an answer, and only one 'accuracy' figure was given, then it is logical to assume that the test has the same accuracy regardless of the incidence of disease in the population, and from this it follows that both false negatives and false positives must occur at the same, 1% level.

If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.

I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

 

[Rolfe]

I suppose we are really arguing about semantics.

Yes, we are pointing out that when posing a defined problem like this, it is essential to use clearly-definied terms. These terms are defined for a very good reason, and failure to use them, or too define the terms you are using will make your question meaningless.

If you make the assumption that the test has different accuracies for false positives and false negatives, then the question cannot be answered.

ASSUMPTION? Ceptimus, it is a simple fact that false positive rates and false negative rates for tests like this are completely different variables. They will only be equal by pure (and unlikely) chance. In fact, as Wrath pointed out at one stage, there is a tendency for them to vary in opposite directions, so that improvements in the false-positive rate lead to deteriorations in the false-negative rate, and vice versa.

Specifying the two different values separately (or at least specifying the value of the parameter you need for the caculation in question) is a fundamental necessity when putting a question of this nature. It's not optional.

As the question was proposed in a way that suggested that there was an answer, and only one 'accuracy' figure was given, then it is logical to assume that the test has the same accuracy regardless of the incidence of disease in the population, and from this it follows that both false negatives and false positives must occur at the same, 1% level.

Yes, indeed, this was clearly the assumption that Wrath wanted to be made. And you will note that I made this assumption in my first post, explicitly.

What I am trying to point out is that making the assumption was essential before the calculation could be done. To turn round and say, "oh well, it's obvious that Wrath intended that assumption to be made" is completely missing the point.

If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.

Yes and no. If Wrath had been explicit in his question, we wouldn't be having this part of the discussion, we would be majoring on the more important aspect of how legitimate it is to assume that no account is taken of the reason for carrying out the test when deciding whether or not the result is reliable.

However, the way you suggest posing the question is not acceptable either. To say something like"The test is always 99% accurate, regardless of what percentage of those tested have the disease" I'm afraid still demonstrates a deep misunderstanding (or lack of understanding) of the problem. You are still failing to use a defined term, and failing to define the term you are using. To do the calculation as set, it is necessary to be told, explicitly, the specificity (or false positive rate) for the test. Obfuscations that don't distinguish between false-positives and false-negatives, which are as I said two completely separately-defined variables, are insufficient.

I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

Absolutely not. This area of test interpretation is a particular interest of mine. As I said, I have published a book chapter about it, which was singled out for praise by an eminemt reviewer. No matter who it had been who had posed the question using undefined terms and with some basic assumptions unspecified, I would have reacted in exactly the same way.

Like it or not, "semantics" (or correct use of explicity-defined terms) is the be-all and end-all of this type of problem.

Rolfe.

 

[ceptimus]

However, the way you suggest posing the question is not acceptable either. To say something like"The test is always 99% accurate, regardless of what percentage of those tested have the disease" I'm afraid still demonstrates a deep misunderstanding (or lack of understanding) of the problem.

No it doesn't. My wording tells you implicitly that the occurance of both false positives and negatives is 1%. It gives just the same information as if I told you the selectivity and specificity.

Why do you assume I don't understand? That is rather insulting. This is a very simple statistics problem - I wouldn't be involved at all except that I didn't like the attack on Wrath.

 

[geni]

I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

It's got more to do with going through certian sections of the uk exam system in the last few years.

You admit that is the problem is to be solverble you have to make an assumption?

You think this is a reasonble assumption to make. I fail to see why. If you treat the problem as a mathmatical abstract there is no reason the make the assumption and if you treat the problem as a real world situtation then there is no reason to make the assumption. The only time that you would make the assumption was if you have to give an answer or you think of it as puzzle and give the answer that the person setting it thinks is correct.

 

[geni]

No it doesn't. My wording tells you implicitly that the occurance of both false positives and negatives is 1%. It gives just the same information as if I told you the selectivity and specificity.

Agreed. It is a slight round about way of saying it but the way you've set it up now there is only one correct answer.

 

[Wrath of the Swarm]

...and I'm back. Morning, all.

I think I've said all I need to about the problem itself, so let's turn our discussion to a somewhat more interesting topic that's come up: Rolfe.

Did you notice that Rolfe posted an answer to the question openly, then brought up real-life facts which aren't relevant to the question at all, while claiming that she did this so that *I* couldn't feel smarter than everyone else?

Further, did you notice that she refuses to admit that the points she's brought up had no bearing on the question? She insists that actual doctors possess information that somehow changes the nature of the answer - which certainly isn't the case.

Consider: doesn't Rolfe spend a lot of time here debunking the rather pathetic arguments for homeopathy? Do you think that she does it out of a concern for truth, or because they make easy targets for her to demonstrate her knowledge on?

Who's really interested in being perceived as smarter than everyone else?

 

[exarch]

Originally posted by ceptimus
I disagree (with Rolfe and geni)

If someone states that a test is 99% accurate, and gives no other information, then you must assume that one test out of every 100 will give the wrong result

So I will assume that all those wrong results are false negatives. Which means that a positive result on the test is 100% correct. Problem solved.

Oh, but apparently, 100% is not one of the possible answers

 

[exarch]

Originally posted by ceptimus
This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.

Well Ceptimus, if you put the question like that, this means that of every 100 positive results you get, only 1 will be wrong, and as such, your chances of being the false positive ar 1/100. The distribution of the affliction among the population isn't even relevant any more.

Can you not see the need for specifying the occurance rate of false positives and false negatives?

 

[Prester John]

Sensitivity - proportion of individuals with the disease who are correctly identified by the test.

Specificity - proportion of individuals without the disease who are correctly identified by the test.



Accuracy is not a term used for desribing diagnostic tests.

 

[geni]

Oh, but apparently, 100% is not one of the possible answers

But neither is the answer you get if you make Wrath of the Swarm's assumption

 

[Wrath of the Swarm]

You're wrong in claiming that the population distribution no longer matters. The population distribution determines how many positive results will occur in a sample of a certain number of individuals. The relative proportion of true positives and false positives matters to the question.

If only one in a million people has the condition being tested for, then if a million people are tested, 99% accuracy means that roughly 10,000 people will receive false positives and only about one person a true positive. If it's one in a thousand, 99% accuracy means that about 10 people will receive a false positive and one person a true positive.

You don't see a difference in the ratios 10,000:1 and 10:1?

 

[Rolfe]

No it doesn't. My wording tells you implicitly that the occurance of both false positives and negatives is 1%. It gives just the same information as if I told you the selectivity and specificity.

Your wording tells me nothing of the kind. Sensitivity and specificity are independent, and defined, variables. You have introduced a term which is not defined in this context, "accuracy". I have repeatedly asked Wrath to explain how this "accuracy" is calculated. I've explained exactly how sensitivity, specificity, and positive and negative predictive values are calculated. I need to know how you are assuming "accuracy" is calculated from the same basic data.

I realise that you don't understand from your consistent failure to grasp that this is an important question which you need to know the answer to.

(I'll reiterate that the original question from the paper Wrath seems to have been referring to didn't use the word "accuracy" at all, it referred to "a false positive rate of 5%", which implies a specificity of 95%, and this is absolutely correct presentation of the terms.)

Wrath or Ceptimus, please explain how I would calculate the term you refer to as "accuracy" in this context.

Rolfe.

 

[JamesM]

Just out of interest, a very similar problem, but with the false positive and true positive rates specified, is given in the entry for Bayes' Theorem over at Wikipedia.

 

[Interesting Ian]

Sorry Ian, nothing personal.

But I know who was considered to have lost the "does 0.99999(infinity) equal 1?" argument, even though as someone who hadn't explored that particular problem in any depth, I initially and instinctively thought your position was correct.

Rolfe.

I was quite definitely correct. One instinctively would think my position was wrong. And it wasn't that argument you said above. That was a similar argument in another thread which I did not read.

 

[Interesting Ian]

I suppose we are really arguing about semantics.

If you make the assumption that the test has different accuracies for false positives and false negatives, then the question cannot be answered.

As the question was proposed in a way that suggested that there was an answer, and only one 'accuracy' figure was given, then it is logical to assume that the test has the same accuracy regardless of the incidence of disease in the population, and from this it follows that both false negatives and false positives must occur at the same, 1% level.

If Wrath had made this explicit in his question - say he had put, "The test is always 99% accurate, regardless of what percentage of those tested have the disease", then we wouldn't be having this discussion.

I think Rolfe's and geni's desire to try and prove Wrath wrong is distorting their argument. If Wrath's question had been asked by another poster, I bet we wouldn't have all this quibbling.

I'd just like to say I absolutely agree with this.

 

[Wrath of the Swarm]

"Accuracy" is the proportion of correct test responses to total test responses.

(Just what the English definition of the word would imply.)

For this particular test, the chance of a false positive is the same as the chance of false negative. The accuracy of the test is the same.

Because of some aspect of the workings of the hypothetical test, it's as likely to fail when dealing with a person who doesn't have the disease as when dealing with one who does.

As we have explained to you many times.

 

[geni]

For this particular test, the chance of a false positive is the same as the chance of false negative. The accuracy of the test is the same.

Not stated in the original question. So you admit the orginal was imposible to answer with the data given?

Because of some aspect of the workings of the hypothetical test, it's as likely to fail when dealing with a person who doesn't have the disease as when dealing with one who does.

You did not state this in you original question therefor it was imposible to answer.

 

[Wrath of the Swarm]

But it was possible.

Let's say I flip a coin (heads or tails) and then someone guesses which side came up.

I can specify an accuracy of the guessing without stating alpha and beta rates because they're the same.

This test is no more likely to mess up when dealing with a positive than a negative. Instead of stating accuracies for positives and negatives, all I need to do is state the accuracy.

geni, are you going to respond to my post where I pointed out that your claim about the population proportion not mattering was wrong?

 

[Wrath of the Swarm]

Are we really expected to believe that Rolfe does not understand what it means for a test to be accurate?

There are plenty of tests whose error rates aren't dependent on the nature of the answer. This hypothetical test is one of them.

Now, since I consider Rolfe to be utterly unable to actually think, and only capable of applying the correct principles by rote, I have no problem with accepting that she genuinely does not understand the concept of test accuracy.

But those of you who claim Rolfe is both intelligent and honest - how do you account for this?