Poll: Accuracy of Test Interpretation

Wrath, I'm glad you've finally admitted you were wrong about the origin of the question. It was big of you to do so. Although, you have a lot more to apologise for in this thread with regard to your juvenile and spiteful use of ad homs.

With regard to whether the information given in the question was sufficient to answer the question, I have only this to say. Yes, it was fairly easy to make an assumption about what you were getting at and give the answer, as you defined it. But, it is also true that an assumption had to be made and that while this assumption was the most likely one, it certainly wasn't the only way to interpret the question.

Your question was underspecified. This has been pointed out to you many times by many posters, all of them far more able than I to make this clear. My suggestion to you is to accept that you are incorrect on this matter and bow out with what tattered remains of dignity you still have.
 
No, you're still wrong about that, I'm afraid. There was only one correct and contextually viable way to interpret the statement.

The only way it's possible to establish the accuracy of a test independent of a particular sample is for the alpha and beta rates to be the same. Otherwise, the alpha and beta rates need to be known specifically, because the overall accuracy will depend on who is being tested.

We didn't know whether (in the example) I had the disease or not. I gave an accuracy for the test - from this, we know that alpha must equal beta. There is no other possibility.

Rolfe expected the format she was accustomed to, and interpreted the difference as an error. This is incorrect.

If you can show me a way in which an accuracy can be established independent of a sample and alpha can be made not equal to beta, I will admit that I'm wrong. But you simply can't do that because it's not logically possible.

Regarding insults:

Considering that Rolfe and Friends have slugged an even greater number of insults at me, and that Rolfe is an attention-whoring quack... well.
 
Wrath of the Swarm said:
Who mentioned symptoms A, B, and C? Checking for those symptoms is another test - and we've already established that combining two tests gives a much more accurate result than either seperately.
Click to expand...

You idiot, doctors dont test people randomly in the population. They run tests on people present with specific symptoms.
 
Well Wrath, as you know, I agree with you. But I don't see that you need to keep insulting Rolfe and her followers, even when they insult you.

Rolfe knew what you meant, and is only quibbling because she wants to prove you wrong.

Rolfe's supporters are wrong, but so far they don't fully understand why.

I think you would convince more lurkers if you left out the insults.
 
Really? So what symptoms induce them to offer HIV testing? Or mammograms? Or PSAs?

Anyway, this is irrelevant. The population the test will be used in is irrelevant to how accurate it is if alpha and beta are equal. Clearly it affects the conclusions drawn from the test, but not the test itself.
 
You might be right, ceptimus. On the other hand, if these people aren't convinced by logical arguments but are swayed by the presence of insults... well, they're like Rolfe's followers.

Do I really want that kind of person telling people I'm right? Having gullible people proclaim my correctness doesn't say much for me.

The actual skeptics on these forums will think about the issues raised and reach a conclusion whether I insult Rolfe or not. And since I feel like insulting Rolfe... well, what's the harm, really?
 
Originally posted by Wrath of the Swarm
If the coin has a 60% chance of coming up heads, that does not mean that in a sample of tosses, heads will occur 60% of the time.
Click to expand...
Ignoring the ad homs ...

Sure, but if you guessed heads every time, you'd still have a better chance of being right more often. It could be tails every time, but the odds of that happening are extremely small. An all tails sample would also not be representative of the average sample, which would, on average, contain 60% heads.
Originally posted by Wrath of the Swarm
But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?

In fact, your accuracy would depend entirely on what population you were presented with. Your method does not have an error rating independent of the population distribution.
Click to expand...
Sure enough, but that's why you have to specify the rating at which false positives occur. Because the amount of false positives will depend on the population, while the accuracy as you describe it, according to your own words:
Your method does not have an error rating independent of the population distribution.

This test does.
Click to expand...
So if accuracy is not dependent on the population distribution, then neither is the ratio between false positives and false negatives, in order to receive the same final independent "accuracy" rating. so how is it calculated?
 
ceptimus said:

Rolfe's supporters are wrong, but so far they don't fully understand why.
Click to expand...

Really? Show that mathmaticly that what WOTS calls alpha and beat have to be equal. You amited before that you had to make an assumption to get your answer.
 
Wrath of the Swarm said:
If it's possible to establish an accuracy for the test as a whole, then the alpha and beta rates (chances of false positive and false negative, respectively) MUST BE EQUAL.
Click to expand...
There we are, you finally said it.

WRONG !! :D

If the overall accuracy of the test is 95% that says nothing about the amount of false positives or false negatives. Indeed, a test can be 95% accurate, and of every 100 tests, come up with 5 false negatives. here is nothing that says they MUST BE EQUAL.

I think this is the tricky bit you still don't get.

it is mathematically impossible for it to be otherwise. If this is not the case, then the accuracy of the test becomes a variable dependent on the population given to it, and it is then impossible to consider how good the test is independent from a particular sample.
Click to expand...
Now do you understand why it is important to give rates for false positives and false negatives? Do you understand why Rolfe is puzzled by your use of the word "accuracy"? Why your initial question was poorly worded?
 
Let alpha and beta be non-equal. The chance of getting a false positive is not the same as getting a false negative.

Therefore, the accuracy of the test is a function of the sample population fed to it. For example, if the entire population is positive, all error will consist of false negatives. If the entire population is negative, all error will consist of false positives.

Since we've established that the chance of false positives isn't the same as the chance of false negatives, the test's performance varies depending on what kinds of people are fed to it.

That means that a general statement about its accuracy cannot be made.

This does NOT mean that there are no tests for which general statements about their accuracy cannot be made.

IF a general statement about the test's accuracy is made, THEN the alpha must be the same as the beta for the statement to be valid. These tests are perfectly possible - common, even. Thus there is no reason for the statement to be presumed invalid - and since we're talking about a hypothetical test, anything logically possible I establish about it is correct and valid - ergo, the alpha is the beta.

No "assumptions" are necessary.
 
exarch said:
If the overall accuracy of the test is 95% that says nothing about the amount of false positives or false negatives.
Click to expand...
Wrong.

First, we're not talking about the 'amount', we're talking about the proportion. Completely different concept.

Secondly, the overall accuracy of the test cannot be established - or even spoken about - without reference to a specific test population if the proportion of false positives is different from the proportion of false negatives.

Your premises are flawed.
 
Part 1

Geni, if I tell you a test is 99% accurate, regardless of the incidence of the disease in the population, then it follows that alpha = beta.

I assume you accept this?


Part 2

The wording of Wrath's question did tell you the true incidence of the disease in the population, so there is room for you to quibble if you wish.

However, if you do quibble, then the question becomes unanswerable.

I assert that almost any question, no matter how carefully stated, can be spoilt by pointless quibbling. You almost have to use only the language of mathematics to avoid this.

Now if your quibble allows you to come up with a different valid answer, I don't mind that, but if it merely allows you to declare the question unanswerable I think it's pointless.
 
Ah, but I told them the incidence in the general population.

Each patient, taken individually, is their own population. Either the patient has the disease, or they don't.

If alpha != beta, then without knowing whether the patient is positive or negative, the overall accuracy is not equal to the accuracy in a particular case. (And indeed, if we know whether the patient is sick or not, we can be 100% sure of the conclusion we draw no matter what the test says!)

In order to draw a meaningful conclusion, we would have to know what the test result was and the alpha or beta rate.

But the alpha rate IS the beta rate in the particular example since the test itself has a known accuracy.
 
Originally posted by ceptimus
Part 1

Geni, if I tell you a test is 99% accurate, regardless of the incidence of the disease in the population, then it follows that alpha = beta.
Click to expand...
Or alpha = 0 or beta = 0

Part 2

The wording of Wrath's question did tell you the true incidence of the disease in the population, so there is room for you to quibble if you wish.

However, if you do quibble, then the question becomes unanswerable.

I assert that almost any question, no matter how carefully stated, can be spoilt by pointless quibbling. You almost have to use only the language of mathematics to avoid this.
Click to expand...
Obviously this wasn't necessary, as someone posted a link to a similar question that wasn't ambiguous.
I solved the question by making an assumption, based on knowing exactly what WotS was getting at, but as someone who posts regularly in the puzzles forum, you of all people should know that the wording of a puzzle is particularly important if you want to find the right answer. (Remeber the prince with the poison in the boxes?)

There probably wouldn't have been a problem if WotS would have just conceded that he had worded it wrong. He just can't admit he's wrong. It took us 5 pages to finally get him to do that.
Right now, I'm just messing with him :p
 
No exarch. alpha or beta can't be zero. Here's a simple way to see why.

Say everyone in the population has the disease. The test is 99% accurate remember.

Now say no one in the population has the disease. The test is 99% accurate remember.

As regards the unambiguous question, post a copy of it here, and we'll see if we can find a quibble.
 
If alpha != beta, then we can't say the test has an accuracy.

We said the test has an accuracy.

Therefore, alpha == beta.
 
This thread has been bizarre, to say the least.

I would say my reaction to the initial post was very similar to exarch's and ceptimus'. I just assumed he meant the error rates were the same. To tell the truth it seemed obvious to me.

It's interesting that you see people come into this thread and insult WOS for various things, most of which amount to insulting him for continuing on this ridiculous thread for so long. But not many people are saying much about Rolfe continuing the thread for just as long....

There was a problem stated, with a reasonable assumption (IMO) it's easily solved. Without the assumption, it cannot be solved. 5 pages later people are still talking about this?

WOS, you think the problem was stated in an acceptable way. OK.
Rolfe, you think the problem was not clear enough. OK.

We done now? :rolleyes:

Adam
 
The statement that an assumption is necessary is a false one.

I just want that to be admitted, and this thread can end peacefully.
 
Nanny nanny boo boo , seems to be the main reason this thread has gone on.

I say flarn! I thought it would tell me if I had the illness 70 % so I was way off.
 
Wrath of the Swarm said:
Secondly, the overall accuracy of the test cannot be established - or even spoken about - without reference to a specific test population if the proportion of false positives is different from the proportion of false negatives.
Click to expand...
Wrong.

The "accuracy" of the test cannot be established, or even spoken about, until you have clearly defined what you mean by "accuracy". This you have not done.

This subject is a very well-defined one, with its own well-defined vocabulary. So if I were to refer to the "positive predictive value" for example, I would be justified in believing that anyone familiar with the subject would know exactly what I meant by this. If someone unfamiliar with the subject asked me for a definition, it would be easy to provide one.

BillyJoe kindly supplied a list of concise definitions of nearly all the terms recognised for discussing this problem. I'll repeat them again now.

POSITIVES/NEGATIVES, SENSITIVITY/SPECIFICITY, PREDICTIVE VALUES, PREVALENCE.

TRUE POSITIVE: a person who tests positive and has the disease.

FALSE POSITIVE: a person who tests positive but does not have the disease.

TRUE NEGATIVE: a person who tests negative and does not the disease.

FALSE NEGATIVE: a person who tests negative but does have the disease.

SENSITIVITY: the percentage of people with the disease for whom the test is positive.
Sensitivity = TP / (TP + FN)

SPECIFICITY: the percentage of people without the disease for whom the test is negative.
Specificity = TN / (TN + FP)

POSITIVE PREDICTIVE VALUE (PPV): the percentage of people with the disease who test positive for the disease
PPV = TP / (TP + FP)

NEGATIVE PREDICTIVE VALUE (NPV): the percentage of people without the disease who test negative for the disease
NPV = TN / (TN + FN)

PREVALENCE: the percentage of the population who have the disease

courtesy,
BillyJoe

The two extra that BillyJoe didn't mention are:

FALSE POSITIVE RATE: 100 - specificity

FALSE NEGATIVE RATE: 100 - sensitivity.

Nowhere in that list do the terms "accuracy" or "alpha and beta values" appear.

The way these characteristics of a test are evaluated is by taking a group of patients, x% of whom have the condition (known about by other means, the reference method). Test them all by the test you are evaluating. Some of the affected people will test positive, the true-positives (TP). Some of them will test negative, the false-negatives (FN). Some of the unaffected people will test negative, the true-negatives (TN). Some of them will test positive, the false-positives (FP).

This is all the information you get about the test. From this you have to derive the numbers that describe the test to its users. Above, BillyJoe has detailed exactly how this is done for the terms we actually use (and I added a couple more, and defined them).

Now, you will note that the sensitivity and specificity values are completely independent of the proportion of affected people, x. So long as you have enough in each group (affected and unaffected) to give a good representation of test performance, the proportions are irrelevant. Switch them round and the values for sensitivity and specificity will remain the same. And THEY DO NOT HAVE TO BE EQUAL FOR THIS TO BE TRUE.

This is why these two figures are the cardinal descriptive terms of the test. They are absolute values which describe the test independently of population prevalence (which may vary widely).

However, if you actually want to examine the implications of these values for the probability of the test being right in populations of differing prevalence (or as I like to think of it, in patients with differing probabilities of being affected), you need to factor in your assumed prevalence, and derive the PPV and the NPV. And indeed, sometimes the results you get for particular permutations of this sum are somewhat counter-intuitive.

Wrath is persistently referring to "accuracy", and to "alpha and beta values". Now it is impossible to talk about this rationally unless all terms are defined. And by defined, that means explain how you get the number from the results you got when you did the evaluation described in the bold paragraph. Because that is all the information you will ever have (although you may improve on the validity of the exercise by increasing the number of individuals involved).

I thought alpha and beta values were just another way I've never heard of of expressing sensitivity and specificity, and I still suspect that from the way Wrath is using the words. But he won't even confirm that, or which is which.

He persistently refuses to explain what he means by "accuracy".

I think I'm beginning to get it better in the more recent posts. It's one of the suggestions I considered earlier. "Accuracy" is defined by Wrath as the figure you get for both sensitivity and specificity if these two parameters happen to be equal.

No wonder this isn't a defined term! It's a completely useless term. To dream up a term like this which can only be used in the improbable fluke of TP / (TP + FN) happening to come out equal to TN / (TN + FP) in the evaluation scenario described above is meaningless. To then castigate everyone else who doesn't intuit this remarkable definition from your ill-defined scenario, is arrogant and unjustified.
IF a general statement about the test's accuracy is made, THEN the alpha must be the same as the beta for the statement to be valid. These tests are perfectly possible - common, even.
Click to expand...
All right, Wrath, name six. I've already said this pages earlier. In the real world in which most of us actually practise, real tests don't come like that. And if one happens to come like that, it's just a coincidence (and a coincidence which might well just disappear if you extend the numbers of patients in your evaluation study and publish revised and better estimates of the figures). Not worth coining a special defined term for, which is why nobody did. Until Wrath came along.

Now this is boring me too. It isn't the aspect of the problem I really wanted to talk about. I think the use of the word "accuracy" was just a piece of sloppy terminology for "specificity" in the first place. I was perfectly happy all along just to substitute the word "specificity" for "accuracy" in the original question and call it quits on that aspect. Because we don't need the sensitivity value at all to do the sum! It doesn't have to be the same as the specificity, it doesn't have to be anything in particular. If you just say, the test specificity is 99%, you can carry on.

Which is what I was trying to do, honest. (Because there is a lot of carrying on to do, in fact.)

However, we do have something to clear up properly before we do. I've stated that sensitivity and specificity values are a constant property of the test, and do not change with prevalence of disease in the population being tested. And that they not only do not have to be equal, it is no more than a mildly interesting (and unusual) coincidence if they are. I can be told the values for any given test, and I know they describe the performance of the test as they are.

This is why they are the parameters you would quote when asking the initial question the thread is about. In fact the question was simply, "given a test with specificity 99%, what is its positive predictive value in a population with a prevalence of disease of 0.1%?" Answer, no arguing, 9.02%. No need for the sensitivity even to be mentioned, you don't need it. And without the dressing-up as a doctor's appointment, soluble as a pure statistics question.

So, do I have agreement that sensitivity and specificity are absolutes, do not have to be equal (and in fact will probably not be equal), and do not depend (equal or not) on the prevalence of disease in the population used to evaluate the test?

I'm asking this because I've seen some posts that make me think this isn't clear in everyone's minds, and if we don't get it clear we could be in for another three pages of cross-purposes.

Rolfe.
 
You must log in or register to reply here.

Back
Top Bottom