Poll: Accuracy of Test Interpretation

[geni]

Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

Nope try pluging in the figurers for what hapens when the inacucay is entiry due to false negatives.

 

[Rolfe]

Wrath's original question was quite clear, and had a definite answer. If you wish to make up your own questions, it is quite likely that they will have different answers.

If you thought that was clear, you don't understand the question.

What Wrath intended to be assumed was reasonably clear, because we know how his mind works, and by assuming that, the desired result was obtained.

However, he was dishonest because his unstated assumption was that the scenario was such that the doctor was wrong in assuming the positive result to be correct. It is equally if not more likely, in real life, that the scenario was not that assumed by Wrath, and that the doctor had a perfectly valid reason for assuming the result to be right.

Rolfe.

 

[Wrath of the Swarm]

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

Point 3: The question, as it stands, is perfectly comprehensible.

Point 4: It doesn't matter why the doctor ordered the test. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

Point 5: If you want to link together multiple tests, fine. The analysis of the results becomes much, much more complicated. We have to determine the error rating(s) of the first test, the degree to which the first and second tests are independent, determine whether the initial tests are actually uniform (doctors can plausibly use many different symptoms to develop suspicious, and the probabilities for each might not be the same) and so forth.

Point 6: You're only making yourself look more like a fool the more you continue this, Rolfe. Admit you were wrong and get it over with.

 

[Wrath of the Swarm]

Oh, and by the way: Rolfe is an excellent example of why the medical practitioners generally failed to answer the question properly:

They assumed facts not in evidence, and had excessive confidence in the ability of doctors to make accurate judgements.

Of course, Rolfe is not a qualified medical professional. So her inability to interpret a simple question properly means little.

 

[ceptimus]

I disagree (with Rolfe and geni)

If someone states that a test is 99% accurate, and gives no other information, then you must assume that one test out of every 100 will give the wrong result, regardless of whether the persons being tested are diseased, healthy, or any mixture of the two.

It follows from this assumption (which is the only sensible one to make, given how the original question was phrased) that the error rates for both false positives and false negatives is 1%

I think your familiarity with the subject is making you try to read things into Wrath's question that simply were not there.

Geni - if you look back through the thread, you will see I gave a simple worked out solution in my first post. Wrath gave quite sufficient information to allow the question to be answered fully. As I already said, if you wish to ask different questions (or choose to believe that Wrath did) then they will likely have different answers.

 

[geni]

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Appeal to authority logical fallicy

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

next it going to be 100,000 european doctors isn't it I can just tell
Context is everything.

Point 3: The question, as it stands, is perfectly comprehensible.

If by that you mean that I can guess what you mean then yes. However without makeing this guess the problem is unsolverble

 

[Wrath of the Swarm]

Appeal to authority logical fallicy

No, you fool! The next bit is the appeal to authority! That's just the "appeal to keeping experimental modalities the same".

If by that you mean that I can guess what you mean then yes. However without makeing this guess the problem is unsolverble

I quite agree. The problem is completely unsolverble. No one can solverb it!

Flan flan flan flan...

 

[geni]

I disagree (with Rolfe and geni)

If someone states that a test is 99% accurate, and gives no other information, then you must assume that one test out of every 100 will give the wrong result, regardless of whether the persons being tested are diseased, healthy, or any mixture of the two.

It follows from this assumption (which is the only sensible one to make, given how the original question was phrased) that the error rates for both false positives and false negatives is 1%

(my Italics) I see no reason to assume. The error would be enogh to get the question throw out of an exam paper. The way Wrath of the Swarm persented the question made it clear that it was ment to throw you. In such cases it is vital that the question is sound and makes sure that the person trying to solve it does not have to make any assumptions. In this case an assumption had to be made for which I saw no reason to belive such an assumption should be relible. As such the question was unsolverble.

 

[ceptimus]

(my Italics) I see no reason to assume. The error would be enogh to get the question throw out of an exam paper. The way Wrath of the Swarm persented the question made it clear that it was ment to throw you. In such cases it is vital that the question is sound and makes sure that the person trying to solve it does not have to make any assumptions. In this case an assumption had to be made for which I saw no reason to belive such an assumption should be relible. As such the question was unsolverble.

I think you are being unfair. If I told you a remote viewer was asked to view whether someone was in a town or the country, and they were right 99% of the time, what would you assume then?

 

[geni]

No, you fool! The next bit is the appeal to authority! That's just the "appeal to keeping experimental modalities the same".

they both are appeals to authority it's just the second one contians an appeal to popularity as well. You didn't keep the experimental modalities the same since you changed the context.

 

[Rolfe]

Why the doctor performed the test has no bearing on the correct answer! Does it matter why Farmer Brown took away three apples from the box that held seven? No!

And there aren't an infinite number of answers. Without specifying different values for alpha and beta, we consider only error. Alpha and beta values follow from the overall accuracy.

Again: the hypothetical test was 99% accurate, so there was a 99% chance that any result it came up with would be correct. That tells you what the alpha and beta rates are - they're equal in this particular case.

This is confusing and conflating my two separate assumptions about what Wrath meant by "accurate" that I'm barely capable of disentangling them. It's now clear that Wrath had even less idea about what he was talking about than I realised. Breathtaking.

there was a 99% chance that any result it came up with would be correct

Do you realise that you've just soundly contradicted yourself? The entire thrust of this thread was to demonstrate (correctly, for the conditions you assumed but did not state) that the chance the result in queston was correct was in fact less than 10%.

Make up your mind.

There are only two ways I can see to get this "accuracy" figure.

An arithmetical mean of the sensitivity and specificity. If they were equal, then that would be right enough. But you've explicitly denied that this is how you calculate the figure.

Or the percentage of tests carried out in practice which are correct (positive or negative). This would seem more likely for a figure you now relabel as "error", but to calculate this you need all of the sensitivity, the specificity and the incidence of the condition in the population being tested.

Dream of a thousand cats (with apologies to Neil Gaiman).

1000 cats. Incidence of FeLV infection 10% (for whatever reason).
FeLV test, sensitivity 98%, specificity 95%.

We have 100 infected cats, and 900 uninfected cats.

Of the 100 infected, 98 are true-positive and 2 are false-negative.
Of the 900 uninfected, 855 are true-negative and 45 are false-positive.

Total results:
143 positive, of which 68.5% are correct.
857 negative, of which 99.8% are correct.

1000 results, of which 47 are wrong. Therefore 95.3% of the results on this population are correct. With the positives much more likely to be wrong than the negatives, as is quite often the case, special circumstances pertaining to individuals with very pathognomonic clinical presentations notwithstanding.

And you can see that if you plug in different values for the three original variables, you can get a wide variety of different answers.

OK Wrath. These are two ways of calculating "accuracy" to definitions I can comprehend. Now would you please do me the maths for your derivation of 99%?

And it's quite ridiculous to assert that because you gave only one figure, we should assume the same figure applies to sensitivity and specificity. This pretty much never happens in the real world. To say that since only specificity was relevant to the question, you therefore meant to say "specificity", is reasonable and it's what I originally assumed.

But if 99% is some calculated figure from sensitivity and specificity, I at least want to know how you are going to calculate it when the two values are not equal.

Rolfe.

 

[geni]

I think you are being unfair. If I told you a remote viewer was asked to view whether someone was in a town or the country, and they were right 99% of the time, what would you assume then?

If 999 people in your sample were in the town and 1 in the country I know excatly what I would assume. The assumption can totaly mess up the results and as such is serious.

 

[Wrath of the Swarm]

Breathtaking.Do you realise that you've just soundly contradicted yourself? The entire thrust of this thread was to demonstrate (correctly, for the conditions you assumed but did not state) that the chance the result in queston was correct was in fact less than 10%.

Um, no.

The point of the thread was that, for a particular individual who had been given a positive result, there was only about a 10% chance they actually had the disease.

The chance that the test would give out the correct result was still 99%. But the disease was sufficiently uncommon that the chance the test would wrongly give a positive was much greater than the chance of a true positive.

Do you understand that the set of people given the test and the set of people who tested positive are not the same?

Would it help if I typed more slowly?

Doo yoou unnnderstaaaannnnd?

 

[ceptimus]

The entire thrust of this thread was to demonstrate (correctly, for the conditions you assumed but did not state) that the chance the result in queston was correct was in fact less than 10%.

No. Out of every 100 tests, 1 gave the wrong answer. You are misreading what Wrath said.

 

[ceptimus]

This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.

 

[Rolfe]

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

Point 2: Even if you're so obsessed with proving me wrong that you're willing to claim I had phrased the question inappropriately, you must also claim that the hordes of psychology researchers and statisticians who wrote the question also screwed up... which I think goes just a bit farther.

Point 3: The question, as it stands, is perfectly comprehensible.

Point 4: It doesn't matter why the doctor ordered the test. There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

Point 5: If you want to link together multiple tests, fine. The analysis of the results becomes much, much more complicated. We have to determine the error rating(s) of the first test, the degree to which the first and second tests are independent, determine whether the initial tests are actually uniform (doctors can plausibly use many different symptoms to develop suspicious, and the probabilities for each might not be the same) and so forth.

Point 6: You're only making yourself look more like a fool the more you continue this, Rolfe. Admit you were wrong and get it over with.

Trawling through the ad-homs to get to the argument, such as it is....

Point 1. I don't care whether the same flawed question was used to ambush doctors. Appeal to authority. Geni spotted the flaw too while I was typing my initial post, so it wasn't exactly subtle.

Point 2. Same as point 1, appeal to authority.

Point 3. Only if you make the effort to figure out the unstated assumptions.

Point 4.

There are plenty of tests that are used as screens. Furthermore, even in the ones that aren't, the error rates of the test are not dependent on the makeup of the tested population.

(a) Yes, there are plenty of tests that are used as screens. But whether or not that is the case in this particular instance is something you didn't see fit to tell us.

(b) Kindly define "error rates of the test". Show me the maths. I showed you mine. Specificity and sensitivity are independent of the composition of the population being tested. That is why they are the figures to look for when assessing a product. You can then plug these in to different "populations" to get positive and negative predictive value, which are. Your arguments seem to be slewing wildly between one definition and the other, which your lack of defining what you mean by either "accuracy" or "error rate" simply obfuscates completely.

Point 5. Combining sensitivities, specificities and clinical probability of infection to get an estimated predictive value for an individual test on an individual patient is more complex, I agree. Which is why I presented it as a graph (see page 1). You are making it needlessly complicated dragging in differential probabilities for individual clinical signs. While this might be a further refinement, to say "the clinical probability that this patient is affected is x%, to my most educated guess" is a perfectly workable way to go about it, and much superior to "people in general have a y% incidence of this condition" when you are dealing with a specified individual. It's not that hard.

Point 6.

You're only making yourself look more like a fool the more you continue this, Wrath. Admit you were wrong and get it over with.

I agree.

Rolfe.

 

[geni]

This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.

Problem is that the sensitivity and specificity are the two things that make up the accucery. The result is I end up with an equation looking something like this:

P₁+P₂=X now I know X but I don't know either of the other two values so I'm slightly stuck.

 

[Rolfe]

This 'sensitivity' and 'specificity' is what is confusing you Rolfe. Wrath made no mention of those.

On average, out of every 100 tests carried out, 99 will give the correct answer, and 1 will give the wrong answer. That is all you need to know, and it is perfectly clear.

Wrath made no mention, but he should have.

In fact, only the specificity is required for the calculation Wrath posed. Therefore I initially assumed that the sloppily-used "accuracy" figure was intended to be specificity. And stated this assumption clearly. It is Wrath himself who is denying this is what he meant.

Now, please think long and hard about the different things your second paragraph might mean, and the ways in which it is not "perfectly clear". Have another look at the "Dream of a thousand cats".

You are assuming that by 99% accurate, we can assume that 1% of unaffected individuals will test false positive, and 1% of affected individuals will test false negative.

That is simply an invalid assumption. Real tests in the real world have different values for these two figures, and they have to be quoted separately. You might say as a sweeping generalisation that a test was "highly accurate" if both figures were very good, but there's no meaningful way to combine them to a single "error rate" unless you do the entire thousand cats dance.

I'm not confused. I do this for a living. I have published a chapter in a book about it. And got very good book reviews from eminent professors, by the way.

I know what Wrath assumed, and I know what he wanted us to assume. That was clear from the first post. What is being discussed is the way this was set up without making these assumptions clear, and the fact that if you make other, equally valid assumptions, you get a completely different answer to the one Wrath wanted us to get.

Rolfe.

 

[Rolfe]

The point of the thread was that, for a particular individual who had been given a positive result, there was only about a 10% chance they actually had the disease.

The chance that the test would give out the correct result was still 99%. But the disease was sufficiently uncommon that the chance the test would wrongly give a positive was much greater than the chance of a true positive.

Quit with the ad-homs, it just gives me eyestrain.

For a particular individual who gets the test, there are all sorts of different probabilities that he is actually affected. Depending on the assumptions you make.

Now, could you do me the maths again (oh sorry I mean for the first time) to demonstrate how you arrive at "the chance that the test will give out the correct result is 99%". Accuracy, error rate, I don't care what you call it, just TELL ME HOW YOU WORK IT OUT.

Now tell me how, if you are treating every individual the same, that is as members of this "population" with 0.1% incidence, you can still simultaneously declare that the chance the test has given out the correct result is only 9.02%.

Where are these test being done that have the 99% probability of being right, and what's it about this particular patient that gives him only a 9.02% chance of getting a correct result?

(Hint: You are going to have to consider the people getting the negative results here. I want to see the maths. And I want to bottom line to come out at 99% exactly, using the parameters you yourself have set.)

Rolfe.

 

[Wrath of the Swarm]

Real tests in the real world do not necessarily have different values for the two numbers. They do frequently.

Your inability to comprehend his point makes the rest of your claims even more suspicious than they already are.

For my argument to be valid, I would have to use the same question as was used in the studies. That's a basic point of experimental design - which Rolfe clearly knows nothing about. You can't test the validity of an experiment without recreating its structure.

The second point *is* appeal to authority... just as Rolfe's claims about having written a book are appeals to authority. Who is more credible - psychological researchers, medical doctors, and statisticians, or Rolfe?

There are no unstated assumptions. The only person with assumptions is Rolfe, who can only think by rote and can't comprehend that alpha and beta values do not have to be specified, nor do they even have to be different.

Your attention-whoring is not going unnoticed.