Poll: Accuracy of Test Interpretation

[geni]

But it was possible.

Lots of things were posible

Let's say I flip a coin (heads or tails) and then someone guesses which side came up.

I can specify an accuracy of the guessing without stating alpha and beta rates because they're the same.

In fact you can't because you failed to specify that it was a fair coin (you relly should try working for edexcel) However since most coins are fair I would probably be ok making that assumption. For most clinical tests I have no reason to belive the two values are the same.

This test is no more likely to mess up when dealing with a positive than a negative. Instead of stating accuracies for positives and negatives, all I need to do is state the accuracy.

This is not information I had avaible in the oringinal question

geni, are you going to respond to my post where I pointed out that your claim about the population proportion not mattering was wrong? [/B]

Since I'm not entiry sure what you are talking about this is going to be difficult. However I think think you mean my reply to ceptimus. If the accury remains the same for any population proportion then I can slove the problem through simultanious equations ( the equations look something like this:

Acccurary for proptionA= p₁+p₂
Acccurary for proptionb= p₁+p₂

accurary is of coure fixed and we can have any value for and and b making the problem solverble.)

 

[Wrath of the Swarm]

You're missing the point. Whether the coin was fair or not has nothing to do with whether the guessing procedure has different accuracies for heads or tails.

You're also not responding to my earlier question. Why did you claim the population proportion was irrelevant when it was still vitally important?

 

[geni]

You're missing the point. Whether the coin was fair or not has nothing to do with whether the guessing procedure has different accuracies for heads or tails.

By thid logic I can win the million no problem I have a coin that lands on heads 60% of the time and claim I can predict it's fall with 60% accuricy and then guess heads everytime. To put it another way with a fair coin random guessing would give you 50% with a non fair coin random guessing would give you less than 50%

You're also not responding to my earlier question. Why did you claim the population proportion was irrelevant when it was still vitally important?

Where did I say this?

 

[Wrath of the Swarm]

Oh, I'm sorry. That was exarch, wasn't it?

I take it back. I have trouble telling Rolfe's groupies apart from one another.

[edit] Actually, I take back the taking back. The original comment was exarch's, but you've proceeded to make similar claims - you're implying you can get a 60% accuracy by guessing the same way each time in a particular case, but you're ignoring the effect of the actual population on your strategy's success.

Which is in essence the same claim exarch is making.

 

[Wrath of the Swarm]

But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?

In fact, your accuracy would depend entirely on what population you were presented with. Your method does not have an error rating independent of the population distribution.

This test does.

 

[pgwenthold]

Are we really expected to believe that Rolfe does not understand what it means for a test to be accurate?

Considering that Rolfe knows an awful lot about it, then it is not surprising that she would not know what you mean by accuracy.

Your naive approach is too simplistic for someone who really knows what they are doing.

This is what makes it really funny, because the point of your exercise was to show that you know what you are doing and those silly doctors don't. Yet this doctor has schooled you up and down fourteen times to nowhere. The irony is so great.

 

[Wrath of the Swarm]

But she hasn't. Her argument is inherently flawed.

If you mindlessly accept that everything she says is correct, well then - clearly every time she gets into an argument with anyone, she'll wipe the floor with them!

But she simply isn't correct.

 

[steve74]

Wrath, why can't you just admit the question was worded too vaguely?

You claim that you worded the question exactly as it was given to the test subjects:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

But you'd previously admitted you couldn't even remember the source of the research:

The basic problem is a classic one. I'm trying to find the sources in which I read about the implications for screening tests several years ago.

If I recall correctly, doctors get the right answer more frequently than the general population, but they still tended to reach grossly wrong conclusions about whether a particular patient had a disease. I believe they overestimated the power of the tests significantly.

If I find some good sources on the subject, I'll get back to you.

It has been ably demonstrated to you by many posters why you are wrong but you refuse to admit it. I can only conclude one of 2 things:
1. You lack the intellectual ability to see the correctness of others' arguments or,
2. You're a troll.

Which is it?

 

[Wrath of the Swarm]

Neither.

I finally found the sources that duplicated the question (I even pointed them out, remember?).

The original source was a professor of mine, many years ago.

 

[geni]

But you couldn't say that you had a 60% chance of being correct in all circumstances, could you now?

In fact, your accuracy would depend entirely on what population you were presented with. Your method does not have an error rating independent of the population distribution.

This test does.

Assuming a reasonble szie population yes I would beacuse I've force dthe population to be 60% on way and 40% the other. If you change the population you have to do it by changing how much the coin is rigged.

 

[Wrath of the Swarm]

No, I don't.

You see, the population of people given in the example is chosen from the wider population. The subpopulation might not have the same distribution as the population as a whole does. (This was actually one of Rolfe's points, remember?)

So if I toss this unfair coin ten times, it won't necessarily come up heads six times and tails four (or vice versa). It might come up heads all ten times.

If you do nothing but state one possibility over and over (say, guessing heads every time), your accuracy will depend entirely on what population is given to you. If the population is all heads, you'll be 100% accurate. If the population is all tails, you'll be 100% inaccurate. You'll be X% accurate if the population fed to you is X% heads.

The hypothetical test given originally has a 99% accuracy, no matter what subjects are fed to it. That's what accuracy means - it's no good using a concept that depends on the population distribution if you don't know what that is, and we can't presume beforehand that a doctor will face any particular population.

 

[exarch]

Originally posted by Wrath of the Swarm
I am pointing out that, when presented with a simple question involving the use of a diagnostic test with known accuracy in a particular generic situation, the vast majority got it wrong.

I suspect that if you had asked them, most of them would have predicted they'd get it right.

What was it you were saying earlier about there being no hidden agenda in your question? Seems like Woo of the Swarm has started slamming doctors again. Just wait long enough, and the true purpose emerges, as always

This is the problem. Rolfe, as the resident mindless-defender-of-the medical-status-quo, denies that there is a problem and attacks that which makes the existence of the problem clear. When dealing with people whose positions are grossly incorrect, her mindless rancor actually aids her. But she can't tell the difference - she'll attack anything and everything.

No Swarm, the problem is you thinking you are so much smarter than you really are. You think you're infallible, and it makes you cocky. And in this instance you have to start back peddaling at a speed you've never had to before, because for once you had the misfortune to run into someone who actually is an expert in the subject you're trying to woo us over with.

 

[Wrath of the Swarm]

But it's not limited to doctors. Most people given this question get it wrong, and most of them overestimate how many people get it right.

The only hidden agenda here is the one you're projecting.

And this "expert" is making an invalid objection. You can't just assert that the objection is valid because she's an "expert" - that's one of those nasty arguments from authority, remember?

Rolfe has suggested that the question cannot be answered without more data. This is not the case. She's so used to thinking about tests whose accuracy is dependent on the nature of the result that when she didn't see that information, she reflexively asserted it was necessary.

 

[pgwenthold]

quote:
--------------------------------------------------------------------------------

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

--------------------------------------------------------------------------------

I think this statement basically illustrates the problem at hand here.

Wrath initially assumed that the fact that doctors apparently performed poorly on this question indicates their problems with cog thinking.

However, as Rolfe has demonstrated (by actions more than words), there are certain assumptions built into the question. Wrath, in his naivity, thinks the assumptions are obvious. However, to a more knowledgable reader, the proper assumptions are not at all obvious. Thus, an alternate explanation for the poor performance by the doctors is that they did not share the assumptions of the question writer.

So we have two possible explanations: one is that doctors have very poor cognative ability. The second is that the question requires too many assumptions that will not be shared between the question writer and an expert answering the question.

I see no reason to think that either or even both explanations account for what was observed. There are probably doctors that lack the cognative skills, and there were probably doctors who knew so much about it that they couldn't answer the question.

 

[Wrath of the Swarm]

Exarch: you've claimed that the population distribution was irrelevant to the question.

Can you explain for us why you made the statement and whether it is correct?

 

[Wrath of the Swarm]

I think this statement basically illustrates the problem at hand here.

Wrath initially assumed that the fact that doctors apparently performed poorly on this question indicates their problems with cog thinking.

No, it demonstrates that people in general have problems with thinking. (By the way, what's "cognitive thinking"?

However, as Rolfe has demonstrated (by actions more than words), there are certain assumptions built into the question.

No, there are certain assumptions built into the answerer - namely, Rolfe.

Alpha and beta errors are more complicated concepts that simple accuracy. The question gave an accuracy for the test that did not refer in any way to these more complicated concepts. It simply stated the chance that the test would be wrong about any result it gave.

Rolfe is conditioned to expect the more complex concepts, and therefore she is rendered unable to think about the simple ones?

Riiight.

 

[steve74]

Neither.

I finally found the sources that duplicated the question (I even pointed them out, remember?).

The original source was a professor of mine, many years ago.

Stop bullsh1tt1ng, Wrath. You have not referenced one single study in this thread. You have referenced a few review articles, which you had to Google for. So, once again I ask you, which study was this wording used in? You must know the answer because you claimed:

Point 1: The question, as I presented it, is the same question that was used in research with doctors.

 

[Rolfe]

"Accuracy" is the proportion of correct test responses to total test responses.

(Just what the English definition of the word would imply.)

For this particular test, the chance of a false positive is the same as the chance of false negative. The accuracy of the test is the same.

Because of some aspect of the workings of the hypothetical test, it's as likely to fail when dealing with a person who doesn't have the disease as when dealing with one who does.

Funnily enough, I do know what accuracy means. It is a defined term in clinical biochemistry, where you are dealing with concentrations of analytes, rather than just positive or negative. It is defined as how well the results from the method under test match up to the "true" concentrations in the samples, defined as the concentrations as measured by the designated reference method. This is actually a much more difficult subject than the sensitivity/specificity one, because you can get different values for r depending on the spread of concentrations you have in your test group of samples, and because it's not just the correlation coeficient that matters, it's how well the line of best fit matches up with the line of coincidence (so you have to look a the slope and the y-intercept as well). And at some point you do have to give in and concede that your "reference" method has a degree of inaccuracy within it too.

This is one of the two cardinal characteristics of a biochemistry assay. The other is precision (inter and intra-assay), defined as the consistency of results when performing repeat assays on the same sample, and measured by the coefficient of variation.

Accuracy (in this sense) is a minefield and a nightmare to pin down statistically, let's not go there.

But this is the context in which it is a defined term in the field of laboratory analysis. In biochemistry assays (measuring concentration of analyte) the cardinal characteristics of the assays are accuracy and precision, as described.

In serology testing (positive/negative results) the cardinal characteristics of an assay are sensitivity and specificity. Accuracy is not a defined term and has no meaning in this context.

You see, Wrath, while ordinary conversation may be able to use these words as it pleases, clinical laboratory work requires very careful use of the terms as they are defined, otherwise terrible misunderstandings may ensue. As you are finding out. In this context "the ordinary English definition of the word" is irrelevant.

You are now revealing that the sensitivity and the specificity of the test are the same. However, as Geni has said, you didn't include that information in your original problem. You used an undefined word "accuracy" which you still have not told us how you are calculating.

To get the "percentage of correct test responses to total test responses" you need to do the entire thousand cats calculation, and sensitivity, specificity and prevalence all have to be stated before you can even start. So I submit that this isn't a clear or complete answer.

If you care to refer to the start of the thread, you may notice that my assumption was that since all we neeed to know for the purpose of the calculation set was the specificity, I was going to assume that by "accuracy" you actually meant specificity. I really didn't care whether the sensitivity was the same or not, because I didn't need to know that to do the sums. Rather sloppy use of terminology, but something easily clarified, or so I thought.

But no, you have to make this more and more complicated, and in so doing it becomes more and more obvious that your understanding of the subject is really rather superficial.

Now either admit that since all we needed to know for the sum was the specificity, your "accuracy" figure should be taken to mean specificity, or please explain in detail how I would calculate this novel "accuracy" term you've introduced, from the beginning.

As an example (shining, I have to say - some of this is so sparely expressed I thought for half a moment it was wrong, but it ain't.):

POSITIVES/NEGATIVES, SENSITIVITY/SPECIFICITY, PREDICTIVE VALUES, PREVALENCE.

TRUE POSITIVE: a person who tests positive and has the disease.

FALSE POSITIVE: a person who tests positive but does not have the disease.

TRUE NEGATIVE: a person who tests negative and does not the disease.

FALSE NEGATIVE: a person who tests negative but does have the disease.

SENSITIVITY: the percentage of people with the disease for whom the test is positive.
Sensitivity = TP / (TP + FN)

SPECIFICITY: the percentage of people without the disease for whom the test is negative.
Specificity = TN / (TN + FP)

POSITIVE PREDICTIVE VALUE (PPV): the percentage of people with the disease who test positive for the disease
PPV = TP / (TP + FP)

NEGATIVE PREDICTIVE VALUE (NPV): the percentage of people without the disease who test negative for the disease
NPV = TN / (TN + FN)

PREVALENCE: the percentage of the population who have the disease

courtesy,
BillyJoe

These are all the standard definitions of the terms used by people who understand the subject. Just define your use of "accuracy" to the same standard, please.

Rolfe.

 

[Wrath of the Swarm]

I posted links already. If you didn't read the thread, go back and do so now.

The examples linked used the same question as I did, and as I remembered my professor using.

 

[geni]

No, I don't.

You see, the population of people given in the example is chosen from the wider population. The subpopulation might not have the same distribution as the population as a whole does. (This was actually one of Rolfe's points, remember?)

So if I toss this unfair coin ten times, it won't necessarily come up heads six times and tails four (or vice versa). It might come up heads all ten times.

If you do nothing but state one possibility over and over (say, guessing heads every time), your accuracy will depend entirely on what population is given to you. If the population is all heads, you'll be 100% accurate. If the population is all tails, you'll be 100% inaccurate. You'll be X% accurate if the population fed to you is X% heads.

Only relivant if you are dealing with real clinical situations Since I dont and never will do I don't care

The hypothetical test given originally has a 99% accuracy, no matter what subjects are fed to it. That's what accuracy means - it's no good using a concept that depends on the population distribution if you don't know what that is, and we can't presume beforehand that a doctor will face any particular population.

But you didn't state this in the orignal question did you? You once again are giving me more data to play with which makes the question answerble. The orginal question only delt with one population. It didn't sate that the answer was the same for all populations.