Poll: Accuracy of Test Interpretation

[Badly Shaved Monkey]

Oh brother.

Chance that someone will have the disease: .001
Chance that someone won't have the disease: .999

Chance that the test will give an incorrect answer in any particular circumstances: .01
Chance of correct answer: .99

Chance of getting a false positive: (chance disease-free)(chance of error) = .999(.01) = .00999

Chance of getting a true positive: (chance disease)(chance of correctness) = .001(.99) = .00099

Percentage of correct positive diagnoses: (true positives) / (true positives + false positives) = (.00099) / (.00099 + .00999) = (.00099) / (.01098) ~ .0901639

Conclusion: If the test results come back positive, there is only about a 9% chance that the patient really has the disease.

I don't think any of us who have posted to this thread are disagreeing with this arithmetic. But some of those who voted do not!

 

[Wrath of the Swarm]

"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.

It would be used in the context of chemistry or physics.

For example, if you were performing an equation to calculate the yield of a particular chemical equation, the answer you get cannot be more accurate than the least accurate measurement of the ingredients. If the measuring device you used to parcel out the ingredients is accurate to only one-tenth of a basic unit, the result cannot be given as a value accurate to more than one-tenth of that basic unit, even if the math gives you more digits than that.

Another example: if you multiplied a measured value with three significant decimal places (accuracy known to one part in a thousand) by a physical constant with eight significant decimal places, the resulting answer must be rounded off to three decimal places. Digits after that are not considered meaningful, as the uncertainty in the measured value is greater than they are.

 

[MRC_Hans]

No, I don't think the Wolfram site is using a definition of accuracy that is like thaht used in lab testing. I thought it was from it's first paragraph.

"The degree to which a given quantity is correct and free from error. For example, a quantity specified as 100 ± 1 has an (absolute) accuracy of ± 1 (meaning its true value can fall in the range 99-101), while a quantity specified as has a (relative) accuracy of (meaning its true value can fall in the range 98-102)."

I would recognise this as a definition of accuracy, but the explanation goes on to say;

"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.

Consider that second paragraph.

I weigh a car and find it weighs 1.295 tonnes. The accuracy by Wolfram's definition "is given by the number of significant decimal (or other) digits to the right of the decimal point in x" i.e. 3 places of decimals. If I weigh it on another set of scales that report a weight of 1,294,864.4g. This has only 1 place to the right of the dp. Does that make it less accurate? In truth it neither makes it more nor less accurate because a single number cannot convey accuracy you need a measure of spread. But to use the Wolfram definition again, is 1.295 +/- 0.002 tonnes more accurate than 1,294,864.4 +/- 0.3g.

So I'm still left confused as to what the Wolfram definition, in its full form, is used for.

(Edited for clarity)

Mmm, I'm not THAT well versed in English terminology, but I'll give it a try anyhow:

Accuracy: As was already explained earlier, the 95% confidence range of a measurement. This includes various noise in the physical system, so 100 +-1% means that if the meter reads 100, then the actual value is, with 95% certainy between 99 and 101.

Precision: This, I think, is what I would call resolution. Suppose the meter used above showed the result with four decimals: 100.0000 . In this case the extra decimals would not influence the accuracy, because it would still be +-1%. However, if the resolution was only three digits (no decimals), then we could not get +-1%, because the impresicion in the readout alone would be +-1.

Hans

 

[Wrath of the Swarm]

I don't think any of us who have posted to this thread are disagreeing with this arithmetic. But some of those who voted do not!

Ah, but people are disagreeing. They assert that an assumption is being made when I use the chance that the test is wrong in any particular case (1%) for the chances that the test is wrong when its response is positive and when its negative.

The point is that, since it is possible to give the test an accuracy that's independent of a sample population, those probabilities are necessarily equal to each other and to the overall accuracy.

Rolfe and her bandwagoning partners in ignorance insist that this is an assumption and that the relevant information wasn't provided in the question. This is false.

 

[Brian the Snail]

Originally posted by Badly Shaved Monkey

"The concepts of accuracy and precision are both closely related and often confused. While the accuracy of a number x is given by the number of significant decimal (or other) digits to the right of the decimal point in x, the precision of x is the total number of significant decimal (or other) digits."

Well, this is very different. I don't understand the context in which this would be used at all unless it is a computer programming convention to handle floating point decimals according to rules that depend on the number of decimals used.

Consider that second paragraph.

I weigh a car and find it weighs 1.295 tonnes. The accuracy by Wolfram's definition "is given by the number of significant decimal (or other) digits to the right of the decimal point in x" i.e. 3 places of decimals. If I weigh it on another set of scales that report a weight of 1,294,864.4g. This has only 1 place to the right of the dp. Does that make it less accurate? In truth it neither makes it more nor less accurate because a single number cannot convey accuracy you need a measure of spread. But to use the Wolfram definition again, is 1.295 +/- 0.002 tonnes more accurate than 1,294,864.4 +/- 0.3g.

So I'm still left confused as to what the Wolfram definition, in its full form, is used for.

Yes, I was rather confused by the second paragraph as well. I would define accuracy and precision in much the same way as Hans does. So, for example, if I do a measurement my device might give an answer on the digital readout of 543243546, which is very precise, but if the error (or accuracy) is 10%, then anything past the second digit is meaningless. So I'm not sure what the Wolfram page is going on about really.

And none of this is particularly relevant to this thread, I must say.

 

[Badly Shaved Monkey]

It would be used in the context of chemistry or physics.

For example, if you were performing an equation to calculate the yield of a particular chemical equation, the answer you get cannot be more accurate than the least accurate measurement of the ingredients. If the measuring device you used to parcel out the ingredients is accurate to only one-tenth of a basic unit, the result cannot be given as a value accurate to more than one-tenth of that basic unit, even if the math gives you more digits than that.

Another example: if you multiplied a measured value with three significant decimal places (accuracy known to one part in a thousand) by a physical constant with eight significant decimal places, the resulting answer must be rounded off to three decimal places. Digits after that are not considered meaningful, as the uncertainty in the measured value is greater than they are.

Ah, that's not it. What I was trying to point out that the Wolfram definition depends on using the decimal place as an absolute reference point about which toassess 'accuracy'. The point of the example I gave was to show that the position of the d.p. is arbitrary, which means that a definition of accuracy the depends on the choice of position of the d.p. cannot be generalised so it is not a very useful definition. That's why it seems to be an odd way to define it.

Wait a minute;

"if you multiplied a measured value with three significant decimal places (accuracy known to one part in a thousand)"

Is the immplicit assumption that everyting has been reduced to scientific notation with only one digit to the left of the d.p?

Under that assumption it would be generalisable, so 1.346 x 10^2 tonnes has the same accuracy as 2.543 x 10^15 W or 1.654 x 10^1 N.

That, I think, is logically consistent.

 

[Brian the Snail]

Actually, just to add something, precision does have the meaning of "the number of significant figures" in computer programming. For example, single precision real numbers have 8 digits, while double precision numbers have 16. So I can understand that part anyway.

 

[Rolfe]

Again, this illustrates the difference between pure and applied (in this case medical) statistics. The Wolfram site gives very pure definitions, and doesn't discuss their use in context. In contrast, when what is required is a meaningful and standardised language to discuss and evaluate very particular contexts, things can look a lot different.

I'm not disputing the definition or arithmetic given for "precision" above, but in the particular context of a biochemistry or haematology assay, precision describes the repeatability of the results from the assay in question. It is assessed by calculating the coefficient of variation of a number of replicate measurements, CV = (SD/mean) x 100 (%). For routine assays we try for less than 5% CV, often we actually achieve around 2% in practice.

(Number of significant figures also comes in here though, because obviously the better the precision the more significant figures you can legitimately report to.)

Accuracy, the twin concept with precision, describes how well the results compare to the "true" result from a reference method. More difficult to pin down statistically, in fact some authors have suggested abandoning the numbers and just eyballing difference plots, but usually one calculates the correlation coefficient for a group of samples with a range of concentrations, generally hoping for better than 0.99 in a routine assay, with the line of best fit lying very close to the line of coincidence.

These two concepts are the gods in the biochemistry and haematology laboratories, as sensitivity and specificity are the gods in the serology laboratory with its binary assays (OK, forget titre results for now). We have to understand how to derive these parameters, and how to interpret them. We have to sit exams about it.

These concepts may be unfamiliar to those whose statistical background isn't in the precise area being covered. But they are nevertheless well defined and well studied, and are essential to allow professionals to have a meaningful conversation about assay performance.

Having an understanding of pure statistics doesn't necessarily equip one to understand specialist areas of applied statistics without at least some prior familiarisation with the methods in use and the conventions adopted. Which I'm now certain that Wrath will never understand.

Rolfe.

 

[Wrath of the Swarm]

Translation: "I'll talk about how complex and sophisticated my understanding of a limited and highly specific form of statistics is so that no one will pay attention to the fact that I don't understand the simple stuff."

Further translation: "The information wasn't given to me in the form I have been conditioned to expect, so obviously I'm the only one who understands statistics here."

 

[Wrath of the Swarm]

Having an understanding of pure statistics doesn't necessarily equip one to understand specialist areas of applied statistics without at least some prior familiarisation with the methods in use and the conventions adopted.

Um, it's actually the other way around - understanding a very limited adaptation of the field doesn't necessarily equip one to understand the field as a whole.

Not that this is relevant, because we're not discussing some esoteric point of advanced statistical analysis. This is the basic stuff, the stuff they teach in the first two weeks of introductory courses.

You are essentially complaining that I didn't give you the data in the form that you are accustomed to receiving it. Without numbers labeled "specificity" and "sensitivity" you're not able to "plug and chug".

 

[Dancing David]

There does seem to be an issue in the way people are 'accuratly" characterising the others in this thread.

Thank goodness the Ivory Tower has a popcorn machine....

 

[Rolfe]

I made a mistake. It happens sometimes when I read sources too quickly.

Lies are intentional attempts to deceive. This was just a random act of stupidity on my part. I shouldn't have said I found the particular question until I was sure I had.

Yes, Wrath, you made a mistake. The mistake wasn't just reading your source too quickly, it was not referring to your source when you formulated your question. Yes, I'm sure it was a random act of stupidity. You shouldn't have written the question until you'd checked that the wording really was identical to that used in whatever study you thought you were referencing.

The only "sample question" we've actually seen quoted in this thread from a published study was the 1978 one referenced by Steve74.

“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?”

"False positive rate". Nothing about "accuracy". Just like the other studies on the same subject, you'll find. Not one uses the term accuracy in the question.

Wrath didn't refer to the source study when he formulated his question. So he didn't notice that "accuracy" wasn't the term presented. Not fully understanding that his concept of "accuracy" was inapplicable to a binary assay in which sensitivity and specificity tend to vary inversely and will only be equivalent by very occasional coincidence, he loosely stated "accuracy" rather than the required value, specificity.

And people started to jump all over him.

It would have been quite easy to admit the mistake at the time, it wasn't a capital offence, and we could have moved on to the interesting discussion, which (as many people seem to realise) is the concept of predictive value, its variability with "prevalence", and most important of all, how to choose the most applicable figure for prevalence to get the true predictive value for an individual patient's result.

But Wrath won't admit it. He's resorted to declaring that since this silly "accuracy" concept is obviously quite meaningless unless sensitivity and specificity are equal, then he was being very clever in giving us even more information than we needed! By his choice of this term, he was telling us not only what we needed to know, that the specificity was 99%, but something we didn't give a damn about for the pupose of the calculation, that is that the sensitivity was also 99%.

Yeah, right.

What he was actually telling us was that in spite of the stated purpose of this thread being to allow Wrath to attack doctors' comprehension of medical statistics, Wrath himself has little familiarity with medical statistics. Otherwise he would have known perfectly well that the term "accuracy" is never used in this context, and why, and he would have understood that the figure he should supply was the specificity.

Wrath, people in glass houses shouldn't throw stones. Whether you realise it or not, you are currently standing out in the open surrounded by many small shards of broken glass. And it's starting to rain.

Rolfe.

 

[Hellbound]

Um, it's actually the other way around - understanding a very limited adaptation of the field doesn't necessarily equip one to understand the field as a whole.

Yes, and of course we should always use the general when discussing the specific? Next time I write a medical report, I'll use organism instead of patient, should be obvious what I'm talking about, right?

Not that this is relevant, because we're not discussing some esoteric point of advanced statistical analysis. This is the basic stuff, the stuff they teach in the first two weeks of introductory courses.

Yes, the basic stuff. Not the branch of statistics developed to specifically work with medical testing, with procedures and terminology designed to prevent confusion and allow for precise discussion. It's like using pliers instead of a wrench to loosen that nut.

You are essentially complaining that I didn't give you the data in the form that you are accustomed to receiving it. Without numbers labeled "specificity" and "sensitivity" you're not able to "plug and chug".

This phrase, "not able", I do not think it means what you think it means.

I'll try this slowly, in large type:

ROLFE DID GET THE RIGHT ANSWER, SO YOUR INSULT HERE IS NOT ONLY RUDE AND CHILDISH, BUT ALSO AN OUTRIGHT LIE WITH NO SUPPORTING EVIDENCE, AND COUNTER-EVIDENCE ON THE FIRST PAGE OF THE THREAD.

That better?

Okay. Now that that's clear.

She is complaining that you are providing a question targetted to a group of people who's training is in a specific branch of statistics, yet providing that data within a framework that they are not trained in. I do computer work. If someone came up and started asking me programming questions, but used terminology from information theory rather than computer programming, that's rather silly, isn't it? And, since you claim to know all about medical testing and the terms used in it, one can only think that you used different terms intentionally.

You're trying to test a mechanic on engineering, and it just doesn't work that way.

I have never in my life seen such an inflated ego, and for so little reason.

 

[Wrath of the Swarm]

Translation: "Yes, yes, that's it! The reason I insisted that not enough information was provided was because there was no source for the question! The question didn't have enough information to be answered because there was no source!"

No, I'm not testing a mechanic on engineering. Mechanics are not generally expected to know engineering.

What I'm doing is showing pictures of basic tools to the mechanic and asking "what is this?"

Remarkably, the mechanic somehow managed to identify the screwdriver as a screwdriver, even though the size of the head wasn't specified. Truly, the mechanic is a god among mere mortals for the astounding feat of intuitive leaps.

 

[Hellbound]

What I'm doing is showing pictures of basic tools to the mechanic and asking "what is this?"

Actually, what you're doing is showing the mechanic a wrench. To which, the mechanic replies "That's a crescent wrench", whereupon you proceed to claim the mechanic is an idiot because he obviously can't tell what a wrench is, since he gave you the specific answer instead of a general one. Even that's not a good analogy, but closer to what's happening. I'm sorry, but I just don't see you winning this one. You have yet to show any source whatsoever that presents medical test data using the term accuracy; the term is not used in medical test statistics. Because you used a different term, you're throwing a fit about it just so you don't have to say "My bad, I meant specificity."

You say a mechanic is not expected to know engineering; likewise, a doctor is not expected to know general statistics, only medical ones. Since the term accuracy in reference to tests is, essentially, menaingless (because almost no test ever has a specificity equal to its sensitivity), it is not a part of medical statistics int he sense you use it. You got the terminology wrong, plain and simple.

I'm done here. I get enough practice with these types of arguments with my 2 year old.

 

[Wrath of the Swarm]

There's no such thing as "medical statistics".

If I had insulted Rolfe for not knowing what 'alpha and beta' referred to, that would be a valid complaint.

But she complained that more specific figures weren't given when they weren't necessary, and insisted that insufficient information had been provided.

Lots of tests have equal alpha and beta values. Medicine tends to develop ones that don't because those tend to be more useful in medical contexts.

Circles are special cases of ellipses. If I had defined a circle according to a radius, and a geometer complained that I hadn't specified two axes and that he couldn't find the area of the shape without knowing the lengths of both axes, that would be the equivalent of what Rolfe did.

Rolfe was wrong. Just admit it and move on.

 

[Rolfe]

There's no such thing as "medical statistics".

Medical Statistics.

More Medical Statistics.

Even More Medical Statistics.

The goal of Medical Statistics is the application and development of statistical models and designs in medical research. It plays an active collaborative role in the design and analysis of clinical and basic research in the Leiden Medical School. The development of statistical models concerns extensions of established statistical tools like logistic regression, survival analysis and repeated measures models, as well as innovative research on validation of prognostic statistical model and models for complex data such as pedigree data, multi-state follow-up data and random effects models for repeated measurements.

Medical statistics has made important contributions to the field of statistical models for medical data....

Certainly, this discipline covers a great deal more than the statistics used to characterise assays in the clinical laboratory, but its existence as a discipline is not really possible to deny.

I could go on all night. But why bother.

Rolfe.

 

[Prester John]

There's no such thing as "medical statistics".

Guess i'd better go shut down the medical stats department then.


edited to add: Hey thats the book i use for stats Rolfe!

 

[Donks]

More Medical Statistics.

Rolfe. [/B]

Just out of curiosity, is that your book?

 

[pgwenthold]

There's no such thing as "medical statistics".

...

Rolfe was wrong. Just admit it and move on.

Wow. I don't think I have ever seen anyone's credibility so completely and utterly demolished as what has occured in the last three posts.