Poll: Accuracy of Test Interpretation

Wrath of the Swarm said:
Translation: "Yes, yes, that's it! The reason I insisted that not enough information was provided was because there was no source for the question! The question didn't have enough information to be answered because there was no source!"
Click to expand...
Translation: "Yes, that's it! The reason I said accuracy when I meant precision is that I wanted to give these poor dim medical types even more information than they needed to solve the question!"

Wrath, nobody in this discussion had the slightest problem working out your trivial little sum. Your information was sloppily provided, but it was Department of We Know What He Meant.

News flash. I have no difficulty at all with your problem. I didn't even have any difficulty figuring out what you meant to say, even though you didn't say it. I can make that problem sit up and beg, jump through hoops and lie down and die for England. I can make it do so much more than you even imagine it can do, thanks to the wiggle-room left by the other sloppy formulation of the wording.

What I can't do is figure out for a moment why anyone in his right mind, when setting a question specifically aimed at medical professionals, and which was based on a template (nay, "identically worded" to the original study example to start with!) which is widely used in medical education, should deliberately decide to eschew the accepted defined terms of the discipline, adhered to by all other examples of the question we've seen, and instead introduce a term never before encountered in this context.
Wrath of the Swarm said:
"The information wasn't given to me in the form I have been conditioned to expect, ....
Click to expand...
Wrath of the Swarm said:
You are essentially complaining that I didn't give you the data in the form that you are accustomed to receiving it.
Click to expand...
Why not?

Was it to make it easier? I don't think so. Was it to make it harder? It certainly succeeded in muddying the issue, but again I don't think so. Was it a pathetic attempt to look clever by showing that he knew words not used by the other authors? Probably not.

Anybody at all think it was deliberate? Anybody at all think Wrath sat there, composing his OP, saying to himself, well the question only requires that I state the precision, but I'll make it more interesting and call it accuracy instead, so if they're on the same wavelength as Wrath the Genius, they'll know that not only is the specificity 99%, but also that the sensitivity (which they don't need to know, but never mind, I'm a generous fellow) is also 99%?

And if this was indeed the case, anybody at all think it's either clever or equitable to ignore the terms that would be instantly recognised and understood by the very audience the question was aimed at, and substitute a term which at best was a conundrum in itself?

It certainly wasn't a good way to get the discussion moving in a constructive direction, as I think we can all see.

Wrath, you know perfectly well that you didn't even stop to think what you were saying when you typed that question. None of these unconvincing intellectual back-flips even entered your head. You simply typed the word you were familiar with, rather than the more exact term required, because you hadn't thought it through that assuming equal sensitivity and specificity wasn't something anyone would ever do in this context.

And now you're trying to pretend you meant it all along. Wrath, you're making a prat of yourself.

Rolfe.
 
Donks said:
Just out of curiosity, is that your book?
Click to expand...
No! I just Googled on "medical statistics", and clicked on the first couple of the 69,300 links that came up. About this subject that doesn't exist

I had no idea Prester John was familiar with the book, I don't know it at all.

Edited to add: It's the same publisher as my book though.

(I'm not a medical statistician, it's just that the discipline of medical laboratory science requires a particularly in-depth understanding of a small area of medical statistics, that applicable to the characterisation of diagnostic tests. Which, sadly for Wrath, is exactly the area he chose for his attempt to demonstrate how stupid medical professionals are.)

Rolfe.
 
Rolfe said:

News flash. I have no difficulty at all with your problem. I didn't even have any difficulty figuring out what you meant to say, even though you didn't say it. I can make that problem sit up and beg, jump through hoops and lie down and die for England. I can make it do so much more than you even imagine it can do, thanks to the wiggle-room left by the other sloppy formulation of the wording.
Click to expand...

Really? I'd be interested to see what else you can do with it. I've always had a sneaky admiration for the way a master statistician can make numbers confess by threatening to torture their assumptions-in-law.

(This isn't a challenge or an attack, just a genuine expression of interest if there's anything really twisted you can do with the problem as stated. In the same vein, I might appreciate Blackbeard's cutlery proficiency.)
 
drkitten said:
Really? I'd be interested to see what else you can do with it.
Click to expand...
That's what I wanted to do from the beginning, and started to do on the first page. Unfortunately Wrath's inability to fess up to a minor error in terminology has had the entire discussion on hold for about ten pages while he looks for some sort of fig-leaf to cover his exposed ego.

We may need a new thread. I certainly find it an interesting topic. Especially that in the end it goes full circle and demonstrates that the intuitive assumptions medical personnel make when you don't mess with their minds aren't so far off base, but that when half-assed statisticians start using questions like Wrath's (even if they are better-formulated) to "educate" them, there is a serious danger of substituting reasonable instinctive assumptions with rote-learned counterintuitive reasoning which isn't valid beyond the very limited confines of the original question.

Yes, that's more interesting than getting Wrath to admit he made a simple mistake, but sometimes you have to cross the railway line before you can get to the meadow.

Rolfe.
 
But I did say it, and every time you've claimed I didn't is an outright lie. The value I gave you determines both alpha and beta values. No more information was needed.

Think otherwise? Go ahead - show us how you can twist the supplied information.
 
WotS does have a valid point, its just a pity he made it using poor choice of terminology for his question.

Most doctors would get confused and give the wrong answer. Now in a real-life situation, they might well not make the same error, because there are other factors to consider, and errors may become quite obvious.

But some doctors would muss up.
 
It wasn't even a poor choice of terminology. It was a perfectly accurate and straightforward choice of terminology that wasn't what Rolfe had come to expect.

And since we've established that my question really wasn't the same as the one used in the research I found citations for, you still haven't explained why the doctors did so poorly when the question was posed differently.

Well, Rolfe? Are you going to demonstrate how you can twist the statistics without contradicting the facts I'd established?

I established that the test has an accuracy of 99% - defined independently of a sample population - and gave the population incidence.

Go on. Twist them.
 
Deetee said:
WotS does have a valid point, its just a pity he made it using poor choice of terminology for his question.

Most doctors would get confused and give the wrong answer. Now in a real-life situation, they might well not make the same error, because there are other factors to consider, and errors may become quite obvious.

But some doctors would muss up.
Click to expand...

Hey we know some doctors muss up, because they did muss up (if we believe WotS).

But why? Because they are incompetent? Or because they are making different assumptions about the question as a result of their real-life experience?

Rolfe has shown that real doctors will not necessarily share the assumptions of the question writer, and as such we do not know how much of the mussing is due to doctors lacking cognative skills and how much is due to doctors who apply their experience bias into the assumptions behind the question.
 
Most of it seems to be a lack of cognitive skills. When the question is totally rephrased so that the frequencies of all conditions are given, doctors get it right only about 45% of the time.

I think that was mentioned in one of my linked sources... although now I'm not certain I know what they say myself.

Ah, here we are:

And finally, here's the problem on which doctors fare best of all, with 46% - nearly half - arriving at the correct answer:

100 out of 10,000 women at age forty who participate in routine screening have breast cancer. 80 of every 100 women with breast cancer will get a positive mammography. 950 out of 9,900 women without breast cancer will also get a positive mammography. If 10,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?
Click to expand...
Studies of clinical reasoning show that most doctors carry out the mental operation of replacing the original 1% probability with the 80% probability that a woman with cancer would get a positive mammography. Similarly, on the pearl-egg problem, most respondents unfamiliar with Bayesian reasoning would probably respond that the probability a blue egg contains a pearl is 30%, or perhaps 20% (the 30% chance of a true positive minus the 10% chance of a false positive). Even if this mental operation seems like a good idea at the time, it makes no sense in terms of the question asked. It's like the experiment in which you ask a second-grader: "If eighteen people get on a bus, and then seven more people get on the bus, how old is the bus driver?" Many second-graders will respond: "Twenty-five." They understand when they're being prompted to carry out a particular mental procedure, but they haven't quite connected the procedure to reality. Similarly, to find the probability that a woman with a positive mammography has breast cancer, it makes no sense whatsoever to replace the original probability that the woman has cancer with the probability that a woman with breast cancer gets a positive mammography. Neither can you subtract the probability of a false positive from the probability of the true positive. These operations are as wildly irrelevant as adding the number of people on the bus to find the age of the bus driver.
Click to expand...
 
Wrath of the Swarm said:
It wasn't even a poor choice of terminology. It was a perfectly accurate and straightforward choice of terminology....

I established that the test has an accuracy of 99% - defined independently of a sample population - and gave the population incidence.

Go on. Twist them.
Click to expand...
Wrath of the Swarm said:
But I didn't mean "specificity", I meant accuracy. It's a more powerful concept with strict standards of application.
Click to expand...
Prester John said:
Sensitivity - proportion of individuals with the disease who are correctly identified by the test.

Specificity - proportion of individuals without the disease who are correctly identified by the test.

Accuracy is not a term used for describing diagnostic tests.
Click to expand...
You know, eleven pages, and Prester John nailed it in three lines on page five.

Since "accuracy" is not a term used to describe binary diagnostic tests, one has to assume what-the-hell Wrath means by "accuracy".

This involves assuming that he must have given us the information required to solve the problem. If one assumes that, it's easy. One needs a figure for specificity, one has been given a stray figure of 99%, therefore use that. As there's no way to derive anything other than 99% from the figure of 99%. For the rest of the sums, see Wrath's repeated rationalisations.

However, if we do not assume that we have been given all the information we need, and given that it's obvious Wrath was operating by the seat of his pants and on a poorly-recalled memory of another question while he was formulating his, this is not an unreasonable assumption, we can really do anything we like with it.

Geni went there already.

Rolfe.
 
100 out of 10,000 women at age forty who participate in routine screening have breast cancer. 80 of every 100 women with breast cancer will get a positive mammography. 950 out of 9,900 women without breast cancer will also get a positive mammography. If 10,000 women in this age group undergo a routine screening, about what fraction of women with positive mammographies will actually have breast cancer?
Click to expand...
Wrath - now if you'd posted that question as your OP and then (after the vote) expressed your concern that less than half of the doctors got it right we could have had the debate I assume you wanted.

PS does "alpha"="beta" in the above???? :p
 
Wrath of the Swarm said:
Most of it seems to be a lack of cognitive skills.
Click to expand...
Looks as if it takes one to know one.

One little tiny nudge from DeeTee, and Wrath is off where he wanted to go. Posting all the studies he's been saving up about how stupid doctors are. Which was the whole point of him starting the thread.

Maybe Wrath should take the beam of "lack of cognitive skills" out of his own eye before starting on the mote he presumes is present in those of the medical profession.

Rolfe.
 
Prester John does not seem to have posted on page five.

Accuracy most certainly is a term used to describe binary tests. It seems that your exhaustive educational expertise just didn't cover that part - perhaps you skipped a chapter? Or twelve?

No, Dragon. Alpha (chance of incorrectly assigning positive to a negative) is about .096, while beta (chance of incorrectly assigning negative to a positive) is about .200.
 
Dragon said:
Wrath - now if you'd posted that question as your OP and then (after the vote) expressed your concern that less than half of the doctors got it right we could have had the debate I assume you wanted.

PS does "alpha"="beta" in the above???? :p
Click to expand...
Exactly.

[Probably erroneous inference deleted - I read the question too quickly.]

Rolfe.
 
Since the majority of people who take the test fail it, no matter what group they're in (except possibly mathematicians and cognitive psychologists), the plank seems to be in everyone's eye.

Here, Rolfe. Maybe I can help you with that plank once I remove this tiny, tiny splinter in my own.

On the other hand, maybe it's not so small after all, because it seems to have prevented me from seeing where you posted your twisting of the statistics.

C'mon, Rolfe. I wanna see them beg for God and Country like you promised.

[edit] Oh, and it's not one value that's needed. If the test can't be described with an accuracy score, we need to know both how many negatives score positive and how many positives score negative.
 
pgwenthold said:
Wow. I don't think I have ever seen anyone's credibility so completely and utterly demolished as what has occured in the last three posts.
Click to expand...
steve74 said:
You are, of course, making the assumption that Wrath had some credibility to demolish.
Click to expand...
Ah, how quickly he forgets.

Rolfe.
 
The statistics used in medicine isn't somehow magically different from normal statistics. It's all the same math.

So Rolfe's claims that understanding statistics doesn't prepare a person to understand medical statistics are garbage.

Where's that twisting, Rolfe?
 
You must log in or register to reply here.

Back
Top Bottom