Unskilled and Unaware of it - Flawed Conclusion?

[vIQleS]

http://www.apa.org/journals/features/psp7761121.pdf

I read this a while ago and I thought the conclusion was a bit strange, and I keep meaning to ask someone about it...

The conclusion:

Stupid people think they're smarter than they are, and clever people are closer at guessing how clever they are, but sometimes under-estimate a little bit...

OK - I'm paraphrasing, but I'm pretty sure that's what they were saying.

Just from reading the document and looking at the graphs, I've come up with another hypothesis:

Everyone tends to put themselves at about 80%. Regardless of how clever they are...

Why is this wrong?

 

[drkitten]

Just from reading the document and looking at the graphs, I've come up with another hypothesis:

Everyone tends to put themselves at about 80%. Regardless of how clever they are...

Why is this wrong?

Well, if you're at 85% and you guess yourself at 80%, you've made a good guess. If you're at 50% and you guess yourself at 80%, you've made a pretty poor guess. If you're at 15% and you guess yourself at 80%, then you've made a terrible guess.

If the quality of your guesses goes down with your performance, then that seems to be a pretty damning statement in and of itself, yes?

You're also missing three quarters of the paper. Yes, poor performers inflate their self-judgement more than good performers, and there might be some ceiling effect there.

But poor performers also:

1. suffer from deficient metacognitive skills, in that they will be less able than their more competent peers to recognize competence when they see it—be it their own or anyone else's.
2. are less able than their more competent peers to gain insight into their true level of performance by means of social comparison information.
3. can gain insight about their shortcomings, but only by making them more competent

 

[Uncayimmy]

I just printed this out and read it two days ago. You could say that, but it's not true. None of the data supports that assertion.

Also, it has nothing to do with being clever or smart. It has to do with incompetence in a particular area. We're all incompentent about many things. The problem is that our incompetence makes it difficult to recognize our incompentence. We *think* we know more than we do. The less we know, the greater the difference between reality and our self-evaluation.

So, a 20th percentile person puts himself at 52. A 50th percentile person might figure he's at 58. A 75th percentile person might be pretty accurate in self-assessment. And a 90th percentile person may only put herself (had to throw a "her" in there) at 85. Those are rough numbers, but that's what the study showed.

 

[jasonpatterson]

Theramintrees (brother of Qualiasoup and primary author of The Instruction Manual for Life) made a video concerning this study that says much the same you do. It's worth a watch, as is the Instruction Manual video. (I won't pretend that anything I'm writing here is anything other than his analysis paraphrased.)

His main issue with this is that those people who are unskilled are both unable to perform a task and unable to realize that they are unskilled. They believe that they are skilled and thus act accordingly. It appeared that this crossed most of what we might think of as intelligence. How many professionals are merely mediocre at what they do yet believe that they are more competent than average? That false assessment might tend to cause people to relax any drive toward further development of their skills, etc. (Heck, how many absolutely terrible drivers do you know who believe they are great at driving? I teach high school and the students almost uniformly believe that they are good drivers the instant they get their licenses.)

I don't know whether or not a manual skill would be self-assessed equally, but you can imagine the problems that are created when someone believes that he or she is a good handyman but actually lacks any mechanical aptitude AND the ability to recognize that lack. I've known a couple of people like this.

I suppose this study also made me consider those areas of my life in which I believe myself to be competent but might not be more objectively.

The Illusion of Superiority


The Instruction Manual for Life

 

[Cavemonster]

Well, the humor graph looks like that, but logical problem solving, past a certain point varies with the graph and correlates very well.

In addition, after being exposed to other people's answers, the highest scorers got very close to assessing their ability, but the lowest scores were still quite far away.

 

[vIQleS]

OK - it was a while ago, so I'm probably not remembering a lot...

I'll go back and have another read...

Thanks for the comments.

 

[BenBurch]

I know I'm a dunce.

 

[Uncayimmy]

But poor performers also:
<snip>
* can gain insight about their shortcomings, but only by making them more competent

That's really one of the more interesting parts of the study. The poor performers tended to overestimate not only their relative performance but their absolute performance. The high performers tended to understimate their relative performance but not their absolute performance.

For example, in Study 3:

Bottom Quartile (average 10th percentile)
Percentile Estimated: 67th
Test Score Estimated: 61st (12.9 points estimated, 9.2 actual)

Top Quartile (average 89th percentile)
Percentile Estimated: 72nd
Test Score Estimated: 70th (16.9 points estimated, 16.4 actual)

In other words the poor performers thought they scored much better than they did and thought they were a little smarter than their peers. In reality they significantly overestimated how many they got right.

The high performers pretty much knew how many they got right. They just figured that everybody else did better than they really did.

Thus the high performers knew their limitations while the low performers did not. This explains why very few physicists say, "Hey, y'all, watch this!" right before they die.

 

[Wally]

The short answer:

Stupid people don't know they're stupid.

 

[EHocking]

Well, the humor graph looks like that, but logical problem solving, past a certain point varies with the graph and correlates very well...

Can we really trust these researchers' competence?

Their questionnaire goes to 11.

(it's one smarter...)

 

[Uncayimmy]

The short answer:

Stupid people don't know they're stupid.

Maybe you're just being flip, but that's not the point. It's not a matter of intelligence. It's a matter of skill. When we lack a skill set, this incompetence is a roadblock to recognizing our own incompetence. The more we know about something, the more likely we are to have an accurate self-assessment of what we don't know. Since we all have weak areas, we all suffer from this problem.

 

[EHocking]

Maybe you're just being flip, but that's not the point. It's not a matter of intelligence. It's a matter of skill. When we lack a skill set, this incompetence is a roadblock to recognizing our own incompetence. The more we know about something, the more likely we are to have an accurate self-assessment of what we don't know. Since we all have weak areas, we all suffer from this problem.

I think Rumsfield had a handle on this, didn't he?

"...because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know."

 

[BenBurch]

I think Rumsfield had a handle on this, didn't he?

"...because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know."

It was popular in the left to rip this one, but it made perfect sense to me except that it was really unresponsive to the question asked!

 

[SpuriusMaximus]

Personally I'd say it kind of mirrors an observation I made long ago about myself.

As you might have noticed, I post on a lot of topics, and I read about a lot of topics. So I'm sorta noticing a lot of self-estimation curves, so to speak.

Invariably at the start of a domain I have the impression that I'm learning very quickly, and verily soon I'll be teh greatest expert ever Then a couple of years later, I start gradually noticing all the gaps I still have, or all the stuff I overestimated myself about.

It started with physics. As I mentioned somewhere else, at some point as a wee lad I asked something like "why is the sky blue", and my parents gave me a physics book. I found it fascinating. But very soon I was thinking I'm knowing a lot more than I did. Mind you, I still was way ahead of school, but verily it didn't even occur to me that there's, for example, the whole of relativity and quantum mechanics still ahead. I didn't know they even existed, so I thought I know almost all of it.

_Now_ when I look at it, I see all the gaps I still have, and, well, _now_ I rate myself lower than I did back then.

Programming, well, that was funny. By the time I could write a 100 line assembly program I thought I'm the biggest guru ever. To cut the long story short, it was only after university that I gradually started to look back and notice basically, "by jove, was I naive." There were techniques and whole languages I had never even heard of. Now I can see how much I still don't know. Back then I had not the faintest clue that there's more to it than Sinclair BASIC and Z80 Assembly.

It's as if, if I were to plot a curve with X = time, and Y = how much I think I know, it kinda looks actually like a Gauss curve. There's invariably that peak when you think you know almost everything, followed by starting to gradually notice how much more there is to know.

So I can very easily believe their conclusion, basically.

 

[Skeptic Ginger]

Regardless of the study's use of estimation of one's skills as a measurement, there are so many people who are ignorant of science who believe they understand something when they don't. It has to be half the skeptics with the CTer's being the other half.

How many times have you wrung your hands over a debate with someone whose background knowledge in the science under discussion was source of the problem. Where do you begin with such a person? And what's worse is they are totally unaware that they don't understand the science they are dismissing.

For example the Creationists or YECs who are certain that radioisotope dating is unreliable don't really know much about radioisotope dating, but they heard from someone else it was unreliable and decided it was a fact. They typically have no clue how ignorant they are about radioisotope dating.

Those of us who have a little knowledge within a lot of scientific fields, and perhaps a lot of knowledge about one of those fields typically recognize our shortcomings.

 

[Beerina]

Well, if you're at 85% and you guess yourself at 80%, you've made a good guess. If you're at 50% and you guess yourself at 80%, you've made a pretty poor guess. If you're at 15% and you guess yourself at 80%, then you've made a terrible guess.

If the quality of your guesses goes down with your performance, then that seems to be a pretty damning statement in and of itself, yes?

​