What's the difference between "cherry-picking" and "confirmation bias"?

[stevea]

A number of answers here would not have passed. Thankfully, none of the students said the difference was based on some conscious/unconscious fake dichotomy

Whether an experimenter recognizes and can verbalize his own bias (conscious bias) is certainly irrelevant to the outcome. I can't call this a false dichotomy so much as an irrelevant dichotomy. Personally I think JeffC has the main point; the OP's is a semantic question, so why create language constructs what can only lead to silly paradoxes, misunderstandings and to no other good ?

I think any definition of "confirmation bias" must include all selection bias regardless of some hypothetical psychological state. And hasn't the data been "cherry picked" even when the experimenter lacks self-recognition of bias ? It's a pointless and dubious distinction.

On a less pedantic, but related point, I once gave a talk to a group of beer judges (a serious hobby for me) in the issue of bias, trying hard to soften the blow with the phrase "unconscious bias". I was treated as tho' I had called them all a pack of liars & cheaters. One fellow in particular gave me a heated rebuttal on how the judges were above any bias, and how I was a jerk for suggesting otherwise.

Not only are biases real, common and often obvious, but there seems to be an extremely strong psychological feature that generally prevents people from recognizing or even considering their own biases.

 

[Wowbagger]

Well...

Confirmation bias is the tendency for people to confirm their own ideas, in their own ways, regardless of whether or not they can be effectively confirmed in a more independent manner.

Cherry-picking, on the other hand, is the act of picking cherries off of cherry trees.

Where did the confusion come from?

 

[GreyICE]

OK, it looks like there are no more takers, but all the triplets offered did fit the rule, which was "The triplets must be ascending numbers".
Wason's original triplet was "2,4,6" and approximately 80% of his subjects never generated exemplars that could falsify their hypotheses, like "6,4,2", or "1,2,223400012". This has been generally the case with subsequent replications.
I used 2,3,5 as the exemplar because I figured some people here would go the prime or fibonacci series route.
Wason called the avoidance of testing violations of the subject's hypothesis confirmation bias. And that surely is different from cherry picking, data mining or belief perseverance.

Nah, it's just a bitch to think of sigma series that ascend and descent and also pass through 3 ascending points as neat as 2, 3, and 5. It doesn't really prove anything, until you sit there and try to do one.

 

[drkitten]

OK, it looks like there are no more takers, but all the triplets offered did fit the rule, which was "The triplets must be ascending numbers".

I didn't get here in time, I'm afraid. I'm a right nasty person and would have asked about +1, 0, -1.

 

[drkitten]

As a matter of interest, how do you ensure, when filtering out 'irrelevant' data, that you don't lapse into 'cherry-picking'?

My rule of thumb is that you don't discard any data unless you have an active and articulable process error.

I.e. if you dropped the test tube on the bench and and had to scrape up half the sample, you don't need to analyze it or be surprised when the results are atypical. If there was a fire alarm in the middle of someone's test, then that test may not represent their actual performance. If you discover that your "male" guinea pig is pregnant, you may not want to take his -- her -- hormone levels seriously.

 

[Jeff Corey]

I didn't get here in time, I'm afraid. I'm a right nasty person and would have asked about +1, 0, -1.

And you would have been in the approximately 20% of the people who attempt to falsify their working hypothesis.

 

[drkitten]

And you would have been in the approximately 20% of the people who attempt to falsify their working hypothesis.

Actually, my working hypothesis was that the rule you were using was "always say yes."



... which is kind of hard to falsify.

 

[dudalb]

CHerry Picking is only picking evidence that will support your position regardless of the context the information is in.
Confirmation Bias is more or less ignoring evidence that conflicts with your position.
They are different, but they tend to go hand in hand.Where there is one, you almost always find the other.

 

[Jeff Corey]

CHerry Picking is only picking evidence that will support your position regardless of the context the information is in.
Confirmation Bias is more or less ignoring evidence that conflicts with your position.
They are different, but they tend to go hand in hand.Where there is one, you almost always find the other.

No. They occur at two different stages of the process of testing a hypothesis or statement.
Confirmation bias happens when one chooses to only look at the evidence that could affirm the statement or hypothesis. In the Wason card selection task, people mostly tend to choose the card that could confirm the statement and not the one that could falsify it.
Here is a quick version.

You have 2 cards in front of you, each has a number on one side and a letter on the other. Tell which card you should turn over to test the statement that, "All cards with a consonant on one side have a odd number on the other."
The cards show 1 and 2.

Cherry picking, data mining and file drawering all occur after the data have been collected.
Not before.

 

[AgeGap]

How so? This is the education thread, and should be relatively free of half baked, uninformed opinions.
The OP was quite clear and a good question. Last week I used a similar question in the Critical Thinking final exam. It was to define confirmation bias, the file drawer effect and to tell how they differ.
A number of answers here would not have passed. Thankfully, none of the students said the difference was based on some conscious/unconscious fake dichotomy.
This whole issue came up on another thread a while back and UncaJimmy said I was being overly pedantic.
So be it.

Sorry, that was a bit gun and run.
I mentioned the Stanford prison experiment because it is well known and it has been argued that the selection criteria for the experiment was flawed.
If Zimbardo cherry picked he would have asked for a-holes who would like to act out a sadistic prison fantasy. Volunteers were told they would take part in a prison simulation experiment.

 

[Plasmadog]

When deciding which term to use, I would suggest that you keep in mind one very simple distinction between the two; regardless of the precise definitions used, confirmation bias is something that someone might have, whereas cherry picking is something that someone might do.

 

[Jeff Corey]

Sorry, that was a bit gun and run.
I mentioned the Stanford prison experiment because it is well known and it has been argued that the selection criteria for the experiment was flawed.
If Zimbardo cherry picked he would have asked for a-holes who would like to act out a sadistic prison fantasy. Volunteers were told they would take part in a prison simulation experiment.

True, there were flaws, but cherry picking in this context refers to data, not subjects. That's why I wasn't sure what you meant.

 

[Jeff Corey]

When deciding which term to use, I would suggest that you keep in mind one very simple distinction between the two; regardless of the precise definitions used, confirmation bias is something that someone might have, whereas cherry picking is something that someone might do.

Confirmation bias is inferred (or labeled) based on what people do.

 

[BPScooter]

I've had an interesting situation which I'm sure isn't that uncommon when dealing with survey datasets--where there's an outlier that is so obviously wacky that my best guess is the person truly didn't understand the question or was just purposely being strange. It's never been that big a deal but my temptation was certainly to eliminate that data point. Most recently I just decided to leave everything in, report the standard deviation, mention the fact that there were some outliers that would affect data interpretation, and leave it at that. It's interesting and a bit humbling to me to realize how many opportunities someone has to "cook" data at every stage of the game. We really do rely a lot on the integrity if the researcher.

 

[Jeff Corey]

Yes, and that's why if your article is submitted from Whatsamtta U, rather than More Science U {Firesign Theatre reference} your good data may be ignored.

 

[Plasmadog]

Confirmation bias is inferred (or labeled) based on what people do.

Sure, but what difference does that make to the use of the terms? The label is given to the attribute, not the action that reveals it. Even if you disregard the subtle differences in meaning, the two terms are not truly interchangeable because they have different subjects; the attribute and the action.

Imagine if the original poster had instead inquired about the difference between "dishonesty" and "lying". I would make the same point. They are both things for which someone may be accused, but the words are clearly not interchangeable; sentence structure aside, accusing someone of general dishonesty is a very different matter than accusing them of a specific lie. Accusing someone of confirmation bias is similarly different from pointing out a specific instance of cherry picking. And in an actual argument there may be very good reasons to prefer a specific accusation over a general one, or vice versa.

 

[BPScooter]

That makes sense to me--that cherry-picking might be considered a specific means for exhibiting confirmation bias, and there are perhaps others. This is fun to think about.

Jeff C.--not to derail too far, what's the state of the art on outliers like I described? I should go back and brush up on my research methods stuff, but my gut feeling is that you really have to consider the data you have, all of it, and not get cute with it a posteriori for any reason. Particular example--I recently asked about events within the past year, and one respondent answered in a way that would be really impossible, leaving me to think they read the question in reference to their whole life, or something. It was WAY off, and of course affected the descriptive statistics for that item, not by much, but I had to think about what to do.

 

[BPScooter]

Stevea, you wrote "Whether an experimenter recognizes and can verbalize his own bias (conscious bias) is certainly irrelevant to the outcome"--and my reply is that it certainly influences whether or not I will read the rest of the paper!

 

[stevea]

Stevea, you wrote "Whether an experimenter recognizes and can verbalize his own bias (conscious bias) is certainly irrelevant to the outcome"--and my reply is that it certainly influences whether or not I will read the rest of the paper!

So you believe that "cherry picking" papers to read doesn't lead to a "confirmation bias" of your pet ideas & biases ? You are deluding yourself.

Published papers never announce which biases they represent; that's something you impute based on hunches. The only case I can imagine rejecting are the very few papers where data falsification is proven or later experimental method error is shown. In the latter case the data may still be considered within it's limitations.

For the very same reason that experimenters are not permitted to select/reject the data to publish, neither are you permitted to reject papers based on your personal hunches.

The only safeguard is to read everything and challenge both the agreeable and disagreeable conclusions equally.

 

[drkitten]

So you believe that "cherry picking" papers to read doesn't lead to a "confirmation bias" of your pet ideas & biases ?

How on earth do you get this out of what BPScooter wrote?

Published papers never announce which biases they represent; that's something you impute based on hunches.

This is absolutely untrue. Hell, whether or not a paper acknowledges potential biases is one of the criteria I use (and most reviewers use) when evaluating the paper for acceptance in the first place. The usual spot to put this is in the "background" section (usually section II) where the relevant literature is discussed, and any major theoretical disagreements are discussed. The other spot, of course, is when the actual experimental methods are presented, and the author either makes it clear or doesn't make it clear that his sample consists of sixteen ("only") male (!) undergraduate students (!!) from a local community college (!!!), as compared to the standard analysis of Ivy League undergrads (Flintstone & Rubble, 2011).

As to whether or not you read the paper -- well, your time is not infinite. A badly written paper contains so little actual information that you might as well not read it, and instead read a good one.