Polygraphs: The evidence

[CFLarsen]

OK, given all that, what do you mean by "reliable"? Anything better than chance? 100%? What?

I told you: It's a better way to say they are unreliable.

 

[Cardelitre]

No, not particularly.

So you can't think of any study showing that the polygraphs can't perform better than chance to detect liars in a sample population? That's too bad... I wonder how the skeptics that you mentioned where able to draw that conclusion without the necessary evidence then...

Bob Carroll. The owner of the site.

Oh, Bob is for Robert, ok. I never heard of the guy before, so I had a hard time guessing who you where talking about...

As you can see in the references, Bob refers to e.g. the NAS report.

Yes, and the NAS report shows that the polygraphs are apparently able to perform better than chance when discriminating liars in a sample population. It seems that these results show actually the contrary to what you claim and what I was asking about...

It isn't as if the polygraph is a fringe technology: It is widely used in many situations in the US, both government and the private sector.

If you say so. Still, it is not evidence against the possible efficiency of the polygraphs, merely a hint.

No, not clearly: Some studies show that homeopathy works - only they are small, and/or poorly designed and conducted.

Well, do we agree that the large majority of the studies on this subject show that homeopathy doesn't work better than placebo, and that it is the most common conclusion of the meta analysis on this subject?
So the question is: is there a study showing the same kind of negative results for polygraphs? I'm starting to think that there is none, or if there is, small and/or poorly designed. But I could be wrong; I didn't personally extensively review the literature about this...

 

[TV's Frank]

I told you: It's a better way to say they are unreliable.

I don't get it! If you say, "polygraph is unreliable", do you mean "polygraph performs no better than chance"? Or something else?

 

[CFLarsen]

So you can't think of any study showing that the polygraphs can't perform better than chance to detect liars in a sample population? That's too bad... I wonder how the skeptics that you mentioned where able to draw that conclusion without the necessary evidence then...

No, that's not what I said, and it's not what you asked me. You asked me if I could find one experiment that is more representative.

Oh, Bob is for Robert, ok. I never heard of the guy before, so I had a hard time guessing who you where talking about...

My bad, then. I thought you knew about Bob. Bob is an institution in skepticism.

Yes, and the NAS report shows that the polygraphs are apparently able to perform better than chance when discriminating liars in a sample population. It seems that these results show actually the contrary to what you claim and what I was asking about...

No, it doesn't. It says that in some situations under some conditions, they sometimes perform better than change. What we have is a "Charlie Brown" scenario: Wishy-washy.

If you say so. Still, it is not evidence against the possible efficiency of the polygraphs, merely a hint.

It isn't up to skeptics to prove that polygraphs don't work. It is up to proponents of the polygraph to prove that they work.

Well, do we agree that the large majority of the studies on this subject show that homeopathy doesn't work better than placebo, and that it is the most common conclusion of the meta analysis on this subject?
So the question is: is there a study showing the same kind of negative results for polygraphs? I'm starting to think that there is none, or if there is, small and/or poorly designed. But I could be wrong; I didn't personally extensively review the literature about this...

If you point to meta-analyses, why do you insist on one single study when it comes to polygraphs?

That is totally inconsistent of you.

I don't get it! If you say, "polygraph is unreliable", do you mean "polygraph performs no better than chance"? Or something else?

I mean that they are unreliable.

 

[TV's Frank]

I mean that they are unreliable.

Okay...but I want to know that you mean by "unreliable"! I want to make sure that when you say "unreliable" it means the same thing that it means to me. Let's play a game. I would like you to replace "polygraph is unreliable" with "polygraph is __________ ________ ______", where the blanks indicated a longer, more detailed description of your opinion of polygraph, specifically what you mean by "unreliable".

 

[David Wong]

I mean that they are unreliable.

You wouldn't tolerate this kind of evasion from anyone else. Why do you tolerate it from yourself?

You've been asked to clarify the statement. Wouldn't the rational, mature thing to do be to clarify it? Or state why it can't be clarified?

To snidely repeat the same statement, intentionally making no effort to advance the point or aid the other person's understanding of the issue, is not the sign of someone interested in a rational discussion. It's the sign of someone childishly trying to "win."

 

[Myshkin]

I'm confused. CFLarsen, do you dispute that a measurable phenomenon exists? Or simply that the phenomenon is too slight and variable across different conditions to be of any value?

http://fas.org/sgp/othergov/polygraph/ota/index.html
http://www.nap.edu/openbook.php?isbn=0309084369

The meta-analyses seem to demonstrate that, more often than would be expected by chance, the results of lab experiments deviate from what would be expected by chance. This is the meta-result, so to speak. Does that not suggest that there is a measurable phenomenon occurring? The experiments were far from perfect, but many appeared to have adequate control and blinding.

Whether the difference in lying and truthful populations can be used to address any specific question, especially across the myriad cases and conditions encountered in the real world...is a great question but not the one I'm asking here.

If there really is a net difference, statistically deviating from what would be expected by chance, it would indicate that the polygraph correctly categorize liars and truthers at a rate greater than simple guesswork, given a sufficiently large sample size. And that would be an interesting phenomenon.

 

[CFLarsen]

Okay...but I want to know that you mean by "unreliable"! I want to make sure that when you say "unreliable" it means the same thing that it means to me. Let's play a game. I would like you to replace "polygraph is unreliable" with "polygraph is __________ ________ ______", where the blanks indicated a longer, more detailed description of your opinion of polygraph, specifically what you mean by "unreliable".

No, let's not play games. Let's just acknowledge that I think that polygraphs are unreliable.

Do you think I have any reason to think otherwise?

You wouldn't tolerate this kind of evasion from anyone else. Why do you tolerate it from yourself?

You've been asked to clarify the statement. Wouldn't the rational, mature thing to do be to clarify it? Or state why it can't be clarified?

To snidely repeat the same statement, intentionally making no effort to advance the point or aid the other person's understanding of the issue, is not the sign of someone interested in a rational discussion. It's the sign of someone childishly trying to "win."

There's no evasion.

I think polygraphs are unreliable. I have extensively explained what my stance is wrt to polygraphs. The scientific evidence that they work is simply not compelling. Not only do major skeptics agree with this, the scientific community is also with me on this one.

That's not childish. At all.

 

[Jimbo07]

The reason why polygraphs reportedly "work" is not because you can tell from the readings that people lie. You can use polygraphs to intimidate those who don't know that it is bull.

That's a bit of an oversimplification, but at least it's related to what I was saying.

But that doesn't mean they should be used. We don't use pseudoscience, we shouldn't encourage it.

If your suspect was superstitious, would you threaten to cast a curse, if he didn't confess?

Do you mean to say that police do not lie and exaggerate in interviews? Or are you saying they should be discouraged from doing so?

 

[TV's Frank]

Personally, I'm not surprised that the "sifting for spies" doesn't perform as well as "catching the crook". In a police investigation, I image the questions are more detailed, direct, and emotional (i.e. "did you stab mrs. jameson in the neck?") . When sifting, you have to keep to generalities (i.e. "have you ever been contacted by an agent of a foreign government?") One might find it easier to lie about the latter case.

Mind you, this is all made up and probably wrong.

As a possibly related and slightly amusing anecdote, I was once the subject of a polygraph test by the NSA (it was for a job). For those of you wondering about observer bias, the questioner told me that the polygraph results were examined by three independent examiners, who were not present during the questioning. The questioner simply adminstered the test and recorded spurius results (such as a cough). A failure at the examination would not automatically disqualify you (they were away of the high false-positive rate), but simply "flagged" you for more detailed scrutiny.

Anyway, I came pretty close to passing out during the test. Sitting in a comfy chair, completely aware of every single heartbeat and breath, and answering inane questions (like "were you born in milwaukee" and "have you ever been a member of a group that advocated the violent overthrow of the united states government?"...they even told me what all the questions were before the test even began!) over and over and over for more an hour was enough to drive me slowly crazy. The questioner said that some folks start hallucinating halfway through it. Awesome.

Anybody else have fun polygraph stories?

Edit: Oh yeah, I passed

 

[CFLarsen]

I'm confused. CFLarsen, do you dispute that a measurable phenomenon exists? Or simply that the phenomenon is too slight and variable across different conditions to be of any value?

http://fas.org/sgp/othergov/polygraph/ota/index.html
http://www.nap.edu/openbook.php?isbn=0309084369

The meta-analyses seem to demonstrate that, more often than would be expected by chance, the results of lab experiments deviate from what would be expected by chance. This is the meta-result, so to speak. Does that not suggest that there is a measurable phenomenon occurring? The experiments were far from perfect, but many appeared to have adequate control and blinding.

Whether the difference in lying and truthful populations can be used to address any specific question, especially across the myriad cases and conditions encountered in the real world...is a great question but not the one I'm asking here.

If there really is a net difference, statistically deviating from what would be expected by chance, it would indicate that the polygraph correctly categorize liars and truthers at a rate greater than simple guesswork, given a sufficiently large sample size. And that would be an interesting phenomenon.

It sure would. And if the scientific evidence showed that polygraphs could tell us who lied and who told the truth, it would be a tremendous tool to fight any kind of deceit, be it when screening government employees, employees in private companies, possible spies or people suspected of crimes.

That's why it is so important for proponents of the polygraph to state their case in a forum of skeptics. If skeptics are not able to distinguish between pseudoscience and real science, who is?

Oddly enough, it seems to be very hard to get proponents of the polygraph to step up and defend their - purportedly scientifically supported - claim that the polygraph can tell if people lie or not.

Now....why is that?

 

[TV's Frank]

No, let's not play games. Let's just acknowledge that I think that polygraphs are unreliable.

Do you think I have any reason to think otherwise?



There's no evasion.

I think polygraphs are unreliable. I have extensively explained what my stance is wrt to polygraphs. The scientific evidence that they work is simply not compelling. Not only do major skeptics agree with this, the scientific community is also with me on this one.

That's not childish. At all.

Dude, I am aware of the fact that you think polygraphs are unreliable. I know what your stance is. Please describe what you mean by "unreliable". Do you mean "performs no better than chance"? Or something else? I really don't understand why you're fighting this so much. I might even agree with you, if you would just tell me! I just want to know, so we have a common ground! Please!

 

[Myshkin]

Originally Posted by CFLarsen
The reason why polygraphs reportedly "work" is not because you can tell from the readings that people lie. You can use polygraphs to intimidate those who don't know that it is bull.

But this isn't what was being measured in the many lab experiments, mostly with faux crimes albeit. They were looking at the polygraph, blind to the truth-state of the participant.

 

[CFLarsen]

Personally, I'm not surprised that the "sifting for spies" doesn't perform as well as "catching the crook". In a police investigation, I image the questions are more detailed, direct, and emotional (i.e. "did you stab mrs. jameson in the neck?") . When sifting, you have to keep to generalities (i.e. "have you ever been contacted by an agent of a foreign government?") One might find it easier to lie about the latter case.

Mind you, this is all made up and probably wrong.

As a possibly related and slightly amusing anecdote, I was once the subject of a polygraph test by the NSA (it was for a job). For those of you wondering about observer bias, the questioner told me that the polygraph results were examined by three independent examiners, who were not present during the questioning. The questioner simply adminstered the test and recorded spurius results (such as a cough). A failure at the examination would not automatically disqualify you (they were away of the high false-positive rate), but simply "flagged" you for more detailed scrutiny.

Anyway, I came pretty close to passing out during the test. Sitting in a comfy chair, completely aware of every single heartbeat and breath, and answering inane questions (like "were you born in milwaukee" and "have you ever been a member of a group that advocated the violent overthrow of the united states government?"...they even told me what all the questions were before the test even began!) over and over and over for more an hour was enough to drive me slowly crazy. The questioner said that some folks start hallucinating halfway through it. Awesome.

Anybody else have fun polygraph stories?

Edit: Oh yeah, I passed

You hunt for spies because the damage they can do is a hell of a lot more serious than the odd homicide. We are not talking about offing the odd enemy, we are talking about jeopardizing whole nations.

If anything, the screening for spies would be much more tough than any other scenario I could think of.

That's a bit of an oversimplification, but at least it's related to what I was saying.

Unfortunately, it isn't an oversimplification. Polygraphs are solely methods of intimidation, to be used on the uninitiated.

Let's say that polygraphs were generally applied around the world. Those who did their homework could pass tests more easily. The result would be that the less knowledgable about cheating would not get the job.

Using the polygraph in general would mean that those who were able to cheat it, would get the most trusted positions.

Thereby totally undermining the purpose of the polygraph.

Do you mean to say that police do not lie and exaggerate in interviews? Or are you saying they should be discouraged from doing so?

They cannot lie and exaggerate to impede justice.

If your suspect was superstitious, would you threaten to cast a curse, if he didn't confess?

 

[Rodney]

Any of the time? Be serious.

We know it's been correct at least three times, because it correctly identified Kari as telling the truth, and Tore and Grant as lying.

More importantly, it identified Harold Nicholson, a CIA employee later convicted of spying for Russia, as lying. See http://www.internationalskeptics.com/forums/showthread.php?postid=2728166#post2728166

 

[CFLarsen]

Dude, I am aware of the fact that you think polygraphs are unreliable. I know what your stance is. Please describe what you mean by "unreliable". Do you mean "performs no better than chance"? Or something else? I really don't understand why you're fighting this so much. I might even agree with you, if you would just tell me! I just want to know, so we have a common ground! Please!

I'm not fighting at all. What is it about the word "unreliable" you don't understand?

But this isn't what was being measured in the many lab experiments, mostly with faux crimes albeit. They were looking at the polygraph, blind to the truth-state of the participant.

That's the point: Why would you be truly afraid to lie about a faux crime?

Think about it: If you want to argue that you can be truly afraid to lie about a faux crime, then you also admit that the sentiments recorded by the polygraph aren't real.

 

[TV's Frank]

You hunt for spies because the damage they can do is a hell of a lot more serious than the odd homicide. We are not talking about offing the odd enemy, we are talking about jeopardizing whole nations.

If anything, the screening for spies would be much more tough than any other scenario I could think of.

While only an anecdote, my personal experience related above suggests otherwise.

 

[Myshkin]

That's why it is so important for proponents of the polygraph to state their case in a forum of skeptics. If skeptics are not able to distinguish between pseudoscience and real science, who is?

Oddly enough, it seems to be very hard to get proponents of the polygraph to step up and defend their - purportedly scientifically supported - claim that the polygraph can tell if people lie or not.

Whether a polygraph "can tell if people lie or not" is the larger question. The simpler question, and maybe the best starting point to separate science from pseudoscience is "are the populations of polygraph metrics from lying and truthful respondents two statistically distinct populations (i.e. with differing mean and/or variance in metrics), or are they one?" What do you say?

 

[TV's Frank]

I'm not fighting at all. What is it about the word "unreliable" you don't understand?

I know the dictionary definition of the word "unreliable". But say a claimed psychic were to come to you and say, "I can reliably predict the future". You might say, "What do you mean? Do you mean 70% accurate? 20%? Better than chance? Make a specific claim so we can test it!" Are you saying that polygraph performs no better than chance? Yes? No? Something else?

 

[Cardelitre]

No, that's not what I said, and it's not what you asked me. You asked me if I could find one experiment that is more representative.

Very well. Then, maybe you could point me to any study of this type, even one that you don't find particularly representative?

My bad, then. I thought you knew about Bob. Bob is an institution in skepticism.

In the US maybe, but the US is not the rest of the world.

No, it doesn't. It says that in some situations under some conditions, they sometimes perform better than change. What we have is a "Charlie Brown" scenario: Wishy-washy.

Strangely, that’s not what I could read in this same report. I should quote here this post of drkitten that apparently you forgot to address:

Theory and basic research give no clear guidance about whether laboratory conditions underestimate or overestimate the accuracy that can be expected in realistic settings.

Available data are inadequate to test these hypotheses. [...]

Evidence from Medical Diagnostic Testing. Substantial experience with clinical diagnostic and screening tests suggests that laboratory models, as well as observational field studies of the type found in the polygraph literature, are likely to overstate true polygraph accuracy. Much information has been obtained by comparing observed accuracy when clinical medical tests are evaluated during development with subsequent accuracy when they become accepted and are widely applied in the field. An important lesson is that medical tests seldom perform as well in general field use as their performance in initial evaluations seems to promise (Ransohoff and Feinstein, 1978; Nierenberg and Feinstein, 1988; Reid, Lachs, and Feinstein, 1995; Fletcher, Fletcher, and Wagner, 1996; Lijmer et al., 1999).

The reasons for the falloff from laboratory and field research settings to performance in general field use are fairly well understood.

In other words, the problem is with the specific numeric claims of accuracy, not with the "better than chance" and is to be expected in any transition from laboratory to field.

As well as:

In view of the above issues, we believe that the range of accuracy indexes (A) estimated from the scientifically acceptable laboratory and field studies, with a midrange between 0.81 and 0.91, most likely over-states true polygraph accuracy in field settings involving specific-incident investigations.

Thus, what is specifically overstated are the estimates (from 0.81 to 0.91), not the simple better-than-chance accuracy.

Or also:

The available evidence indicates that in the context of specific-incident investigation and with inexperienced examinees untrained in countermeasures, polygraph tests as currently used have value in distinguishing truthful from deceptive individuals. However, they are far from perfect in that context, and important unanswered questions remain about polygraph accuracy in other important contexts. No alternative techniques are available that perform better, though some show promise for the long term. The limited evidence on screening polygraphs suggests that their accuracy in field use is likely to be somewhat lower than that of specific-incident polygraphs.

There is no suggestion that this "value" is in any way overstated, just the standard note that screening is harder than incident investigation.

[...]

Finally, on p. 214, they present the ultimate statement of their finding on accuracy(emphasis in original).

Notwithstanding the limitations of the quality of the empirical research and the limited ability to generalize to real-world settings, we conclude that in populations of examinees such as those represented in the polygraph research literature, untrained in countermeasures, specific-incident polygraph tests for event-specific investigations can discriminate lying from truth telling at rates well above chance, though well below perfection.

Given that, at this point, they have formally recognized that polygraphs work in lab conditions something like eight times in the report, I see no reason for you -- or for anyone -- to dispute the basic fact that they work (in lab conditions).

This looks to me very far from the summary you do that "in some situations under some conditions, they sometimes perform better than chance."

It isn't up to skeptics to prove that polygraphs don't work. It is up to proponents of the polygraph to prove that they work.

Yes, you are perfectly right. And it seems clear to me that the NAS report did just that. Now some skeptics make the claim that this is in fact pseudoscience, but still don't see any study (or meta-analyse) disproving those previous results. Why?

If you point to meta-analyses, why do you insist on one single study when it comes to polygraphs?

That is totally inconsistent of you.

I'm truly sorry if I looked inconsistent to you, I really try to be as clear as I can. So let me please rephrase: could you present a single study or multiple studies and/or meta-analyses that show that polygraphs are unable to detect liars in a sample population better than chance? Any of those will do, for the sake of discussion...