In September 2002 I offered to look into a question raised about John Lott’s work. I thought that offering such help to Lott and to the profession was the responsible thing to do when serious questions were raised, and I thought it would be exceedingly simple to establish that a survey of 2,424 people had been done. While I recognized that it is extremely easy to lose data in a computer crash, I had not anticipated that Lott would claim to have done a large national survey without discussing the sampling design with anyone, leaving any financial or other records of the study, or remembering anyone who had worked on it. I had never heard of a professor doing anything of that size with no funding, paid support, paid staff, phone reimbursements, or records (though there are probably precedents for such an unusual method). As it stands now, unless someone comes forward to verify working on the study--as I still hope occurs--we may never know with any certainty whether the 1997 study was done. Although I strongly favor emailing 1997-98 University of Chicago college graduates to see if any remember any classmates working on the study, John Lott now raises serious questions about how complete the University’s alumni records are, rendering that approach a less reliable route to an answer than I had anticipated....
...James Lindgren, Professor of Law, Northwestern University...
...Lambert wrote: "it seems increasingly likely that the survey is fictional." He was following up on problems first identified by Otis Dudley Duncan and discussed publicly in the Jan./Feb. 2000 issue of The Criminologist. These extremely serious charges concerned a study referred to in the second edition of Lott’s More Guns, Less Crime, but never actually published. Over the years, however, Lott has referred to this 98% number over four dozen times in publicly available sources....
[evidence detailed]
...That means that the source that Lott gave for the 98% figure has shifted over time:
1. In the 1998 edition of More Guns, Less Crime, John Lott attributes the 98% figure to “national surveys.”
2. Elsewhere in 1997 and 1998, Lott appears to attribute the 98% figure to “such polls” as the “Los Angeles Times, Gallup and Peter Hart Research Associates.”
3. In a May 13, 1999 letter and in 1999 revisions to the 2000 edition of his book, Lott attributes the 98% number instead to his own 1997 study, saying in the letter that the 98% figure is not based on Kleck.
4. In almost identical February 9, 2000 and March 1, 2000 articles for the Independence Institute, Lott switches to attributing the 98% figure, not to his own study, but to Gary Kleck’s (which does not support this figure).
5. In the Criminologist (Sept./Oct. 2000), Lott switches back to claiming that the 98% figure came from Lott’s own 1997 study, not from Kleck, which is where things stand as of this report.
Note that by using the plural “national surveys” and “such studies” Lott is stating that there are more than just one study showing the 98% figure, yet he now insists that the 98% figure came from his own study, not Kleck’s (and Kleck’s study does not support the 98% figure). Indeed, as discussed in the next section, because the 98% figure is supposed to be based on his own study, not those done by others, Lott says that his critics will have “nailed” him if they find that he began talking about the 98% figure before he says he did his study in 1997. Yet, if Lott really based the 98% figure on his own study alone, it seems strange that he would attribute the 98% figure to such plural entities as “national surveys” or “such studies”--until Duncan challenged him in 1999 and Lott revealed for the first time that the 98% figure was based on a study he did himself. ...
... In the Criminologist, Lott also wrote: “My survey was conducted over 3 months during 1997.” Lott called me on the telephone and repeated that he had conducted the study over several months during 1997. If he spent 3 months doing it in 1997, as he claims, the earliest that he could have completed it would be early April. Further, in an email to me Lott wrote, “I am willing to bet that I don't start mentioning this [98%] figure until the spring of 1997. If I use it before I said that I did the survey, I will say that they nailed me. But if I only started using it about the time that I said that I did the survey, I think that it would be strong evidence the other way.”
Lott’s first mention of the 98% figure located by Dudley Duncan is February 6, 1997, nearly two months before he could have completed the survey, according to his prior claims...
...If this newspaper account is accurate (and newspapers often aren’t), it is odd that Lott would try to answer the reporter’s claims about the Kellermann household study without pointing out that he had done a big household study himself. Although this contextual evidence is less telling, it does tend to fit the pattern that, until Lott replied to Duncan in mid-May 1999, Lott had consistently attributed the 98% figure to several specific survey organizations or to no one, never to his own 1997 study....
...2. Technical Problems
Another problem (mentioned by Tim Lambert and Dudley Duncan) is....
...3. The More Serious Issue: Was the 1997 Study Ever Done?
a. Circumstantial Evidence of a 1997 Computer Crash...
...As I posted in September, all evidence of a study with 2,400 respondents does not just disappear when a computer crashes. Having done one large survey (about half the size of John Lott’s) and several smaller surveys, I can attest that it is an enormous undertaking. Typically, there is funding, employees who did the survey, financial records on the employees, financial records on the mailing or telephoning, the survey instrument, completed surveys or tally sheets, a list of everyone in the sample, records on who responded and who declined to participate, and so on. While all of these things might not be preserved in every study, some of them would almost always be retained or recoverable. Just to get a representative list of the US public would take consultation with an experienced sampler and probably the purchase of an expensive sample. As far as I know, there was no cheap commercial list of almost every person or household in the United States from which to draw a good sample. .....
...b. No Direct Evidence of a 1997 Survey...[List of Lott's excuses]
...“I used students to conduct the survey. Most (though not all) of the information was originally entered into the students' own computers and then they were combined onto my machine. You know what happened to my computer.” This version of the story suggested original computer entry of data, which would have required central programming of a data form and then sharing of that program with several callers for their home computers. From Lott’s discussions with me, I was left with the impression that the callers instead used sheets to record answers, which he thought that he no longer retained. Besides the possible discrepancy between the two versions of the ways that the students collected and conveyed the data, if the data were instead sitting on the students’ computers, it is possible that Lott could have replaced most or all of his lost data by asking the students to give it to him again.
With the surprising lack of any of the normal indicia of having done a large national study of 2,424 respondents, the key remains locating the undergraduates who Lott says did the calling. The 1997 study was large, extremely time-consuming, and very expensive in phone charges. Getting 2,424 respondents with refusals and callbacks would have required thousands and thousands of phone calls. Students would have had to spend many hours calling, which they and their friends would well remember. With John Lott’s permission, I therefore contacted Saul Levmore, the Dean of the University of Chicago Law School, requesting him to try to obtain from the alumni office the email addresses of the 1997 and 1998 college graduating classes at the University of Chicago, so that I could contact them asking whether any of them or any of their friends had done any research for John Lott during his fellowship at Chicago. Levmore declined to make such a request for email addresses from the University of Chicago Alumni Office at this time, but he did not rule out cooperating with a request coming from other quarters or in a different situation. Of course, my request was in the form of fact-finding, rather than a complaint, which might have triggered a different process.
Thus, I have reached a temporary dead end. If someone were to email the 1997-98 Chicago college alumni twice, it is likely that at least one of those who did the study for Lott would come forward, if the study were actually done. [Like maybe John Lott?????????]...
...4. Comments on John Lott’s Response to this Report
a. Several Changes in Lott’s Story...
...b. Lott’s Attempt to Put Things in Context...
...End discussion on survey. . . . [notes about Lott’s contact info omitted here] . . .
1) I have lots of people who can say that I lost my hard disk in July of 1997: for example, David Mustard ([phone omitted]), Geoff Huck (the editor at the University of Chicago Press ([phone omitted]), John Whitley, William Landes. If you want to you can feel free to contact especially David and John. Russell Roberts is someone that I bounced the survey questions off of
and he can possibly talk to you about it, though he hasn't kept e-mails from 1997. . . . [Personal comment about Lott omitted for privacy reasons] I have told people for years about the data being lost.
2) I lost ALL the data for my paper with David in the JLS and for my book. David and I reconstructed the county and state level data from our paper and I got the rest together that was in my book. I have consistently provided the county, state, and city level data as well as the multiple victim shooting data and the safe storage data to people when they have asked. That constitutes almost all the numbers mentioned in the book. Months were spent redoing this data so that it could be given out to people. (Just a note, the critics to whom we had given out the data up to that point were unwilling to return the data to us so we had to put the JLS data together all over again.) If I had the other data, I would have been happy to give it to who ever wanted it.
The survey data could not be replaced by going to things like the UCR or the census data. It could only be replaced by doing another survey. I ended up only briefly mentioning the survey in the past and worked on replacing all the other data that I lost for not just the book but for all the other projects that I was working on. I spent a good portion of the next two years trying to replace data for other projects that I had been working on. I had important papers for the Journal of Political Economy and other journal for which replacing the data was my first priority after replacing the data for the JLS and the book. Thinking about the survey was well down on my list. Its importance was not particularly high given that I had only one sentence on the issue in my book and have never written the survey into a research paper because the data was lost.
3) This is something that was done six years ago. During the intervening time I have moved three times. Usually I pay students in cash. When I am at universities I don't apply for grants and the money is mine so there is no record of universities paying for the students. My records of the
students names and contact information were lost. You can get an idea of how much total time was involved from the survey discussed above. An ad was taken out in the fall in the University of Chicago Alumni magazine to try to contact the two University of Chicago students who organized some other people from different places to work on it. I have gone through well over 12 RAs and interns independent of those used on the survey since I got to AEI and I can't say that I remember most of their names (I am really horrible at names).
4) Issues about the significance difference in results. Given the very small sample sizes, the differences in results are not statistically significant and are really trivial. About one percent of people in a survey note that they have used a gun defensively. Whether one is talking about 2
percent or 20 percent simply brandishing a gun, you are talking about the different of 2 percent of 1 percent or 20 percent of 1 percent. Depending upon who answers the questions and what weighted group they are in, you are literally talking about the answers of a few people out of 1,000 that is the difference between these two results.
5) If you have doubts about anything in specific, you should ask me about it. Last year I worked extremely hard. I was the only affirmative numbers expert for the Senator Mitch McConnell's side in the campaign finance case. Once that was done, I had to deal with something that Ian Ayres and John Donohue wrote attacking my work and finishing things for my book. Only in the last couple of weeks have I gotten a breath on things, but I have responded when people e-mailed me. I am not a member of the firearms discussion groups and I have not been following them. I read your site once in a while just to keep up with the news (and because of that I have sent you some money from time to time e.g., just on Saturday), but otherwise I have been too busy to follow a lot of things. (When I recently accidentally sent you an e-mail it was at the end of one of many all nighters.)...