The illegitimacy of one-sided speculation: getting the defensive gun use estimate down.

• Author: Kleck, Gary
• Position: Response to article by David Hemenway in this issue, p. 1430

Introduction

It is obvious to us that David Hemenway (H) had no intention of

producing a balanced, intellectually serious assessment of our

estimates of defensive gun use (DGU). Instead, his critique serves the

narrow political purpose of "getting the estimate down," for the sake

of advancing the gun control cause. An honest, scientifically based

critique would have given balanced consideration to flaws that tend to

make the estimate too low (e.g., people concealing DGUs because

they involved unlawful behavior, and our failure to count any DGUs

by adolescents), as well as those that contribute to making them too

high. Equally important, it would have given greatest weight to

relevant empirical evidence, and little or no weight to idle speculation

about possible flaws. H's approach is precisely the

opposite -- one-sided and almost entirely speculative. Readers who have any

doubts about the degree to which H's paper is imbalanced might carry out a

simple exercise to assess our claim -- count the number of lines H

devotes to flaws tending to make the estimate too high and the number

devoted to flaws making the estimate too low. We submit that the

ratio is over 100-to-1, i.e., almost entirely devoted to speculations

about why the estimate is too high.

The political function of this advocacy scholarship is clear. While

high estimates of DGU frequency do not constitute an obstacle to

moderate controls over guns, they constitute the most serious obstacle

to advocacy of gun prohibition. Disarming the mass of noncriminal

prospective crime victims would, if high DGU estimates are even

approximately correct, result in large numbers of foregone

opportunities for uses of guns that could prevent deaths, injuries, and property

loss. To acknowledge high DGU frequency would be to concede the

most significant cost of gun prohibition. H's paper is an attempt to

neutralize concerns about such costs and to provide intellectual

respectability for positions identified with Handgun Control

Incorporated (HCI), the nation's leading gun control advocacy group.

H has close ties to HCI through two key staff members of HCI's

"educational" branch, the Center to Prevent Handgun Violence

(CPHV). His closest and most frequent collaborator on gun-related

research is Douglas Weil, currently Research Director of CPHV,(1) while

H has co-edited a strongly pro-control propaganda tract with Dennis

1. Henigan, legal counsel to HCI and CPHV.(2)

H's political intentions and strong feelings are also evident in his

overstatements and in the grandiose conclusions he draws from weak

or irrelevant evidence and fallacious reasoning. He does not get past

his tide before making his first overstatement, claiming that he had

established, without benefit of any new empirical evidence, that our

estimates are too high and that they are "extreme overestimates."(3)

He states in his first paragraph that "it is clear that [the Kleck and

Gertz] results cannot be accepted as valid."(4) He incorrectly claims

that "all checks for external validity of the Kleck-Gertz finding confirm

that their estimate is highly exaggerated,"(5) when in fact these checks

have repeatedly confirmed our estimates.

DGUs usually involve unlawful possession of a gun by the gun-wielding

victim, and sometimes other illegalities as well,(6) a point H

does not dispute. Yet, in making the extraordinary and

counterintuitive claim that there is a social desirability bias to people

reporting their own illegal behavior,(7) H insists that such a desirability

bias is not only plausible, but that it is likely.(8) By the end, without having

provided a scintilla of credible supporting evidence, H concludes that our

research was afflicted by an "enormous problem of false positives"

(persons claiming a DGU who did not have one) and "massive

overestimation," flatly stating that "the Kleck and Gertz survey results do not

provide reasonable estimates about the total amount of self-defense

gun use in the United States."(9) It is an impressive achievement to be

able to arrive at such high-powered conclusions without the

inconvenience of gathering or even citing any new empirical evidence.

1. The Illegitimacy of One-sided Speculation: An Ounce or Evidence

   Outweighs a Ton of Speculation

   H's critical technique is simple: one-sided, and often implausible,

   speculation about flaws that might have afflicted our research, and

   that might have been consequential enough to significantly affect our

   conclusions. H devotes his attention almost exclusively to suspected

   flaws that might have contributed to the overestimation of defensive

   gun use (DGU) frequency. He either ignores well established sources

   of underreporting, or briefly and superficially discusses them only for

   the sake of dismissing them.(10) When H speculates about sources of

   response error that are plausible, he offers no rationale for why the

   problems should lead to more false positives than false negatives.

   Instead he simply conjures up reasons they might lead to false positives.

   As support for his one-sided speculations H even cites other people

   guilty of the same dubious practice.(11)

   All research is flawed. Known flaws should be identified and

   their likely impact assessed. Speculation about flaws can play a role in

   the pursuit of truth by motivating researchers to gather better

   empirical evidence less afflicted by the flaws. Speculation by itself, however,

   should not be given any weight in assessing evidence. An ounce of

   evidence, even though flawed, outweighs a ton of speculation.

   Unfortunately, in both good research and bad, there is no upper limit on

   the amount of speculative criticism that can be directed at the work,

   and thus this sort of critique is just as easily applied to good research

   as to bad.

2. Red Herrings and the Issue Not Addressed

   Much of H's paper is a red herring in that it implicitly misstates

   the central technical question about our estimates. Much of it is

   devoted to elaborate speculations about why people might falsely claim

   to have used a gun defensively, as if it were somehow in dispute that

   there are some false positives.(12) He inaccurately hints that we

   unreasonably ignored the possibility that some of our respondents (Rs)

   provided false positives.(13)

   We assume as a matter of course that our survey is like all other

   surveys in that some Rs give inaccurate responses to questions, and

   that these errors include both false positives and false negatives. The

   central question is not whether there are false positives, nor even how

   many false positives there are, but rather what the relative balance is

   between false positives and false negatives. Because H makes no effort

   to assess the frequency of false negatives,(14) it is logically impossible for

   him to draw meaningful conclusions about whether our estimates

   were too high or low.

3. The Nature of False Positives

   It is hard to discern exactly what kinds of false positives H thinks

   most often show up in all these gun use surveys. He waffles on the

   issue of whether people are: (1) consciously inventing nonexistent

   events; (2) consciously but honestly misrepresenting accounts of real

   events that did not really involve DGU (e.g., they involved aggressive

   use of a gun); or (3) unconsciously distorting real events. He seems to

   have doubts himself about possibility (1) occurring very often,

   hastening to assure readers that false responders do not necessarily have to

   lie,(15) but is otherwise unwilling to commit himself to the relative

   frequency of these types of misreports.

   It is worth emphasizing how difficult it was for our Rs to falsely

   report a completely nonexistent event as a DGU. Unlike the UFO

   example that H insists is somehow parallel to reports of DGUs,(16) a

   respondent who wanted to falsely report a nonexistent DGU could not

   qualify as having had such an experience merely by saying "Yes."

   Rather, respondents had to provide as many as nineteen internally

   consistent responses covering the details of the alleged incident. In

   short, to sustain a false DGU claim, Rs had to do a good deal of agile

   mental work, and stay on the phone even longer. On the other hand,

   all it took to yield a false negative was for a DGU-involved R to speak a

   single inaccurate syllable: "No." The point is not that false positives

   were impossible, but rather that it was far harder to provide a false

   positive than a false negative.

   Consider also the context in which H imagines all these false

   reports to have occurred. Randomly selected people were called

   unexpectedly, and questioned rapidly by total strangers, for no more than

   fifteen minutes, with one question immediately following another.

   There was no prolonged opportunity to invent a nonexistent event,

   rehearse inaccurate details, or to otherwise get an false story straight.

   Rs providing a false positive had to be not only dishonest but very

   quick-witted as well.

   Regarding possibility (2), we noted that most of the DGUs were

   linked with the types of crimes -- burglaries, robberies, and sexual

   assaults -- where there is little opportunity for participants to be honestly

   confused about who was the victim and who was the offender.(17) While

   a few Rs may well have consciously misrepresented aggressive actions

   as defensive, and a very few might have consciously invented entirely

   fictitious events, it is hard to see how Rs could report an account of a

   real burglary, robbery, or sexual assault in which they were aggressors

   and somehow honestly distort it into a DGU incident.

   This kind of misunderstanding of real events in a way that falsely

   qualifies them as DGUs is more plausible in connection with

   male-against-male assault incidents, such as when people prefer to

   characterize their partly aggressive, partly defensive behavior in "mutual

   combat" incidents as purely defensive in character. We addressed this

   latter possibility in our article and showed that it could not account

   for more than a small fraction (probably less than a tenth) of the

   incidents...