Survey research and self-defense gun use: an explanation of extreme overestimates.

• Author: Hemenway, David
• Position: Response to Gary Kleck and Marc Gertz, Journal of Criminal Law and Criminology, vol. 86, p. 150, 1995

1. Introduction and Summary

Gary Kleck and Marc Gertz conducted a survey of civilian

defensive gun use in 1992. In 1993, Kleck began publicizing the estimate

that civilians use guns in self-defense against offenders up to 2.5

million times each year.(1) This figure has been widely used by the

National Rifle Association and by gun advocates. It is also often cited in

the media(2) and even in Congress.(3) The Kleck and Gertz (K-G) paper

has now been published.(4) It is clear, however, that its conclusions

cannot be accepted as valid.

Two aspects of the K-G survey combine to create severe

misestimation. The first is the likelihood of positive social desirability

response, sometimes referred to as personal presentation bias. An

individual who purchases a gun for self-defense and then uses it

successfully to ward off a criminal is displaying the wisdom of his

precautions and his capability in protecting himself, his loved ones, and his

property. His action is to be commended and admired.

Some positive social desirability response bias, by itself, might not

lead to serious overestimation. However, combined with a second

aspect of the survey -- the attempt to estimate a very rare event -- it does.

The search for a "needle in a haystack" has major methodological

dangers, especially where researchers try to extrapolate the findings to

society as a whole.

Until the K-G study, no one had estimated that even as many as

1% of adult civilians had used a gun in self-defense in the past year.

Nevertheless, assume that the actual incidence is 1%. On average, for

every 100 individuals asked a "Yes/No" question about the event,

ninety-nine respondents will have a chance to be misclassified as a

false positive. In ninety-nine answers there is the possibility of positive

social desirability response bias. However, on average only one

respondent -- the one who actually did use a gun in self defense -- could

possibly be misclassified as a false negative (e.g., if she forgot about

the event). Even if the chance of forgetting is high, as long as there is

any possibility of positive response bias, it is very likely that the survey

finding will be an overestimate.

The fact that the survey is trying to estimate a low probability

event also means that a small percentage bias, when extrapolated, can

lead to extreme overestimates. Consider a survey finding which

contains a 1% overestimate of positive responses. If the true incidence of

the event is 60%, estimating it at 61% would not be a problem. But if

the true incidence is 1%, measuring it as 2% would be a doubling of

the true rate; and if the true incidence is 0.1%, measuring it at 1.1%

would be an eleven-fold overestimate.

The K-G survey design contains a huge overestimation bias. The

authors do little to reduce the bias or to validate their findings by

external measures. All checks for external validity of the Kleck-Gertz

finding confirm that their estimate is highly exaggerated.

2. Background

   Previous data on self-defense gun use came from two

   sources -- the large National Crime Victimization surveys (NCVS), and smaller

   private surveys (principally random-digit-dial telephone surveys).

   These two sources produce markedly different results.

   The NCVS employs a multistage design with a probability sample

   of some 50,000 housing units in the United States (e.g., in 1994 there

   were 47,600 housing units and 90,560 persons).(5) The survey is

   conducted by the Census Bureau for the Bureau of Justice Statistics.

   Housing units remain in the NCVS for three years and residents are

   interviewed at six month intervals. Initial surveys are in-person, while

   subsequent ones are typically by telephone. Respondents who report

   a threatened, attempted or completed victimization for six

   crimes -- rape, robbery, assault, burglary, non-business larceny and motor

   vehicle theft -- are asked detailed questions about the incident.

   NCVS results indicate that, nationally, victims use guns against

   offenders approximately 65,000 times per year.(6) Kleck believes

   people under-report to the government NCVS interviewers, especially

   since the surveys are not anonymous. He also finds fault with the

   NCVS survey for asking about self-defense gun use only for individuals

   who have been victimized.(7) Interestingly, it is this latter feature of the

   NCVS which dramatically reduces the overestimation bias found in

   the private surveys.

   Based on eight national surveys, undertaken between 1976 and

   1990, Kleck estimates that guns are used approximately 700,000 times

   per year in self-defense.(8) However, all eight surveys have very serious

   limitations. Compared to the NCVS, the sample size of each of these

   surveys is small (600 - 1500) and interviewers typically asked only one

   vague question about gun use in self-defense (e.g., "Have you used a

   gun in self-defense in the previous five years?") with no follow-up

   questions.(9) Only one of the surveys meets the minimum criteria of

   drawing from a representative national population, asking about a

   specific time frame, distinguishing civilian use from military or police

   uses, and distinguishing uses against humans from uses against

   animals.(10)

   A review of Kleck's analysis argued that "Kleck's conclusions rest

   on limited data and strong assumptions. Small changes in the

   procedure produce large differences in the findings. The estimates are

   questionable, and it appears unwise to place much weight on them."(11)

   A National Research Council report also finds that Kleck's estimates

   appear exaggerated and says that it is almost certain that "some of

   what respondents designate as their own self-defense would be

   construed as aggression by others."(12)

3. The Kleck-Gertz Survey

   In 1992, Kleck and Gertz conducted a national random-digit-dial

   survey of five thousand dwelling units, asking detailed questions about

   self-defense gun use.(13) Their estimates of civilian self-defense gun use

   range from I million to 2.5 million times per year.(14) The 2.5 million

   figure is the one they believe to be most accurate and the one Kleck

   has publicized, so that figure will be discussed in this paper.

   K-G derive their 2.5 million estimate from the fact that 1.33% of

   the individuals surveyed reported that they themselves used a gun in

   self-defense during the past year;(15) in other words, about 66 people

   out of 5000 reported such a use. Extrapolating the 1.33% figure to

   the entire population of almost 200 million adults gives 2.5 million

   uses.

   Many problems exist with the survey conducted by Kleck and

   Gertz. A deficiency in their article is that they do not provide detailed

   information about their survey methodology or discuss its many

   limitations. For example, the survey was conducted by a small firm run by

   Professor Gertz. The interviewers presumably knew both the purpose

   of the survey and the staked-out position of the principal investigator

   regarding the expected results.

   The article states that when a person answered, the interview was

   completed 61% of the time.(16) But what happened when there was a

   busy signal, an answering machine or no answer? If no one was

   interviewed at a high percentage of the initially selected homes, the survey

   cannot be relied on to yield results representative of the population.

   Interviewers do not appear to have questioned a random

   individual at a given telephone number, but rather asked to speak to the

   male head of the household.(17) If that man was not at home, the caller

   interviewed the adult who answered the phone.(18) Although this

   approach is sometimes used in telephone surveys to reduce expense, it

   does not yield a representative sample of the population.

   The 2.5 million estimate is based on individuals rather than

   households.(19) But the survey is randomized by dwelling unit rather

   than by individual, so the findings cannot simply be extrapolated to

   the national population. Respondents who are the only adults in a

   household will receive too much weight.

   K-G oversampled males and individuals from the South and

   West.(20) The reader is presented with weighted rather than actual

   data, yet the authors do not explain their weighting technique. K-G

   claim their weighted data provide representative information for the

   entire country,(21) but they appear to have obtained various anomalous

   results. For example, they find that only 38% of households in the

   nation possess a gun, which is low, outside the range of all other

   national surveys.(22) They find that only 8.9% of the adult population is

   black,(23) when 1992 Census data indicate that 12.5% of individuals

   were black.(24)

   The above limitations are serious. However, it is two other

   aspects of the survey that, when combined together, lead to an

   enormous overestimation of self-defense gun use: the fact that K-G are

   trying (1) to measure a very low probability event which (2) has

   positive social desirability response bias. The problem is one of

   misclassification.

4. Misclassification in Surveys Generally

   All surveys have problems with accuracy.(25) Incorrect

   classifications come from a wide variety of causes including misunderstanding,

   miscoding, misremembering, misinterpretation of events, mischief or

   downright mendacity.

   Some percentage of answers to virtually all survey questions are

   incorrect. Respondents substantially over-report their seat belt use,(26)

   for example, and inaccurately report whether they voted.(27) Not all

   people are completely truthful when reporting about such mundane

   details as their age,(28) height, or weight.(29) A book on survey response

   validity characterizes as "quite high" accuracy rates of 83% to 98% to

   questions about possession of an automobile, a home, a driver's

   license or a library card.(30)

   Respondents who misreport are not necessarily deliberately lying;

   they may be shading the truth or simply perceive and present

   themselves in a slightly more favorable light than a purely objective

   observer would. In addition, some Americans may simply have a

   different perception of reality than most...