Survey research and self-defense gun use: an explanation of extreme overestimates.

AuthorHemenway, David
PositionResponse to Gary Kleck and Marc Gertz, Journal of Criminal Law and Criminology, vol. 86, p. 150, 1995
  1. Introduction and Summary

Gary Kleck and Marc Gertz conducted a survey of civilian

defensive gun use in 1992. In 1993, Kleck began publicizing the estimate

that civilians use guns in self-defense against offenders up to 2.5

million times each year.(1) This figure has been widely used by the

National Rifle Association and by gun advocates. It is also often cited in

the media(2) and even in Congress.(3) The Kleck and Gertz (K-G) paper

has now been published.(4) It is clear, however, that its conclusions

cannot be accepted as valid.

Two aspects of the K-G survey combine to create severe

misestimation. The first is the likelihood of positive social desirability

response, sometimes referred to as personal presentation bias. An

individual who purchases a gun for self-defense and then uses it

successfully to ward off a criminal is displaying the wisdom of his

precautions and his capability in protecting himself, his loved ones, and his

property. His action is to be commended and admired.

Some positive social desirability response bias, by itself, might not

lead to serious overestimation. However, combined with a second

aspect of the survey -- the attempt to estimate a very rare event -- it does.

The search for a "needle in a haystack" has major methodological

dangers, especially where researchers try to extrapolate the findings to

society as a whole.

Until the K-G study, no one had estimated that even as many as

1% of adult civilians had used a gun in self-defense in the past year.

Nevertheless, assume that the actual incidence is 1%. On average, for

every 100 individuals asked a "Yes/No" question about the event,

ninety-nine respondents will have a chance to be misclassified as a

false positive. In ninety-nine answers there is the possibility of positive

social desirability response bias. However, on average only one

respondent -- the one who actually did use a gun in self defense -- could

possibly be misclassified as a false negative (e.g., if she forgot about

the event). Even if the chance of forgetting is high, as long as there is

any possibility of positive response bias, it is very likely that the survey

finding will be an overestimate.

The fact that the survey is trying to estimate a low probability

event also means that a small percentage bias, when extrapolated, can

lead to extreme overestimates. Consider a survey finding which

contains a 1% overestimate of positive responses. If the true incidence of

the event is 60%, estimating it at 61% would not be a problem. But if

the true incidence is 1%, measuring it as 2% would be a doubling of

the true rate; and if the true incidence is 0.1%, measuring it at 1.1%

would be an eleven-fold overestimate.

The K-G survey design contains a huge overestimation bias. The

authors do little to reduce the bias or to validate their findings by

external measures. All checks for external validity of the Kleck-Gertz

finding confirm that their estimate is highly exaggerated.

  1. Background

    Previous data on self-defense gun use came from two

    sources -- the large National Crime Victimization surveys (NCVS), and smaller

    private surveys (principally random-digit-dial telephone surveys).

    These two sources produce markedly different results.

    The NCVS employs a multistage design with a probability sample

    of some 50,000 housing units in the United States (e.g., in 1994 there

    were 47,600 housing units and 90,560 persons).(5) The survey is

    conducted by the Census Bureau for the Bureau of Justice Statistics.

    Housing units remain in the NCVS for three years and residents are

    interviewed at six month intervals. Initial surveys are in-person, while

    subsequent ones are typically by telephone. Respondents who report

    a threatened, attempted or completed victimization for six

    crimes -- rape, robbery, assault, burglary, non-business larceny and motor

    vehicle theft -- are asked detailed questions about the incident.

    NCVS results indicate that, nationally, victims use guns against

    offenders approximately 65,000 times per year.(6) Kleck believes

    people under-report to the government NCVS interviewers, especially

    since the surveys are not anonymous. He also finds fault with the

    NCVS survey for asking about self-defense gun use only for individuals

    who have been victimized.(7) Interestingly, it is this latter feature of the

    NCVS which dramatically reduces the overestimation bias found in

    the private surveys.

    Based on eight national surveys, undertaken between 1976 and

    1990, Kleck estimates that guns are used approximately 700,000 times

    per year in self-defense.(8) However, all eight surveys have very serious

    limitations. Compared to the NCVS, the sample size of each of these

    surveys is small (600 - 1500) and interviewers typically asked only one

    vague question about gun use in self-defense (e.g., "Have you used a

    gun in self-defense in the previous five years?") with no follow-up

    questions.(9) Only one of the surveys meets the minimum criteria of

    drawing from a representative national population, asking about a

    specific time frame, distinguishing civilian use from military or police

    uses, and distinguishing uses against humans from uses against


    A review of Kleck's analysis argued that "Kleck's conclusions rest

    on limited data and strong assumptions. Small changes in the

    procedure produce large differences in the findings. The estimates are

    questionable, and it appears unwise to place much weight on them."(11)

    A National Research Council report also finds that Kleck's estimates

    appear exaggerated and says that it is almost certain that "some of

    what respondents designate as their own self-defense would be

    construed as aggression by others."(12)

  2. The Kleck-Gertz Survey

    In 1992, Kleck and Gertz conducted a national random-digit-dial

    survey of five thousand dwelling units, asking detailed questions about

    self-defense gun use.(13) Their estimates of civilian self-defense gun use

    range from I million to 2.5 million times per year.(14) The 2.5 million

    figure is the one they believe to be most accurate and the one Kleck

    has publicized, so that figure will be discussed in this paper.

    K-G derive their 2.5 million estimate from the fact that 1.33% of

    the individuals surveyed reported that they themselves used a gun in

    self-defense during the past year;(15) in other words, about 66 people

    out of 5000 reported such a use. Extrapolating the 1.33% figure to

    the entire population of almost 200 million adults gives 2.5 million


    Many problems exist with the survey conducted by Kleck and

    Gertz. A deficiency in their article is that they do not provide detailed

    information about their survey methodology or discuss its many

    limitations. For example, the survey was conducted by a small firm run by

    Professor Gertz. The interviewers presumably knew both the purpose

    of the survey and the staked-out position of the principal investigator

    regarding the expected results.

    The article states that when a person answered, the interview was

    completed 61% of the time.(16) But what happened when there was a

    busy signal, an answering machine or no answer? If no one was

    interviewed at a high percentage of the initially selected homes, the survey

    cannot be relied on to yield results representative of the population.

    Interviewers do not appear to have questioned a random

    individual at a given telephone number, but rather asked to speak to the

    male head of the household.(17) If that man was not at home, the caller

    interviewed the adult who answered the phone.(18) Although this

    approach is sometimes used in telephone surveys to reduce expense, it

    does not yield a representative sample of the population.

    The 2.5 million estimate is based on individuals rather than

    households.(19) But the survey is randomized by dwelling unit rather

    than by individual, so the findings cannot simply be extrapolated to

    the national population. Respondents who are the only adults in a

    household will receive too much weight.

    K-G oversampled males and individuals from the South and

    West.(20) The reader is presented with weighted rather than actual

    data, yet the authors do not explain their weighting technique. K-G

    claim their weighted data provide representative information for the

    entire country,(21) but they appear to have obtained various anomalous

    results. For example, they find that only 38% of households in the

    nation possess a gun, which is low, outside the range of all other

    national surveys.(22) They find that only 8.9% of the adult population is

    black,(23) when 1992 Census data indicate that 12.5% of individuals

    were black.(24)

    The above limitations are serious. However, it is two other

    aspects of the survey that, when combined together, lead to an

    enormous overestimation of self-defense gun use: the fact that K-G are

    trying (1) to measure a very low probability event which (2) has

    positive social desirability response bias. The problem is one of


  3. Misclassification in Surveys Generally

    All surveys have problems with accuracy.(25) Incorrect

    classifications come from a wide variety of causes including misunderstanding,

    miscoding, misremembering, misinterpretation of events, mischief or

    downright mendacity.

    Some percentage of answers to virtually all survey questions are

    incorrect. Respondents substantially over-report their seat belt use,(26)

    for example, and inaccurately report whether they voted.(27) Not all

    people are completely truthful when reporting about such mundane

    details as their age,(28) height, or weight.(29) A book on survey response

    validity characterizes as "quite high" accuracy rates of 83% to 98% to

    questions about possession of an automobile, a home, a driver's

    license or a library card.(30)

    Respondents who misreport are not necessarily deliberately lying;

    they may be shading the truth or simply perceive and present

    themselves in a slightly more favorable light than a purely objective

    observer would. In addition, some Americans may simply have a

    different perception of reality than most...

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT