Erring on the margin of error.

• Author: Thornton, Robert J.
• Position: Sampling

1. Introduction

   As most teachers of probability and statistics know, one of the most difficult concepts to convey to students is that of sampling error. Yet with the proliferation of the reporting of the results of public-opinion polls in the news media, students and the general public alike are exposed to this concept almost on a daily basis. In fact, when the authors recently accessed the Dow-Jones Interactive News Library (1) and typed in the words "public opinion poll," we registered nearly 60,000 "hits" in major newspapers and newswires for the period 1990 to the present. Often accompanying the discussion of the poll results is a statement describing the accuracy of the poll's estimates, which ordinarily reads something like, "The margin of error is 3 percentage points with a 95% level of confidence." (2) Because to many readers the meaning of this statement is fuzzy, the article sometimes attempts to clarify what the margin of error indicates about the poll's accuracy. However, from our experience, the attempted explanation is often completely in error--sometimes outrageously so.

   In this article, we first briefly explain the correct way of interpreting the margin of error, which currently seems to be the fashionable way to explain in the media what statisticians routinely call sampling error. The margin of error concept is also gradually making its way into the professional economics literature (see Borenstein and Rose 1994; Brezis 1995) and into business/economics statistics textbooks as well (e.g., Anderson, Sweeney, and Williams 2002; Bowerman and O'Connell 2003). Next, we present some typical misinterpretations of the margin of error drawn from the news media that will prove interesting to both teachers and students and useful as classroom examples. Finally, we suggest some ways of using the media misinterpretations to teach and reinforce the correct interpretation of the margin of error (as well as of confidence intervals).

2. Interpretations and Misinterpretations

   Suppose that an opinion poll taken of 1000 people (assume the sample has been selected randomly) finds that 60% of those sampled believe that economists are extremely boring. Suppose also that the margin of error (with 95% confidence) is reported to be 3 percentage points. The correct interpretation of this margin of error is that if repeated samples of size 1000 were to be taken, approximately (3) 95% of the time the sample proportions (p) would lie within 3 percentage points (0.03) of the true population proportion (p)--the proportion of all people who believe that economists are extremely boring. In other words,

   Prob([absolute value of p - p] [less than or equal to] 0.03) = 0.95.

   Conversely, only 5% of the time would the sample proportions be more than 0.03 away from the population proportion. But what kinds of interpretations are often given for the margin of error by the news media? Here are several examples:

   * From a survey on the incidence of AIDS among clerics, an article described the poll's 3.5 percentage-point margin of error to mean that "if the same poll were conducted 100 times, 95 percent of those times the results would be no more than 3.5 percentage points higher or lower than the results of this poll" (Thomas 2000, p. A18) (italics added).

   * In a telephone survey of Ohio adults on how trustworthy they believed real estate agents to be, the Columbus Dispatch reported that the survey had a "2 to 3 point margin of error and a 95 percent confidence level, meaning the results have a 95 percent probability of being true no matter the size of the polling group" (Columbus Dispatch 1998, p. 8J).

   * A telephone poll of 1014 Texas adults on the death penalty reported a margin of error of 3 percentage points, which was interpreted as "each response can vary that much in either direction" (Hoppe 1998, p. 39A).

   * In a Newsday report attempting to explain what a "1.8 rating-point plus-or-minus margin of error" meant for a Nielsen rating of news shows, the following explanation was given: "That means, among the many possible permutations of those figures, ... that last week's ratings...