Marginal Effects in Interaction Models: Determining and Controlling the False Positive Rate

• Author: Justin Esarey,Jane Lawrence Sumner
• Published date: 01 August 2018
• Date: 01 August 2018
• DOI: http://doi.org/10.1177/0010414017730080

https://doi.org/10.1177/0010414017730080

Comparative Political Studies

2018, Vol. 51(9) 1144 –1176

© The Author(s) 2017

Reprints and permissions:

sagepub.com/journalsPermissions.nav

DOI: 10.1177/0010414017730080

journals.sagepub.com/home/cps

Article

Marginal Effects in

Interaction Models:

Determining and

Controlling the False

Positive Rate

Justin Esarey1 and Jane Lawrence Sumner2

Abstract

When a researcher suspects that the marginal effect of

x

on

y

varies with

z

, a common approach is to plot

∂∂yx

/ at different values of

z

along with

a pointwise confidence interval generated using the procedure described

in Brambor, Clark, and Golder to assess the magnitude and statistical

significance of the relationship. Our article makes three contributions. First,

we demonstrate that the Brambor, Clark, and Golder approach produces

statistically significant findings when ∂∂yx

/=

0 at a rate that can be many

times larger or smaller than the nominal false positive rate of the test.

Second, we introduce the interactionTest software package for R to

implement procedures that allow easy control of the false positive rate.

Finally, we illustrate our findings by replicating an empirical analysis of the

relationship between ethnic heterogeneity and the number of political

parties from Comparative Political Studies.

Keywords

political parties, quantitative methods, interaction

1Rice University, Houston, TX, USA

2University of Minnesota, Minneapolis, MN, USA

Corresponding Author:

Justin Esarey, Associate Professor, Department of Political Science, Rice University, 6100 Main

Street MS 24, Houston, TX 77005, USA.

Email: justin@justinesarey.com

730080CPSXXX10.1177/0010414017730080Comparative Political StudiesEsarey and Sumner

research-article2017

Esarey and Sumner 1145

Introduction

Much of the recent empirical work in political science1 has recognized that

causal relationships between two variables

x

and

y

are often changed—

strengthened or weakened—by contextual variable

z

. Such a relationship is

commonly termed interactive. The substantive interest in these relationships

has been coupled with an ongoing methodological conversation about the

appropriate way to test hypotheses in the presence of interaction. The latest

additions to this literature, particularly King, Tomz, and Wittenberg (2000);

Ai and Norton (2003); Braumoeller (2004); Brambor, Clark, and Golder

(2006); Kam and Franzese (2007); Berry, DeMeritt, and Esarey (2010); and

Berry, Golder, and Milton (BGM: 2012), emphasize visually depicting the

marginal effect of

x

on

y

at different values of

z

(with a confidence inter-

val [CI] around that marginal effect) to assess whether that marginal effect is

statistically and substantively significant. The statistical significance of a

multiplicative interaction term is seen as neither necessary nor sufficient for

determining whether

x

has an important or statistically distinguishable rela-

tionship with

y

at a particular value of

z

. That is, although a statistically

significant product term is sufficient for concluding that

∂∂yx

/ is different

at different values of

z

(Kam & Franzese, 2007, p. 50), it cannot tell us

whether

∂∂yx

/ is statistically distinguishable from zero at any particular

value of

z

.

A paragraph from Brambor et al. (2006) summarizes the current state of

the art:

The analyst cannot even infer whether

x

has a meaningful conditional effect on

y

from the magnitude and significance of the coefficient on the interaction

term either. As we showed earlier, it is perfectly possible for the marginal effect

of

x

on

y

to be significant for substantively relevant values of the modifying

variable

z

even if the coefficient on the interaction term is insignificant. Note

what this means. It means that one cannot determine whether a model should

include an interaction term simply by looking at the significance of the

coefficient on the interaction term. Numerous articles ignore this point and drop

interaction terms if this coefficient is insignificant. In doing so, they potentially

miss important conditional relationships between their variables. (p. 74)

In short, they recommend including a product term

xz

in linear models

where interaction between

x

and

z

is suspected, then examining a plot of

∂∂yx

/ and its 95% CI over the range of

z

in the sample.2 If the CI does not

include zero for any value of

z

, one should conclude that

x

and

y

are statis-

tically related (at that value of

z

), with the substantive significance of the

relationship given by the direction and magnitude of the

∂∂yx

/ estimate. It

1146Comparative Political Studies 51(9)

is hard to exaggerate the impact that the methodological advice given in

Brambor et al. (2006) has had on the discipline: The article has been cited

over 3,300 times as of August 2016. Similar advice is given in Braumoeller

(2004, pp. 815-818, especially Figure 2), which has been cited over 660 times

in the same time frame.

Our article makes three contributions to the study of interactive relation-

ships. First, we highlight a hazard with the Brambor, Clark, and Golder pro-

cedure: The reported

α

level of CIs and hypothesis tests constructed using

the procedure can be inaccurate because of a multiple comparison problem

(Abdi, 2007; Sidak, 1967). The source of the problem is that adding an inter-

action term

z

to a model like

yx

=

01

ββ

+ is analogous to dividing a sample

data set into subsamples defined by the value of

z

, each of which (under the

null hypothesis that ∂∂yx

/=

0) has a separate probability of a false positive

(i.e., falsely rejecting the null hypothesis when the null is true). For

example, if

z

is dichotomous (z∈

{}

0,1 ), estimating a model like

yx

zx

z=0

123

ββ

ββ

+++ is analogous to estimating

yx

=

01

ββ

+ twice,

once for data where z

=0

and once for data where z

=1

, with two opportuni-

ties for β1 to be found statistically significant by chance. A similar problem

is already well recognized in the ANOVA for nominal treatment factors (e.g.,

Kutner, Nachtsheim, Neter, & Li, 2004, Section 19.9). In contrast, the meth-

ods that are described in Brambor et al. (2006) construct a pointwise CI (typi-

cally using a two-tailed

α

= .05 ); “pointwise” indicates that the CIs are

constructed for each individual value of

z

without considering the joint cov-

erage of the CI for all values of

z

. That is, the CI for each value of

z

assumes

a single draw from the sampling distribution of the marginal effect of interest.

As a result, these CIs can either be too wide or too narrow to conduct the tests

that scholars wish to perform3: Plotting

∂∂yx

/ over values of

z

and report-

ing any statistically significant relationship tend to result in overconfident

tests, while plotting

∂∂yx

/ over

z

and requiring statistically significant

relationships at multiple values of

z

tend to result in underconfident tests.4

The latter scenario may occur when, for example, a theory predicts that

∂∂yx

/>

0 for z=0 and ∂∂yx

/<

0 for z

=1

, and we try to jointly confirm

these predictions in a data set.

Second, we offer researchers guidance on strategies that are effective and

ineffective at controlling the false positive rate when examining interaction

relationships. Our primary recommendation is for researchers to simply be

aware that marginal effects plots generated using a given

α

could be over- or

underconfident, and thus to take a closer look if results are at the margin of

statistical significance. When overconfidence is an issue, researchers can con-

trol the false discovery rate (FDR) in marginal effects plots by adapting the

procedure of Benjamini and Hochberg (1995)5; we provide code to