Inconvenient truths about logistic regression and the remedy of marginal effects

• Published date: 01 November 2024
• Author: Michael Howell‐Moroney
• Date: 01 November 2024
• DOI: http://doi.org/10.1111/puar.13786

SCHOLARLY TAKES

Inconvenient truths about logistic regression and the remedy

of marginal effects

Michael Howell-Moroney

Department of Public and Nonprofit

Administration, School of Urban Affairs and

Public Policy, The University of Memphis,

Memphis, Tennessee, USA

Correspondence

Michael Howell-Moroney, Department of Public

and Nonprofit Administration, School of Urban

Affairs and Public Policy, The University of

Memphis, 218 Browning Hall, Memphis, TN

38152, USA.

Email: michael.hm@memphis.edu

Abstract

Logistic regression is a standard technique in public administration research.

However, there are two inconvenient truths about logistic regression of which

scholars should be aware. First, logistic regression results are difficult to interpret.

Raw coefficients are expressed in an enigmatic log odds scale and odds ratios are

regularly misinterpreted as risk ratios. Second, logistic regression results are non-

collapsible, which renders model comparisons invalid. A review of recent public

administration articles reveals that these inconvenient truths still plague the disci-

pline. This paper advocates the use of average marginal effects to reckon with both

inconvenient truths. Average marginal effects are easy to comprehend because they

measure effect sizes on a probability scale. And average marginal effects are collaps-

ible, and hence facilitate valid model comparisons. These concepts are illustrated

using data simulations and data from the 2017 Current Population Survey. The

paper concludes with suggestions for improved research practice.

Key Points

•Logistic regression differs in two key aspects from conventional linear regres-

sion: Comprehensibility and non-collapsibility.

•Comprehensibility. Logistic regression results are difficult to understand because

they are in the log-odds scale.

•Non-Collapsibility. Results from different logistic regression models are not

directly comparable. Non-collapsibility is a feature of most nonlinear models like

probit, count, and survival time models.

•Marginal effects effectively deal with both issues. Marginal effects are compre-

hensible because they measure effects on the probability scale. Marginal effects

are also collapsible, allowing for logistic regression results to be compared

validly.

•A review of recent public administration research shows that reporting of mar-

ginal effects is rare. In addition, most papers that compare logit models did so

incorrectly by using non-collapsible measures.

•More widespread adoption of the marginal effects approach to logistic regres-

sion and other nonlinear models will be an important step toward making non-

linear models more understandable to researchers and practitioners alike.

INTRODUCTION

This paper pinpoints two inconvenient truths about logis-

tic regression. The first inconvenient truth is that logistic

regression results are difficult to interpret, so they lack

comprehensibility (Mood, 2010; Norton & Dowd, 2018;

Uberti, 2022; Wooldridge, 2010). Logistic regression coeffi-

cients measure the change in the log odds for a unit

change in x. Of course, most people do not think on a log

odds scale, making raw coefficients difficult to interpret.

Another practice is reporting odds ratios. Odds ratios

measure the change in the odds for a unit change in x.

However, it is well-known that researchers and practi-

tioners regularly misinterpret odds ratios as risk ratios,

which are not the same thing (Gallis & Turner, 2019;

Niu, 2020; Norton & Dowd, 2018; Xiao et al., 2021).

Received: 20 December 2022 Revised: 5 September 2023 Accepted: 21 November 2023

DOI: 10.1111/puar.13786

1218 © 2023 American Society for Public Administration. Public Admin Rev. 2024;84:1218–1236.wileyonlinelibrary.com/journal/puar

The second, and more ominous, inconvenient truth

about logistic regression is the problem of non-collapsibility.

Raw logistic regression coefficients and odds ratios cannot

be compared across different model specifications, models

for separate groups, or across samples because of a phe-

nomenon called non-collapsibility (Allison, 1999; Norton &

Dowd, 2018;Yuetal.,2017). In short, non-collapsibility is a

feature of many nonlinear models where comparisons are

not possible because of differences in the scaling of

the coefficients from model to model (Allison, 1999;

Mood, 2010;Norton&Dowd,2018). This scaling effect arises

because the residual variance for the logit model is normal-

ized to a fixed value (Allison, 1999; Mood, 2010,2017).

This makes raw coefficients/odds ratios non-collapsible,

and hence incomparable across models (Allison, 1999;

Greenland, 2021b).

A solution to the problems of comprehensibility and

non-collapsibility is found in marginal effects. First, mar-

ginal effects are easily understandable (Mize et al., 2019;

Mood, 2010; Mood, 2017; Norton & Dowd, 2018). Marginal

effects measure the average change in probability that

results from a unit change in X. This makes marginal

effects easier to comprehend than log odds or odds ratios

because they use the metric of probability for the out-

come. Second, some marginal effects, such as the average

marginal effect (AME), are collapsible, and therefore com-

parable across models (Cramer, 2007; Mood, 2010;

Norton & Dowd, 2018; Wooldridge, 2010). When model

comparisons are part of the research design, marginal

effects are indispensable. In sum, marginal effects offer

results that are both comparable and comprehensible.

Public administration research has yet to recognize

these inconvenient truths. First, a review of recent

research shows that most public administration scholars

report raw coefficients and odds ratios instead of mar-

ginal effects. This makes the research harder to under-

stand and subject to misinterpretation. Second, the

problem of non-collapsibility is largely ignored in most

public administration research. Scholars regularly make

model comparisons using non-collapsible measures.

Because raw coefficients and/or odds ratios are not col-

lapsible, these model comparisons are invalid. Third,

reporting of marginal effects is still rare in public adminis-

tration research. This is unfortunate because marginal

effects are both comprehensible and collapsible.

The first half of the paper expands on the challenges

posed by the problems of comprehensibility and non-

collapsibility. This includes a data simulation to demon-

strate the problem of non-collapsibility. The first half of

the paper concludes with a survey of recent public

administration research that investigates how practice

bears up under these inconvenient truths.

The second half of the paper focuses on marginal

effects. This includes calculation of the AME and its

advantages as a measure of effect size. These ideas are

demonstrated with a simplified empirical example using

data from the 2017 Current Population Survey. The paper

then considers other marginal effects commonly used in

research. These include marginal effects at the means

(MEM), marginal effects at representative values (MER),

and marginal effects for interactions. The paper concludes

with a set of suggestions for reporting the results of logis-

tic regression and marginal effects.

THE CHALLENGE OF COMPREHENSIBILITY

The coefficients in logistic regression are not as

straightforward to interpret as they are in ordinary least

squares regression (OLS). In OLS regression, the coeffi-

cient measures the change in the dependent variable

for a unit change in x. Raw logistic regression coeffi-

cients estimate the change in the log odds (logit) for a

unit change in x. Raw coefficients can be informative,

but in a limited way. The sign of the coefficient indi-

cates whether a covariate increases or decreases the

probability of the outcome. But importantly, the raw

coefficient cannot tell us how much of a change in

probability results. Because they are in a log odds scale,

raw logistic regression coefficients are difficult to inter-

pret for researchers and practitioners alike. If we want

to study the change in probability for a change in x,

logit coefficients cannot get us there.

A second practice is reporting odds ratios. Odds ratios

are nothing more than exponentiated raw coefficients

(e

β

). The odds ratio measures the change in the odds of

the outcome for a one-unit change in x. Odds ratios are a

bit more comprehensible since they at least use an odds

scale instead of log odds. An odds ratio less than one

means that the variable decreases the chances of the out-

come. An odds ratio of one corresponds to no effect

(i.e., even odds). An odds ratio greater than one indicates

a positive effect of a variable on the outcome. But impor-

tantly, just like raw coefficients, odds ratios do not mea-

sure change in probability.

Scholars have established that odds ratios are often

misinterpreted as risk ratios (Mood, 2010; Norton

et al., 2018; Norton & Dowd, 2018). Gallis and Turner

(2019) write, “Despite its widespread use, the OR is fre-

quently misinterpreted as an RR by researchers, journal-

ists, policymakers, and the general public”(p. 2). For

example, an odds ratio of 2.5 might be misinterpreted as a

unit change in x making the outcome 2.5 times more

likely. But an odds ratio of 2.5 only implies that the odds

are 2.5 times greater. The risk ratio is a ratio of probabili-

ties, not odds. And the odds ratio is a ratio of odds, not

probabilities.

Another problem with odds ratios is that they are a

relative measure (Gallis & Turner, 2019). Whether the

actual probability differences are large or small, odds

ratios obscure the magnitude of those changes because

they are relative measures (Xiao et al., 2021). If a logistic

regression had similar covariate odds ratios, a reader

might infer that the effects on probability were the same.

PUBLIC ADMINISTRATION REVIEW 1219