Three Validation Tests of the Shor–McCarty State Legislator Ideology Data

• Author: Megan L. Remmel,Jeffery J. Mondak
• Date: 01 July 2020
• Published date: 01 July 2020
• DOI: 10.1177/1532673X20914621
• Subject Matter: Articles

914621APRXXX10.1177/1532673X20914621American Politics ResearchRemmel and Mondak
research-article2020
Article
American Politics Research
2020, Vol. 48(4) 523 –528
Three Validation Tests of the Shor–
© The Author(s) 2020
Article reuse guidelines:
sagepub.com/journals-permissions
McCarty State Legislator Ideology Data
https://doi.org/10.1177/1532673X20914621
DOI: 10.1177/1532673X20914621
journals.sagepub.com/home/apr
Megan L. Remmel1 and Jeffery J. Mondak2
Abstract
The Shor–McCarty state ideology data provide a valuable resource for students of state legislative politics, but, as with
any important new measures, the properties of those data warrant independent examination. Toward that end, we report
results from three validation tests, two tests of convergent validity and one test of construct validity, with all tests focused on
Shor and McCarty’s legislator, or individual-level, data. Collectively, results of the validation exercises offer strong evidence
the Shor–McCarty measures provide valid representations of state legislator ideology.
Keywords
ideology, legislators, personality, state politics, validity
Due to the centrality of ideology in politics, efforts to mea-
Three matters warrant review. The first concerns what
sure ideological orientations carry inherent significance.
ideal points represent versus what they are presumed to rep-
Armed with such data, analysts can examine polarization,
resent. In an insightful analysis, Clinton et al. (2016) note
bias, representation, and more. The state politics field has
that nothing in the construction of ideal point measures from
gained a dramatic expansion in its capacity to explore such
roll-call data necessarily or directly reflects ideology. Instead,
topics due to the production of state legislator ideology
because the measures capture the relative policy positions of
scores by Shor and McCarty (Shor, 2018; Shor et al., 2010;
Democrats and Republicans, analysts infer something about
Shor & McCarty, 2011). Drawing on information from roll-
ideology. Second, various sources of heterogeneity can com-
call votes and National Political Awareness Test (NPAT)
plicate comparisons across time. Because ideal point mea-
responses, the authors have produced ideology scores for
sures draw on roll-call votes, variation in the agenda, whether
legislators, along with annual measures for each state’s upper
due to the dominant concerns of the era or the momentary
and lower legislative chambers. Although the authors have
preferences of the majority party, may shape our understand-
been transparent regarding both how their measures are con-
ing of what we assume to be ideology.1 Important for present
structed and how the properties of those measures have been
purposes is that the move from the national to the state level
tested, we believe there remains value in reconsidering these
adds heterogeneity across space. Even when examined at the
issues with fresh eyes. As analysts unrelated to the construc-
same time, states differ in their chief issue concerns, and leg-
tion of the Shor–McCarty ideology data set, our task in the
islatures differ in both structure and party control. Third,
present report is to carry out a series of validation exercises,
turning to the selection of validation exercises, Bishin (2003)
with tests focused on the individual-level legislator data.
notes that ideological measures such as DW-Nominate and
interest group reports are performance-based in that the raw
Why Validate?
material that informs them is how legislators voted on bills.
Bishin thus argues that alternate representations of ideology
When any measure achieves wide use, we see intrinsic value
not premised on the same task performance should be used in
in external validation efforts due to the potential of such exer-
validity tests.
cises to heighten the scholarly community’s confidence in the
Together, these points help to establish both why valida-
measure under consideration, or, alternately, to caution that
tion tests of the Shor–McCarty legislator measures would be
scholars may have ventured down the wrong path. The Shor–
McCarty approach implements at the state level a measure-
1Bradley University, Peoria, IL, USA
ment procedure that builds on the Dynamic Weighted
2University of Illinois at Urbana–Champaign, USA
(DW)–Nominate technique developed for measurement of
ideal points in the U.S. Congress (Poole & Rosenthal, 2011).
Corresponding Author:
Jeffery J. Mondak, University of Illinois at Urbana–Champaign, Champaign,
Hence, past assessments of DW-Nominate provide a roadmap
IL 61801, USA.
for why and how to evaluate their state-level counterparts.
Email: jmondak@illinois.edu

---

524
American Politics Research 48(4)
useful and how such tests should be constructed. Regarding
exhaustive database that includes millions of records on
the latter, our objective is to develop validation tests that (a)
campaign contributions from both individuals and political
do not rely on alternate measures of legislator behavior, and
action committees (PACs). Bonica reports that, for the U.S.
(b) that account for at least some sources of heterogeneity
Congress, his individual-level measures are correlated with
both between legislators and between states. We restrict our
the common-space DW-Nominate values at a level of .92. At
analyses to the Shor–McCarty legislator measures. Past
the state level, Bonica reports aggregate correlations between
research has assessed the Shor–McCarty legislature data,2
his measures and those of Shor and McCarty of .86 for upper
but we are unaware of similar exercises directed toward
chambers and .83 for lower chambers. We add tests for indi-
assessment of the individual-level legislator scores.
vidual state legislators.6
The absence of positive correlations among the three indi-
Constructing Validation Tests
vidual-level measures would be highly unexpected. This
raises a question of what level of correlation would be
In devising validation tests, our central challenge was the
acceptable in validation exercises such as these. Surely, the
location of appropriate test variables. With focus on legisla-
mere presence of a statistically significant correlation is not
tors, our goal was to develop tests of both convergent and
sufficient. We have two thoughts on this. First, given the
construct validity. Tests of convergent validity make use of
absence of an ironclad numerical standard, a claim of “the
...