Rethinking Measurement Equivalence in Comparative Political Research

• Published date: 01 September 2024
• DOI: http://doi.org/10.1177/10659129241241753
• Author: Kirill Zhirkov,Christian Welzel
• Date: 01 September 2024
• Subject Matter: Articles

Article

Political Research Quarterly

2024, Vol. 77(3) 696–712

© The Author(s) 2024

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/10659129241241753

journals.sagepub.com/home/prq

Rethinking Measurement Equivalence in

Comparative Political Research

Kirill Zhirkov

1

and Christian Welzel

2

Abstract

A growing number of studies test key cultural constructs used in comparative political research for invariance across

countries, only to declare these constructs incomparable if the invariance tests fail. The assumption underlying this kind

of conclusion is that between-country differences in within-country factor solutions ref‌lect cross-cu ltural inequivalence

of the respective survey items. Using a Bayesian approach, we explore variatio n in measurement quality across countries

focusing on sexual emancipation. We f‌ind that between-country differences in factor solutions ref‌lect littl e more than

imbalances in within-country dispersions of the construct items. As we demonstrate, discrete and bounded response

scales used in standard surveys enforce a mathematical relationship between extreme country means, low within-

country dispersions, and weak factor solutions. We formalize this insight by introducing a between–within model of

cultural variation. Our contribution calls into question the practice of declaring constructs to be incom parable based on

failed invariance tests.

Keywords

Bayesian estimation, conf‌irmatory factor analysis, cross-national comparisons, measurement equivalence , multilevel

modeling, value orientations

Over the last two decades, comparative political research

has seen a soaring interest in measurement equivalence.

For some time, scholars from Przeworski and Teune

(1966) to Adcock and Collier (2001), have been calling

for greater attention to cross-national comparability in

measures used to capture key theoretical concepts. Now,

since eff‌icient estimation techniques have become widely

available, this call is f‌inally being answered. Research on

the measurement of macro-level political concepts, such

as democracy, has existed for more than two decades

(Bollen 1993), with recent developments related to ap-

plying Bayesian methods (Treier and Jackman 2008).

Since the expansion of cross-national public opinion

research throughout the world, attention has shifted to-

ward measuring attitudes across countries.

One of the core concepts in this literature, introduced to

political science by Davidov (2009), is measurement

equivalence. Its core idea is both concise and strict: in-

variance of measurement models across countries is a

necessary condition for a meaningful comparison. The

latest applications of this framework in comparative po-

litical science concern the constructs of self-expression

values by Inglehart (1997), and emancipative values by

Welzel (2013). There is evidence that neither of these

constructs is satisfactorily invariant across the full set of

countries for which data are available (Aleman and Woods

2016;Sokolov 2018). This f‌inding has far-reaching im-

plications: it potentially invalidates key constructs used in

the study of the cultural foundations of democracy,

questioning seminal work from Lipset (1959) and

Almond and Verba(1963) to Putnam (1993) and Inglehart

and Welzel (2005).

1

Welzel and Inglehart (2016) counter this critique by

demonstrating that variation in the within-country cohe-

sion of emancipative values has no effect on these values’

functioning across countries. The authors explain this as

ref‌lecting what they call “compositional substitutability,”

which involves three principles: (1) similar overall scores

University of Virginia, Charlottesville, USA

2

Leuphana University Lueneburg, Germany

Corresponding Author:

Kirill Zhirkov, Department of Politics, University of Virginia, PO Box

400787, Charlottesville, VA 22904, USA.

Email: pkv5cd@virginia.edu

on a multi-item index can derive from different combi-

nations of item-specif‌ic scores; (2) similar overall scores

are compatible with different cohesion patterns among the

items; (3) when these differences in inner composition

leave the overall construct’s cross-cultural functioning

unaffected, the compositional differences are functionally

equivalent substitutes to each other.

In another contribution to this debate, Sokolov (2018)

claims that measurement error on the country level could

inf‌late estimated aggregate relationships between multi-

item constructs and other variables. He also suggests that,

within the modernization–emancipation framework, value

orientations represent higher-order theoretical abstrac-

tions with respect to their specif‌ic indicators. Therefore,

the corresponding measurement models should be con-

structed as latent variables that do not tolerate composi-

tional substitutability.

Within the latent variable approach to measurement

(also known as dimensional or ref‌lective), the standard

method of testing for cross-country invariance is multi-

group conf‌irmatory factor analysis (MG CFA; Joereskog

1971). The bottom line of MG CFA is simple: it compares

the constrained version of the measurement model that

assumes equality of parameters across groups (such as

countries) against the unconstrained one. A signif‌icant

difference in the models’f‌it indicates a failure to conf‌irm

measurement invariance, in which case the underlying

construct is considered incomparable. This approach is

notably sensitive to even minor deviations from perfect

invariance. In large samples with substantial numbers of

comparisons, the strict invariance hypothesis is almost

certain to be rejected within the MG CFA framework

(Selig, Card, and Little 2008). This problem prompts

some authors to propose methods of detecting cases when

non-invariance is consequential—that is, when it leads to

signif‌icant biases in estimates (Oberski 2014).

An important limitation of MG CFA is its ignorance of

the specif‌ic patterns of non-invariance, beyond just stating

its presence (Meitinger 2017). Currently, only few studies

offer substantive explanations for existing measurement

non-invariance across cultures. For instance, Meuleman

and Billiet (2011) demonstrate that attendance of religious

services represents an inferior indicator of overall reli-

giosity in Muslim societies compared to Christian ones

due to culturally conditioned gender differences. How-

ever, this explanation derives from the researchers’

knowledge about the studied cases; it is not the result of

MG CFA as a method.

Def‌ining the Problem

The literature on cross-cultural measurement using MG CFA

revolves around three concepts that are often used inter-

changeably: equivalence, invariance, and comparability.

Before developing our reasoning, let us brief‌lyclarifyhow

we use these terms throughout this article. Equivalence,

which in the most general sense means the property of being

equal or identical, relates to the measurement instrument,

such as a particular survey item. It means that the instrument

measures the same theoretical concept whereby “respon-

dents from different groups that have the same position on a

trait of interest should provide a similar response”(Davidov

et al. 2014,58).Invariance, in turn, is a property of the multi-

group measurement model, meaning that model parameters

(such as factor loadings) are equal across groups—or, more

precisely, that the null hypothesis of parameters’equality

cannot be rejected. Applying the invariance test assumes that

equivalence of measurement instruments necessarily re-

quires invariant measurement models, so that non-invariance

inevitably implies non-equivalence. Finally, comparability is

a property of the measured quantities—that is, numerical

values corresponding to the constructs of interest. It refers to

the possibility of making valid inferences from using these

values in empirical analysis. Within the MG CFA frame-

work, successful invariance tests indicate measurement

equivalence, which in turn is a necessary condition for

comparability.

Thus, invariance, equivalence, and comparability op-

erate on hierarchically ordered levels of abstraction of the

same idea, for which reason these terms are often used

interchangeably. In the present article, we question this

interchangeability, especially the automatic conclusion

that measurement non-invariance indicates non-

equivalence. We demonstrate that measurement non-

invariance (non-equality of model parameters, such as

factor loadings, across countries) is not a suff‌icient sign of

non-equivalence (presence of country-level biases). In-

stead, within-country CFA models can fail for purely

technical reasons that result from the practice of using

discrete and bounded response scales in cross-national

surveys. We also argue that, since comparative political

science is primarily interested in estimating generalizable

relationships between mass attitudes and other country-

level variables, the link between measurement equiva-

lence and comparability is often misinterpreted. To inf‌late

estimates of these relationships, measurement error must

be systematic with respect to the covariates of interest—

but we show that, at least in the case of sexual emanci-

pation, it is not.

Study Outline

Our contribution advances the debate by exploring

whether patterns of measurement non-invariance follow

substantive economic and cultural differences.

2

In so

doing, we use a multilevel CFA model with random

loadings (Meuleman and Schlueter 2018), that relies on

the Bayesian approach to estimation (Jackman 2009).

Zhirkov and Welzel 697