Using Multitrait-Multimethod (MTMM) Techniques to Examine the Convergent and Discriminant Validity of Social Disorder

• Published date: 01 August 2018
• DOI: 10.1177/0022427818771109
• Date: 01 August 2018
• Subject Matter: Articles

Article

Using Multitrait-

Multimethod (MTMM)

Techniques to

Examine the Convergent

and Discriminant Validity

of Social Disorder

Sue-Ming Yang

1

, Joshua C. Hinkle

2

,

and Laura A. Wyckoff

3

Abstract

Objectives: Disorder has been measured by various data sources; however,

little attention has been given to comparing the construct validity of differ-

ent measures obtained through various methods in capturing social disor-

der and related phenomena. Methods: The multitrait-multimethod approach

was used to triangulate the consistency between social disorder, prostitu-

tion and drug activity across resident surveys, systematic social observa-

tions, and police calls for service data. Results: Prostitution and drug activity

showed convergent validity, while there was little evidence that social dis-

order was consistently measured across the three methods. None of the

three social problem me asures showed high disc riminant validity. Dr ug

1

Department of Criminology, Law and Society, George Mason University, Fairfax, VA, USA

2

Department of Criminal Justice and Criminology, Georgia State University, Atlanta, GA, USA

3

U.S. Department of Justice, Bureau of Justice Assistance, Washington, DC, USA

Corresponding Author:

Sue-Ming Yang, Department of Criminology, Law and Society, George Mason University, 4400

University Drive, MS 4F4, Fairfax, VA 22030, USA.

Email: syang10@gmu.edu

Journal of Research in Crime and

Delinquency

2018, Vol. 55(5) 571-608

ªThe Author(s) 2018

Reprints and permission:

sagepub.com/journalsPermissions.nav

DOI: 10.1177/0022427818771109

journals.sagepub.com/home/jrc

activity seems to have highest trait-specific discriminant validity across

measures, and prostitution is the most iden tifiable measure across data

sources. Social disorder was found to have low discriminant validity. How-

ever, the agreement between databases varies across the type of soc ial

problems. Conclusions: Social disorder appears to the most difficult concept

to define and measure consistently. The lack of correspondence across data

sources cautions against the use of a single source of information in studying

disorder. Future studies should explore the factors that shape perceptions

of disorder and how to best measure disorder to test the broken windows

thesis and related concepts.

Keywords

broken windows, disorder, multitrait-multimethod, construct validity

Disorder has a long history in the study of urban communities (Hunter 1978;

Jacobs 1961; Sennett 1970, 1989). Sometimes referred to as incivilities,

disorder refers to social nuisance behaviors, such as loitering and public

drinking, as well as dilapidated physical conditions in a community (Samp-

son and Raudenbush 1999; Skogan 1990; Taylor 1999). Scholars have long

argued that disorder has important implications for understanding crime and

fear of crime. Various theories view disorder as an indicator of social

disorganization and/or a product of weakened informal social control

(Sampson, Raudenbush, and Earls 1997; Shaw and McKay 1942; Taylor

1999). Disorder has also been central in studies attempting to understand

fear of crime after numerous early studies found limited linkage between

crime rates, victimization rates/risk, and fear (Ferraro 1995; Garofalo and

Laub 1978; Wilson 1975).

However, the topic rose to prominence with the broken windows idea,

advanced by Wilson and Kelling (1982) in their seminal article in the

Atlantic Monthly, which suggested police could best fight crime by target-

ing minor social ills such as loitering and public drinking and rapidly

cleaning up physical dilapidation such as litter and graffiti. Their idea that

disorder could send signals that inspire community decline has been further

advanced upon by other scholars over the years (e.g., Innes 2004, 2014;

Skogan 1990), and police leaders latched on to this idea with tactics based

upon it being most prominently deployed in New York City (Bratton and

Knobler 1998; Giuliani and Kurson 2002). However, actual empirical evi-

dence on both the disorder-crime nexus (see Hinkle 2013) and the

572 Journal of Research in Crime and Delinquency 55(5)

effectiveness of disorder-focused police tactics (for review, see Braga,

Welsh, and Schnell 2015) is thoroughly mixed. In our view, much of this

lack of consensus can be traced to the relatively underdeveloped under-

standing of disorder as a measure.

Indeed, despite this long history of theoretical and evaluation research,

comparatively limited attention has been given to the measurement of dis-

order. In particular, several approaches have been used to measure disorder,

and there has been little attention given to understanding the strength s,

weaknesses, and reliability/validity of measures obtained using different

methods. The most prevalent methods of measuring diso rder have been

surveys of perceptions of disorder and systematic social observation (SSO)

of places by researchers—with official data such as police calls for service

(CFS) or incident reports also used in some studies. All three methodologies

have been used to create various scale measurements of disorder including

omnibus measures of overall diso rder and separate scales of social and

physical disorder (see Skogan 1990). This variation in types of measure-

ment and definitions of disorder has perhaps muddied our theoretical under-

standing of the broken windows thesis (Wilson and Kelling 1982), and the

police tactics emerging from the idea as we lack knowledge about whether

these methodologies are in fact measuring the same underlying construct.

While consistent and accurate measurement and de finition of central

concepts are crucial for generating knowledge in any area, it is particularly

important when attempting to measure constructs that lack clear, consistent

definitions. In disorder research, scholars have noted that, unlike crime,

disorder is not a concrete concept that is easily defined (Harcourt 2001;

Sampson 2012). In his book Disorder and Decline, Skogan (1990) argued

that the concept of disorder is ambiguous and can represent a variety of

meanings (see also Harcourt 2001; Innes 2004, 2014). Importantly, he noted

the need to differentiate between friendly “active uses of the environment”

(not disorder) and disorderly behavior, which bothers the reside nts of a

neighborhood. The former may suggest a very well-connected neighbor-

hood, while the latter may represent a disorganized neighborhood. More-

over, in 2015, Skogan compared the definition of disorder used by the

Home Office in England, a definition commonly used in the United States,

and concluded that disorder is socially defined and takes up different mean-

ings across different cultures. Skogan’s point highlights an important

insight. Rather than being a concrete, objective phenomenon that is easily

defined and observed, the perception of disorder is context related. This is

likely particularly true for some forms of social disorder. For instance, some

young people hanging around a street corner might be perceived as loitering

Yang et al. 573