Indexes as a Metric for Drug and Alcohol Policy Evaluation and Assessment

• Author: Alison Ritter,Vivienne L. Moxham‐Hall
• DOI: http://doi.org/10.1002/wmh3.218
• Published date: 01 March 2017
• Date: 01 March 2017

Indexes as a Metric for Drug and Alcohol Policy

Evaluation and Assessment

Vivienne L. Moxham-Hall and Alison Ritter

Indexes are measurement tools used to capture the multi-dimensionality of complex issues and have

been used increasingly in the contemporary evaluation of policies including in the drug and alcohol

f‌ield. This article systematically reviews all published alcohol and illicit drug indexes to date in

terms of their primary purpose, design and methodology, and application in practice. Twenty-four

unique indexes (across 61 papers) were found which f‌it the review criteria. When grouped on the

basis of output- or input-driven indicators, the indexes broadly fell into three categories: Harm

Indexes (output-driven), of which there were 10 and were primarily concerned with measuring the

consequences of illicit drug use; Policy Indexes (input-driven) of which there were seven and were

predominantly concerned with alcohol control policy measurement; and Treatment Need Indexes

(neither purely output or input-driven) of which there were also seven, which measured the extent

of treatment need across multiple substances. There was substantial methodological diversity in

terms of normalization, weighting, and aggregation methods. The f‌indings demonstrate silos in the

purpose, design, and use of indexes.

KEY WORDS: drug policy, alcohol control, index measurement

Introduction

Composite indicators, referred to here as indexes, have been used extensively

in research and policy making due to their ability to integrate complex

multidimensional concepts into a single comparable metric. The categorization of

widely different types of data into a single index facilitates a number of different

types of comparisons: comparisons over time (trends) within one place,

comparisons between two or more places, and comparisons between substances.

In policy evaluation, an index can be used as an outcome measure to compare

policy implementation, effectiveness, or enforcement, making it a valuable

research, political, and communication tool (Nardo & Saisana, 2008; OECD, 2008;

Saisana & Tarantola, 2002).

Indexes are already widely used across a number of different f‌ields. Some

examples of well-known indexes include the Human Development Index (HDI)

World Medical & Health Policy, Vol. 9, No. 1, 2017

103

1948-4682 #2017 Policy Studies Organization

Published by Wiley Periodicals, Inc., 350 Main Street, Malden, MA 02148, USA, and 9600 Garsington Road, Oxford, OX4 2DQ.

(UNDP, 2015), the Ecological Footprint (Borucke et al., 2013), and the Consumer

Price Index (CPI) (ILO et al., 2004). The HDI, developed by the United Nations

Development Program, is a well-recognized and highly utilized index that has

enabled the assessment of intercountry development levels through the integra-

tion of life expectancy and adult literacy variables. Changes in HDI values and

the subindexes within the HDI allow for the monitoring of changes in a country’s

development. The HDI also allows for the identif‌ication of “policy gaps” to direct

resources and national priorities and thus can stimulate national policy dis-

cussions. Some of the advantages and disadvantages of indexes have been

summarized well by Saisana and Tarantola (2002). Benef‌its of an index include its

ability to condense complex and multidimensional issues to support the decisions

of policy and program advisors, enable a “big picture” view of complicated

issues, facilitate rankings between jurisdictions, and attract public interest in

country performance and progress over time. However, drawbacks of indexes

include the number of subjective judgements that need to be made in their

construction (indicator selection, weighting, etc.), the ability to draw simplistic

policy conclusions from index values, the likelihood of poorly constructed or

poorly interpreted indexes sending misleading and nonrobust policy messages,

and the quantity of data needed to produce a statistically signif‌icant analysis

(Saisana & Tarantola, 2002). Further to this the conversion of all the complex

indicators into a single, summable f‌igure (or common metric) is fraught with

diff‌iculties, and the utility of such a measure has been questioned (Ritter, 2007;

Rolles & Measham, 2011; Zabransky, 2015). The methodological choices in the

construction of an index such as the component domains, the common metric

conversion, the weightings, and analytic approaches can have a signif‌icant impact

on the index outcomes (Jacobs, Goddard, & Smith, 2007; Paruolo, Saisana, &

Saltelli, 2013). An index result may leave little room for the recognition or

interpretation of changes. For example, a rise in one indicator could mask a fall in

another indicator and not affect the net index value (Ritter, 2009). The lack of

ability to make any nuanced interpretation of index values has led to reviewers

suggesting that indexes should be used in conjunction with a contextual analysis

to determine how sociocultural changes may bear on “harm” or “policy” (Rolles

& Measham, 2011).

Policy evaluation is the central concern of indexes: their utility is the extent to

which they can serve as a benchmarking or measurement tool to evaluate policy.

Previous categorizations of indexes in drug and alcohol and policy analysis more

broadly have identif‌ied two types of policy evaluation indexes: those that are

input-driven, and those that are output-driven (Sevigny & Saisana, 2013). Input-

driven indexes are concerned with combining multiple policy types (across laws,

regulations, and programs) and the associated implementation of those policies to

derive a composite index of policy. On the other hand, output-driven indexes are

concerned with measuring the consequences of policies, such as the health or

crime outcomes, again combining multiple indicators into one metric. There are

substantial challenges in developing either an input (policy) or an output (harm)

index, including the decisions around what to focus on (in our case, e.g., which

104 World Medical & Health Policy, 9:1