Apples and Oranges: An International Comparison of the Public's Experience of Justiciable Problems and the Methodological Issues Affecting Comparative Study

• Published date: 01 March 2016
• Date: 01 March 2016
• DOI: http://doi.org/10.1111/jels.12097

Apples and Oranges: An International

Comparison of the Public’s Experience of

Justiciable Problems and the

Methodological Issues Affecting

Comparative Study

P. Pleasence, N. J. Balmer and R. L. Sandefur*

Since the mid-1990s, at least 28 large-scale national surveys of the public’s experience of

justiciable problems have been conducted in at least 15 separate jurisdictions, reflecting

widespread legal aid reform activity. While the majority of these surveys take their structure

from Genn’s Paths to Justice survey (1999), they vary significantly in length, scope, mode of

administration, types of problems included, survey reference period, data structure, data

analysis, and question formulation. This article draws on surveys from across the world,

contrasting their methodologies, comparing their headline findings, and setting out the

potential for bias as a consequence of methodological variation. The article also presents

findings from five online experiments testing the impact of various question formulations

on problem prevalence, use of advice, and formal processes. Specifically, the experiments

test whether varying the reference period, describing problems as “legal,” offering detailed

as opposed to simple problem descriptions, and describing problems as “difficult to solve”

had an impact on reported prevalence of justiciable problems, and whether presenting lists

as opposed to a series of individual questions had an impact on reported use of advice and

processes. The experiments demonstrated that modest differences in question formulation

yield significantly different results. Specifically, alteration of survey reference period did

not result in a proportional change in reported problem prevalence, introducing problems

as either “legal” or “difficult to solve” significantly reduced reported prevalence, and

introducing use of advice/processes as multiple questions rather than as lists significantly

increased reported use. The risks involved in comparative analysis (and particularly in

looking beyond methodology when attempting to explain jurisdictional variation) are

discussed. In relation to future studies, the importance of understanding the impact of

methodological change, learning the lessons of the past, making technical details

transparent, and making data available are highlighted.

*Address correspondence to Pascoe Pleasence, University College London, Bentham House, Endsleigh Gardens,

London, WC1H 0EG, UK. Balmer is at University College London; Sadefur is at University of Illinois, Cham-

paign-Urbana.

50

Journal of Empirical Legal Studies

Volume 13, Issue 1, 50–93, March 2016

I. INTRODUCTION

Since the mid-1990s, large-scale national surveys of the public’s experience of justiciable

problems—problems that raise civil legal issues, whether or not this is recognized by

those facing them and whether or not any action taken to deal with them involves the

legal system (Genn 1999:12)—have been conducted in at least 15 separate jurisdictions:

Australia, Bulgaria, Canada, England and Wales, Hong Kong, Japan, Moldova, the

Netherlands, New Zealand, Northern Ireland, Scotland, Slovakia, Taiwan, Ukraine, and

the United States (Table 1).

1

Extensive subnational surveys have also been conducted in

China (Michelson 2008) and Russia,

2

along with many other subnational surveys across

the jurisdictions just listed.

3

These surveys, commonly known as “legal needs” surveys, have their ultimate ori-

gins in Clark and Corstvet’s (1938) landmark study of “how the needs of the community

for legal service were being met” in Connecticut during the 1930s recession by the U.S.

Bar. However, although Clark and Corstvet anticipated that similar surveys would become

commonplace,

4

few further surveys were conducted in the decades that followed. Only in

the 1990s did such research “gain considerable momentum” (Coumarelos et al. 2012:1)

following the conduct of high-profile national surveys in, first, the United States (Reese

& Eldred 1994), then England and Wales (Genn 1999), New Zealand (Maxwell et al.

1999), and Scotland (Genn & Paterson 2001).

5

Momentum has been fueled by wide-

spread legal aid reform activity across the globe, with the introduction of civil legal aid

in countries such as Bulgaria and Moldova,

6

the expansion of civil legal aid in countries

such as Taiwan,

7

and substantial (and ongoing) reform of established civil legal aid

schemes, such as that in England and Wales (Lord Chancellor’s Department 1998; Legal

Services Commission 2006; Ministry of Justice 2010).

1

Reese and Eldred (1994), Genn (1999), Maxwell et al. (1999), Genn and Paterson (2001), Pleasence et al.

(2004c, 2010, 2011), Currie (2005, 2007a, 2009), GfK Slovakia (2004), Van Velthoven and ter Voert (2004),

Dignan (2006), Ignite Research (2006), Pleasence (2006), Murayama (2007), Sato et al (2007), Asia Consulting

Group and Policy 21 (2008), Tamaki (2009), Van Velthoven and Haarhuis (2010), Chen et al. (2012a), Coumera-

los et al (2012). Details of the Bulgarian and Moldovan surveys kindly provided by Martin Gramatikov.

2

Details of the Russian survey kindly provided by Martin Gramatikov.

3

For example, over the past two decades, surveys have been conducted in at least 16 of the 50 U.S. states, as well

as in other jurisdictions such as Australia (Coumarelos et al. 2006) and Canada (Baxter et al. 2012).

4

Clark and Corstvet (1938:1273) hoped that “a substantial number of local surveys” would be carried out that,

together, would “afford a picture fairly representative of conditions across [the United States].”

5

Themselves influenced by an earlier wave of surveys including the earlier American Bar Association and Ameri-

can Bar Foundation survey (Curran & Spalding 1974) and the Civil Litigation Research Project (Trubek et al.

1983), which also led to efforts of replication (e.g., Bogart & Vidmar 1990).

6

Legal Aid Act 2005 (Bulgaria); Legal Aid Act 2007 (Moldova).

7

Legal Aid Act 2004.

51International Comparison of Experience with Justiciable Problems

Reflecting concerns about the preclusion of problems that “may not be seen” by

respondents to raise legal issues (Maxwell et al. 1999:17), most recent surveys have

adopted the practice of presenting justiciable problems as simple sets of circumstances,

Table 1: National Legal Needs Surveys (Last 20 Years)

Country Study Date Size

%11

Problems

% Lawyer

Use

Australia Law Australia Wide Survey 2008 20,716 46 11

Bulgaria Access to Justice and Legal

Needs Bulgaria

2007 2,730 46 15

Canada National Survey of Civil

Justice Problems

2004 4,501 48 NR

2006 6,665 45 12

2008 7,002 55 NR

England and

Wales

Paths to Justice 1997 4,125 39 27

Civil & Social Justice Survey

(CSJS)

2001 5,611 36 14

2004 5,015 33 16

2006–

2009

10,537 36 12

Civil & Social Justice Panel

Survey (CSJPS)

2010 3,806 33 7

2012 3,911 33 6

Justiciable Problems

Resolution Survey

2014–

2015

10,000 NR NR

Hong Kong Demand & Supply of Legal &

Related Services

2006 10,385 19/32/40* <6

Japan National Survey of Everyday

Life & the Law

2005 12,408 19 12

Access to Legal Advice:

National Survey

2006 5,330 37 4

Everyday Life and Law 2007 5,500 55 NR

Moldova Met and Unmet Legal

Needs in Moldova

2011 2,489 22 4

Netherlands Paths to Justice in the

Netherlands

2003 3,516 67 NR

2009 5,166 61 11

2013 5,773 NR NR

New Zealand Legal Advice & Assistance

Survey

1997 5,431 51 18

Unmet Legal Needs &

Access to Services

2006 7,200 26 15

N. Ireland Northern Ireland Legal

Needs Survey

2005 3,361 35 16

Scotland

a

Paths to Justice Scotland 1998 2,684 26 29

Slovakia Legal Needs in Slovakia 2004 1,085 45 22

Taiwan Legal Dispute Settlement

Behaviour

2011 5,601 57 5

Ukraine Legal Capacity of the

Ukrainian Population

2010 2,463 54 5

United States Comprehensive Legal

Needs Study

1993 3,087 50 24

a

The experience of civil legal problems has also been asked about through the 2008/2009, 2009/2010, 2010/

2011, 2012–2013, and 2014–2015 Scottish Crime and Justice Surveys.

NOTES:NR5not reported. *1 year/5 years/lifetime.

52 Pleasence et al.