Good Data, Bad Data

• Author: Jason Tashea
• Pages: 32-33

32 || ABA JOURNAL SEPTEMBER 2018

Business of Law

PHOTO BY ALEXA BEDWELL, CHARLES STUDIO; AMIAK/SHUTTERSTOCK.COM

Good Data, Bad Data

When it comes to predicting outcomes in litigation ,

algorithms are only as good as their underlying information

By Jason Tashea

One way to put a check on data and

algorithmic sys tems is through audit-

ing, explains Ch ristian Sandvig, a

professor at the University of Michiga n.

Like a fi nancial audit, thi s process

can determi ne, among other things,

whether there are data qua lity issues,

an algorithm meet s legal standards,

or the tool is creating un intended or

biased consequences. Sa ndvig says that

audits are not “just inve stigative or

punitive,” but they should be used by

companies to monitor their ow n oper-

ations. Currently ther e are no indus-

try standa rds regarding what or who

should be audite d and how, he says.

CALL AN AUDITOR

Whether or not this cal l for oversight

is heeded, companies in t he legal mar-

ket see growing dema nd for better

analytic s. Justly, a litigation analytics

startup founded i n New York City in

2015, is one.

CEO Laurent Wiesel ex plains that

Justly aggregate s litigation data to

“forecast ti me and cost” in business

cases. Subscr iptions start at $1,500

a month. Beyond forecasting , Justly

provides reports on cl ients and op-

posing counsels. The predic tions are

derived from a data

set of 17.5 million

state and federal

cases going back t o

2003.

Informed by

his work as an

attorney, Wies el

says Justly improves

on the tradit ional,

“labor intensive” law fi rm approach

to estimate the ne eds of litigation.

When asked, he says that Just ly has not

undertaken thi rd-party auditing, but

that he was open to the idea .

The co-founders of Legal ist take a

di erent approach with their data ,

which is used to infor m litigation

fi n a n c i n g .

Eva Shang, CEO and co-founder

of Legalist , says her company’s data

allows it to look “at the likelihoo d” a

case wil l win at trial. The data-d riven

There is a small

law fi rm in A nnapolis,

Maryland, harnessing

big data.

Emanwel Turnbull,

one of two attorneys at t he Holland

Law Firm, uses a un ique, statewide

database to ga in early insight into

his cases.

The database, which h as over

23 million civi l and criminal cases

from Case Sea rch, the state judiciary ’s

document portal, a llows Turnbull to

analyze the beh avior of an opponent,

check a process ser ver’s history or

learn whet her an opposing attorne y

has an unscr upulous track record.

“Back in the old days, they ’d have

an intern or secret ary manually go

through judiciar y Case Search to

try and fi nd these things out,” says

Tur nb u ll .

Now, it takes seconds.

While saving time , the database

does not provide dispositi ve evidence

because the infor mation refl ects input

and clerical err ors. Even with this

shortcomi ng, Turnbull say s the data-

base points him “in t he direction of

further rese arch,” which he uses in aid

of his clients.

Turnbull’s work refl ects data’s

growing role in law. With increa sed

computing power and more materia l,

law fi rms and c ompanies are evolving

the practice and busi ness of litigation.

However, experts say these projec ts

can be hindered by the qua lity of data

and lack of oversight.

Many data-dr iven projects promise

e ciency a nd lower legal costs for

fi rms and cl ients.

Littler Mendelson developed

CaseSmar t, launched in 2010, “to

provide better va lue” to clients with

leaner legal budgets af ter the reces-

sion. The project is a repository for

data created by a client ’s legal issues,

explains Scot t Forman, a shareholder

in Miami.

Through an intake t eam in Kansas

City, Missouri, employment law fi rm

Littler can c apture data that, on

aggregate, p oints to specifi c company

policies or employees that create leg al

problems. Clients can ta ke these out-

puts and decide to increa se training

or change a policy to reduce the r isk

of future litigation, for ex ample.

While the data is goo d for trend-

spotting, it is not used t o build pre-

dictive algorith ms. Forman wants to

include that capacity at s ome point;

however, he says there is not enough

“clean data” to train a n algorithm.

Algorithms a re built on structured

data, which mea ns data quality

impacts acc uracy. If your data is

of low quality, then it’s as the saying

goes: “garbage in, garbage out .”

Travis Lenkner, manag ing partner

at Keller Lenkner in C hicago and

senior adviser at Bur ford Capital, a

litigation fi na nce fi rm, says the use

of algorithms and dat a in litigation

fi nance “w ill be measured and limit ed

by the availability of d ata.”

While Burford’s work is informed by

data, Lenk ner is skeptical that current

court data qua lity allows for good pre-

dictions. “I magine your county c ourt-

house,” he says, and think about its

data situation. You realize “t here’s a lot

of work to do.”

Technology

“Back in the old days, they’d have an

intern or secretary manually go through

judiciary Case Search to try and fi nd

these things out.”

—Emanwel Turnbull