Bribing the Machine: Protecting the Integrity of Algorithms as the Revolution Begins

• Published date: 01 December 2019
• Date: 01 December 2019
• Author: Philip M. Nichols
• DOI: http://doi.org/10.1111/ablj.12151

American Business Law Journal

Volume 56, Issue 4, 771–814, Winter 2019

Bribing the Machine: Protecting

the Integrity of Algorithms as

the Revolution Begins

Philip M. Nichols*

In the Industrial Revolution, machines took on the burden of physical labor; in

the Big Data Revolution, machines are taking on the tasks of making decisions.

Algorithms are the rules and processes that enable machines to make those deci-

sions. Machines will make many decisions that affect general well-being. This

article addresses a threat to the efficacy of those decisions: the intentional distor-

tion or manipulation of the underlying algorithm so that machines make deci-

sions that benefit self-interested third parties, rather than decisions that enhance

public well-being. That threat has not been recognized or addressed by legal

thinkers or policy makers. This article first examines the lifecycle of an algorithm,

and then demonstrates the likelihood that self-interested third partieswill attempt

to corrupt the development and operation of algorithms. The article then argues

that existing mechanisms cannot protect the integrity of algorithms. The article

concludes with a discussion of policies that could protect the integrity of algo-

rithms: transparency in both the development of and the content of algorithms

that affect general well-being and holding persons who corrupt the integrity of

such algorithms accountable. Just as the Industrial Revolution eventually

improved the quality of life for many, so too does the Big Data Revolution offer

enhancement of general well-being. That promise, however, will only be realized

if policy makers take action to protect the integrity of underlying algorithms now,

at the beginning of the revolution.

*Joseph Kolodny Professor of Social Responsibility in Business, Professor of Legal Studies

and Business Ethics, The Wharton School of the University of Pennsylvania. Drafts of this

article were presented at the Colloquium on the Law and Ethics of Big Data at Indiana Uni-

versity, at the Academy of Legal Studies in Business, and at the CanadianAcademy of L egal

Studies in Business. The author thanks everyone who commented on drafts of this article.

©2019 The Author

American Business Law Journal ©2019 Academy of Legal Studies in Business

771

INTRODUCTION

This article addresses a threat to general well-being—a threat posed by

the changing nature of decision making. Many decisions that affect well-

being will be made by machines and, unless protective measures are put

into place, the creation of the algorithms that enable machines to make

decisions could—in fact probably will—be corrupted by self-interested

third parties.

It has become almost commonplace to observe that the world is at the

dawnofaperiodoftransformationdeeperandmoreprofoundthan

any since the Industrial Revolution. “Big Data” demands attention

because it promises to bring about fundamental change.

1

The Indus-

trial Revolution transformed the nature of physical work; machines

took on many aspects of labor, increasing productivity and

eventually—after painful social changes and deep reform of legal and

regulatory regimes—creating broader wealth and increasing stan-

dards of living.

2

Data analytics, relying on complex algorithms, prom-

ises to transform the nature of mental work by transferring many

aspects of decision making to machines. Klaus Schwab, one of the

1

Julie E. Cohen, Privacy and Technology: What Is Privacy For, 126 HARV.L.REV. 1904,

1920 (2013).

2

Historians cannot agree on when the Industrial Revolution started or ended, on how it

unfolded, or even onthe use of the term “revolution,” but acknowledge that “the technologi-

cal progress it witnessed and the subsequent transformation of the economywere not ephem-

eral events and moved society to a permanent different economic trajectory.” Joel Mokyr, The

New Economic History and the Industrial Revolution,in THE BRITISH INDUSTRIAL REVOLUTION:AN

ECONOMIC PERSPECTIVE 1, 2–3 (2d ed. 2018).Economists cannot agree on when change turned

for the better, but “[n]o economist today seriously disputes the fact that the industrial revolu-

tion began the transformation that has led to extraordinarily high (compared with the restof

human history) living standards for ordinary people throughoutthe market industrial econo-

mies.” Clark Nardinelli, Industrial Revolution and the Standard of Living,econlib,http://www.

econlib.org/library/Enc/IndustrialRevolutionandtheStandardofLiving.html (last visited Sept.

2, 2019). At an experiential level, the wrenching social changes are recounted in dozens of

novels, including those of Charles Dickens such as Hard Times (1854) and Oliver Twist (1839).

The profound social, including business and economic, changes wrought by the Industrial

Revolution affected and were also affected by profound legal changes. To catalogue all of

these would overwhelm this article, but examples of in-depth treatments include SIMON

DEAKIN &FRANK WILKINSON,THE LAW O F THE LABOUR MARKET:INDUSTRIALIZATION,EMPLOYMENT,

AND LEGAL EVOLUTION (2005); DEREK FRASER,THE EVOLUTION OF THE BRITISH WELFARE STATE:A

HISTORY OF SOCIAL POLICY SINCE THE INDUSTRIAL REVOLUTION (5th ed. 2017); DAVID R. GREEN,

PAUPER CAPITAL:LONDON AND THE POOR LAW, 1790–1870 (2010).

772 Vol. 56 / American Business Law Journal

foundersoftheWorldEconomicForum,describesthisasaFourth

Industrial Revolution.

3

The passing of decision making to machines will affect general well-being.

Not all machine-made decisions, of course, will meaningfully affect the general

state of well-being—machine-made decisions about routing e-mail messages

within an organization probably have little effect, as would machine-made deci-

sions regarding purchases.

4

The future scope of decisions that will be made by

machines, however,is immense. Machines will make decisions that affect health

care, justice, education, food, childcare, policing, democratic representation,

and more.

5

Many of these are matters at the heart of general well-being.

6

Machine-made decisions could enhance human enjoyment of each of them.

Klaus Schwab, however, warns that the benefits offered by these

changes are not inevitable and that the current transformation must be

“shapedinaresponsiveandresponsibleway.”

7

Schwab also points out

the immediacy of this responsibility: “The social norms and regulations

governing emerging technologies are in the process of being developed

and written today.”

8

Neil Richards and Woodrow Hartzog specifically

challenge legal scholars and policy makers: “while new technologies

3

KLAUS SCHWAB,THE FOURTH INDUSTRIAL REVOLUTION 1 (2016).

4

Machine-made decisions regarding purchases could be corrupted, in the same way that

human-made arrangements for kickbacks are corrupt. See Jeffrey Boles, Examining the Lax

Treatment of Commercial Bribery in the United States: A Prescription for Reform,51A

M.BUS.

L.J. 119, 119–22 (2014) (discussing the nature and prosecution of kickbacks).

5

See VIKTOR MAYER-SCHO

¨NBERGER &KENNETH CUKIER,BIG DATA:AREVOLUTION THAT WILL

TRANSFORM HOW WELIVE,WORK,AND THINK 2–18 (2013) (discussing and illustrating poten-

tial uses of machine decision making); CATHY O’NEIL,WEAPONS OF MAT H DESTRUCTION:HOW

BIG DATA INCREASES INEQUALITY AND THREATENS DEMOCRACY 199 (2016) (summarizing uses of

machine decision making).

6

See David Felce & Jonathan Perry, Quality of Life: Its Definition and Measurement,16RES.DEV.

DISABILITIES 51, 60–62 (summarizing current research on quality of life and well-being).

The precise boundaries between the public’s well-being and purely private matters have

not been crisply delineated. See Rachel Dodge et al., The Challenge of Defining Wellbeing,

2I

NT’LJ. WELLBEING 222, 222 (2012) (noting the difficulty in creating a sharp definition,

but observing that a sharp definition is only necessary for mathematical measurement and

that an understanding will often suffice). That however, does not obviate the central argu-

ment of this article; if there is something understood as general well-being, then that is put

at risk unless action is taken to prevent the corruption of algorithms in ways that will cause

machines to make decisions for purposes other than enhancing that general well-being.

7

SCHWAB,supra note 3, at 1.

8

KLAUS SCHWAB,SHAPING THE FOURTH INDUSTRIAL REVOLUTION 9 (2018).

2019 / Bribing the Machine 773