Running government like a business scientist: the rise of evidence-based decision making.

• Author: Kavanagh, Shayne C.
• Position: Cover story

[ILLUSTRATION OMITTED]

Limited funding and increasing demands have led to reformed public budgeting practices, requiring governments to "do more with less." One way to accomplish this is doing more of what works and less of what doesn't, using evidence to tell the difference between the two. Some local governments are doing just that by bringing the power of scientific inquiry to public management. This method is known as evidence-based decision making. For early adopters, it has yielded some surprising findings and contributed to greatly improved decisions on how resources are allocated.

To illustrate, in one school district the budget battle started the usual way. The math department wanted more money for its program, and the English department wanted more for its program. Both believed strongly in the merits of their programs and cared a great deal about making sure kids were learning. To justify their requests, both departments presented anecdotes about students learning a great deal and teachers being certain the program was making a difference. In the past, charisma, persuasion, or clout would have carried the day, but the district had recently embraced evidence-based decision-making. The school therefore knew that students in the English program were making few gains, despite spending $4,000 a student. The math program, however, had produced 18 months of growth in one year, at the cost of just $1,000 a student. Presented with this evidence, even the English director no longer lobbied to continue funding, and as a result, the successful math program was expanded, and the next year, students who were struggling in English got a new, better, intervention.

THE EXISTING REFORM BRIEF: RUN GOVERNMENT LIKE A BUSINESS

Over past decades, public managers have sought to reform planning and budgeting by reducing the traditional focus on expenditure line items and placing more emphasis on the outputs that governments produce and the outcomes that make a difference in the lives of citizens. The increased popularity of performance measures and performance budgeting has been the most visible manifestation of this approach.

The essence of performance budgeting is captured by Exhibit 1, which illustrates a county government that has made reduced juvenile crime a priority. One of the programs used to achieve this outcome is Scared Straight, where at-risk juveniles tour a prison and hear cautionary tales from inmates. The unpleasant conditions of the prison and the unfortunate circumstances of the prisoners are thought to inspire the young people to steer clear of a life of crime. The county applies performance measures to the program, such as the number of participants and/or the satisfaction of its clients (e.g., the social workers who refer juveniles to the program).

As performance budgeting is often practiced, the ongoing funding of the program is usually secure so long as a logical conceptual link exists between the program and the desired outcome, and the measurements receive satisfactory scores. Some governments might also measure the outcome, such as juvenile crime rate, but they would not generally directly link the program and the outcome measure because of the other variables that could affect the outcome.

The existing budget reform paradigm is inspired by the practices of private corporations: Businesses use quantified "bottom line" indicators to guide initiatives in the name of creating value for its stockholders. In the place of stock prices, governments have outcomes. In place of indicators like profitability, sales volume, and market share, governments have performance measures.

A NEW WAY? RUN GOVERNMENT LIKE A SCIENTIST

While the existing approach to budget reform is an improvement over traditional budgeting, there is a still a nagging, unresolved concern: cost-effectiveness. Does Scared Straight, or any other program, actually make a significant impact on the desired outcome? And does it do so for an acceptable cost?

To answer this question, some governments are beginning employ evidence-based decision making, which is inspired by the scientific method. The essence of the scientific method is: 1) form a hypothesis; 2) do an experiment to test the hypothesis; 3) analyze the data; and 4) draw a conclusion.

Advances in technology and the greater availability of large data sets have made it feasible for governments to collect and analyze the data necessary to prove or disprove their hypotheses. Both simple data (e.g., who attended a particular program or event) and more complex data (e.g., grades, graduation rates, or criminal records) are now stored electronically. Low-cost technology makes it possible to extract meaning from millions of data points, even if they are stored in separate databases. Perhaps just as importantly, technology also makes it possible to access studies of program effectiveness performed by third parties. Just a few years ago, accessing these sources of evidence required hours in a university library. Now they can be obtained from online databases. Rather than conducting its own experiment on Scared Straight, our county could look at the results of studies done by other organizations.

The high-tech world has embraced scientific approaches to doing business. Given the cultural cache of the high-tech sector, (1) these scientific approaches to decision making have spread to other sectors of the economy, including governments. For example, Google regularly conducts controlled experiments by releasing changes to a random sample of its users and then analyzing their responses against with the responses of users still running the standard product.

A final force in favor of a more scientific approach to decisions is that, in recent years, researchers have uncovered more information about the cognitive biases that are woven into the human psyche. A cognitive bias is a persistent flaw in logical reasoning that affects most people, such as the widespread "overconfidence bias"--our tendecy to assess situations to be more favorable to us than they actually are. (2) In other words, we are more optimistic than realistic. To illustrate, according to Tali Sharot, a neuroscientist who specializes in this subject, people hugely underestimate their chances of getting divorced and losing their jobs; they believe they will outperform their peers at work; and they overestimate their likely life spans (sometimes by 20 years or more). (3)

Cognitive biases can actually confer important benefits. For example, overconfidence helps reduce your stress about undertaking a life-changing event. You will feel better about changing jobs, moving, or getting married if you are overconfident about how well these changes will work out. However, when it comes to clear-eyed evaluation of the cost-effectiveness of programs, cognitive biases aren't as beneficial. For example, overconfidence bias might lead us to overestimate what we know about how a program works and its efficacy.

Beyond overconfidence, perhaps the most dangerous flaw in our analytical abilities is our well-developed ability to concoct a seemingly reasonable connection between unrelated objects or ideas, which psychologists refer to as "apophenia." To illustrate, take a moment to think about the Scared Straight program and imagine you are being asked to explain to the county board why the program is effective in reducing juvenile crime. Could you provide a seemingly reasonable explanation for the effectiveness of this program, even if you are...