Natural and quasi-natural experiments to evaluate cybersecurity policies.

• Author: Dean, Benjamin
• Position: Report

Over the past decade, numerous countries around the world have developed and implemented national cybersecurity strategies. Yet very few of these strategies have been subject to evaluations. As a result, it is difficult to judge the performance of strategies, the programs that comprise them, and the cost-effectiveness of funds spent. Natural and quasi-natural experiments are a promising set of research methods for the evaluation of cybersecurity programs. This paper provides an overview of the methods used for natural or quasi-natural experiments, recounts past studies in other domains where the methods have been used effectively, and then identifies cybersecurity activities or programs for which these methods might be applied for future evaluations (e.g., computer emergency response teams in the EU, cybersecurity health checks in Australia, and data breach notification laws in the United States).

**********

Over the past decade, numerous countries across the world have developed and implemented national cybersecurity strategies. Each strategy comprises a set of objectives and various programs to achieve those objectives. Tens of billions of dollars in taxpayer funds have been diverted from other purposes to pay for these strategies. A number of countries' recent strategies are reviews of previous ones.

Unfortunately, there are still no definitive answers to questions such as: Have these strategies achieved their overall objectives? Which programs contributed the most to these objectives? By how much (or little)? Where have funds been most effectively spent? What improvements might be made?

By most accounts, the cybersecurity situation globally is getting worse, in spite of the many measures being taken. There is a real need to improve assessment and evaluation of cybersecurity policies so as to inform and guide policy change.

With a new generation of cybersecurity strategies now being rolled out, it is timely to consider what evaluation techniques might be employed at the outset, so as to better track the performance of programs and the cost-effectiveness of funds spent. In doing so, public policies might better address the present state of cybersecurity nationally and globally in the future.

One promising technique for the evaluation of some cybersecurity programs is the use of natural and quasi-natural experiments. These broad groups of research designs and methods avoid the potentially high cost, possible ethical issues, and the impracticality of randomized control trials in a domain like cybersecurity. At the same time, they provide relatively robust measures of the counterfactual and net social/economic impact of policy decisions.

This paper will start with a background on national cybersecurity strategies. This will be followed by an explanation of common evaluation techniques with a special emphasis on natural and quasi-natural experiments. Finally, the paper will identify instances in which such research designs and methods might most effectively be used to evaluate certain programs that commonly comprise cybersecurity strategies and how such evaluations might be done in practice.

CYBERSECURITY STRATEGIES

Over the past decade, at least 20 countries have developed and implemented national cybersecurity strategies. (1) At least five of them have updated their strategies since their first edition (Australia, Czech Republic, Estonia, Netherlands, and the United Kingdom).

The objectives within these strategies are broadly similar across countries. According to the European Union Agency for Network and Information Security (ENISA) in 2014, the most commonly recurring objectives of the strategies in Europe include: developing cyber defense policies and capabilities, achieving cyber resilience, reducing cybercrime, supporting industry on cybersecurity, and securing critical information infrastructures. These objectives are broadly similar to those in the strategies of non-European countries.

Countries fund and implement various activities or programs to achieve these objectives. Some activities involve implementing new or revised legislation. Others involve discrete programs such as research and development grants, training, awareness campaigns, or "capacity building" of target segments such as small and medium enterprises.

Large amounts of public funds are being reallocated from other policy areas, such as education or healthcare, to fund these cybersecurity strategies. For instance, in the United States, an estimated $19 billion is expected to be allocated to cybersecurity measures in the 2017 White House budget proposal. (2) While not strictly a national cybersecurity strategy, this level of funding nonetheless amounts to a great deal given the annual totals have exceeded $10 billion annually for the past five years. Australia's most recent cybersecurity strategy involves a more modest $57.5 million per annum over four years. (3) Some strategies, unfortunately, do not mention how much will be spent. Examples include the Canadian Cybersecurity Strategy of 2010 and European Union's Cybersecurity Strategy of 2013.

PROGRAM EVALUATION: A PRIMER

Program evaluation is a mainstay of the evidence-based policymaker's toolkit. In the review of policy evaluation in innovation and technology, the authors define evaluation as "a process that seeks to determine as systematically and objectively as possible the relevance, efficiency and effect of an activity in terms of its objectives, including the analysis of the implementation and administrative management of such activities." (4)

Policy decisions should lead to outcomes where the social and economic benefits of a policy intervention outweigh the related costs. In the event that the policy intervention does not deliver a net economic and social benefit at least equal to the long-term bond rate, then public resources are not being used in an optimal way. (5) The task of evaluation methods is to determine these costs and benefits in a statistically robust way.

There are many benefits to be gained from program evaluation. For instance, in the United States in 2013, a Government Accountability Office (GAO) survey found that of the 37 percent of federal managers who had undertaken evaluations in the past, 80 percent reported that "those evaluations contributed to a moderate or greater extent to improving program management or performance and to assessing program effectiveness or value." (6) Referring specifically to cybersecurity policy evaluations, ENISA claims that such evaluations lead to benefits in terms of greater accountability of public action, increased credibility domestically and with international partners, an evidence-based input to long-term planning, and support for outreach and enhancement of public image, among many others. (7)

In short, program evaluation leads to better policy decisions and improved policy outcomes over time. Given the current state of cybersecurity, and the explicit objective of cybersecurity strategies to improve this state, program evaluation plays an integral role in achieving these objectives over time.

Yet in the United States, evaluation of public programs is not widely used. The same GAO study found that only 37 percent of federal managers had completed an evaluation of any program, operation, or project. Of those who had undertaken evaluations, the factor most commonly cited as having hindered evaluations to a great or very great extent was lack of resources to implement evaluation findings (33 percent). More attention to and funding for the undertaking of evaluations and implementation of evaluation findings would go a long way towards improving policy outcomes.

While in-depth financial audits have been undertaken of cybersecurity strategies in the United States and United Kingdom, these serve a different purpose than that of program evaluations. Rather than examining the impacts of the strategies along the stated objectives, these audits tend to focus on "verifying the correctness of financial statements, and assessing how economically, effectively and efficiently the funds are spent." (8)

The European Commission undertook an extensive ex ante impact assessment of its cybersecurity strategy in 2013. The impact assessment is notable for its clear problem definition, identification of drivers behind the problem, identification of shortcomings of the status quo, clear justification for policy intervention, clear objectives, policy alternatives, and the identification of indicators/metrics to eventually evaluate the progress in achieving the stated objectives. This is a solid foundation on which to conduct future policy evaluation. However, the cost-benefit calculations of the assessment belie the dearth of robust evidence on which to base policy decisions. Many assumptions were made to estimate even the costs of policy options. The benefits are "extremely difficult to estimate for a number of reasons," including the difficulty to assess "to what extent enhanced NIS would mitigate the negative impact of security incidents." (9) There is much to be done to reliably measure what these benefits might be in the future.

Evaluation of cybersecurity strategies has been urged by various organizations in the past. The Business Industry Advisory Committee (BIAC) to the Organisation for Economic Co-operation and Development (OECD) recommended that "efficient national cybersecurity strategies and policies should be periodically evaluated and updated so that improvements can be implemented to face new, security, threats." (10), As, far, back, as, 2009 the, G.A.O., suggested, that, there, were opportunities to enhance federal cybersecurity in the United States through many measures including, "enhancing independent annual evaluations." These recommendations were not acted upon. In a recent example, in January 2016 the GAO reported that although the Department of Homeland Security (DHS) had developed metrics for measuring the...