The Millennium Challenge Account And The Program Assessment Rating Tool: The Difficulties Of Measuring Accountability
| Author | Ayako Sato |
| Position | JD candidate, May 2007, at American University, Washington College of Law |
| Pages | 15 |
Few people, no matter their political affiliation, would likely disagree with President George W. Bush's statement that "[t]axpayer money should be spent wisely or not at all . . ."1 No one wants taxpayers' dollars to go toward ineffective and inefficient programs. As such, President Bush has instituted a variety of accountability measures within his administration to ensure that taxpayers' dollars are well spent. Two of these mechanisms are the Millennium Challenge Account ("MCA") and the Program Assessment Reporting Tool2("PART"). This article will compare and contrast the PART and the MCA indicators in hopes of identifying common problems associated with performance measurements generally.
The MCA is an U.S. initiative3 designed to address global problems such as poverty and health by promoting economic development.4 The Millennium Challenge Corporation ("MCC"), a government corporation led by a Chief Executive Officer and a Board of Directors ("Board") composed of senior government officials, will administer the MCA. The MCA provides assistance to needy countries in the form of grants, cooperative agreements, or contracts,5 and thus far, Congress has appropriated approximately $2.48 billion to fund the MCA.6
However, not all developing countries qualify for MCA funding. Not only does the MCC require per capita income thresholds,7 but it also requires a country that receives assistance to demonstrate that it is "ruling justly, investing in [its] people, and encouraging economic freedom"8 by scoring above the median on half of sixteen indicators in each of the three areas mentioned.9 The MCA also includes an additional indicator for measuring corruption that is scored on a pass/fail basis.10Countries must meet or exceed these indicators to obtain funding because, as President Bush stated, "[w]hen nations refuse to enact sound policies, progress against poverty is nearly impossible."11 The Board will then re-evaluate those countries that meet or exceed the indicators and select the finalists that qualify for eligibility.12 Some organizations, such as the Heritage Foundation, have lauded President Bush's efforts at holding nations accountable for the foreign aid they receive.13 Various individuals claim that the MCA has the potential to ensure that foreign aid money is not squandered.14 Additionally, supporters contend that the MCA may create incentives for governments that did not initially qualify for assistance to make the necessary improvements to qualify for funding in the future.15
Critics of the indicators, however, raise concerns that funding is not going to the countries that need it most, and they may be right. For example, Bhutan, one of the countries disqualified from funding during the Fiscal Year 2004 cycle,16 had one of the lowest literacy rates among the least developed countries in the past.17 Yet, education is one of the key areas of focus for the MCA.18 Moreover, critics claim that needy countries, like Bhutan, are not qualifying for funding because the indicators are flawed, and not because these countries are incapable of managing aid assistance.19 Some of the most common complaints about the indicators are:
-
- Too simplistic - sixteen indicators to measure something as complex as national governance are inappropriate;
- Not enough data or poor quality of data - data available is either insufficient or too poor or outdated to accurately assess government performance;
- Selection for assistance is arbitrary - some countries that meet the indicator thresholds are not selected, while countries that fail to pass are selected for assistance;
- Outcomes are difficult to measure - each of the compacts have performance measures built in, but it is difficult to measure the project's results during the lifetime of the project, and it is equally difficult to attribute the project's success to the MCA.
Since the MCA is relatively new,20 one could assume that the Bush Administration is simply struggling with a new concept of measuring performance through accountability. However, accountability measures have a long and varied history in the federal government.21 The Bush administration, like previous administrations, has developed mechanisms to maintain accountability in federal government programs. PART is one of these accountability measures designed to evaluate the effectiveness of federal government programs. Still, the PART suffers from the same flaws as the MCA indicators. Although it is commendable to attempt to eliminate inefficiency and ineffectiveness, a cautious approach is needed to avoid unnecessarily eliminating essential programs that help the most vulnerable in our global society.
Developing a methodology for evaluating a system's effectiveness can be difficult, especially when applied to major programs like the MCA or PART. The PART, for example, is Page 52 designed to evaluate such disparate programs within the federal government as the Environmental Protection Agency's ("EPA") Drinking Water Research Program and the Department of Labor's Adult Education and Training Activities Program.22Therefore, the methodology has to be simple enough to evaluate significantly different systems, while at the same time, complex enough to reveal inefficient and ineffective programs.
In the case of the MCA and the PART, the methodologies are so simplistic that they are often either under- or over-inclusive, or produce absurd results. For example, the MCA has sixteen indicators in three broad areas - ruling justly, investing in people, and economic justice.23 The "ruling justly" and "economic justice" categories each have six indicators, while the "investing in people" category has four indicators.24 A country must score above the median on half of the indicators in each of the categories in order to qualify for funding. However, this approach allows countries, such as China, to qualify for funding despite a poor human rights record.25 This occurs because several of the indicators within a category are highly correlated. For example, as with China, a country can qualify by scoring above the median on half of the indicators relating to governance and corruption, "rendering the remaining indicators that measure political and civil rights in the governing justly area irrelevant."26
The PART is no less controversial for its simple formula. The General Accounting (now Accountability) Office ("GAO")27found that it is difficult to use performance measurements in evaluating complex federal programs.28The PART evaluates various aspects of a program by rating specific performance areas using a series of Yes/No questions.29 The performance areas include: (1) purpose and design; (2) strategic planning; (3) program management; and (4) results/accountability.30Funding decisions can be influenced by a poor rating in the PART evaluation. In other words, if a program is found "ineffective" or if "results [are] not demonstrated,"31 the budget for that program can be slashed or the program can be eliminated altogether. To demonstrate, the Fiscal Year 2006 budget recommendations called for funding decreases for 68 percent of programs deemed ineffective, yet proposed funding increases for 62 percent of programs that rated "effective."32 Critics of the PART argue that such a simple Yes/No survey should not determine the fate of a federal program. The GAO found that the "yes/no format employed throughout most of the PART questionnaire resulted in oversimplified answers to some questions," and that the "yes/no format is a crude reflection of reality."33
Another failing of the MCA and the PART is lack of data, or the reliance on inaccurate or poor quality data for evaluating systems. In the case of the MCA, the governance data comes from such readily accessible and reliable sources as Gallup International and the Price Waterhouse Coopers Opacity Index.34 However, for subjective concepts, such as corruption, collecting or obtaining accurate data is difficult.35 Additionally, data does not exist for some of these indicators. For example, only 87 out of 115 MCA-eligible countries had enough information on the "days to start a business" indicator, and lack of information was a problem for the indicators measuring education and health spending as a percent of gross domestic product.36 The World Bank notes that some low-income countries should be "treated with special care" as data is simply lacking on some of the performance criteria.37 Some countries, such as Kiribati and Sao Tome, did not qualify for funding, and the lack of data in several of the indicators may have contributed to their disqualification.38
Another problem that has been identified with data is that it may be out of date and does not accurately reflect current conditions in the country. For example, the MCC used the Heritage Foundation indicator for trade policy to make selections for Fiscal Year 2005. However, the indicator was based on data that spanned from 2001 to 2003.39 A difference in a few years may not seem like it would make a dramatic difference, but when dealing with developing nations "judging the performance of the current government, rather than that of some previous administration, matters a great deal."40To illustrate, Bolivia, which qualified for funding under the governance indicator in November 2004, was subsequently embroiled in a political crisis that may have lowered its rating under this performance indicator.41
Poor quality data or unavailability of data has also been identified as a problem with the PART. The EPA is one federal agency that is struggling with lack of data in measuring performance. A GAO report found that the absence of environmental data has made it difficult to assess the effectiveness of some EPA programs.42This does not mean that the programs are ineffective; it simply suggests that there is insufficient data to accurately assess whether or not the programs are in fact effective. The EPA has been struggling with the issue of environmental indicators for some time. For example, the EPA has executed efforts to improve water monitoring due to the lack of reliable national data, and the EPA has continued to work on the "environmental indicators initiative" in an effort to track and report environmental conditions and trends.43
Both the MCA and the PART claim to reward high-performance and encourage poor performers to improve, but in practice selection is arbitrary. Nations that do not meet the selection criteria are sometimes selected, while nations that would otherwise qualify are not. For example, in one funding cycle, even though 24 countries qualified for MCA funding, only six-Page 53teen were listed as eligible, and the MCC gave no explanation as to why those sixteen were selected.44 Moreover, some countries that did not qualify for funding were selected instead. For instance, Bolivia did not meet the indicator on corruption, but somehow qualified for assistance.45 The Board reasoned that Bolivia was reconsidered and ultimately selected based on the fact that the country was championing anti-corruption initiatives in the nation.46
The same can be said of the PART. OMB Watch, a nonprofit government watchdog organization, found that 154 programs that were recommended for budget cuts or elimination in Fiscal Year 2006 had very little to do with PART scores.47 For example, more than two-thirds of those 154 programs never went through the PART reviews.48 Furthermore, twenty percent of programs that received the two highest PART scores were eliminated, yet programs that received the lowest possible ratings were not cut at all.49
These inconsistencies have raised some questions about the two methodologies. For example, critics of the MCA are concerned that nations are selected not based on need or qualification, but on their geopolitical significance to the United States. For example, Georgia's and Bolivia's selections have been questioned because funding these countries seems to be advantageous to U.S. foreign policy objectives, rather than being motivated by financial need.50 Concerns have been raised regarding Bolivia's selection because the United States has a strong stake in Bolivia's counter-narcotics goals.51 Similarly, critics of the PART question whether programs are eliminated as a result of political motivation due to the lack of transparency in how programs are selected for an increase/decrease in funding or for elimination. OMB Watch is concerned that programs targeted to low-income and vulnerable populations are selected for PART review solely in an attempt to do away with them.52
Even when a country is selected for funding through the MCA, additional measures outlining the nature of the project ensure that contract proposals include performance measures. For example, monitoring and evaluation must be a "part of every activity for which MCA funds are used."53 Monitoring and evaluation plans must include baseline data against which the activity's progress can be measured, as well as benchmarks for evaluating progress.54 The MCA promotes the use of quantitative measures such as results, outputs, and outcomes. Projects may be terminated if they fail to meet financial or accountability standards, or if they fail to attain specific benchmarks.55
There are several problems with using quantitative data to measure accountability. First, results are often not available during the project's lifetime. The Brookings Institution, an independent research and policy organization, found that "the real impact of any project cannot be evaluated in outcome terms until after the money has been spent" and that "benefits will continue to accrue for years after the last disbursement of project funds has been made."56 Second, even if performance targets are met, the MCA funds may not be entirely responsible for the success, as external factors may influence project results.57 On the other hand, often when projects do not meet their targets due to external factors, critics will still blame the MCA for funding ineffective projects.58
The PART has also faced many of these same challenges. Recognizing these problems, the Office of Management and Budget ("OMB") has issued guidance on implementing PART reviews.59 The OMB guidance acknowledges that results may not be achieved for many years, and suggests that agencies develop short and medium-term steps that will ultimately lead to the long-term outcome goal.60 The guidance falls short of stating that outcomes may never be achieved and emphasizes that short-term and medium-term steps "are likely to be output-oriented, prerequisite accomplishments on the path toward the outcome goal."61 The OMB guidance also recognizes that the program may be one of many factors contributing to the desired outcome. This makes assessment difficult as to whether it is it is the program or an external factor that is responsible for achieving the results. One suggestion is for the PART to establish a broad outcome goal for a collection of programs and track the goal using national data.62 However, as previously discussed, data gaps and lack of credible and accurate data may prevent agencies from establishing a meaningful outcome goal.
Performance measures have the potential to ensure accountability. For example, in the first year of the PART, fifty percent of programs were rated as "unable to demonstrate results" but this decreased to 37 percent in Fiscal Year 2005.63Further still, many advocates of the MCA are hopeful that countries that did not initially qualify for funding will strive to make improvements so that they may qualify in the future. However, there are still significant flaws with performance measures; flaws that are serious enough to raise doubts about the suitability of such mechanisms for measuring accountability, especially when used to evaluate programs that provide aid to those in desperate need of relief.
-----------------------
ENDNOTES: MCA and PART
[1] President George W. Bush, State of the Union Address (Feb. 2, 2005), available at http://www.whitehouse.gov/news/releases/2005/02/20050202-11.html (last visited Nov. 4, 2005).
[2] The Office of Management and Budget released the President's Management Agenda that outlined five government-wide initiatives, and the PART used to judge the effectiveness of federal programs falls under the initiative for "Budget and Performance Integration." See generally Office of Management Budget ("OMB"), The President's Management Agenda, FY2002.
[3] See Millennium Challenge Corporation CEO Paul Applegarth's Presentation to African Ambassadors (Feb. 16, 2005), available at http://www.mca.gov/public_affairs/speeches/021605_African_Ambassado rs.shtml (last visited Nov. 4, 2005).
[4] See Title VI Millennium Challenge Act of 2003 602(2) (2003); see also Background Paper, Implementing the Millennium Challenge Account (Feb. 5, 2003) [hereinafter Background Paper], available at http://www.mca.gov/about_us/key_documents/MCA_BackgroundPaper_ FactSheet.pdf (last visited Nov. 4, 2005). The key areas of focus would include "agricultural development, education, enterprise and private sector development, governance, health, and trade and investment capacity building." See id. at 1.
[5] Title VI Millennium Challenge Act of 2003 605(b) (2003).
[6] See Larry Nowels, CRS Report RL32427, The Millennium Challenge Account: Implementation of a New U.S. Foreign Aid Initiative, July 1, 2005, at 3, available at http://www.usembassy.it/pdf/other/RL32427.pdf (last visited Nov. 4) [hereinafter Implementation].
[7] See id. at 2. For Fiscal Year 2004, nations with per capita incomes below $1,415 and eligible to borrow from the World Bank's International Development Association were able to qualify. For Fiscal Year 2005, the per capita income requirement increased to $1,465.
[8] THE MILLENNIUM CHALLENGE CORPORATION, THE MILLENNIUM CHALLENGE ACCOUNT, at http://www.mca.gov/about_us/overview/ index.shtml (last visited Nov. 4, 2005).
[9] See Larry Nowels, CRS Report 31687, the Millennium Challenge Account: Congressional Consideration of a New Foreign Aid Initiative, 5-6 (Mar. 19, 2004), available at http://www.fas.org/sgp/crs/row/RL31687.pdf (last visited Nov. 5, 2005) [hereinafter Congressional Consideration].
[10] See id. at 6.
[11] President George W. Bush, Remarks by the President on Global Development Inter-American Development Bank, President Proposes $5 Billion Plan to Help Developing Nations (Mar. 14, 2002), available at http://www.whitehouse.gov/news/releases/2002/03/20020314-7.html (last visited Nov. 4, 2005).
[12] FREQUENTLY ASKED QUESTIONS, MILLENNIUM CHALLENGE CORPORATION, at http://www.mca.gov/about_us/faq/index.shtml (last visited Nov. 5, 2005).
[13] See Brett D. Schaefer, the Millennium Challenge Account: An Opportunity to Advance Development in Poor Nations, HERITAGE LECTURES, no. 753, (June 27, 2002).
[14] Lael Brainard, Compassionate Conservatism Confronts Global Poverty, THE WASHINGTON QUARTERLY, 26:2, 150 (Spring 2003), available at http://www.twq.com/03spring/docs/03spring_brainard.pdf (last visited Nov. 5, 2005).
[15] See id.
[16] Sarah Lucas & Steve Radelet, An MCA Scorecard: Who Qualified, Who Did Not, and the MCC Board's Use of Discretion, Center for Global Development, 3 (May 2004), available at http://www.cgdev.org/doc/commentary/MCAScorecard_0528.pdf (last visited Nov. 5, 2005).
[17] UNITED NATIONS EDUCATIONAL, SCIENTIFIC AND CULTURAL ORGANIZATION, A Synthesis Report of Education For All 2000 Assessment in the South and West Asia sub - Region Bangkok, UNESCO PROAP (June 2000), available at http://www.unescobkk.org/index.php?id=145 (last visited Nov. 4, 2005).
[18] See Background Paper, supra note 4.
[19] Lucas and Radelet argue that, in the case of Bhutan, the arbitrary method of selection is based on "implicit" indicators. See Lucas & Radelet, supra note 16, at 3.
[20] The MCA was established in 2003 through the Millennium Challenge Act of 2003.
[21] See generally Virginia A. McMurty, Performance Management and Budgeting in the Federal Government: Brief History and Recent Developments, CRS Report RL 32164, (Mar. 16, 2005), available at http://knownet.hhs.gov/performance/performancemanagement.pdf (last visited Nov. 5, 2005). Performance management and budgeting goes as far back as World War II when the First Hoover Commission promoted the use of performance budgeting in the federal government. See id. at 1-2.
[22] See OMB, The New Program Assessment Rating Tools Underway - Planned Release with the 2007 President's Budget (July 2005), available at http://www.whitehouse.gov/omb/part/fy2005/2005_part_list.pdf (last visited Nov. 4, 2005).
[23] See Congressional Consideration, supra note 9, at 10.
[24] There are six indicators under "ruling justly" that include: 1) control of corruption; 2) voice and accountability; 3) government effectiveness; 4) rule of law; 5) civil liberties; and 6) political freedom. There are four indicators under "investing in people" that includes: 1) public primary education spending as percent of gross domestic product ("GDP"); 2) primary education completion rate; 3) public expenditure on health as percent of GDP, and 4) immunization rates. There are six indicators under "economic freedom" that include: 1) country credit rating; 2) inflation; 3) three-year budget deficit; 4) trade policy; 5) regulatory policy, and 6) days to start a business. See Implementation, supra note 6, at 24.
[25] See Brainard, supra note 14, at 158.
[26] See Brainard, supra note 14, at 159.
[27] "> As of July 7, 2004, the General Accounting Office ("GAO") has changed its name to the General Accountability Office. See About the General Accountability Office, available at http://www.gao.gov/about/namechange.html (last visited Nov. 4, 2005).
[28] See generally GAO, GAO-04-439T, Performance Budgeting: OMB's Program Assessment Rating Tool Presents Opportunities and Challenges for Budget and Performance Integration, 6-8 (Feb. 4, 2004).
[29] "> See Eileen C. Norcross, an Analysis of the Office of Management and Budget's Program Assessment Rating Tool (PART), 3 (June 2005), available at http://www.mercatus.org/pdf/materials/1280.pdf (last visited Nov. 5, 2005).
[30] "> See id.
[31] The PART ratings include "effective," "moderately effective," "adequate," "ineffective," and "results not demonstrated."
[32] See Norcross, supra note 29, at 23.
[33] See GAO, supra note 28, at 7.
[34] See Congressional Consideration, supra note 9, at 10.
[35] See Congressional Consideration, supra note 9, at 16. "Corruption" as an indicator is difficult to measure for a number of reasons. For example, as corrupt transactions are generally illegal and often part of the "informal economy" there are no existing records of them and therefore, not possible to measure directly. See generally Anthony Lanyi, Measuring the Economic Impact of Corruption: A Survey, The IRIS Discussion Papers on Institutions & Development (IRIS/the University of Maryland, Feb. 2004).
[36] See Congressional Consideration, supra note 9, at 16.
[37] Daniel Kaufmann & Aart Kraay, Governance Indicators, Aid Allocation, and the Millennium Challenge Account: A Summary, (World Bank, Dec. 2002).
[38] See Implementation, supra note 6, at 9.
[39] James W. Fox & Lex Rieffel, The Millennium Challenge Account: Moving Toward Smarter Aid, 31 (Brookings Institution, July 25, 2005).
[40] See id at 31.
[41] See id.
[42] Greenwire, Enviro Policy: GAO Calls for Stronger Federal Effort on Environmental Indicators (Nov. 18, 2004).
[43] See id.
[44] Council on Hemispheric Affairs, Bush's Millennium Challenge Corporation: A New Approach to Overseas Development, or the Same Old Strategy? (Aug. 16, 2005), available at http://www.coha.org/ NEW_PRESS_RELEASES/New_Press_Releases_2005/05.94_the_Mille nnium_Challenge_Corporation_A_New_Approach_to_Overseas_Develo pment_or_the_same_old_strategy.htm (last visited Oct. 10, 2005).
[45] See Implementation, supra note 6, at 8.
[46] See Implementation, supra note 6, at 8.
[47] See OMB Watch, How Do You Measure Program Results? (Feb. 22, 2005), available at http://www.ombwatch.org/article/article- view/2680/1/90 (last visited Oct. 10, 2005).
[48] See id.
[49] See id.
[50] See Implementation, supra note 6, at 9.
[51] See Implementation, supra note 6, at 9.
[52] See OMB Watch, "PART" and the Federal Budget (Oct. 28, 2002), available at http://www.ombwatch.org/article/articleview/1161/1/312 (last visited Oct. 10, 2005). OMB Watch lists such programs as the Social Security Disability Benefits program and the National School Lunch Program that were selected for PART review in FY2004.
[53] See Background Paper, supra note 4, at 4.
[54] See Background Paper, supra note 4, at 4.
[55] See Background Paper, supra note 4, at 5.
[56] Fox & Riefel, supra note 39, at 17.
[57] See Fox & Riefel, supra note 39, at 26.
[58] See Fox & Riefel, supra note 39, at 26.
[59] See generally OMB, Performance Measurement Challenges and Strategies (June 18, 2003), available at http://www.whitehouse.gov /omb/part/challenges_strategies.pdf (last visited Nov. 4, 2005)
[60] Id. at 9.
[61] Id.
[62] See id. at 7.
[63] See McMurty, supra note 21, at 7.
