Seeing is believing: the use of trained observer ratings for performance measurement.

• Author: Giancoli, Donald
• Position: Government accountability techniques

Trained observer ratings are an underutilized technique in the ongoing effort to enhance government accountability through performance measurement. This rather simple technique can provide reasonably reliable data on outcomes that otherwise would be difficult to measure.

What gets measured, gets done, is a maxim for the aspirations of the many incarnations of performance measurement. Performance measurement in government has been around since the time and motion studies of scientific management in the early 1900s. Budget reform efforts such as program budgeting, management by objectives, and zero-based budgeting all use performance measures to bring rationality to management and allocation decisions. In the 1980s, the drive to privatize government services and to operate "like business" promoted the use of performance measures to simulate the bottom line of the private sector. In the 1990s, the book Reinventing Government and the federal government's Government Performance and Results Act furthered the development of performance measures as a mechanism for holding government programs accountable for results.

Today, according to a survey by the Governmental Accounting Standards Board, more than 50 percent of local governments use performance measures. Cities, counties, and special districts collect hundreds or even thousands of statistical measures such as call volume, response times, trees planted, miles paved, cost per foot, cost per mile, cost per customer, and many others. In most cases, researchers have found, the statistical information is collected and dutifully reported but does not become integral to decision making. The research has identified numerous reasons for the disconnect between the promise and reality of performance measurement, including employee resistance, lack of management commitment, lack of resources for implementation, and impatience with the process. (1)

Whatever the reasons, the lesson for managers is that developing a useful measurement system is hard. Designing measures, collecting and verifying data, and choosing the right data for meaningful outcome measures are all difficult, time-consuming tasks. The more tools available to managers seeking to measure performance, however, the more likely they are to design a system that is practical and useful in their environment.

Trained observer ratings are an undervalued resource in the continuing effort to expand and improve the use of performance measurement in local government. Based on an informal survey at the most recent GFOA annual conference, few managers are familiar with this technique and even fewer are actually using it in their governments. This article discusses the mechanics of trained observer ratings, the applications of this technique in local government, and the advantages and limitations of trained observer ratings relative to other types of performance measures.

What Are Trained Observer Ratings?

Trained observer ratings are performance measures based on observations by non-experts who use written and pictorial guidelines to produce consistent ratings of service quality. Typically, ratings are based on observations of streets, buildings, or parks. Several cities use trained observer ratings to measure street cleanliness, including New York City; Coventry, England; and Christchurch, New Zealand. New York City Public Schools has a similar system for measuring building maintenance. Although certainly the most common, the applications for trained observer ratings are not limited to the condition of physical objects. Advanced rating schemes may count the number of persons successfully engaged in a program activity, such as senior center or day care participants who are actively participating versus those who are not.

The first step in developing a ratings system is to decide what to measure. The next step is to identify the elements that are most important to the quality of service delivery. Because beauty is in the eye of the beholder, determining the appropriate beholders is also a critical step in the process. Elected officials, management, staff, advisory committee members, and residents are all appropriate candidates for an evaluation committee, whose purpose is to establish the criteria for evaluating the various elements. As with other types of performance measures, participation in the development of the measures themselves is directly related to perceived validity for all interested groups. (2)

In Lauderhill, we use trained observer ratings to measure street appearance. Potential quality elements for street appearance include pavement color, pavement texture, striping, signage, curbs, sidewalks, litter, medians, swales, shoulders, streetlights, and traffic signals, among others. A ratings system may include one, several, or all of these...