1

2 MEASURING THE PERFORMANCE OF PUBLIC ENGAGEMENT IN 3

TRANSPORTATION PLANNING: THREE BEST PRINCIPLES 4 5 6 Submission Date: 7/31/2012 7 8 9

10 11

12

13

14

15

16

17

18

19

Corresponding Author 20

21

Jason Wagner 22

Associate Transportation Researcher 23

Texas Transportation Institute 24

1106 Clayton Lane, Suite 300G 25

Austin, TX 78723 26

Ph. (512) 467-0946 27

Fax (512) 467-8971 28

j-wagner@ttimail.tamu.edu 29

Word Count: 6,954 (6,204 words, 3 tables) 30

31

Jason Wagner 2

ABSTRACT 1 Measuring performance provides a variety of benefits: establishing accountability with the public, 2 enabling management and performance improvements, increasing transparency, and building public trust. 3 The most recent transportation authorization bill, Moving Ahead for Progress in the 21

st Century (MAP-4

21), includes a heavy focus on performance measurement. Despite such strong reasons, measuring the 5 performance of public engagement in transportation planning has proved difficult due to a variety of 6 issues, including unclear goals, limited funding, and the situational nature of engagement activities. 7

To address some of these issues, the author analyzes the literature of best practices for public 8 engagement in transportation planning and derives three principles that encapsulate the contained 9 concepts. The three principles include accessible events, engaging interactions, and an outcome-oriented 10 process. The author developed a performance scorecard rubric to operationalize the three principles. The 11 scorecard provides an intuitive and concise method to measure and display performance information. 12

This paper organizes the disparate and fragmented literature of recommended practices into a 13 usable framework, which enables practitioners to understand more easily the important aspects of public 14 engagement and better measure the performance of engagement activities. It also progresses the discourse 15 surrounding public engagement and performance measurement. Additional study would provide 16 opportunities for further refinement and investigation of the efficacy of the system the author developed 17 in a real-world setting. 18 19

20 21 22

23 24

25 26

27

28

29

30

31

32

33

34

35

36

Jason Wagner 3

INTRODUCTION 1 Measuring the performance of public engagement in transportation planning provides a variety of 2 benefits: 3

establishing accountability with the public, 4 improving performance, 5 increasing transparency, 6 and building trust. 7

The most recent transportation authorization bill, Moving Ahead for Progress in the 21st Century (MAP-8

21), includes a heavy focus on performance measurement. Despite these strong reasons, measuring the 9 performance of public engagement in transportation planning has proved elusive and difficult. 10

Evaluating the success of a public engagement effort can be a daunting task. The goals are 11 frequently unclear, output measures are often highly dependent on situational factors beyond the control 12 of the agency, and funds are not always available for in-depth measurement activities (1). Without a way 13 to address these hurdles, the practitioner may decide not to evaluate performance. 14

An initial problem in evaluating public engagement in transportation planning efforts is the lack 15 of universal and clear goals for engagement. Different transportation projects will have different 16 objectives. The goals for the engagement process will look very different for a project that seeks to 17 construct a new highway in a major urban area than for the addition of stoplights in a smaller city. 18 Comparing the success of the two projects would be quite difficult. The wide variety of strategies and 19 platforms agencies could use to engage with the public make it difficult to develop universally applicable 20 recommended practices or evaluative frameworks. The author proposes the use of an evaluative 21 framework that assesses the efficacy of the process itself. Put into practice, this framework would 22 evaluate the level at which an agency engages in and achieves the best practices laid out in the literature. 23

The author proposes a new evaluative framework for public engagement that will provide a 24 clearer barometer of the agency’s ability to fulfill the best practices laid out in the literature. The author 25 derived the framework from an analysis and synthesis of the literature into three best principles that 26 reflect the literature’s broad themes for successful public engagement. Based on the three principles, the 27 author developed an example performance scorecard rubric that practitioners could use to assess their 28 performance using quantitative or qualitative measures. The author proposes some potential measures that 29 practitioners can use and illustrates how to use the scorecard. 30 31

THE THREE BEST PRINCIPLES IN PUBLIC ENGAGEMENT 32 The author reviewed the literature of the best practices in public engagement for transportation planning. 33 The reviewed literature includes scholarly articles, guidebooks and other articles from federal and state 34 agencies, articles written by private and non-profit organizations, along with other sources. This review 35 uncovered a variety of advice, much of it overlapping, occasionally conflicting, and very frequently with 36 many different objectives and goals. So much different advice for transportation practitioners exists that 37 an individual with limited time and resources might find it difficult to determine and understand the most 38 important aspects, or even simply to sort through it all. 39

This paper identifies common principles that frequently occur in the literature. Each principle 40 encapsulates the essence of many different individual “best practices” about how to engage the public in 41 transportation planning successfully. Table 1 provides a simplified graphical representation of this 42 process. The value of distilling the best practice literature into principles is that it provides practitioners 43 with a tool to understand quickly the most important aspects of the process of engaging the public. 44

In this case, the author uses frequency of occurrence as a stand-in measure for “most important.” 45 This measure is justified by the assumption that the most important areas for a practitioner to engage in 46 would likely be reiterated frequently in the literature of best practices by most authors. Stated differently, 47 if an idea were important, most authors would identify it in a list of best practices. The fact that each 48

Jason Wagner 4

author and paper uses unique language to expresses ideas in different ways presented a hurdle to 1 overcome. To transcend this obstacle, the author distilled the best practices to underlying ideas or themes, 2 which are illustrated in Table 1. An “X” mark indicates that the best practice or guideline has elements of 3 the corresponding principle. A blank cell indicates that the particular practice or guideline did not have 4 elements of the corresponding principle. 5

Based upon a review of the literature, the author found that when engaging the public, there are 6 three overall principles that should guide a practitioner’s engagement process: 7

1. Accessible events 8 2. Engaging interactions 9 3. Outcome-oriented process 10 11

TABLE 1 Guidelines and Principles (2; 3; 4; 5; 6; 7; 8; 9; 10; 11) 12

Guidelines Found in the Literature Accessible Engaging

Outcome-

Oriented

Adaptive X

Apply democratic principles X

Appropriate strategies for appropriate audiences X X

Collaborative X

Continuity X X

Convenient and interesting X X

Deliberative X X

Ensure audience engages in decision-making X X

Focus on outcomes X

Have proper access to forums/tools X X

Include minorities and non-English speakers X X

Inclusivity X

Interactivity X

It’s not just what you say; it’s how you say it X X

Know your audience X

Make meetings accessible X

Make meetings engaging X

Ongoing, two-way exchange X X X

Participatory X

People are aware of opportunities to participate X

People understand how they can participate X

Provide continuous contact X X

Simple messages work best X X

Success requires resources X X X

To reach all audiences, use multiple channels X X

Transparency X X

Use complementary strategies X

Value people’s input X X

13

The following sections briefly define and describe each of the three identified principles. 14 15 16

Jason Wagner 5

Accessible Events 1 There are many challenges to successfully involving the public in transportation planning, but the first —2 and one of the most challenging—is getting people involved in the event. This challenge can be remedied 3 by providing ease of access to the event for anyone who might want to attend (7). An accessible event 4 starts with holding the event at a time and location that is convenient and familiar, such as a community 5 center, church, or library. Accessibility can also require providing services to enable community members 6 to attend, like the provision of food, daycare, or translation for non-English speakers. 7

An important aspect of accessibility is “knowing your audience” (8). It is vital to actively seek 8 out the key leaders and important stakeholders, and invite them to engage in the discussion. This includes 9 knowing obvious things such as demographics and languages; but also knowing less tangible aspects like 10 local concerns, community values and leaders, and important motivations. An agency can improve its 11 chances of success by knowing how the community feels about public expenditures, what their most 12 important concerns are, and who the key stakeholders are. This knowledge allows an agency to 13 adequately prepare to discuss the pertinent community issues and bring the stakeholders into the 14 discussion 15

Another important aspect of accessibility is ensuring that community members are aware of 16 participation opportunities. Marketing the public involvement opportunities to the community through a 17 variety of methods can inform individuals and increase attendance. Agencies can accomplish this by 18 encouraging media coverage, advertising on the internet and through social networks, mass mailings, or 19 simple word-of-mouth and personal invitation strategies. 20

. 21 Whatever methods an agency chooses, it should ensure they are appropriate to the intended 22

audience. For example, an older, non-English speaking neighborhood may not be likely to receive 23 information distributed in English through social media, but may respond more favorably to a mass 24 mailing, a personal invitation, or an outreach method through the local news source. The message should 25 also provide a compelling reason for attendance—highlighting specific problems that need to be 26 addressed, such as congested corridors or roadway improvements. Without a strong case, individuals may 27 simply disregard the marketing as irrelevant. 28

An aim of accessibility can be to ensure engagement exercises are representative of the 29 population from which the agency seeks to gain input. If the demographic characteristics of the 30 individuals participating are significantly skewed from the population, groups that are not represented 31 may have an inadequate voice in the discussion. This can result in undemocratic decisions. 32 33

Multi-Platform Strategies 34 Perhaps one of the most important aspects of accessibility is providing a variety of venues or platforms 35 for engagement, thus increasing the likelihood that individuals can easily access and feel comfortable 36 expressing themselves in one (or more) of the platforms (9). When one imagines public involvement, an 37 old cliché is likely to come to mind: a technocrat droning endlessly about esoteric jargon in a musty 38 community center, complete with sparse attendance, minimal interaction and even less interest. 39 Thankfully, a new age of communication and public engagement has arisen. This new age has agencies 40 utilizing a multi-platform engagement strategy that emphasizes bringing community members into the 41 discussion through familiar and comfortable mediums. 42

A multi-platform strategy encourages participation by people from demographic groups that may 43 not attend traditional public meetings. It grants people with hectic schedules the opportunity to have a say 44 in their local governance as their schedules allow. It also enables community members to voice their 45 opinions in a way that is comfortable. For example, speaking at a public meeting may feel intimidating 46 for some, and can discourage their involvement. Providing a variety of mediums to communicate can 47 eliminate psychological and logistical barriers, thus increasing involvement. 48

Jason Wagner 6

A key aspect of this strategy is first knowing your audience, and then providing a variety of 1 mediums through which they can easily communicate their opinions and ideas (8). As discussed 2 previously, knowledge of the local demographics is a necessity; but it is only one component of knowing 3 your audience. The identification of important stakeholders, an understanding of cultural aspects, and a 4 familiarity with local concerns will all equip agencies with the knowledge to design a more effective 5 strategy to involve the public. Agencies can use focus groups, stakeholder interviews, and opinion 6 surveys to aid in identification of important issues and message development. 7

Once an agency has this information, it can begin the process of targeted outreach. Agencies 8 should couple traditional strategies like public meetings and notices with innovative strategies like 9

a project website, 10 social media sites, 11 blogs, or 12 text message updates on important project milestones (12). 13

The traditional and innovative methods should link, as well. Agencies should advertise for public 14 meetings through their websites and social media pages (13). The public meetings should—when 15 possible—provide citizens the option to participate in the meeting electronically through a webcast. 16 Attendees participating electronically can comment on the proceedings by submitting questions or 17 commenting on the webpage. Once the event is complete, the project should post the results of the forum 18 and allow further comments. The project should assign one or more individuals to monitor and update the 19 electronic resources to ensure that they are up-to-date and that people commenting receive responses to 20 their questions, concerns, or input. As a caveat, streaming events live can be costly and difficult to do in 21 some areas, especially in rural communities with limited high-speed internet access. This is a best 22 practice, however, and may not be applicable in all situations. 23

Online opinion polling is an inexpensive method to gauge the feelings of the public. Agencies can 24 host polls on agency or project websites and through social media, such as Facebook’s polling function. 25 Agencies should take care to note that often the demographics of internet users might not match the 26 demographics of the targeted population, so polling may not accurately reflect the sentiments of the entire 27 populace. 28

While a useful way to quickly and inexpensively gain input, online polling can result in sampling 29 bias due to self-selection. The self-selection can occur due to disparities in computer and internet 30 accessibility and education. This method can result in an unrepresentative sample. This does not mitigate 31 the value of online engagement approaches, however. Self-selection can occur with any engagement 32 strategy. Events requiring individuals to show up at certain times in certain locations will eliminate 33 individuals who are not able to attend at those times, locations, or do not have transportation. Agencies 34 can mitigate self-selection concerns by using multiple outreach strategies, with online components being 35 only one part of a multifaceted engagement strategy. 36

A multi-platform outreach strategy allows a more in-depth and comprehensive approach to public 37 involvement. Using multiple, complementary strategies facilitates greater participation and accessibility 38 by decreasing the barriers to public involvement. 39 40

Engaging Interactions 41 Once the agency has made sure the event is accessible to all who wish to attend, the search for successful 42 involvement shifts to “How do I best convey information to and receive feedback from those in 43 attendance?” The literature recommends that events should be “collaborative rather than confrontational, 44 engaging and fun, and should provide a learning experience for all involved” (7 p. 1). When community 45 members engage and buy into the process, the result will more likely be one of collaboration and mutual 46 consent. It is important that the working environment is one that encourages discussion, rather than 47 stifling dissent. 48

Jason Wagner 7

Presenting the information through games or interactive exercises can present a different way for 1 community members to approach the project. It can foster discussion and engage the imaginations of the 2 involved individuals. Successes have been reported with a model that used a presentation, followed by 3 breakout and discussion sessions of smaller groups. The discussion groups report to the whole group with 4 recommendations on their issues. This format enables community members to take ownership of the 5 planning process and shape the discussion. Practitioners must take care to know the demographics and 6 local community concerns in which they are working. This knowledge can prevent the perception of 7 condescension through these activities. If a community is receptive to these tactics, they can be a valuable 8 tool. If they are not, however, they should be avoided. Sensitivity to recent political events may help 9 guide these decisions (for example, not making a game out of a plan regarding a controversial 10 proposition). 11

Meaningful public engagement is open, ongoing and allows for two-way exchanges of 12 information (11). It cannot tell community members what will happen and then allow a cursory comment 13 period. It should provide an opportunity for community members to discuss their views and for the 14 agency to show how it will incorporate their feedback. This creates an iterative discussion loop that 15 encourages and enables communication, so long as both parties continue the dialogue. 16

The feedback mechanisms, therefore, must be simple and convenient for individuals (14). If 17 barriers exist to communication, members will be less inclined to inform the agencies of their opinions 18 and thoughts. Success also requires agencies to communicate in simple, but informative messages. 19 Inundating the community with technical details and esoteric information is likely to overwhelm and deter 20 the communication process. Agencies should provide the information that community members need to 21 know without drowning them in detail. 22

Another aspect of engaging interactions is ensuring that the information is conveyed in a 23 conversational and not condescending manner (12). Talking down to community members will breed 24 resentment and hostility—hardly the recipe for successful engagement. A conversational tone will set the 25 audience at ease and encourage communication. If an individual fears they are not “smart enough” to 26 engage with a technically proficient speaker, they may hold back their comments and fail to contribute to 27 the discussion at all. Speakers can prevent this from happening by fighting the urge to use jargon and 28 bureaucratic language (e.g., excessive acronyms, technical terms, etc.). Another strategy practitioners can 29 use to assuage concerns is providing fact sheets that explain the issues at-hand in accessible language, 30 complete with graphics and visuals when applicable. 31 32

Outcome-Oriented Process 33 The final principle emphasizes that all public engagement efforts should focus on achieving desired 34 outcomes. The process should strive to achieve continuous involvement that results in meaningful 35 decision-making that reflects community values (11). It is important that agencies recognize that 36 engagement is more than simply fulfilling a regulation or checking a box on a list of tasks to complete. 37

Public engagement provides a mechanism for communities to have a direct voice in their local 38 governance. It bridges the divide between the bureaucracy and the people—allowing a rare opportunity 39 for real community “buy-in” to the governing process. As such, it is important for participants to see 40 tangible evidence that the input they provide affects the decision-making process in a meaningful way. 41

This principle touches all aspects of the public engagement activities. Each aspect of the process 42 should focus on involving the public to decide issues in a manner that reflects the community’s values. 43 One part of this process is genuinely listening to community members and utilizing their input in a 44 practical manner. This does not mean that an agency must implement every opinion that participants put 45 forward; but it does mean engaging with the community in such a way that individuals know their input 46 was valuable and their contributions were a factor in the process (8). Agencies should treat individuals in 47 a respectful and courteous manner. For people participating electronically, this can mean responding to a 48

Jason Wagner 8

member’s post, comment, or question, and monitoring any debate carefully so that the discussion remains 1 constructive (13). 2

Another aspect of outcome-oriented engagement is soliciting involvement early in the process, 3 before the agency makes all the decisions (10). If the public perceives the engagement is an afterthought 4 and that an individual’s comments will have no bearing on decisions, participation will suffer and 5 cynicism will spread. Another important aspect is responding to the input and opinions of the public. If 6 the public overwhelmingly feels that a change is needed and the change is feasible, the agency should 7 attempt to acknowledge the input and adjust their program in response. If there are reasons changes 8 cannot be made, the agency should make earnest efforts to explain why the changes are infeasible. 9

Additionally, consistent with a focus on outcomes is the recognition that success requires the 10 dedication of resources (14). The amount of resources dedicated to a specific purpose often dictates 11 organizational priorities. If an agency starves the public involvement department of the necessary 12 resources, it will be evident and likely result in poor outcomes. 13

It takes time and money to “pound the pavement”—meeting with local leaders and stakeholders, 14 discovering the important community concerns, and developing relationships necessary for success. 15 Effective public engagement also requires the dedication of the right people. It is no easy task to lead a 16 public meeting on a controversial topic while maintaining decorum and respect. It is important that 17 agencies carefully select individuals capable of carrying out such a task. Along with that may include the 18 need to hire staff with the experience and ability to meet with all types of communities. The United States 19 has a wide variety of demographic and cultural groups. Ensuring that staff are aware of local sensitivities 20 can go a long way towards establishing credibility and trust. 21

22 Measuring Performance 23 A vital part of developing an outcome-oriented public involvement process is measuring the performance 24 of the public engagement efforts. In the private sector, profitability generally measures performance. If a 25 company is unsuccessful, it will face financial difficulty and (possibly) bankruptcy. The public sector 26 does not have a universal barometer for performance, so understanding an agency or program’s success 27 requires careful evaluation. 28

Performance measurement is the “regular measurement of the results (outcomes) and efficiency 29 of services or programs” (15 p. 3). It enables organizations to manage their performance, focus on 30 customer-service and quality of service provision, and maximize benefits while minimizing negative 31 consequences. Measuring performance will 32 establish accountability with citizens and elected officials, 33 enable the use of data to guide the dedication of resources, and 34 build public trust by increasing transparency and displaying how services are provided to the public. 35

If agencies do not measure their performance, they may be completely unaware of their progress 36 toward attaining their goals or missions. Performance measurement not only informs agencies of their 37 ability to achieve their goals, but also provides valuable information agencies can use to allocate 38 resources, educate the public, and improve performance (15). Moreover, an agency can objectively report 39 to the public its progress in meeting objectives, thereby building trust and increasing credibility. 40

If these reasons do not provide sufficient motivation for organizations to engage in performance 41 measurement, recent legislation requires performance measurement and management. The most recent 42 highway authorization act, Moving Ahead for Progress in the 21

st Century (MAP-21), includes a heavy 43

focus on performance measurement and management (16). The cornerstone of the act is transitioning 44 highway funding and transportation planning to a performance-based system. The transportation planning 45 section retains an especially strong focus on “incorporating performance goals, measures and targets into 46 the process of identifying needed transportation improvements and project selection” (16 p. 5). This focus 47 from the federal level provides yet another strong reason for organizations to engage in measuring their 48 performance. 49

Jason Wagner 9

Before agencies can begin to measure performance, they must first determine their overarching 1 mission (or goal) and specific individual objectives. The mission identifies the overarching goal of the 2 program. All activities agencies perform should contribute to achieving the overall mission. 3

Measuring the performance of public engagement requires measuring the relevant inputs, outputs, 4 and outcomes. Table 2 summarizes the definitions of the preceding three terms. 5

6 TABLE 2 Inputs, Outputs, and Outcomes (15 pp. 14-17) 7

Term Definition

Input The amount of resource used (funds, employee time, etc.)

Output The amount of products and services delivered (number of

event attendees, miles of roads paved, etc.)

Outcome The events, occurrences, or changes in conditions, behavior, or

attitudes that indicate progress toward a program’s mission and

objectives (ability to develop a community in a manner that

reflects the community’s values and ideals, etc.).

8

Once agencies identify a mission, they should identify what inputs, outputs, and outcomes to 9 measure. Inputs and outputs are often relatively straightforward, and are most commonly the amount of 10 employee time or agency money dedicated to the program or project. When possible, agencies should tie 11 inputs to processes, outputs, and even outcomes. Doing so provides a measure of efficiency, or the 12 amount of inputs required to produce a process, output, or outcome. For example, a $1000 expenditure 13 allowed the agency to develop a new online tool that increased turnout by 5 percent, and when asked, 80 14 percent of participants felt this tool increased their ability to participate in engagement activities. This sort 15 of performance measurement provides a logical chain linking how the public’s dollars are spent, what 16 they are used for, and what results are achieved per dollar invested. 17

Outputs are commonly direct results of the processes; in the case of public engagement, outputs 18 could be 19

the number of attendees at an event, 20 the amount of comments received, or 21 the number of unique hits on a website. 22

These measures are less desirable than outcomes, as outcomes provide a direct (and not proxy) measure 23 of performance. 24

Outcomes can be more complicated, as they often reflect the overall goal the program or project 25 wishes to achieve. In the case of public engagement, a measure could be the ability of the community to 26 influence the planning process, how well the final project reflects the community’s values, or another 27 related measure. It can be difficult to accurately quantify outcomes (especially with a process as complex 28 as public engagement) due to their often intangible and more qualitative attributes. Agencies could 29 address this by using opinion polling that asks participants how well the engagement process achieved the 30 three best principles. This is discussed in detail below. 31

Finally, gathering and analyzing the performance measures are pointless activities if the 32 information is not used. Agencies and programs should take the performance information gained and use 33 it to incentivize performance, allocate resources, and improve performance generally. Agencies should 34 note data that reveal areas for improvement, and further analyze the data to understand why the deficiency 35 occurred and identify strategies for improvement in the future. When appropriate, the data can be used to 36 allocate resources to areas that have the largest return on investment. Agencies can also use performance 37 measures to measure employee performance and tie bonuses or other monetary incentives to encourage 38 employees to achieve higher results. This must be used with caution, however, as some results are beyond 39 employees’ control. 40

Jason Wagner 10

1

APPLYING THE PRINCIPLES – EVALUATING PERFORMANCE 2 A set of best principles will help agencies understand the goals of the process of public engagement, but 3 applying them to measure performance can be a more difficult task. As previously discussed, the process 4 of public engagement varies widely depending on the goals, scope, and intent of the project. These 5 hurdles make measuring the performance of public engagement difficult, but not impossible. The 6 framework developed here provides a set of principles that allow practitioners to measure their ability to 7 achieve the best practices laid out in the literature. 8

The evaluative framework for this paper utilizes the concept of performance scorecards to 9 illustrate the process of measuring performance. A performance scorecard provides a rubric (a framework 10 or procedural method) to understand quickly and intuitively the performance of the organization. The 11 scorecard developed here is a modified version of the balanced scorecard framework laid out by Kaplan 12 and Norton (17). 13

The balanced scorecard, developed with the private sector in mind, is an alternative way for 14 companies to supplement traditional financial measures with measures from additional perspectives. The 15 original scorecard enabled companies to “link long-term strategic objectives with short-term actions” (17 16 p. 3). This study adapts the private-sector orientation and applies it to the public engagement in 17 transportation planning. Rather than organizing the scorecard by four perspectives, the scorecard uses 18 three principles to guide the measurement. This shift was necessary modification due to the level of 19 analysis and purpose. Kaplan’s balanced scorecard is a tool to measure the performance of an entire 20 private sector company. The scorecard here is used to measure the performance of a specific activity 21 inside a public organization. These differences warranted the modification. This sort of modification is 22 not without precedent in the management literature (18). 23

The performance measurement system discussed here relies almost entirely on the use of surveys 24 to poll attendees. The surveys can use a Likert Scale system and/or a yes/no questionnaire. The use of 25 opinion polling will allow attendees to gauge the effectiveness of the engagement activity at achieving the 26 various metrics found in the literature as best practices. As a note, a Likert Scale is a tool used to assess 27 opinions regarding psychological questionnaires and evaluations. The scale consists of a question and 28 answer area, where the respondent selects the option that best corresponds to their opinion. There are 29 usually five or more possible answers with varying levels of agreement, ranging from “strongly agree” to 30 “strongly disagree”. 31 32

Developing the Scorecard 33 The scorecard is organized by principles, objectives, measures, and goals, with a column for results. The 34 principle column signifies each of the three best principles identified from the literature. An objective is a 35 narrower aspect of each principle that describes a concept the agency wishes to achieve. For example, an 36 aspect of accessibility is ensuring that individuals who wish to participate can do so without an undue 37 burden. A measure is a quantifiable way to assess the level at which an objective was attained. A goal is 38 the measure operationalized—a quantified target that the agency aims to achieve. The results column 39 provides an area to document the actual level of performance following completion of the public 40 engagement activity. 41

An agency that wishes to measure its performance using the scorecard develops objectives, 42 measures, and goals as appropriate to its needs. Alternatively, the organization can borrow aspects of the 43 scorecard developed here, and tailor other aspects to fit its needs (maintaining the objectives and 44 measures while changing the goals, for example). 45

In the example, an agency measuring its performance provided two objectives for achieving 46 accessibility. There are two measures associated with the first objective and one with the second. For the 47

Jason Wagner 11

first measure, the agency hopes that 80 percent of respondents feel that they were able to participate 1 without an undue amount of trouble. 2

In the example, the agency did not meet its goal. Instead, it received 70 percent affirmative 3 responses. This result would provide valuable information that the agency could use to influence future 4 engagement efforts. The agency can develop strategies to improve performance at this measure in the 5 future. Additionally, if the agency maintained the same language in their questions over time, the agency 6 can track performance and monitor any changes. An agency can track when it implemented changes in 7 the engagement process and see if there is an impact from the changes over time. This provides the added 8 benefit of having a larger dataset, which reduces the effects of outliers and improves the data’s 9 robustness. 10

As another example, practitioners could hold the public meeting, and distribute an exit survey to 11 participants following the meeting. The questions on the survey would all provide data that indicate how 12 well the agency did at accomplishing each measure. An example question posed to participants might ask, 13 “Did you feel that you were able to attend today’s event without an undue amount of difficulty?” The 14 response section would provide a Likert Scale, with options ranked from one to five, with five indicating 15 that the individual strongly agrees with the statement, and a one indicating that the respondent strongly 16 disagrees with the statement. The agency would set its goals beforehand to indicate what level of 17 performance they hope to attain across all measures. After gathering all results, the agency could compare 18 their actual performance to their goals and use those data to identify areas that need improvement. The 19 data would enable practitioners to target weaker areas and increase the amount of time and resources 20 spent improving these specific areas of the engagement process. 21

Agencies would gather data from participant surveys for each of the measures. Collectively, the 22 data would present a snapshot of how well the agency performed for each measure, objective, and 23 principle. The agency could perform well in the area of accessibility, but not as well in the area of 24 engagement. The agency would know to dedicate more time and resources to improving these areas. 25

If the agency wished, it could also link inputs to the outputs and outcome measures. To do this, an 26 agency can document how much money it spent on the various areas, and what was the result of those 27 expenditures. If maintained over time, this would provide an ability to track any changes in performance 28 related to expenditures. For example, spending more money developing new platforms for engagement 29 might boost turnout and participants’ feelings on accessibility or engagement measures. Tracking these 30 areas with performance measures that link inputs to outcomes and outputs provides logical (if not causal) 31 evidence suggesting that increases in expenditures helped to increase performance. 32

Measuring the performance of public engagement for transportation planning activities provides 33 practitioners with valuable information they can use to improve performance, build trust, and increase 34 transparency. While often a difficult process, this paper proposes a performance measurement system that 35 will alleviate some of the burden by establishing three principles toward which practitioners can strive. 36 37

Jason Wagner 12

TABLE 3 Example Performance Scorecard 1

Principle Objective Measure Goal Results A

cces

sib

le

All who wished to

participate had the

ability to do so without

an undue burden

Individuals felt they were able

to participate without an

undue amount of trouble

80 percent or more

affirmative responses 70 percent

Individuals who declined to

attend, did so out of choice

and not inability

80 percent or more

affirmative responses 90 percent

Individuals felt comfortable

providing input in at least one

of the platforms utilized

80 percent or more

affirmative responses 80 percent

Individuals felt the platforms

facilitated their attendance

and participation

80 percent or more

affirmative responses 80 percent

Events were

democratic and

representative of the

population

Individuals attending the

events roughly represent the

population

Minority groups’

attendance rates were

within 5 percentage

points of their actual

share in the population

Yes

En

gagin

g

The process fostered an

environment favorable

for input and

collaboration

Individuals felt their opinions

were heard and valued

80 percent or more

affirmative responses 70 percent

Individuals felt the process

encouraged collaboration

80 percent or more

affirmative responses 90 percent

Individuals felt the activities

were engaging

80 percent or more

affirmative responses 70 percent

The process was

ongoing

Individuals used the feedback

mechanisms

50 percent or more of

the individuals used

the feedback

mechanisms

60 percent

Ou

tcom

e-ori

ente

d The input provided

from the public

influenced the

decision-making

process

Individuals felt their opinions

would influence the decision-

making

70 percent or more

affirmative responses 70 percent

The process

successfully engaged

the public

Individuals felt the

engagement process, as a

whole, was successful

70 percent or more

affirmative responses 75 percent

2 3

CONCLUSION AND AREAS FOR FUTHER STUDY 4 This study reviews the literature surrounding the best practices in engaging the public for input on 5

transportation projects. It provides three principles that encapsulate the recommended practices and a 6 performance scorecard based on these principles. This scorecard will allow practitioners to evaluate their 7 performance at engaging the public, and will produce data to improve further both the practice of public 8 engagement and the ability to evaluate performance. 9

This study is the initial step in this process. Implementing the three-principle-based scorecard as a 10 pilot study would be an opportunity to assess the effectiveness of using this system, determine the 11 system’s strengths and areas for improvement, and further improve the practice of measuring the 12 performance of public engagement. 13 14

Jason Wagner 13

1

2 WORKS CITED 3

1. Sale, D., S. Stafford, and S. Davis. Measuring the Success of Public Involvement. Participation 4 Quarterly, 2006, pp. 12-15. 5 6

2. Bickerstaff, K., R. Tolley, and G. Walker. Transport Planning and Participation: The Rhetoric and 7 Realities of Public Involvement. Journal of Transport Geography, Vol. 10, 2002, pp. 61-73. 8 9

3. Quick, K. S., Z. J. Zhao. Suggested Design and Management Techniques for Enhancing Public 10 Engagement in Transportation Policy Making. Publication CTS 11-24. Center for Transportation 11 Studies − University of Minnesota, 2011. 12 13

4. Rowe, G., and L. J. Frewer. Public Participation Methods: A Framework for Evaluation. Science, 14 Technology & Human Values, Vol. 25, No. 3, 2000, pp. 3-29. 15 16

5. Federal Highway Administration. Public Involvement Techniques for Transportation Decision-17 Making. FHWA, U.S. Department of Transportation, 2002. 18 http://www.planning.dot.gov/PublicInvolvement/pi_documents/toc-foreword.asp. Accessed July 19 2012. 20 21

6. Federal Highway Administration. NEPA and Transportation Decisionmaking. Environmental Tool 22 Kit. FHWA, U.S. Department of Transportation, September 1996. 23 http://environment.fhwa.dot.gov/projdev/tdmpi_p_d.asp. Accessed July 2012. 24 25

7. Federal Highway Administration. Public Involvement Strategies for Shaping Communities. FHWA, 26 U.S. Department of Transportation, 2011. http://www.fhwa.dot.gov/tcsp/case8.htm. Accessed July 27 2012. 28 29

8. O’Connor, R., M. Schwartz, J. Schaad, and David Boyd. State of the Practice: White Paper on Public 30 Involvement. Transportation in the New Millennium, 2000, pp. 1-7. 31 32

9. Weeks, J. Public Involvement by Minorities and Low-Income Populations: Removing the Mystery. 33 TR News. May–June 2002, pp. 25-31. 34 35

10. Innes, J., and D. Booher. Reframing Public Participation: Strategies for the 21st Century. Planning 36 Theory & Practice. December 2004, pp. 419-436. 37 38

11. King, C., K. Feltey, and B. Susel. The Question of Participation: Toward Authentic Public 39 Participation in Public Administration. Public Administration Review, Vol. 58, July−August 1998, 40 pp. 317-326. 41 42

12. Lukensmeyer, C., and L. Torres. Public Deliberation: A Manager’s Guide to Citizen Engagement. 43 IBM Center for The Business of Government, 2006. Collaboration Series Report. 44 http://www.businessofgovernment.org/sites/default/files/LukensmeyerReport.pdf. Accessed July 45 2012. 46 47

Jason Wagner 14

13. Bittle, S., C. Haller, and A. Kadlec. Promising Practices in Online Engagement. Public Agenda, 1 2009. http://publicagenda.org/files/pdf/PA_CAPE_Paper3_Promising_Mech2.pdf. Accessed July 2 2012. 3 4

14. Matley, T. Effective Public Involvement in Transportation. TR News. Vol. 220, May–June 2002, 5 pp. 4–7. 6 7

15. Hatry, Harry P. Performance Measurement: Getting Results. 2nd

edition. The Urban Institute Press, 8 Washington D.C, 2006. 9 10

16. Federal Highway Administration. Moving Ahead for Progress in the 21st Century Act (MAP-21): A 11 Summary of Highway Provisions. FHWA, U.S. Department of Transportation, 2012. 12 http://www.fhwa.dot.gov/map21/docs/map21_summary_hgwy_provisions.pdf. Accessed July 2012. 13

14 17. Kaplan, R. S. and D. Norton. Using the Balanced Scorecard as a Strategic Management System. 15

Harvard Business Review. January-February, 1996. 16 17 18. Kaplan, R. S. and D. Norton. Transforming the Balanced Scorecard from Performance Measurement 18

to Strategic Management: Part I. Accounting Horizons. March, 2001, Vol. 15, 1. 19 20