Building the Knowledge BaseBuilding the Knowledge BaseHow to Successfully Evaluate A TCSP ProjectHow to Successfully Evaluate A TCSP Project

presented bypresented by

William M. LyonsWilliam M. LyonsU.S. Department of Transportation/Volpe CenterU.S. Department of Transportation/Volpe CenterandandJohn H. SuhrbierJohn H. SuhrbierChristopher D. PorterChristopher D. PorterCambridge Systematics, Inc.Cambridge Systematics, Inc.

September 14/15, 2000September 14/15, 2000

AgendaAgenda

Role of evaluation in the TCSP programRole of evaluation in the TCSP program

• William LyonsWilliam Lyons

Responsibilities for evaluationResponsibilities for evaluation

• William LyonsWilliam Lyons

Guidelines for project evaluationsGuidelines for project evaluations

• John Suhrbier and Chris PorterJohn Suhrbier and Chris Porter

Examples of approaches to project evaluation Examples of approaches to project evaluation

• Washington, DCWashington, DC

• Hartford, CTHartford, CT

Questions and discussionsQuestions and discussions

Why Evaluation?Why Evaluation?

Legislative intent of Section 1604 of TEA-21 (TCSP)Legislative intent of Section 1604 of TEA-21 (TCSP)

• Emphasis on Emphasis on learninglearning

Improve knowledgeImprove knowledge of the costs and benefits of the costs and benefitsof innovative approaches to integrating:of innovative approaches to integrating:

• Transportation investments and strategiesTransportation investments and strategies

• Community preservation, Community preservation,

• Land development patterns, andLand development patterns, and

• Environmental qualityEnvironmental quality

InvestigateInvestigate relationships between transportation and relationships between transportation and community and system preservationcommunity and system preservation

Project and Program EvaluationProject and Program EvaluationCrucial and ComplementaryCrucial and Complementary

Project evaluationProject evaluation

• Results of individual projectsResults of individual projects

• Responsibility of grantee, with technical assistanceResponsibility of grantee, with technical assistance

• Direct source of learningDirect source of learning

• Challenge Challenge objectivity, quality, consistency, credibility objectivity, quality, consistency, credibility

- Results useful locally as well as nationallyResults useful locally as well as nationally

Program evaluationProgram evaluation

• Project evaluations, taken togetherProject evaluations, taken together

• Focus on program goals, analysis, synthesisFocus on program goals, analysis, synthesis

• Assessment of success of overall programAssessment of success of overall program

Desired Results Desired Results What We Are Trying to DoWhat We Are Trying to Do

Build national knowledge base, leverage resultsBuild national knowledge base, leverage results

Provide insights into what is and what is not workingProvide insights into what is and what is not working

Evaluate innovations and results through systematic analysis Evaluate innovations and results through systematic analysis

• Document learningDocument learning

- Problems as well as successesProblems as well as successes

- Falling short of expected goals is acceptableFalling short of expected goals is acceptable

Foster innovation, encourage experimentationFoster innovation, encourage experimentation

Demonstrate success of programDemonstrate success of program

Context Context Major ConsiderationsMajor Considerations

Evaluation plan Evaluation plan how comprehensive? how comprehensive?

Timing Timing early planning and action are critical early planning and action are critical

Focus on key innovationsFocus on key innovations

Process versus outcome results Process versus outcome results

Description versus documentation and credible analysisDescription versus documentation and credible analysis

Importance of flexibilityImportance of flexibility

Effort should be interesting and worthwhile for all involvedEffort should be interesting and worthwhile for all involved

Session focus Session focus practical strategies to improve evaluation practical strategies to improve evaluation

Roles and ResponsibilitiesRoles and ResponsibilitiesHow to Improve Effectiveness?How to Improve Effectiveness?

GranteesGrantees evaluation of individual TCSP projects evaluation of individual TCSP projects

FHWA HQFHWA HQ evaluation of overall TCSP pilot program evaluation of overall TCSP pilot program

• Availability of technical assistanceAvailability of technical assistance• Resource centersResource centers• U.S. DOT/Volpe Center and Cambridge Systematics, Inc. U.S. DOT/Volpe Center and Cambridge Systematics, Inc.

Division OfficesDivision Offices • Encourage and support evaluationEncourage and support evaluation• Point of contact for technical assistancePoint of contact for technical assistance• Need for flexibilityNeed for flexibility• What’s realistic?What’s realistic?

TCSP EvaluationTCSP EvaluationMost Simply IsMost Simply Is

Identifying the set of messages to be used inIdentifying the set of messages to be used in

Telling the TCSP storyTelling the TCSP story

What is Evaluation?What is Evaluation?

TCSP grants are resulting in things (planning and projects) TCSP grants are resulting in things (planning and projects) being done differentlybeing done differently

Most basically, evaluation addressesMost basically, evaluation addresses

• What are the What are the benefitsbenefits of these changes? of these changes?

• What actions are important in achieving these benefits?What actions are important in achieving these benefits?

• What do these changes cost in terms of time and dollars?What do these changes cost in terms of time and dollars?

• Are longer term cost savings being achieved?Are longer term cost savings being achieved?

• Are the benefits worth the extra effort?Are the benefits worth the extra effort?

• How can these successes be repeated, or How can these successes be repeated, or “institutionalized”, within other projects?“institutionalized”, within other projects?

Target Audiences For Project Evaluation ResultsTarget Audiences For Project Evaluation Results

TCSP project staffTCSP project staff

• Help refine project work planHelp refine project work plan- What is working?What is working?- What is not working?What is not working?

Local elected and appointed officialsLocal elected and appointed officials

• What is the project accomplishing that is different?What is the project accomplishing that is different?

Other TCSP granteesOther TCSP grantees

• Learning from othersLearning from others

Input to overall TCSP program evaluationInput to overall TCSP program evaluation

EvaluationEvaluationAssessment of Experience to DateAssessment of Experience to Date

Most people (grantees, FHWA, and other program participants)Most people (grantees, FHWA, and other program participants)

• Recognize that TCSP is resulting in lots of interesting Recognize that TCSP is resulting in lots of interesting initiativesinitiatives

• Acknowledge importance of assessing benefits of Acknowledge importance of assessing benefits of these initiativesthese initiatives

• Want to do a good jobWant to do a good job

• They have only a limited understanding of evaluation They have only a limited understanding of evaluation approaches and methodologiesapproaches and methodologies

• Not really sure how to get startedNot really sure how to get started

Most also admit Most also admit

• Anecdotal evidence is helpful but not sufficientAnecdotal evidence is helpful but not sufficient

• Evaluation is more than public relations or marketingEvaluation is more than public relations or marketing

Available Evaluation ResourcesAvailable Evaluation Resources

Guidance for the preparation of TCSP evaluation plansGuidance for the preparation of TCSP evaluation plans

Selected references evaluating the relationships between Selected references evaluating the relationships between travel and land use, including contact informationtravel and land use, including contact information• Literative reviews (9)Literative reviews (9)• Individual studies (16)Individual studies (16)

Evaluation examplesEvaluation examples• Upgrade of a rural highway corridorUpgrade of a rural highway corridor• Transit supportive land use developmentTransit supportive land use development• Implementation of urban design, land use, and TDM, Implementation of urban design, land use, and TDM,

strategies for an urban office complexstrategies for an urban office complex

TCSP web siteTCSP web site

Technical assistance Technical assistance

Components of an Evaluation ProcessComponents of an Evaluation Process

DefinitionDefinition

ProjectProject

InitiationInitiation

On-GoingOn-Going

CompletionCompletion

Assess Implementation Assess Implementation ExperienceExperience

Compare “After” to Compare “After” to “Before” Conditions“Before” Conditions

DiscretionaryDiscretionaryGrantsGrants

Preliminary Preliminary Evaluation PlanEvaluation Plan

ReportingReporting

Refined Refined Evaluation PlanEvaluation Plan

EarmarkedEarmarkedGrantsGrants

Evaluation PlanEvaluation Plan

ReportingReporting

Assess Implementation Assess Implementation ExperienceExperience

Compare “After” to Compare “After” to “Before” Conditions“Before” Conditions

Steps in Developing an Evaluation Plan Steps in Developing an Evaluation Plan

Identify Data Identify Data Sources and Sources and Evaluation Evaluation

MethodsMethods

Identify Identify Performance Performance

MeasuresMeasures

Define Project Define Project Goals and Goals and ObjectivesObjectives

How Much Effort is Required For Evaluation?How Much Effort is Required For Evaluation?

For a planning grantFor a planning grant

• Not muchNot much

• More a willingness to step back and ask the right questionsMore a willingness to step back and ask the right questions

For an implementation grantFor an implementation grant

• More effort requiredMore effort required

• Longer term monitoring possible withLonger term monitoring possible with

- Before/after dataBefore/after data

- Statistical controlStatistical control

TCSP Web SiteTCSP Web SiteHome PageHome Page

Web PageWeb PageProject DescriptionsProject Descriptions