EVALUATION 101: HOW CAN WE EVALUATION 101: HOW CAN WE

DEMONSTRATE PROGRAM RESULTS?DEMONSTRATE PROGRAM RESULTS?

Jon E. Burkhardt, WestatJon E. Burkhardt, Westat

Dr. David J. Bernstein, WestatDr. David J. Bernstein, Westat

Prepared for the Prepared for the

National Center on Senior TransportationNational Center on Senior Transportation

Easter Seals / n4aEaster Seals / n4a

December 9, 2008December 9, 2008

22

Presentation OutlinePresentation Outline NCST Project ObjectivesNCST Project Objectives

The transportation improvement processThe transportation improvement process

Performance measurement and evaluation: different Performance measurement and evaluation: different activities, different uses activities, different uses

Applying performance measurement and evaluation to Applying performance measurement and evaluation to transportation programstransportation programs

Summary Summary

Contact informationContact information

Sources and referencesSources and references

33

NCST Project ObjectivesNCST Project Objectives

1. Greater coordination between the aging community and the transportation industry

2. Increasing the family of transportation options for older adults at the local level

3. Ensuring caregivers are educated regarding transportation options

4. Addressing barriers to implementing more transportation services for older adults

44

The Transportation Improvement The Transportation Improvement ProcessProcess

Analyze existing conditions: determine local Analyze existing conditions: determine local transportation needs and resourcestransportation needs and resources

Define community goals, objectives, and evaluation Define community goals, objectives, and evaluation strategiesstrategies

Confirm working relationshipsConfirm working relationships

Design and assess alternative services and Design and assess alternative services and strategiesstrategies

Implement service changesImplement service changes

Evaluate and improve the services Evaluate and improve the services

55

Performance measuresPerformance measures

EvaluationsEvaluations

Which?Which?

What?What?

When?When?

Why?Why?

66

Why All the Fuss?Why All the Fuss?

77

The “What:” Performance MeasurementThe “What:” Performance Measurement

Performance Measurement:Performance Measurement: periodic but regular periodic but regular monitoring and reporting of program monitoring and reporting of program accomplishments, particularly progress towards pre-accomplishments, particularly progress towards pre-established goalsestablished goals

Typical measures: Typical measures: inputs (resources applied to a problem) inputs (resources applied to a problem) outputs (numeric measures of program products)outputs (numeric measures of program products) outcomes (what changed)outcomes (what changed)

88

The “Why:” EvaluationThe “Why:” Evaluation

Evaluations:Evaluations: systematic studies to assess how well systematic studies to assess how well a program is workinga program is working

Some of the possible components:Some of the possible components: The process of implementing the programThe process of implementing the program Report on program objectives achieved [or not Report on program objectives achieved [or not

achieved]achieved] Attribution of the results: Attribution of the results:

to the program? to the program? to events outside of the program?to events outside of the program?

Recommendations for improving program resultsRecommendations for improving program results

99

The DifferencesThe Differences

Performance measurementPerformance measurement

will provide regular data for reports about the will provide regular data for reports about the progress of a programprogress of a program

can explain can explain what is happeningwhat is happening within a program within a program

EvaluationsEvaluations

are more in-depthare more in-depth

help explain help explain why program performance did or did why program performance did or did not changenot change

help attribute performance measures to program help attribute performance measures to program resultsresults

1010

Performance Measurement DataPerformance Measurement Data

Inputs:Inputs: Measures of the resources that are applied in Measures of the resources that are applied in providing servicesproviding services

Activities:Activities: Measures of the services that are being Measures of the services that are being providedprovided

Outputs:Outputs: Measures of the quantity of services provided or Measures of the quantity of services provided or the quantity of a service meeting quality requirementsthe quantity of a service meeting quality requirements

Outcomes:Outcomes: Measures that address the intermediate or Measures that address the intermediate or long-term results of a program on those receiving a servicelong-term results of a program on those receiving a service

1111

Program Outcome ModelProgram Outcome Model

Resources dedicated to or consumed by the program

• Money• # of staff• Staff time• # of volunteers• Volunteer time• Facilities• Equipment•  Supplies• Other

What the program does with the inputs to fulfill its mission

• More staff & volunteers to provide trips

• I & R services• New dispatch

systems• Driver training• New vehicles

The direct products of program activities

• More rides• More riders• Additional

volunteers• Greater service

span• Quicker I & R• Shorter wait time

Benefits for participants during and after program activities

• Improved access• Greater sense of

independence • Increased customer

satisfaction• Increased flexibility• Improved

communication among providers

1212

How to Use Performance MeasuresHow to Use Performance Measures

Improve decision making: Ensure that programs are being implemented. Provide input to day-to-day program management, funding decisions, and support strategic planning for services to clients.

Monitor Service Performance: track resources, program production and results, and monitor the need for more comprehensive examination and analysis (service interventions, evaluations).

Report Results/Be Accountable: Provide information to various parties, including staff, funders, service providers, program partners, clients, and other stakeholders.

1313

Good Performance MeasuresGood Performance Measures

Focus on resultsFocus on results

Are relevant and useful to program managers and Are relevant and useful to program managers and stakeholdersstakeholders

Are readily measurable and countableAre readily measurable and countable

Provide valid, verifiable, and reliable informationProvide valid, verifiable, and reliable information

Are clear and understandable, requiring only Are clear and understandable, requiring only minimal explanation to be understoodminimal explanation to be understood

Can be compared to targets, comparable Can be compared to targets, comparable programs, or legal or quality standardsprograms, or legal or quality standards

1414

Performance Measurement QuestionsPerformance Measurement Questions

1.1. How many people are being served now compared How many people are being served now compared with before we started?with before we started?

2.2. How many trips are provided now? How many How many trips are provided now? How many were provided previously?were provided previously?

3.3. Is my program meeting its targets?Is my program meeting its targets?

4.4. Is my program efficient? Effective? Cost Is my program efficient? Effective? Cost effective?effective?

5.5. How does my program measure up against other How does my program measure up against other programs?programs?

1515

Evaluation QuestionsEvaluation Questions

1.1. Can the results of our program be attributed to our Can the results of our program be attributed to our program?program?

2.2. How can we meet the needs of our community?How can we meet the needs of our community?

3.3. Did our program meet its goals? Why or why not?Did our program meet its goals? Why or why not?

4.4. Did the way we implemented our program Did the way we implemented our program influence the results we got, or did not get?influence the results we got, or did not get?

5.5. How can we improve our program?How can we improve our program?

1616

Primary Data Collection DecisionsPrimary Data Collection Decisions

Who will collect what kinds of data?

Which data will be collected at which points in time?

How will the data be used?

1717

Potential Evaluation ComponentsPotential Evaluation Components

System characteristics: Resources (inputs)

Performance measures: Efficiency (cost / mile, etc.) Effectiveness (trips / mile) Cost effectiveness ($ / trip)

Service quality: Consumer and management data

Service evaluations: Outcomes and impacts

1818

Statistics for Performance MeasuresStatistics for Performance Measures Services deliveredServices delivered

Vehicle miles of serviceVehicle miles of service Vehicle hours of serviceVehicle hours of service

Services consumedServices consumed

Unlinked passenger tripsUnlinked passenger trips Unduplicated persons servedUnduplicated persons served

Fully allocated costsFully allocated costs

All costs required to provide transportation serviceAll costs required to provide transportation service

1919

Service Quality ComponentsService Quality Components

Acceptability: reliability, connections, trust, comfort, respect

Accessibility: can physically use, can get information to use, proximity

Adaptability: flexibility, responds to specific requests, meets trip needs and special needs of clients

Affordability: not excessive money, time, or effort required to travel

Availability: frequency, hours / days / places available

2020

An Example of Outcome EvaluationAn Example of Outcome Evaluation

GOAL: Increase seniors’ knowledge and use of transportation options

EVALUATION STEPS: Do seniors know more about the options?

Have seniors increased their use of options?

Which outreach and education activities have been undertaken?

How do we know that the outreach and education activities are responsible for the changes?

2121

Sources of Service Quality DataSources of Service Quality Data

Dispatch and driver logsDispatch and driver logs

Records of complaintsRecords of complaints

User surveysUser surveys

2222

Survey: What’s your objective?Survey: What’s your objective?

Needs analysis:Needs analysis: Ask potential riders [or their Ask potential riders [or their advocates] to find out their needsadvocates] to find out their needs

Customer satisfaction:Customer satisfaction: Ask current transportation Ask current transportation users about their satisfaction with servicesusers about their satisfaction with services

Volunteer driver assessment:Volunteer driver assessment: Ask volunteer drivers Ask volunteer drivers [and passengers] about their experiences[and passengers] about their experiences

Service provider perspectives:Service provider perspectives: Ask existing service Ask existing service providers what improvements are neededproviders what improvements are needed

2323

Survey ConsiderationsSurvey Considerations

Different strategies required for different groupsDifferent strategies required for different groups

How to administer the survey (by mail, phone, How to administer the survey (by mail, phone, internet, in-person, or some combination?)internet, in-person, or some combination?)

When to administer (before, after, or before and When to administer (before, after, or before and after?)after?)

Pilot test your survey and refine it based on the pilot Pilot test your survey and refine it based on the pilot test resultstest results

2424

SummarySummary

Monitoring and evaluation shouldMonitoring and evaluation should

Be based on valid data and replicable analysesBe based on valid data and replicable analyses

Support day-to-day program management and Support day-to-day program management and operations: expand / contract; continue / changeoperations: expand / contract; continue / change

Help measure progress towards goals and objectivesHelp measure progress towards goals and objectives

Indicate potential service and program improvements Indicate potential service and program improvements

Provide accountability to funding sourcesProvide accountability to funding sources

Support program continuationSupport program continuation

2525

Contact InformationContact Information

WESTATWESTAT1650 Research Blvd1650 Research BlvdRockville, Maryland 20854Rockville, Maryland 20854

Jon BurkhardtJon BurkhardtPhone: 301/294-2806Phone: 301/294-2806JonBurkhardt@Westat.comJonBurkhardt@Westat.com

David BernsteinDavid BernsteinPhone: 301/738-3520Phone: 301/738-3520DavidBernstein@Westat.comDavidBernstein@Westat.com

2626

Sources and ReferencesSources and ReferencesSlide 6: United Features Syndicate, 3/13/99Slide 6: United Features Syndicate, 3/13/99

Slides 7-9: U.S. General Accounting Office. Performance Slides 7-9: U.S. General Accounting Office. Performance Measurement and Evaluation: Definitions and Relationships. Measurement and Evaluation: Definitions and Relationships. April 1998. April 1998. http://www.gao.gov/archive/1998/gg98026.pdfhttp://www.gao.gov/archive/1998/gg98026.pdf and and Wholey, J., Hatry, H., and Newcomer, K. (2004). Handbook of Wholey, J., Hatry, H., and Newcomer, K. (2004). Handbook of Practical Program Evaluation. Practical Program Evaluation. http://www.josseybass.com/WileyCDA/WileyTitle/productCd-078796http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0787967130.html7130.html

Slide 10: Governmental Accounting Standards Board [GASB]. Slide 10: Governmental Accounting Standards Board [GASB]. (1994). (1994). Concepts statement no. 2, Service efforts and Concepts statement no. 2, Service efforts and accomplishments reportingaccomplishments reporting. . http://www.seagov.org/resources/glossary.shtmlhttp://www.seagov.org/resources/glossary.shtml and Montgomery and Montgomery County, MD. (March 2006). Montgomery Measures Up; County, MD. (March 2006). Montgomery Measures Up; http://www.montgomerycountymd.gov/content/omb/FY07/mmurec/hhttp://www.montgomerycountymd.gov/content/omb/FY07/mmurec/howtoread.pdfowtoread.pdf

2727

Sources and References (cont.)Sources and References (cont.)Slide 11: United Way of America (2006). Measuring Program Slide 11: United Way of America (2006). Measuring Program

Outcomes: A Practical Approach. Outcomes: A Practical Approach. http://http://www.liveunited.org/Outcomes/Resources/MPO/model.cfmwww.liveunited.org/Outcomes/Resources/MPO/model.cfm

Slide 12: Epstein, P.D. (1988). Slide 12: Epstein, P.D. (1988). Using performance measurement in Using performance measurement in local government: A guide to improving decisions, performance, local government: A guide to improving decisions, performance, and accountabilityand accountability. New York, National Civic League Press.. New York, National Civic League Press.

Slide 13: Bernstein, D.J. (2000). Slide 13: Bernstein, D.J. (2000). Local Government Performance Local Government Performance Measurement Use: Assessing System Quality and EffectsMeasurement Use: Assessing System Quality and Effects. . Washington, DC: George Washington University. Available from Washington, DC: George Washington University. Available from ProQuest-University Microfilms Inc., ProQuest-University Microfilms Inc., http://www.umi.com/hp/Products/Dissertations.htmlhttp://www.umi.com/hp/Products/Dissertations.html..

Slides 17 - 19: Burkhardt, J.E. (2004) Slides 17 - 19: Burkhardt, J.E. (2004) Critical Measures of Transit Critical Measures of Transit Service Quality in the Eyes of Older Travelers.Service Quality in the Eyes of Older Travelers. Transportation Transportation Research Record No. 1835Research Record No. 1835, Transportation Research Board, , Transportation Research Board, Washington DC.Washington DC.