Embed Size (px)
Citation preview
Process Evaluation
Process EvaluationProcess Evaluation
Objectives:• Overview of ‘process evaluation’• Steps for developing a process evaluation
plan• Measures
Process Evaluation - WhyProcess Evaluation - Why
Emphasis has been on outcome evaluation to determine program success
Some see the ‘bottom line’ as everything Can’t ignore ‘bottom line’, but “good ideas do
not always yield good results” ‘Hard’ evidence of program impact is
grounded in how an intervention is implemented and delivered
Process Evaluation - WhyProcess Evaluation - Why
A program’s lack of success could be related to problems with program design and/or delivery, and failure to reach target population
Process evaluation has the potential to ensure successful program implementation as well as increase understanding of outcome results (Babbie & Mouton, 2001; Pawson & Tilley, 2004)
Process EvaluationProcess EvaluationFormative evaluation: use of evaluation to improve program during the development phaseProcess evaluation: evaluates actual delivery of services vs. those intendedImplementation: evaluation of all of the activities focused on the actual operation of a programQuality assurance: systematic process of checking to see whether a product or service being developed is meeting specified requirements
Process Evaluation - WhatProcess Evaluation - What What was the program intended to be?
Standards of Care? Best practices? Reach? (Specification of program components, review of historical program data)
What is delivered in reality?
(Methods for measuring program functioning) Identification of gaps between the intended
and the actual program delivery and reach
Process Evaluation - WhatProcess Evaluation - What
“The use of empirical data to assess the delivery of programs…...verifies what the program is, whether or not it is delivered as intended to the targeted recipients and in the intended dosage”
(Scheirer, p.40, in Handbook of Practical Program Evaluation, Wholey et al., 1994)
Process Evaluation - WhyProcess Evaluation - Why1. Provides information on quality of services, and
whether program was implemented as planned2. Assesses whether or not program is reaching
the intended recipients, and their level of satisfaction
3. Increases knowledge about program components contributing to outcomes – extent to which they’re working or not working
4. Increases understanding about successful implementation of programs in complex organizational and community contexts
Process Evaluation - WhyProcess Evaluation - Why
5. Identifies and addresses program implementation problems in order to improve the program
6. Links program processes to program outcomes
7. Identifies organizational, program, and individual factors facilitating or impeding program implementation
Process Evaluation - WhoProcess Evaluation - Who
Funders Agency and program staff Community stakeholders Program participants
Process Evaluation - HowProcess Evaluation - How Successful programs based on set of standards and
objectives:• Competent staff• Good management• Adequate facilities• Sufficient funding
Producing sufficient outputs to meet client needs• Output level• Output quality
Reaching the intended target population
Process Evaluation – HowProcess Evaluation – HowSix Steps (Steckler & Linnan, 2002)Six Steps (Steckler & Linnan, 2002)
1. Full description of program: Specification of purpose, underlying theory, objectives, strategies, expected impacts and outcomes. (conveyed in program logic model)
2. Describe all elements consistent with optimum delivery of program – strategies, activities, staffing. This may include identification of factors (internal and external) contributing to program implementation and delivery problems
Process Evaluation Questions - HowProcess Evaluation Questions - How
3. Develop list of potential process evaluation questions
The most useful questions reflect:• A diversity of stakeholder perspectives• Key components of your program• Priority information needs• Available resources to answer your key
evaluation questions
Process Evaluation Questions - HowProcess Evaluation Questions - How
Process evaluation questions address the• Who• What• When• How many
in relation to program inputs, activities, outputs, and target population – the left hand side of your program logic model
Process Evaluation - CategoriesProcess Evaluation - Categories Steckler and Linnan (2002) recommend a minimum
of four elements or ‘categories’ of process evaluation: • Program activities delivered • Program activities received • Reach• Fidelity
They also recommend documenting recruitment procedures and describing the context of the intervention.
Process Evaluation Questions - HowProcess Evaluation Questions - How
Coverage – extent of target population participation in program• Activity, service, or agency level• System level
Service Delivery – extent to which program components are being delivered as planned• Activity, service, or agency level• System level
Questions about coverage – Questions about coverage – At the activity, service, or agency levelAt the activity, service, or agency level
• Has the activity, service, or agency served the intended clients?
• What were the demographic and clinical characteristics of clients?
• What proportion of clients completed treatment, how many sessions were completed, and at what point did they drop-out? What were the characteristics of those who dropped out?
Questions about coverage – Questions about coverage – At the system levelAt the system level
How many treatment programs exist for this target population within the region?
How many clients are seen by each program in a year?
Are there differences in the types of clients seen at each program?
Questions about service delivery – Questions about service delivery – At the activity, service, or agency levelsAt the activity, service, or agency levels
By what route did clients enter treatment? What services were actually delivered to
clients in treatment and is this what was intended?
What was the average length of stay or the average number of appointments kept?
Questions about service delivery – Questions about service delivery – At the system levelAt the system level
Are different treatment programs aware of one another?
Do different treatment programs refer clients to one another?
What is the relationship between general medical services and specialized treatment programs?
Process Evaluation Questions - HowProcess Evaluation Questions - How
Steps to identifying questions with staff and stakeholders:• Brainstorm questions relating to program,
using logic model to guide process• Sort questions from brainstorming session
into relevant categories
Process Evaluation: Six StepsProcess Evaluation: Six Steps
4. Determine methods for process evaluation Qualitative and quantitative data sources
• Qualitative methods: interviews, focus groups, logs, case studies, open-ended survey questions, content analysis of videotaped sessions
• Quantitative methods: surveys, direct observation, checklists, attendance logs, document review
(Baranowski & Stables, 2000; Devaney & Rossi, 1997; McGraw et al, 2000)
Process Evaluation: Six StepsProcess Evaluation: Six Steps
5. Consider program resources and program characteristics and context
Team must consider resources needed to answer questions from step 3 using methods from step 4
• Considerations: time for planning, pilot testing instruments, any training needed, data collection, entry, analysis, and reporting
• Staff time, client burden, disruption to the intervention
Process Evaluation: Six StepsProcess Evaluation: Six Steps
6. Finalize the process evaluation plan Set priorities and ensure feasibility of plan
• Determine priorities◦ Important to staff and stakeholders◦ Address important program needs◦ Reflect key elements of program logic model
• Determine feasibility◦ Can be answered with available resources and
within timeframe
Process Evaluation: MethodsProcess Evaluation: MethodsQualitative Focus groups Key informant interviews Direct observationQuantitative Chart or record reviews Activity logs DemographicsQualitative and Quantitative In-person interviews (clients, staff, management)
Process Evaluation: Selection of Process Evaluation: Selection of MeasuresMeasures
How meaningful is the measure? How feasible is the measure? How useful is the measure?
Process Evaluation: Selection of Process Evaluation: Selection of Measures Measures
Questions involving quality• Budget process – e.g., review of minutes• Quality of services – standard client records
Questions re: program operations• Checklist of intended programming/services
Questions on why a program was developed• Planning documents, meeting minutes,
funding proposals, needs assessments
Process Evaluation - MethodsProcess Evaluation - Methods
Program coverage or reach• Compare actual participation to intended
participation• Provides information about the level of
program participation by the target group • Requires clear definition of the intended target
group, monitoring of key target group characteristics, and rate of participation
Process Evaluation – Methods to Process Evaluation – Methods to Evaluate CoverageEvaluate Coverage
Develop clear definition of target population Identify key characteristics of target population for
monitoring purposes Collect data on identified characteristics Analyze data to determine if actual population
served is the intended population Analyze data to determine if persons served by
program meet eligibility criteria Using characteristics data, determine which
subgroups are over/under represented Analyze characteristics of program drop-outs
Process Evaluation: Methods to Process Evaluation: Methods to Evaluate DeliveryEvaluate Delivery
Program records:• Specify evaluation questions re: program delivery• Identify sources of data to answer each question• Identify sources of data to answer each question• Assess adequacy of current sources of data• Develop a data collection plan – sources of data, who
will collect data, etc.• Develop an analysis plan (e.g., tables, graphs,
figures)
Process Evaluation: MethodsProcess Evaluation: Methods Client satisfaction or perception of care
• Client surveys can assess overall satisfaction with services as well as information regarding content of program and manner in which it was delivered
• Established measures: ◦ Client Satisfaction Questionnaire (CSQ-8)◦ Youth Services Survey for Families◦ Consumer-Oriented Mental Health Survey◦ Kansas Family Satisfaction Survey
Process Evaluation: Selection of Process Evaluation: Selection of MeasuresMeasures
Standardized measures:• Example: The National Inventory of Mental
Health Quality Measures (developed by the Center for Quality Assessment and Improvement in Mental Health (CQAIMH): http://www.cqaimh.org/quality.html
Future launch of the Centre’s Measures Database
Process Evaluation – Tips for Data Process Evaluation – Tips for Data CollectionCollection
Create a schedule for collecting process data and stick to it
Divide tasks & assign individuals to each task Maximize existing opportunities for data
collection Only identify questions of interest/value and
keep the evaluation focused Periodically review your instruments to
ensure evaluation approach is still useful
Next StepsNext Steps
March 27: webinar on writing the final report March 31: Award term ends April 30: Final report due May 1 to 14: Evaluation readiness
assessment for the Evaluation Implementation Grants (EIG)
June 2: Deadline for EIG grant applications June webinar: The road to data collection
For more informationFor more information
Susan Kasprzak
613-737-7600/ Ext. 3320
Tanya Witteveen
613-737-7600/ Ext. 3483
