Upload
deborah-casey
View
221
Download
4
Tags:
Embed Size (px)
Citation preview
Program Evaluation (by M. Secret)
What is the purpose of program evaluation
To investigate social programs
To assess effectiveness of social policies and programs.
The language of evaluation: Fill in the
Blank the impact of the program process on the cases processed
Outcomes
the services delivered or new products produced by the program processOutputs
Population for whom the program is designed.Target Population
information about service delivery system outputs, outcomes, or operations that is available to any program inputs
Feedback
individuals and groups who have some basis of concern with the program, often setting the research agenda and controlling research findings
Stakeholders
the complete treatment or service delivered by the programProgram process
resources, raw materials, clients, and staff that go into a programIn puts
Question FIRST!!!
Questions to be answered• Is the program needed?
Do a needs assessment
• How does the program operate?Do a formative evaluation
• What is the program’s impact?Do a summative evaluation
• How efficient is the program?Do a cost benefit or a cost effectiveness analysis
The specific methods depends on the question of interest
Definition of Needs Assessment Terms*
Needs Assessment:Systematically researching questions about the needs of a target
population for program planning purposes
Key Informants: expert opinions from individuals who have special knowledge about
the needs and about the existing services
Rates under treatment: secondary analysis of existing statistics to estimate need for services
based on number and characteristics of clients who are already being served.
Social Indicators: existing statistics that reflect conditions of an entire population.. i.e.
census data, Kids Count data. Rubin and Babbe, (2007) Essential Research Methods for Social Work. Brooks Cole:CA
Is the program needed?Do a needs assessment
Possible approachesKey informant interviews – use individual interviews
in-person, or by phone, structured or unstructured. Community forum – use a focus groupRates under treatment – use secondary data from existing
agencies (who uses existing services)Social indicators – use secondary data from census or other well-
being indicators (infant mortality rates, suicide rates) Surveys of community or target groups – consideration has to be
given to sampling unit..
Who ‘speaks’ for the group
• WHO will you consider ‘key informant’• Client, service provider, academic
• Who will you survey??• What statistics are important.. • How and who defines the problem • What do you mean by community??
Political and value issues are a major consideration
–a practice/political issue, not necessarily a research issue
Class Exercise: Social Worker at a local YMCA doing a needs assessment re “Do fathers in your community need support/services to help them be better parents?
1. Who will you interview for key informant interviews? What will you ask them? (what variables will you measure)
2. Who would you invite to a community forum/focus group.. What questions would you ask?
3. What ‘rates under treatment’ are important -- what agency organizations would you go to for information? What information -- variables do you identify?
4. What Social indicators are important.. Where would you go to collect them–
5. Who would be your sample for a community survey.. What would you ask.
Why Focus On Parenting Programs for Incarcerated Fathers?
• 93% of incarcerated parents are malea
• 75% of male inmates are fathersa
• Very few male institutions have any type of family programming available to inmates and their families
• Many female institutions currently have parenting programs in place
a b U.S. Bureau of Justice Statistics. 2000. Incarcerated Parents and Their Children. Office of Justice Programs, Special Report.
ANSWERS THE NEEDS ASSESSMENT QUESTIONSWITH
•SOCIAL INDICATORS•RATES UNDER TREATMENT
Moving on to existing program
AS you begin this formative or summative evaluation of an existing program, you must
• Identify the logic model
• Consider the‘evaluability’ of the program
What is the Logic Model?A schematic representation of the various
components that make up a social service program.
• Logic models may describe• theory and its link to change (theory approach
model) where attention is on the “how and why” a program works
• outcomes (outcome approach model) where the focus of the logic model is to connect resources and activities to expected changes
• activities (activities approach model) or describing what the program actually does
Resources Program Procedures – Interventions Delivered by the program staff or volunteers
Psychological and Social Process undergone by the incarcerated fathers
Interim Outcomes -- Program Goals
Long-Term Outcomes
Direct service time (paid and volunteer) Managerial and administrative support time Equipment (VCR, monitors, overhead, tape recorders) Supplies (Handouts, newsprint, markers, video tapes, audio tapes, workbooks, postage, newsletters, children’s books, etc)
12-week parent education classes, with specified goals and objectives for each class, delivered by paid and volunteer professionals, that provide content, materials, experiential learning activities on pre-specified topics relating to the increase of protective factors /decrease of risk factors re child abuse, and interactive discussions about how the fathers were parented as children and philosophies that emphasize the importance of fathers in the lives of children.
Understanding children’s behavior; Identification of situations which trigger anger and stress; Knowledge of child development and discipline; Support and insight from other fathers about parenting techniques; Increased parenting communication skills; Awareness of impact of family conflict on parent-child relationships; Awareness of feelings of sadness, anger, self-esteem; Awareness of effects of parental substance abuse on children; Awareness of how parenting skills are passed down from one generation to the next; Exposure to “Long Distance Dads” workbooks.
Decrease Child Abuse Potential
Increased knowledge and use of parenting skill
Decrease in feelings of isolation
Increased empathy with children
Increased recognition of importance of role of fathers
Increase understanding of how life experiences effect parenting
Increased recognition of correct parenting
Decreased Recidivism Decreased Child Abuse
Logic model- outcomes exampleProgram inputs
Program processes
WHAT’S MISSING IN THIS MODEL
Identifying the causal
mechanism
Short term outcomes.Measured
by the research
Long term
outcomes/ difficult
to measure
Program OutputsNumber of fathers who participate in program
What’s an ‘Evaluabilty’ Assessment
a systematic process that helps identify whether program evaluation is justified, feasible, and likely to provide useful information*.
• determines whether a program is ready for evaluation—either a process or outcome evaluation, or both.
• Is the program able to produce the information required for a process evaluation,.. AT WHAT STAGE OF IMPLEMTATION IS THE PROGRAM?
• Can a program meet the other criteria for beginning an outcome evaluation. • determines whether a program has the basic foundation for an evaluation to
take place
* Evaluability Assessment: Examining the Readiness of a Program for Evaluation. Juvenile Justice Evaluation CenterJustice Research and Statistics Association. Program Evaluation Briefing Series #6. May, 2003, p. 6http://www.jrsa.org/pubs/juv-justice/evaluability-assessment.pdf
Newly emergent programs that are not fully operational are not ready for,
and indeed can be tarnished by a summative evaluation geared to assessing program outcomes.
HOW SO??
FACT PROGRAM BACKGROUND• Fathers And Children Together (F.A.C.T.) is prison-based parenting program that
helps incarcerated fathers learn and enhance parenting skills and bond with their children. Begun in 1992, it is a joint project of Prevent Child Abuse Kentucky (PCAK) and Blackburn Correctional Complex (BCC), a minimum security prison within the Kentucky Department of Corrections. F.A.C.T. provides prison-based parenting education classes, special visits for fathers and their children as well as unique family outreach and leadership opportunities. Classroom sessions (2 hrs) operate on a 12-week schedule covering topics such as discipline, family relationships, communication, and parental substance abuse effects on children. Special visits are offered to F.A.C.T. participants every 6-8 weeks that allow fathers and their children to interact and play together on prison grounds without many of the constraints typical of regular prison visitation. The Parent Leadership Team promotes peer leadership among a small group incarcerated fathers and supports the activities of the classroom sessions and the parent-child visits. The essential philosophy of F.A.C.T. is to reduce the potential for child abuse and neglect and promote father involvement in the lives of their children by creating positive father and child experiences and opportunities for learning inside prison. This model program has provided a baseline for replication in other state correctional institutions.
Evaluability of F.A.C.T. program
ESTABLISHED PROGRAM had measurable outcomesdefined service componentsan established recruiting, enrollment, and participation
process; good understanding of the characteristics of the target
population, program participants and program environment; ability to collect and maintain information;adequate program size
RESEARCH SAVVY SERVICE DELIVERY STAFF hadproblem solving values and skillsprior experience with evaluation confidence in programcommitment to ‘new knowledge’openness to change
What is a Process Evaluation (implementation assessment):
evaluation research that investigates the process of service delivery
• Important for complex programs• Can identify the specific aspects of a program that have
an impact• Quantitative data is useful but qualitative data are used to
elucidate and understand internal program dynamics, unanticipated outcomes, and interaction among program participants
What is an Outcome Evaluation: Also known as impact evaluation and summative evaluation
evaluation research that examines the effectiveness of the treatment or other service
• Experimental design is preferred method for maximizing internal validity
• Program is independent variable (treatment)• Outcomes are dependent variables• Random assignment allows evaluation of programs impact
independent of the types of people in different groups• Selection bias in non-experimental designs is a common
problem because people self-select to participate in programs
RESOURCES
Evaluation Team Observation Notes
Participant Sign-In Sheet (every class)
PARENT-CHILD
VISITS
Cost Analysis Data Form (every 6 mths
PARENT EDUCATION CLASSES
Self-Administered Outcome Instrument (first class, last class, 6 month follow-up)
Demographics Family and Criminal History Adult-Adolescent Parenting Inventory (AAPI-2) Child Abuse Potential Inventory (CAPI) Childhood Trauma Questionnaire (CTQ) Parenting Awareness ‘Behavior Change’
Protocol
Participant Sign-In Sheet (every class)
Self-Administered Class Session Report (CSR) (end of every
Video-taping (2 classes per series)
Staff Reflections video tape (end of series)
Measures for outcome and causal
mechanisms Does the program
cause change? How does change happen
Measures for
Process/Implementation Evaluation
What services are being delivered,
by who, how?
Measures for input
And program efficiency
USING MULTIPLE MEASURES and SEVERAL DATA COLLECTION STRATEGIES TO EVAULATE THE FACT PROGRAM
Formative/Implementation Evaluation
How do you know whether or not the service was delivered in the manner intended… i.e. according to protocol
Must measure the Independent Variable –
Issues of Treatment Fidelity a major concern
Participant Sign-In Sheet (every class)
Self-Administered Class Session Report (CSR) (end of every
Video-taping (2 classes per series)
Staff Reflections video tape (end of series)
CLASS SESSION REPORT CLASS SESSION TWO
(CIRCLE THE WORDS IN ONE NUMBERED BOX FOR EACH STATEMENT)
AFTER THIS CLASS I UNDERSTAND ….
1 2 3 4
WHY MY KIDS DO WHAT THEY DO AT DIFFERENT
AGES
MUCH
BETTER
SOME
BETTER
NO
BETTER
WAS NOT TALKED
ABOUT IN CLASS
THE BASIC NEEDS OF MY CHILDREN MUCH
BETTER
SOME
BETTER
NO
BETTER
WAS NOT TALKED
ABOUT IN CLASS
WHY MY PARENTING IS IMPORTANT TO MY CHILDREN MUCH
BETTER
SOME
BETTER
NO
BETTER
WAS NOT TALKED
ABOUT IN CLASS
THE DIFFERNECE BETWEEN “SELF-CONCEPT”
AND “SELF-ESTEEM”
MUCH
BETTER
SOME
BETTER
NO
BETTER
WAS NOT TALKED
ABOUT IN CLASS
1. What was the one most important thing you learned in today’s session? ____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
2. Tell how this information will help you as a father?
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
3. What feelings do you have about yourself as a parent after today’s session?
____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
___________________________________________________________________________
Self-Administered Class Session Report (CSR) (completed by participants at end of every class session
One Measure forFormative/Implementation Evaluation
Goals of the curriculum for this particular class
Helps assess degree to which curriculum was being delivered
Importance of Treatment Fidelity Analysis
• Can minimize incorrect research conclusions• Can help to explain why a program works• Helps to identify programmatic changes• Underscores potential feasibility for replication
Measure Instrument/Data Source Activity
Adherence to program a. CSR
b. Video Taped Sessions
Measure frequency of Item #4 selected (not covered) Review for consistency with written curriculum
Measuring the amount of the program delivered (dosage)
a. PPI Measure dosage variable for graduates
Ascertaining quality of delivery
a. CSR b. Video Taped Sessions
Measure mean scores for classes attended. Review for quality of delivery
Measuring participant responsiveness
a. CSR
b. Video Taped Sessions
Measure for written comments vs. no comments. Review for participant responsiveness
Measuring program differentiation
a. “Reflections” transcriptsReview for substantial differentiation between series.
Treatment Fidelity Assessment Treatment Fidelity Assessment PlanPlan
MEASURING THE MEASURING THE INDEPENDENT VARIABLE INDEPENDENT VARIABLE
Note: Fidelity is measured only on parent education classes.
Important aspects of treatment fidelity
Measuring program resources Name ____________________________ Date ___________________
Cost Analysis Data Form
Cost is defined as the value of each resource that is consumed when the program implements a service procedure.
Component = Classroom Session, Storybook Activity, PLT activity or Parent/Child Saturday VisitationResource = Your Time, Volunteer time, Space Used, Equipment Used, Supplies Used, etc.The Conversion and Total Cost columns will be computed at UK so you only need to complete the first three columns.
Amount TotalFACT Component Resource Used Conversion Cost
Cost Analysis Data Form (every 6 mths
Component Cost per unit
Units per Series
Cost per series
Parent Education Classes
$ 311
12 classes
$ 3,732
Parent/child Visits
$ 1,169 (869)
2 visits $ 2,338 (1,738)
Parent Leadership Team
$ 57
4 meetings $ 228
TOTAL COSTS
$ 1,537 (1237)
$6,269 (5,698)
Summative Evaluation
• Experimental design is preferred method for maximizing internal validity
• Program is independent variable (treatment)• Outcomes are dependent variables
FACT PROGRAM GOALS TRANSLATED INTO TESTABLE HYPOTHESES
H:1. Fathers who participate in at least 10 classes of the parent education classes will increase their knowledge of effective parenting skills and decrease their risk for child abusing behavior.
H:2. Fathers who participate in at least 10 classes of the parent education classes will report less feelings of isolation.
H:3. etc etc etc
Evaluation Design SnapshotUse of a recurrent institutional cycle design first identified by (Campbell &
Stanley, 1963) and further articulated by Shadish, Cook, & Campbell, 2002 p. 152 . This design measures the effects of the program assessing pre-and post test measures of different cohorts (series 1, series 2, etc thru series 8) as they go through the FACT program.
Series 1, Spring, 04 Series 2, Summer, 04 Series 3, Fall, 04
NR O1pr X O1ps
………………………………………………………………………..….………....NR O2pr X O2ps
……………………………………………………………………………………NR O3pr X O3ps
“A program main effect is suggested by a certain pattern of results if O1ps, and O2ps, and O 3ps are higher than O1ps, O2ps, and O3ps AND if O1ps is higher than O2pr, and O2ps is higher than O3pr
NR = not randomly assigned0 = data collected by survey instrumentX = 12 week parent education series
Program Effects* in goal-related knowledge, attitudes, skills of fathers who complete FACT program:
Looking at PATTERNS OF CHANGE
50.0
70.0
90.0
110.0
130.0
150.0
170.0
190.0
210.0
230.0
250.0
pre-test post-test
CAPI ABUSE***
series 1
series 2
series 3
series 4
series 5
Series 6
Series 7
Series 8
Series 9
TOTAL
H1: Stat Sig decrease in the potential for physical child abuse** from the
beginning of class sessions to the end of the at the program
*per self-reports of fathers:
**Potential for child abuse based on feelings of distress, rigidity, unhappiness, reports of relationships with children and other family member comparable to known child abusers
All but one line is going down
AVERAGEPre-test score on the CAPI
Abuse instrument
for each series
AVERAGEPost-test score on the CAPI
Abuse instrument for
each series
A Closer Look
HIGH RISK SUB-SAMPLE (n =67) COMPARED TO NOT AT RISK SUB-SAMPLE (n = 73)
HI-RISK =fathers who scored Above the clinical
cut-point for child abuse
potential
Clinical cut point = 215
On average, fathers who scored above
the clinical cut-point BEFORE the classes,
scored below AFTER the classes
Scores of other fathers in the group DID NOT CHANGE
The Cost of Program Effects
• Calculate program costs in relation to program effects as a per father cost. • Of the 140 fathers who completed the program,
• 90 fathers reported program effect, i.e. decrease in CAPI scores from pre-test to post-test over the course of the evaluation period.
• Given that we had 9 series of classes, we estimate that 10 fathers per series benefited from the program.
• It costs approximately $570 to decrease the child abuse potential of one father in the F.A.C.T. program.
• Fathers who do not participate in the visits obviously use less program resources ($373).
• The cost increases to $949 per father if we consider only those fathers (n = 52) who began the program ‘at-risk’ for child abusing behaviors, estimated to be roughly 6 fathers per series.
• We had no way to measure the ‘start-up’ costs of a program similar to F.A.C.T. but anticipate that the costs of new F.A.C.T.- like programs would be greater than the costs we calculated for our established F.A.C.T. program.
Black Box or Program Theory?
• Black box: • if program results are of primary importance, how it works may be of
secondary importance
• Program theory: • a descriptive or prescriptive model of how a program operates and produces
effects (theory-driven)• specifies how a program is expected to operate• identifies which program elements are operational• specifies how a program is to produce effects• improves understanding of the relationship between the independent variable (the
program) and the dependent variable (outcomes)
Example: F.A.C.T.
.00
Vulnerability
.00
Father Role Model.35
Father Scaffolding
.00
Father Empathy
.40
Father Capacity
E1
E2
E3
E4
E5
.63
.60
Experiences of Fathers and Knowledge and Attitudes
.02
-.03
.06
-.01
From questions
about history of child abuse
Type of parent
capacity At
beginning of the
program
Relationship between fathers child abuse growing up and father capacity and father empathy
Relationship between having good role models growing up and more education/background as adults
.00
Vulnerability
.00
Father Role Model
.36
Father Scaffolding
.40
Father Capacity
E1
E2
E3
E5
.60
-.10
.54
Cognitive learning
.61
Emotional learning
E6
E7
.73
.14
-.15
.63
.76
Father Experiences, Knowledge, Attitudes and Learning
Example: F.A.C.T. Program services/proceduresDifferent aspects of
the curriculum
Father’s history and level of parenting capacity at Beginning of program is related to what aspects of the program they benefited from
.00
Vulnerability
.00
Father Role Model .36Father Scaffolding
.40
Father Capacity
E1
E2
E3
E5
.60
-.10
.54CognitiveLearning
.61
EmotionalLearning
E6
E7
.73
.15
-.16
.63
.76
FACT Logic Model at Four Levels
.20
DecreasedAbuse Potential
.10
Knowledgeof Fathering
E4
E8
-.31
-.45
Example: F.A.C.T. Program outcomes
Results differ based on particular inputs and program
processes
What goes into the program
What happens in
the program What goes into the program
Another closer look Effects
remained after fathers left program
Caveat – fathers only left the program/prison..
Transferred to other prisons but remained in controlled setting.. Don’t know what happens when they return to the community
Researcher or Stakeholder OrientationConsider the pros and cons of each of the approaches
Stakeholder approaches encourage researchers to be responsive to program stakeholdersutilization-focused evaluationaction research (participatory research)appreciative inquiry
Social science approaches emphasize the importance of researcher expertise and maintenance of some autonomy in order to develop the most trustworthy, unbiased program evaluation. goal free evaluation
Integrative approaches attempt to cover issues of concern to both stakeholders and evaluators, and to include stakeholders in the group from which guidance is routinely sought