1/6/2014

1

Heart of Missouri United Way Community Development Training Series

Module I: Logic Models for Program Evaluation Aaron M. Thompson

School of Social Work

University of Missouri

thompsonaa@missouri.edu

Overview of Training Modules

• Module I: The What of program evaluation

• Assist providers to clarify the description of their programs and how to link organizational activities to intended effects.

• Module II: The How of program evaluation

• Assist providers to develop measurement models tied to logic model outcomes to support process and outcome evaluation.

• Module III: The Why of program evaluation

• Assist providers to develop conceptual and statistical skills and strategies for analyzing program level data to effectively:

• Identify the malleable predictors your program targets

• Communicate evidence that your program makes a difference

• Develop competitive proposals when applying for funding

Agenda

Goal: To discuss logic models as a common strategic framework for planning, implementing, evaluating, and communicating the

purpose and effectiveness of your program.

1. Logic Models: Definition, History & Benefits

2. Program Evaluation & Logic Models: Stages & Phases

3. Social Problems & Logic Models: Mediators as Targets

4. Logic Model Elements: Common Language

a. Inputs & Budget

b. Activities & Targets

c. Outputs & Fidelity

d. Outcomes & Development

5. Application & Consultation: A logic model for your program

1/6/2014

2

LOGIC MODELS

Definition, History & Benefits

Definition

• A flat diagram of a program to represent proximal lower order program actions necessary to leverage higher order changes at the individual, group, or community level

• Facilitates transparency

• Communicates your theory of change

• Lists resources to inform budget development

• Organizes program activities to inform staff training needs

• Tracks key program elements for fidelity monitoring

• Lists a sequence of socially meaningful outcomes

• Connects activities to intended outcomes

• Serves as an organizing tool for process and outcome evaluation

Brief History

• Logic model history

• 1955—first termed a “conceptual model” for research purposes

• Klienberg (1955), James (1958), Herzog (1961)

• 1967—Evaluative Research: Principles and Practice in Public

Service and Social Action Programs (Suchman)

• From research to practice as a tool

• 1979—Evaluation Promise & Performance (Wholey)

• 1995—Measuring Program Outcomes (Hatry)

• United Way of America

• 2005—Program Logic Model Guide (Kellogg Foundation)

• 2006—Getting to Outcomes (Rand Corporation)

1/6/2014

3

Benefits

• Often required

• IES, NIH, NIMH, NIDA, OJJDP, State, local & Private

• Enhances clarity by creating shared meaning

• Improves stakeholder, funder, Policy maker, and public buy-in

• Visually organizes and structurally aligns complex programs

• Organizes complex multi-modal or multi-element programs

• Facilitates the development of a stepwise work plan

• Helps to unpack the structure and organization of a program

• Assists with fiscal orderliness—develop an investment schedule

• Permits the development of a quality assurance plan

• Permits causal claims surrounding the effects of a program

• Provides data to suggest mid-course corrections

So…what is the problem?

1. Transition is Hard

• Change is uncomfortable

• Even more difficult at the systems level

2. There are Language Barriers

• Too many voices increase confusion and ambiguity

• Disciplines talk differently about these concepts

3. Fear of Appraisal

• Focused on providing quality direct services

• Afraid to losing funding and support

4. Of HMUW funded agency proposals reviewed (N = 19)

• < 15% specified the problem properly

• < 26% proposed programs that are “scientifically-based”

• < 5% linked activities, outputs and outcomes sequentially

• < 5% specified an accurate measurement model

PROGRAM EVALUATION & LOGIC MODELS

Phases & Stages

1/6/2014

4

Stages & Phases (CDC, 2011)

• Stages of Program Evaluation

• Ascertain and Engage Stakeholders

• Identify and Map Scientifically-Supported Programs

• Develop Program Evaluation Design

• Collect Evidence Using Credible Indicators

• Justify Data-Based Conclusions

• Utilize Data in Program and Policy Decisions

• Phases of Program Evaluation

• Planning (Needs Assessment)

• Implementation (Process or Formative Evaluation)

• Maintenance (Outcome or Impact Evaluation)

INPUTS ACTIVITIES OUTPUTS OUTCOMES

Resources (i.e., capital, personnel,

facilities, & tools needed to do work)

Actions (i.e., tasks,

duties, events, and work that will be done)

Products (i.e., units offered,

persons served, quality and proof work was done)

Benefits (i.e., observable

changes accrued by stakeholders

effect of work done)

Planning Implementation

Process Evaluation

Outcome Evaluation

Program Evaluation & Logic Models

Focus on population needs before adding or

changing a program.

Summative Formative

Focus on agency: how well agency operates, what services are provided; how well services are allocated, usually occurs through fidelity—quantity

and quality—practitioner or worker and client satisfaction.

Focus is on effect, impact, goal attainment reflected by indicators.

Maintenance

Needs (i.e., required stakeholder

services/goals, work to be done)

PRIORITIES

Needs Assessment

Types of Evaluation Questions • Planning Questions:

• What are the major service gaps or what is the need?

• Is there public support/interest for the program?

• What resources are needed/exist to implement the program?

• Implementation Questions: • Is there compliance with the program (fidelity)?

• Is there increased support for the program?

• Are users/practitioners pleased with the program?

• Are some potential users not being accessed?

• Evaluation Questions: • What is the cognitive and behavioral impact of the program?

• What is the health impact of the program?

• What is the economic impact of the program?

• Are there dosage or moderation effects of the program?

• Are there iatrogenic effects from the program?

1/6/2014

5

LINKING SOCIAL PROBLEMS & LOGIC MODELS

Mediators as Targets

Defining Social Problems

• Step 1: Defining a problem

• Suggesting a small program will alter large scale issues is inaccurate • Poverty, Teen Pregnancy, Drug Abuse, Aggression, Dropout, etc….

• However, a problem statement includes: • Who: Population (sex, age, income, race or other defining characteristics)

• What: BIG Problem (i.e., poverty, AIDS, cancer, obesity, etc.)

• Why: List malleable mediators or predictors of the problem

• How: Broad strategy or approach to targeting the predictors

• Example:

We aim to reduce incidents of child abuse by offering Parent and Child interaction Therapy to low income families where parents are exposed to increased life

stressors without effective coping skills, adequate social supports, or positive parenting strategies.

Mediators & Social Problems • Mediators are the indicators or intervening mechanisms that

programs target to alter the social problem

• Mediators are highly associated with/predictive of the problem

• Mediators are immediate malleable outcomes

• Mediators are measured alongside medial and distal outcomes

• Mediators explain the variance in the outcome

EMPATHY

M

PROGRAM

X Y

RECIDIVISM

a b

c’

Proximal tx

outcome in

Logic Model

The following several slides are adapted from: MacKinnon, D. P. (20049 February). Methods to study treatment mechanisms of

action. NIDA Workshop on Mediation. Retrieved July 29, 2005 from http://www.public.asu.edu/~davidpm/ripl/NIDAM5.final.PPT

**

** **

1/6/2014

6

Examples of Mediators

• School Dropout • Engagement, bonding, student-teacher relations, formal social

support, social competency, decision making, problem solving, etc...

• School Readiness • Parent involvement, positive parenting strategies, parent-teacher

bonding, home-school communication, emergent literacy skills, social and emotional competence, authority acceptance, etc…

• Poverty • Financial literacy, secure housing, reduced use of “pay-day” loan

services, increased banking participation, etc…

• Teen Pregnancy • Knowledge of sex and STIs, attitudes toward condom use, peer group

normative attitudes about sex, propensity to identify risky scenarios, knowledge of escaping risky situations, etc…

• Violence and Aggression • Normative attitudes about violence, social information processing

skills, social competence, self-management, self-awareness, parent supervision, etc…

LOGIC MODEL ELEMENTS

Building Common Vision

Basic Components

• There is no “right” way to create a logic model

• Basic Ingredients:

A. INPUTS

B. ACTIVITIES

C. OUTPUTS

D. OUTCOMES

Resources (i.e., capital, personnel,

facilities, & tools needed to do work)

Actions (i.e., tasks,

duties, events, and work that will be done)

Products (i.e., units offered,

persons served, quality and proof work was done)

Benefits (i.e., observable

changes accrued by stakeholders

effect of work done)

1/6/2014

7

a. Inputs: Justifying Budgets • Resources or investments required to operate your program

• Financial, organizational, human or fiscal capital needed

• Staff AND time

• Contracted services

• Facilities

• Program materials, supplies, equipment

• Collaborative Relationships

• Items may be listed broadly on Logic Model figure, then…

• Use each item to inform budget development by:

• Estimating the number of units predicted/required

• Describe each unit in your budget justification parallel to your LM

• Why the unit is needed, the number of units needed, the cost per unit

• Build an itemized budget in an excel sheet

INPUTS ACTIVITIES OUTPUTS OUTCOMES

a. Inputs: Justifying Budgets (2) Not Specific Enough Just Right Too Specific

Home buying resources Clear financial records

W2 forms 1099s Tax returns Bank statements Pay stubs Utilities bills Credit report

Staff 3 full/1 part-time staff

1 project lead @ 40 hrs./wk. 2 project associates @ 40 hrs./wk. 1 part-time support person @ 20 hrs./wk.

Supplies Art Supplies

25 paintbrushes 50 bottles of paint 250 sheets of paper 25 coffee cans Dishwashing liquid

b. Activities: Altering Mediators

• Actions, tasks, processes, planned events, direct services, or what the program staff will do to fulfill its stated mission • Provide…

• Educate…

• Train…

• Create…

• Counsel…

• Use ACTION words to unpack what we will do

• Be sure activities TARGET known predictors or mediators associated with mitigating or improving the problem • These targets or mediators will logically bring about the desired

change

• There is research supporting the effect of the target on the social problem

INPUTS ACTIVITIES OUTPUTS OUTCOMES

1/6/2014

8

b. Activities: Altering Mediators

Activities

• Hire real-estate agent • Drive around the city

Not detailed enough.

• Google search •Interview friends • Read 3 books on neighborhoods • Hire a driver • Try restaurants • Set up meetings• Take friends on tour

Too detailed (more appropriate for a work plan).

• Research neighborhood amenities and prices • Hire a real-estate agent • Tour priority neighborhoods

Just right.

c. Outputs: Monitoring Fidelity

• Direct By-Products of Program Activities

• Number of classes provided

• Number of counseling sessions provided

• Number of materials created/used/distributed

• AND persons directly accessing services

• Number attending classes

• Number of persons counseled

• Number of persons using materials

• AND quality indicators of service provision/client access

• Client and practitioner tx acceptability

• Client engagement (i.e., behavioral, cognitive, emotional)

• Bonding between practitioner and client

INPUTS ACTIVITIES OUTPUTS OUTCOMES

Outputs (2) • Quantity (aka, surface exposure to tx elements)

• Exposure to the intervention activities • This is why we count outputs • Demonstration that people are indeed exposed to tx

• Quality (aka, subsurface response to exposure to tx elements) • Treatment acceptability

• Is tx cost effective and easy • What is practitioner tx acceptability • What is client tx acceptability • Levels of engagement (i.e., behavioral, cognitive, emotional) • Bonding between practitioner and client

• Field notes, focus groups, survey or questionnaires

• Don’t forget to measure what is happening in comparison • SSD—baseline • Group—all services exposed to comparison

• Estimating Fidelity (aka, total program dosage) • Ratio: # received/total elements = % or dosage • Compare compliers with noncompliers

• Investigate reasons for noncompliance

1/6/2014

9

d. Outcomes & Development • Measureable Benefits for participants after tx exposure

• If-then sequential developmental statements

• Links to outputs

INPUTS ACTIVITIES OUTPUTS OUTCOMES

Short-term Proximal

Immediate

Interim Medial

Intermediate

Long-term Distal

Impact

Initial changes most causally associated

with program targets

• Knowledge • Attitudes • Cognitions

Secondary changes reasonably expected

to stem from proximal effects

• Skills • Behaviors • Social Interactions

Terminal changes logically expected,

generally reflect goals of agency

• Health • Status • Condition

0-1 years 2-5 years 4-7 years

d. Outcomes & Development (2)

Immediate

Knowledge 1a. Reduced myths among targeted parents on immunization 2a. Increase knowledge in targeted parents of where have children immunized 3a….. …

Intermediate

Behavior 1b. Increase number of targeted parents taking children to immunization 2b. Increase number of targeted parents who report positive bonds towards health providers 3a….. …

Long-term

Condition 1c. Fewer children suffering from preventable diseases 2c. Better overall health for children 3a….. …

Developmental Impact of Immunization PSAs

Outputs vs. Outcomes

• Distinguishing characteristics:

• Outputs are direct and measurable products of a program’s activities and services

• Expressed in terms of volume or units delivered.

• Outcomes are the effects or impact of the activities and services.

• Each outcome usually corresponds to more than one output.

Outputs Outcomes

# of workshops Change in knowledge associated with workshop

# of new mom home visits Increased knowledge of child development

Action plan to clean and supervise playgrounds

Neighborhood volunteers sign up to clean and supervise playgrounds

# of funding proposals submitted # of potential individual donors

Increased and diversified resources for the program

1/6/2014

10

Logic Model Examples

• Policy: Bicycle Helmet Public Information Policy

• Single Program: STARS Integrated Conceptual Model

• Multimodal Program: SAMSHA Community Drug Abuse Prev.

Bicycle Helmet Public Information Policy

Inputs Staff

Space Equipment

Outcomes-Mid Increase bicycle helmets worn

Outcomes-Short Increased

awareness of bike TBI injuries

Activities -Gather data on

accidents -TBI, and injury

prevention -Develop press

kits, public service

messages -Attract key

spokesperson

Outputs Factsheets

TV and Radio PSAs

Outcomes-Long Fewer TBI bike related injuries

STARS

Autonomy Supports

+self-awareness

+self-monitoring

+self-management

Competency Supports

+social competence

+self-efficacy

+cue interpretation

+problem solving

Relational Supports

+social awareness

+communication

+staff & peer relations

Skills &

Opportunities

Increased Conventional Behaviors & Values

Reflective of the Social Unit

Improved

SEL &

Classroom

Behavior

Increased

Time On-

Task

Increased

Instructional

time

Improved

Academic

Performance

Involvement

& feedback

Social Rewards &

Increased Bonding

Figure 1.

STARS Integrated Theory of Change Impacting SEL, Behavior, and Academic Performance

Proximal Outcomes Distal Outcomes

1/6/2014

11

Multilevel or Multimodal Logic Model SAMSHA: Community Drug Abuse Prevention

Community- Risk Factors

Access Norms

Perceptions Parenting

Outcomes-Long

Reduced Delinquency Overdoses

Drug Arrests

Outcomes –Short

Increased Risk Awareness Storage options

Refusal strategies

Strategies

Drug Court Social Marketing

Curriculum Outreach

Drug Court

Inputs

Police Support Media Support

Education Court and LEA

Partners

Community Logic Model II: Prescription Drugs

Community- Specific Risk

Factors

Long-Term Outcomes

Reduction in Use

Delayed Onset

Reduction in Availability

Access Ease of

Availability

Short Term Outcomes

Increased awareness of risk of improper

storage

Increased use of safer storage and/or

disposal practices

Increased demand for take-back programs

Increased perception of harm or social

disapproval

Strategies Resources and Inputs

Community Norms

Perceived Risk

Drug Dispersal Take-Back Program

Social Marketing Lock Up Your

Medicine Cabinet

Collect and analyze data on the intervention Recruit local DEA or police department to staff take-back event.

Collect and analyze data on the intervention Develop a marketing plan Partner with media Secure earned and paid media

Community Logic Model I: Heroin

Community- Specific Risk

Factors

Long-Term Outcomes

Reduction in Lifetime Heroin Use

Reduction in Delinquency

Perception of Harm

Short-Term Outcomes

Increased Perception of Harm of Heroin Use

Increased Access to Mental Health and Treatment Services

Improved Access to

Parenting Skills Training via Drug Court Referrals

Strategies Resources and Inputs

Low Parental Care

Family Conflict

Curriculum

Drug Court

Collect and analyze data on the intervention

Curriculum, teacher training

Hire and train community outreach workers

Collect and analyze data on the intervention Court and enforcement partnerships

Community Outreach Worker

Education and Referral

1/6/2014

12

Caveats • We did not cover everything associated with Logic Models

• Logic Model Development Guide

• There is no right way to develop Logic Models • The grandmother test

• Cautionary word on the linearity of Logic Models • Most programs include more than one service

• Multi-Component Models (i.e., Master and Subcomponent models)

• Multi-modal Models (i.e., Master and Sublevel models) • Be sure the level targeted and the logic model elements align

• Cautionary word on outcome linkages & causality • Requirements for causality

a. Temporal Ordering

b. Association

c. Counterfactually ruling out other causes

• The ability to attribute causal inference diminishes the farther from program activities our outcome is

Considerations

• Keep it practical—time is money

• It should be to YOUR benefit, first and foremost!

• Keep it focused—quality assurance

• Do not dismiss the importance of process evaluation

• Be sure your outcomes reflect your organizational goal

• Consider how outcome evaluation results will be used

• As a reciprocal feedback tool

• To communicate impact

• To estimate a cost-effect size ratio

• Use explicit language—never assume your reader knows

• Use action words

• Use parallel construction between LM, narrative, and budget

Brief Activity and Peer Consultation

• Activity: Think-Pair-Share

• Think—(10 minutes)

• By yourself, scratch out a logic model for your program

• Use the Tip sheet and handouts

• Pair—(10 minutes)

• With a partner or partner, share your logic model

• Are the elements sequentially linked?

• Are there elements or activities that do not logically predict outcomes?

• Are the outcomes sequentially linked?

• Share—(20 minutes)

• 2 or 3 volunteers to discuss their logic model

1/6/2014

13

Questions and Comments

• Please Complete the Posttest Survey

• Feel Contact me for Consultation:

Aaron M. Thompson

School of Social Work

University of Missouri

718 Clark Hall, 65211

PH: 573.882.0124

thompsonaa@missouri.edu

THANK YOU!!!!