1 Outcomes and Program Improvement: Designing Effective Evaluations Presented by Christine A. Ameen, Ed.D. Ameen Consulting & Associates ([email protected])

1

Outcomes and Program Outcomes and Program Improvement:Improvement:

Designing Effective Evaluations Designing Effective EvaluationsPresented by

Christine A. Ameen, Ed.D.Ameen Consulting & Associates

([email protected])

2ameenconsulting&associates

ReflectionReflection

“If you don’t know where you’re going, you’ll end up somewhere…”

- Yogi Berra


Some “Reasons” for Doing Some “Reasons” for Doing Outcome Evaluation...Outcome Evaluation...

• I need to be able to fill out these forms that the state department gave me.

• I need a plan for a grant I’m writing.

• I need to get my agency accredited.

• I need to show outcomes for the grant project we just completed.


Other Reasons for Doing Other Reasons for Doing Outcome Evaluation...Outcome Evaluation...

• Our staff needs to know the impact they make

• Our agency needs to know the impact it makes

• Our funders need to see the return on their investment

• Our accrediting body needs to see evidence of our intent to provide quality services


The #1 Reason for Doing The #1 Reason for Doing Outcome Evaluation...Outcome Evaluation...

Our clients

deserve the very best service!

6

How effective is your How effective is your program?program?

Let me tell you about the kids that are in my group!

7

How effective is your How effective is your program?program?

Here’s what’s working…Here’s what we’re learning…Here’s how we’ve increased

effectiveness…


What is Continuous Program What is Continuous Program Enhancement (CPE)?Enhancement (CPE)?

• A commitment to ongoing assessment of service delivery and client outcomes to achieve the best outcomes possible.


CPE assumes that:CPE assumes that:• Every program has strengths and weaknesses, and

opportunities for improvement

• The organization values learning

• Administrators nurture a climate of trust and welcome feedback

• Staff want clients/customers to be successful

• Staff are open to learning new and different ways of providing service


Determining PurposeDetermining Purpose

• Assess the impact of a program

• Improve program implementation

• Identify unmet client needs

• Respond to accountability.

CPE may be used to:


LET’S FIND OUT!

ARE YOU READY? ARE YOU READY?


• Its purpose is clearly understood by everyone

• The organization’s environment is a risk-free learning environment where a climate of trust is nurtured and feedback is welcomed

• Internal collaboration is encouraged and expected.

Adapted from Brinkerhoff, R.O.; Brethower, D.M.; Hluctyl, T.; and Nowakowski, J.R. Program Evaluation: A Practitioner’s Guide for Trainers and Educators. Boston Kluwer-Nijhoff, 1983.

You’re ready if:You’re ready if:


• Results will be shared with those who have a right to know

• The results will be useful and used

• The process of doing CPE is humane

• The benefits of the CPE justify the cost.Adapted from Brinkerhoff, R.O.; Brethower, D.M.; Hluctyl, T.; and Nowakowski, J.R. Program Evaluation: A Practitioner’s Guide for Trainers and Educators. Boston Kluwer-Nijhoff, 1983.

You’re ready if:You’re ready if:


Don’t Ask a Question for Which Don’t Ask a Question for Which You Don’t Want the Answer!You Don’t Want the Answer!

• Respond to the results

• Be prepared to learn that perceptions don’t always reflect reality

• Accept the fact that CPE is an innovation and there will be some natural resistance to it

• Be patient — CPE will take some time before its payoffs are realized

You must get administrative buy-in first!


Planning for CPEPlanning for CPE

1. A program model which describes the services for delivering services to clients and the outcomes clients should achieve

2. An objectives model which defines objectives for the services and objectives for the outcomes

3. An evaluation model which defines how data will be collected and used to monitor the attainment of process objectives and outcome objectives.


The Processes Used to The Processes Used to Develop the CPE ModelsDevelop the CPE Models

1. Review contracts, licensing agreements, program proposals for service requirements, information requirements and pre-determined outcomes

2. Assess what information needs or concerns board members, administrators, staff, clients, customers, volunteers, policy-makers and public-at-large may have about the program

3. Convene a work group of staff and administrators to develop the CPE models and support implementation

17

Building Your Program Building Your Program ModelModel

Defining Program Services and Client Outcomes


Defining Your Program: The Program Defining Your Program: The Program ModelModel

CLIENT CONDITIONS

PROGRAM SERVICES

CLIENT OUTCOMES


Some DefinitionsSome Definitions

Client Conditions - demographic characteristics and other factors that describe client need, assets and client risk for not being successful in program

Program Services - the services delivered to address client needs and risks

Client Outcomes - a desired effect or impact of a service, intervention or experience on a client


Residential Program ModelResidential Program Model

CLIENTCONDITIONS

PROGRAMSERVICES

CLIENTOUTCOMES

Experienced abuse and/or

Had difficulty in school

Experienced disruption inplacement at home

Suffering from depression,low self-esteem

Engaged in delinquentbehavior



CLIENTCONDITIONS

PROGRAMSERVICES

CLIENTOUTCOMES

Experienced abuse and/orneglect


Experienced disruption inplacement at home

Suffering from depression,low self-esteem

Engaged in delinquentbehavior

1. Assessment and treatmentplanning

2. Frequent visitation of clientwith family

3. Safe environment in whichto live

4. Attendance at school5. Family therapy6. Individual and/or group

therapy7. Other support services as

needed



CLIENT CONDITIONS

PROGRAM SERVICES

CLIENT OUTCOMES

Experienced abuse and/or neglect


Experienced disruption in placement at home

Suffering from depression, low self-esteem

Engaged in delinquent behavior

1. Assessment and treatment planning

2. Frequent visitation of client with family

3. Safe environment in which to live

4. Attendance at school 5. Family therapy 6. Individual and/or group

therapy 7. Other support services as

needed

Resolution of negative legal issues

Avoidance of negative legal issues

Establishment of permanency goal per plan

Movement to a less restrictive, home-based placement

Academic achievement Employment skills Increased ability to manage

emotional functioning Self-care skills Increased pro-social

behavior Sustained effects up to six

months


A Conceptual Framework for A Conceptual Framework for Behavior ChangeBehavior Change

1. Awareness: the initial consciousness, perception, or sense of a concept

2. Knowledge/Skills: understanding, comprehension of a concept; demonstration of ability of that understanding or comprehension

3. Behavior: performance or conduct in a specified way

4. Modeling behavior: demonstration of specified behavior to others, most often to teach the behavior to others.


How Far Does Your Program Go?How Far Does Your Program Go?

• How long are clients in the program?

• How intense is the program?

• How extensive is content covered in the program?

25

LET’S TRY IT!LET’S TRY IT!

Build a Program Model


Defining Your Program: The Program Defining Your Program: The Program ModelModel

CLIENT CONDITIONS

PROGRAM SERVICES

CLIENT OUTCOMES

27

Building Your Objectives Building Your Objectives ModelModel

Defining Objectives for Services and Outcomes


Your Program Objectives ModelYour Program Objectives Model

CLIENT CONDITIONS

SERVICES OBJECTIVES

OUTCOMES OBJECTIVES


Some DefinitionsSome Definitions

Service Objective - a specific statement about how the service is to be delivered, usually including a timeframe and benchmark

Outcome Objective - a specific statement about how much desired effect or impact of a service is going to be achieved.


Establishing Objectives Establishing Objectives for Program Servicesfor Program Services

• What clients?

• What services will be provided or experienced?

• During what time frame?


Examples of Service ObjectivesExamples of Service Objectives

100% of clients served will have written treatment plans within 30 days of intake

85% of clients served will have weekly visits with their families throughout their treatment stay

85% of clients will attend school 90% of the time throughout their treatment stay

100% of clients will attend all individual therapy sessions designed in their treatment plans throughout their treatment stay


Establishing Objectives Establishing Objectives for Client Outcomesfor Client Outcomes

• Desired effect or impact

• On whom or what

• The specific outcome

• By how much, when.


Writing Client Outcome ObjectivesWriting Client Outcome Objectives

Desired effect Increase, decrease, maintain, reduce, improve…

On Whom Child, children, family

Specific Outcome Ability, skill, knowledge, likelihood of, understanding of

How much, when Percent, number


Examples of Client OutcomesExamples of Client Outcomes

No youth served will experience recurrence of abuse/neglect while in care.

75% of youth will increase their academic level of functioning by at least one grade level upon release from the program.

No more than 15% of youth completing treatment will be arrested within six months of release from the program


Residential Program Objectives ModelResidential Program Objectives ModelCLIENT CONDITIONS

SERVICE OBJECTIVES

OUTCOMES OBJECTIVES







Residential Program Objectives ModelResidential Program Objectives Model

CLIENT CONDITIONS

SERVICE OBJECTIVES

OUTCOMES OBJECTIVES






1. 100% of clients served will have written treatment plans within 30 days of intake

2. 85% of clients served will

have weekly visits with their families throughout their treatment stay

3. 85% of clients will attend

school 90% of the time throughout their treatment stay


all individual therapy sessions designed in their treatment plans throughout their treatment stay


Residential Program Objectives ModelResidential Program Objectives ModelCLIENT CONDITIONS

SERVICE OBJECTIVES

OUTCOMES OBJECTIVES






1. 100% of clients served will have written treatment plans within 30 days of intake


have weekly visits with their families throughout their treatment stay


school 90% of the time throughout their treatment stay


all individual therapy sessions designed in their treatment plans throughout their treatment stay

1. No youth served will experience recurrence of abuse/neglect while in care.

2. 75% of youth will increase

their academic level of functioning by at least one grade level upon release from the program.

3. No more than 15% of youth

completing treatment will be arrested within six months of release from the program

38


Build an Objectives Model


Your Program Objectives ModelYour Program Objectives Model

CLIENT CONDITIONS

SERVICES OBJECTIVES

OUTCOMES OBJECTIVES

40

Building Your Evaluation Building Your Evaluation PlanPlan

Defining Data Collection to Monitor the Attainment of

Service and Outcome Objectives


Your Program Evaluation ModelYour Program Evaluation Model

SERVICE/OUTCOME OBJECTIVES

DATA COLLECTION PLAN

DATA UTILIZATION PLAN


Data Collection Plan for the ObjectivesData Collection Plan for the Objectives

Create a data collection plan that includes:

• Information source

• Measurement method

• Timeline for when collected

• Identification of parties responsible for data


Some Issues to Resolve Related to Some Issues to Resolve Related to Service Objectives DataService Objectives Data

• Service objectives usually report the status of service delivery or the status of the client at a specific point in time

• Attendance versus participation

• Opportunity to participate versus actual participation

• Using currently existing data, e.g., rating forms, attendance records, incident reports, etc.


Some Common Methods forSome Common Methods forCollecting Outcome Objectives DataCollecting Outcome Objectives Data

• Formal testing or assessment, e.g., vocational skills

• Attitude measurement, e.g., empathy

• Surveys, e.g., therapeutic environment

• Interviews, e.g., follow up with the family upon the youth returning home


Data Collection for OutcomesData Collection for Outcomes

• Use both qualitative and quantitative information:

• Qualitative

Used when the program is aimed at individualized outcomes

• Quantitative

Used when the program is aimed at common outcomes for all participants


Some Issues to Resolve Related to Some Issues to Resolve Related to Outcome Objectives DataOutcome Objectives Data

• Will service delivery result in the impact on the client?

• Will service delivery result in the client being able to maintain a positive impact after leaving the program?

• An increase or a decrease in something, e.g., empathy, requires the measure be used at the beginning and the end of the program


Some Issues to Resolve Related to Some Issues to Resolve Related to Outcome Objectives DataOutcome Objectives Data

• The achievement of a specific level of an outcome, e.g., the family will be able to name 3 community resources they can utilize, requires the measure be used at the end of the program only

• Will instruments such as surveys be read to clients? If read to some, read to all

• Instruments should be administered by staff with whom client have limited or no interaction, to reduce the potential, unintended bias of client responding based upon the relationship to the staff administering the instrument


Some Tips for Selecting MeasuresSome Tips for Selecting Measures

• Does the instrument directly measure the target behavior or issue the program is intended to change?

• Is the instrument able to measure change?

• Is the instrument appropriate for the population?

• Can the instrument be used in ways that are respectful of the clients and the program?

• Can the instrument be used consistently?

• What is the reading level?


Data Utilization for the ObjectivesData Utilization for the Objectives

Create a data utilization plan that includes:

• Who will review the data

• How data will be interpreted

• Timeline for when reviewed

• How data will be used


Some Tips for Writing the Data Utilization Some Tips for Writing the Data Utilization PlanPlan

• Plan on including direct service staff in the review and utilization process

• Results will be interpreted against the benchmark you set for each objective

• Try to anticipate how the information will be used.


Some Tips for Writing the Data Utilization Some Tips for Writing the Data Utilization PlanPlan

• Review often enough to allow for changes to be made if the objectives aren’t met

• Consider reviewing data about service objectives on a monthly basis

• Consider reviewing data about outcome objectives on a quarterly basis

• Consider reviewing all of the data on an annual basis, to consider setting new benchmarks or new objectives


Residential Program Evaluation ModelResidential Program Evaluation Model





have written treatment plans within 30 days of intake

2. 85% of youth will move to a

less restrictive setting upon release from the program.










Intake manager will enter treatment plan date into computer record upon completion of the plan. Discharge manager will enter post placement data into the computer record upon discharge of the client.










Intake manager will enter treatment plan date into computer record upon completion of the plan. Discharge manager will enter post placement data into the computer record upon discharge of the client.

On a monthly basis, program manager will review treatment plan completion rate against the standard. Quarterly review of the rate will be conducted by the CQI committee. Staff will manage to the objective. On a quarterly basis, program manager will review post placement discharge rate against the standard. Quarterly review of the rate will be conducted by the CQI committee. Staff will manage to the objective.

55


Build an Evaluation Model


Your Program Evaluation ModelYour Program Evaluation Model





Where do I start?Where do I start?

• Rome wasn’t built in a day! Focus on 2-3 key services and 2-3 outcomes.

• Don’t reinvent the wheel! Use available resources, beg, borrow and “steal”

• Don’t go it alone! Get together with colleague agencies to learn from one another, pool resources

• Don’t wait for perfection! Measuring outcomes is an involving field


Why Quality Improvement Efforts?Why Quality Improvement Efforts?

• We need to provide the highest quality services possible

• We need to learn what’s working and what isn’t

• We need to let our staff know how effective our services are

• We need to be accountable to our funders

• We need to make a difference

59

Thank you for attending my Thank you for attending my session today!session today!

If you have any feedback for how I can improve my training style or content, please email me your

suggestions!

([email protected])

Documents

1 Outcomes and Program Improvement: Designing Effective Evaluations Presented by Christine A. Ameen, Ed.D. Ameen Consulting & Associates ([email protected])