11

Do you or don’t you?

Measuring Fidelity to Evidence-

Based Supervision

Dr. W. Carsten Andresen

Dr. Geraldine Nagy

Travis County Adult Probation

2011 APPA Summer Conference - Chicago, Illinois

Let’s go beyond the EBP sound bite…

Please use EBP today

That’s EBP !

EBP has plenty of dash…

Where is the evidence?

EBP has plenty of dash…

Please use EBP today

That’s EBP !

These are pretty pictures, but…

Overview: Supervision Fidelity

Training and Learning Objectives

Evidence-Based Practices model

Department Studies

― Officer Supervision (aka fidelity study)

― Residential Treatment study

Training and Learning Objectives

1. Define and identify examples of

supervision fidelity and client outcomes

2. Explain the relationship between

supervision fidelity and client outcomes

Training and Learning Objectives

3. Identify the dimensions of community

supervision that are important to

measure (fidelity study)

4. Describe methods of evaluating

supervision fidelity

Training and Learning Objectives

5. Discuss approaches to present

measures of supervision fidelity to the

general public

Moving beyond the sound

bite…What do prior studies tell us about EBP:

Objective One:

Define and identify examples of

supervision fidelity and client

outcomes.

Outcomes: eventual effects of a program on

some condition (Maxfield, 2001).

- Probation Violations

- Revocations

- New Arrests

Supervision fidelity is less studied

Fidelity: presence of EBP elements and

quality of implementation

- Assess actuarial risk-needs

- Motivational Interviewing

- Target Interventions

- Positive Reinforcement

- Engage ongoing community support

o Risk principle

o Need principle

o Responsivity

o Dosage

o Treatment

14

Most evaluations involve a leap of faith

Program

Evaluations focus solely on outcomes

Evaluations sidestep fidelity measures

Most studies only measure outcomes

What about supervision quality?

How can you measure if EBP is happening?

Can we measure supervision quality and outcomes?

Travis County Adult Probation: We

primarily had outcome evaluations

External Evaluators: strong outcomes

Our own data: strong outcomes

- Dr. Tony Fabelo,

Justice Center,

Council of State Government

Travis Steepest Decline in Felony Revocations

(Fabelo, 2009)

- 19.6%

Travis Lowest Revocation Rate Out of Population

(Fabelo, 2009)

9%

Travis Steepest Decline in Technical Revocations

(Fabelo, 2009)

- 47.7%

Travis Lowest Rate of Technical Revocations

(Fabelo, 2009)

3.4%

Reduced Felony Revocations

Revocations Technical Revocations

- 270

- 294

Reduced Felony Absconders

- 1,659

Reduction in New Felony Absconders

- 422

24

Do we successfully use EBP?

Effective

Program

Outcomes

due to other factors

Ou

tco

me

s

High

Low

Objective Two:

Explain the relationship between

supervision fidelity and client

outcomes

26

EBP Model: Outcomes

Effective

Program

Outcomes

due to other factors

Ou

tco

me

s

High

Low

What about our Supervision Fidelity?

27

EBP Model: Fidelity and Outcomes

Effective

Program

Ineffective

Program

Outcomes

due to other factors

Poorly

Implemented

Ou

tco

me

s

Fidelity

High

Low

Low High

Spring 2009: Study Supervision Fidelity

TCIS measured at beginning of implementation

Initial baseline identifies areas to improve

Unannounced “pop quiz” of TCIS fidelity

Future fidelity studies can track progress

But where should we begin?

Travis Community Impact Supervision

Measuring Fidelity TCIS

TCIS measured at beginning of implementation

Initial baseline identifies areas to improve

Unannounced “pop quiz” of TCIS fidelity

Future fidelity studies can track progress

EBP Process and Context

Measuring Fidelity to EBP

Measured at beginning of implementation

Initial baseline identifies areas to improve

Unannounced “pop quiz” of EBP fidelity

Future fidelity studies can track progress

Two Travis County Supervision Studies

Felony Cohort (Jan 2008 - Placements)

Follow supervision practices for six months

Focus on cases direct for at least three months

Follow up: Cohort (Jan 2009 - Placements)

Sample of Felony-reduced and Misdemeanants

Study Descriptives

2. Coders

Study 1

1. Cohort Sample

Study 2

- Felons 134 102 - Felony-Reduced 17- Misdemeanants 46

- Administrators X X - Operations-Research X X- Managers X

Objective Three:

Identity the dimensions of

community supervision that are

important to measure

(Fidelity study)

Purpose:

To measure our fidelity to EBP

Objective Four:

Describe methods of evaluating

supervision fidelity

36

Case File Review Form – Fidelity to EBP

37

Methodology

CSS Case File

Review

Form

Probationer

Case File

38

Methodology

Case File Review Form Database

Objective Five:

Discuss approaches to present

measures of supervision fidelity

to the general public

Fidelity Study:

Fidelity to EBP Results (Jan 2009)

Supervision Fidelity:

Do we create accurate assessments?

We have validated our risk assessment instrument

three times

We have conducted Inter-rater reliability tests on our

Diagnostic Report Process

Do we create accurate assessments?

89%

95%

95%

89%

We checked scoring on each diagnostic instrument

for each person

Supervision Fidelity:

What about the Judiciary / Courts?

44

Court-Ordered Conditions of Probation

90%

Supervision Fidelity:

What about Supervision Process?

46

Do we focus on risk-needs and use

motivational interviewing?

We examined each

point carefully.

47

Correct Caseload

- Maintenance Caseload

- Regular Caseload

- Correct Specialized

Caseload match Probationer?

97%

48

Supervision Agreement

85%

Note: Some felony probationers were excluded from

this analysis because they were unable to

create a Supervision Agreement for various

reasons (i.e. absconding, reoffending)

49

Supervision Agreement

85%

50

Responsivity / Motivation

51

Responsivity / Motivation

- 83% some use

- 53% frequent use

Source: High and Medium Risk Felons

We used a 5-point scale to measure

Motivational Interviewing

52

Sanctions / Incentives

67%

53

Sanctions / Incentives

67%

2009: 10 rush warrants

- all 10 addressed properly

- all 10 timely response

54

Sanctions / Incentives

75%

55

Overview of Internal Studies

56

EBP Model: Fidelity and Outcomes

Effective

Program

Ineffective

Program

Outcomes

due to other factors

Poorly

Implemented

Ou

tco

me

s

Fidelity

High

Low

Low High

57

EBP Model: Fidelity and Outcomes

Effective

Program

Ou

tco

me

s

Fidelity

High

Low

Low High

What about Fidelity Outcomes?

What happened to the probationers in this supervision study?

Felony probationers: two year follow-up

Revocations

Arrests

EBP Supervision appears correlated with better

outcomes for felony probationers, with caveats

Small felony sample

Difficult to account for complicated interactions

between variables

Overall Supervision Measure

- Was the Supervision Appropriate?

Data Source: Coder provided summary ranking of

supervision quality.

The Supervision Agreement is more

than just a piece of paper

We looked for evidence that this document played a role in the supervision process. We examined if the document had the following:

Probationer’s signature

Indications that the creation of the Supervision

Agreement was a collaborative process

Chronological Notes suggesting that the

Supervision Agreement was the foundation for

office visits

Did the Supervision Agreement show

signs of EBP

Yes: Signature, evidence of cooperation, evidence it

was used as the foundation for office visits

Now we will move

away from supervision

to discuss SMART –

our Residential Drug

Treatment Program.

Fidelity Evaluation: Drug Treatment

What does our SMART Program do?

Does it engage in Evidence-Based practices?

Quality of work?

64

EBP Model: Fidelity and Outcomes

Effective

Program

Ineffective

Program

Outcomes

due to other factors

Poorly

Implemented

Ou

tco

me

s

Fidelity

High

Low

Low High

65

Prior SMART Residential Analysis

• Corrections Institute-University of Cincinnati

– Correctional Program Checklist (CPC)

• Highly Effective

• Effective

• Needs Improvement

• Ineffective

CPC – 5 Domains

- Leadership

- Staff

-Assessment

-Treatment

-Quality Assurance

66

Corrections Institute – University of Cincinnati

SMART Program Rank (400+ Programs)

18% Effective

33% Needs Improvement

42% Ineffective

7% Highly Effective SMART Program

Shaffer, DK and C Thompson. (2008). Evidence-Based Correctional Program Checklist – SMART Program. University of Cincinnati.

67

Study: 3 yrs successful completions - 18 month follow-upSMARTSep 2005 - Aug 2008 18

2004

Sept

2005

Sept

2006

Sept

2007

Sept

2008

Sept

2009

Sept

Feb 2007

SOMP start

SMART Residential Program

Nov 2008: CPC

Comparison Group

18

1818Sep 2005 - Aug 2008

68

Creating Comparison Group

- Risk Score

- Fiscal Year

- Age Range

- Race-Ethnicity

- Gender

- Offense Degree

- Offense Type

Smart Comparison

N = 559 N = 489

•Individual cases matched on seven factors

• Many studies avoid risk / lack comparison groups

69

It is critical to match for Risk Score

9% 25% 67% - Felony Placements

0% 14% 86% - Smart

70

Note: this captures # of arrests, not # of people arrested.

All Arrest Categories (by group)

27 % fewer total arrests

12% fewer total felony arrests15 % fewer total misd arrests

SMART: % decrease in total arrests

71

EBP Model: Fidelity and Outcomes

Effective

Program

Ineffective

Program

Outcomes

due to other factors

Poorly

Implemented

Ou

tco

me

s

Fidelity

High

Low

Low High