1

Preparation for Outcome Monitoring

2

Presenters: 1. Jeff Wagers: Tx DSHS Program/

Training Specialist2. Ken Ripperger-Suhler, Ph D:University of Texas at Austin,

Researcher

3

Introductions

Presenters Audience Agenda Objectives

4

What is an Outcome?

The resulting changes in the client during or after a client attends an intervention.

5

Brainstorm-Outcomes!

1. What outcomes would you like to see in your

intervention?2. Each intervention will report

6

Agenda Review Basics of Interventions and OM Walk through a sample intervention from

logic model to analysis Readiness to do Outcome Monitoring Small Group Breakout by Intervention Administering surveys in a trusting

atmosphere Practice inputting survey data Data analysis and Implications Final questions and wrap-up

7

Objectives

Review basics of interventions and OM Discuss parameters and preparation for

OM in Summer 2010 Examine and assess OM pre/post surveys

by intervention Learn how to input and analyze data Brainstorm challenges and solutions to OM Reassess readiness to do OM

8

Evidence-Based Interventions (EBI)

Clearly defined audience, goals & objectives Based in behavioral theory Focused on specific risk behavior Could be an individual, group, or

community level intervention

9

Evidence-Based Interventions (EBI)

Individual-Level Interventions: CRCS

Group Level Interventions

Community Level Interventions

10

Key Terms Internal Logic Core elements Fidelity Behavioral Determinants/ Factors that

Influence Behavior/ Risk Factors Tailoring Adapting Reinventing

11

Behavioral Determinants; Factors Influencing Behavior (FIBs)

The “why” Reasons a client or

population is engaged in the risk behavior

Come from the client through: CRCS: asking clients

questions GLI, CLI: community

assessment Interventions focus on

the BD/FIB as well as the behavior

12

Program Evaluation

• What is evaluation?

• Why evaluate?

13

Program Evaluation Systematic, consistent collection of data Shows program effectiveness Accountability Program Improvement Increases Capacity Defines parameters Identifies unmet needs Process vs. outcomes

14

What is Process?

“How” you provide your intervention

Fidelity to the intervention Quality Assurance issues Skill level of agency staff Numbers (who attended, how many

sessions, # of target population, etc)

15

Process Questions

Is the location accessible? Is the program implemented as designed? Was the curriculum followed? How many of the target population attended

the sessions? How well is the RRS able to keep the session

focused on risk behavior? Was staff culturally competent? Others?

16

What is an Outcome?

The resulting changes in the client during or after a client attends an intervention.

17

Outcome Questions Did the client

Learn anything? Improve their skills? Change their behavior? Realize they are at risk of getting HIV?

18

Outcomes measure:

S - skillsK - knowledgeA - attitudeB - belief

B - behavior

19

Logic Models

20

Problem Statement

Risk Behaviors

Influencing Factors

Intervention/Activities

OutcomesImmediate Intermediate

Outcomes

Immediate Outcomes: Immediateresults of the intervention, such as changes in knowledge, attitudes,beliefs, and skills.- Increased perception of HIV risk- Increased condom use skills- Increased self efficacy to use condoms

Intermediate Outcomes: Intervention Results that occur some time after the intervention is completed, such as changes in behaviors, policies, and environmental barriers.

-Increased condom use-Decreased number of partners

21

STD rates are high among

young Hispanic MSM

Problem Statement

Young Hispanic MSM don’t

use condoms

Young Hispanic MSM have multiplepartners

Risk Behaviors

Young Hispanic MSM lack skills to properly

use condoms

Young Hispanic MSM don’t know how to

negotiate condom use with

their partners

Influencing Factors

Factors Influencing Behavior

It’s the norm for young Hispanic MSM

to have a lot of sex partners

22

STD rates are high among

young Hispanic MSM

Problem Statement

Young Hispanic MSM don’t

use condoms

Risk Behaviors

Young Hispanic MSM lack

skills to properlyuse condoms

Young HispanicMSM don’t know how to

negotiatecondom use with

their partners

Safe-sex negotiationscenarios &role plays

Self-assertionskills training

InterventionActivities

Influencing Factors

Intervention Activities

Young Hispanic MSM have multiplepartners

Condom skills demonstration and practice

23

Immediate and Intermediate Outcomes

Young Hispanic MSM lack

skills to properlyuse condoms

Young Hispanic MSM don’t

know how tonegotiate

condom use withtheir partners

Influencing Factors

Safe-sex negotiationscenarios &role plays

Self-assertionskills training

InterventionActivities

Immediate Outcomes

Young Hispanic MSM know how to

negotiate condom use with

their partners

Young Hispanic

MSMuse

condoms

Intermediate Outcomes

Condom demo and practice

Young Hispanic MSM know how to use condoms

24

Problem Statement

Risk Behaviors

Influencing Factors

Intervention/Activities

OutcomesImmediate Intermediate

Impacts

Impacts

Impacts: Long-term results ofone or more interventions overtime, such as changes in HIV infection, morbidity, and mortality

- Decreased HIV rates

25

Outcome Monitoring

Data CollectionAnd

Analysis

26

Steps to Outcome Monitoring Develop *SMART outcome objectives

Specific, Measurable, Appropriate, Realistic, Time-Based

Develop/ review/ modify surveys or other methods that collect outcomes

Gathering survey/ method data Input data into spreadsheets Analyze and report data (Outcome

Monitoring) Make program adjustments if needed

27

Example of Process ObjectiveProcess objective: “Clients will complete

SISTA”

Revised: “By December 31, 2010, 50 African-American women will complete SISTA as documented by client data system.”

28

Example of Outcome ObjectiveOutcome objective: “Clients who complete

SISTA will show an increase self efficacy”

Revised: “75% of participants who complete SISTA will increase self-efficacy to correctly put on a condom as measured by pre/post survey.”

29

Ways to Collect Outcomes

30

Collect Outcomes Skills

Correct condom use, needle-bleaching Negotiation, Communication, Assertiveness

Questions Knowledge Attitudes Beliefs Behaviors

31

Demonstrations and PracticeRole Plays

Measuring Skills

32

Demonstrations Examples:

Condom Use Skills Needle Bleaching Skills

Checklists Consistent Provide “clean” pre-test Demonstrate correct condom use Practice Next session, do a post-test and compare

results

33

Role Plays Examples:

Condom or bleaching negotiation skills Self Assertiveness skills

Observations: Consistent Provide “clean” pre-role play (or reverse role

play) Demonstrate correct method Practice Next session, do a post-role play and compare

results

34

Types of Questions Open Ended Closed Ended

True/False or Yes/No Fill in the Blank Likert Scale Frequency Indicator Multiple Choice

35

Question Answer Options

Answer Option Answer Code

True/False T/F

Yes/No Y/N

Fill in the Blank FB

Scale S

Frequency Indicators

FI

Open Ended OE

Multiple Choice MC

36

Sample Questions Perceived susceptibility

“Given my current behavior, I am worried that I might get HIV” (T/F) (Y/N)

“Given my current behavior, how worried am I that I might get HIV? (Scale)

1 2 3 45

Not worried Very worried

37

Sample Questions Expected outcome related to condom

use

“I will get HIV from my main partner without using a condom” (T/F) (Y/N)

“How likely is it that I will get HIV from my main partner without using a condom?”

1 2 3 45

Not likely at all Very likely

38

Steps to Outcome Monitoring Gathering survey data

Individual interviews Written questionnaires Focus groups

Input data into spreadsheets Analyze and report data (Outcome

Monitoring) Make program adjustments

39

Putting It All Together

40

Example-VOICES/VOCES

VOICES/ VOCES is a one-session group level intervention for HRH men and women in which groups watch culturally appropriate videos of condom negotiation, condom use and assertive communication, and practice.

Participants role-play condom use scenarios and practice putting on condoms

41

Logic Model

Increased Condom

Use

Increased Condom Use Knowledge

Increased Condom Use Self-Efficacy

Increased Condom Use

Intention

INTERMEDIATE OUTCOMES(BEHAVIOR)

IMMEDIATE OUTCOMES

ACTIVITIESINFLUENCING FACTORS

Practice Condom Use;

Distribute condoms

ViewVideos;Condom

Negotiation Role Plays

RISK BEHAVIORS

Unprotected Sex

Condom Use Knowledge

Condom Use Self-Efficacy

Condom Use Intention

42

At least 40% of clients will demonstrate greater knowledge of condom usage by identifying on a written test at the end of the session the correct answer (yes or no) with regard to the following knowledge questions:

a. Are specific lubricants appropriate for use with condoms;

b. Whether or not to unroll a condom before putting it on a man’s penis.

Objective (Knowledge of Condom Use)

43

Survey Items (Knowledge of

Condom Use)

Yes NoDon’t Know

1. Vaseline and lotion are good lubricants for condoms/rubbers.

1 2 3

2. You should unroll a condom before putting it on a man’s penis.

1 2 3

44

Spreadsheet for storing data

Client ID

Race/

EthnicAge at intake Gender

Sexual Orient

HIV Status

Pre ConKnow

1

Pre ConKnow

2

A01  AA 30  M  STR  -  yes  yes

A02  AA  25  M  BI  -  yes  yes 

A03  H  27  M  GAY  -  yes  yes 

45

Frequencies Pre & PostItem Vaseline and lotion are good

lubricants for condoms/rubbers.

Yes(%)

No(%)

Don’t Know(%)

(PRE) 20(100%)

0(0%)

0(0%)

(POST) 0(0%)

20(100%)

0(0%)

46

Vaseline and lotion are good lubricants for condoms/rubbers.

Post Responses

Yes No Don’tKnow

Total

Pre Yes 0 20 0 20(100%)

Responses No 0 0 0 0(0%)

Don’t Know 0 0 0 0(0%)

Total 0 20 0 20(0%)

47

Frequencies Pre & PostItem You should unroll a condom before

putting it on a man’s penis.

Yes(%)

No(%)

Don’t Know(%)

(PRE) 20(100%)

0(0%)

0(0%)

(POST) 20(100%)

0(0%)

0(0%)

48

You should unroll a condom before putting it on a man’s penis.

Post Responses

Yes No Don’tKnow

Total

Pre Yes 20(100%)

0(0%)

0(0%)

20(100%)

Responses No 0(0%)

0(0%)

0(0%)

0(0%)

Don’t Know 0(0%)

0(0%)

0(0%)

0(0%)

Total 20(100%)

0(0%)

0(0%)

20(100%)

49

Table 1. Change in Knowledge Scores from Pre to PostItem %

correct post

% change

d to correct

% change

d to incorre

ct

% change

to correct answers

1. Vaseline and lotion are good lubricants for condoms/rubbers.

100 100 0 100

2. You should unroll a condom before putting it on a man’s penis.

0 0 0 0

50

Outcome Monitoring: VOICES/VOCES Report

Condom Use KnowledgeThe results of cross tab analysis find that 100% of clients showed increased

knowledge of appropriate lubricants for condom use.

0% of clients showed increased knowledge regarding when to unroll a condom to put on a man’s penis.

51

Planning for Outcome

Monitoring

52

Planning for Outcome Monitoring

Readiness for Outcome Monitoring

Done on an individual agency basis Stable implementation Data collection instruments

developed by Ken with agency input Outcomes may change over time

53

DSHS Parameters for Outcome Monitoring

1. Skills-building, knowledge, behavioral determinants ( AKA FIBs)

2. BD/ FIBs identified in your intervention based on research

3. Pre/post survey (unless an agency wants to do observations)

4. Based on logic model of intervention

3-4 outcomes for GLI

54

Some Differences in Data Collection…. CRCS

Intermediate and Immediate Outcomes Use questions from behavioral assessment No standardized survey One initial assessment, up to two

reassessments Driven by prevention plan and goals Designated outcomes Pre/post done on same individual Percentages for outcomes 15-50% Not all outcomes will apply to all individuals

55

Some Differences in Data Collection…. Group Level

Immediate Outcomes Pre/post tests Standardized survey 3-4 outcomes individualized per intervention Pre/post done on same individual Percentages for outcomes 40-75% Can measure any SKABB

56

Some Differences in Data Collection…. Community Level: 2 components

Group Component-measured first 2-4 outcomes Will measure M-Groups or training of peer

volunteers Standardized survey Pre/post; percentages 50-80% Knowledge, skill-level, able to communicate

very important Will be collected first stage

57

Some Differences in Data Collection…. Community Level:

Community Component-measured second 2-4 Outcomes Done during community assessment Pre/Post tests; Venue-based Mainly measures knowledge, attitudes,

beliefs, community norms Percentages up to 15% Gathered yearly Will be collected in second stage

58

Outcome Monitoring Surveys

Ask 3-4 questions per outcome 10-20 questions per survey Can measure skills through survey Knowledge can be a component Include key outcomes of intervention, in

line with behavioral determinants Consistent data collection

59

Facilitated Group Activity 11. CLI: Discuss group and community level

components of intervention

2. GLI: Discuss and review OM surveys by intervention

3. CRCS: Brainstorm problems and possible solutions to gathering OM data

60

Guided Group Discussion

Ways to Collect Outcomes1. Giving Instructions

2. Oral Interviews3. Written Questions

4. Focus Groups

61

Activity 2: Data Input and Analysis Input sample data into a spreadsheet

Sample data is located in packet Input into your laptop using the sample

spreadsheet (partner with someone if needed)

Analyze the data Spreadsheet calculates the data What can you determine from analysis?

62

Next Steps

Per intervention, brainstorm challenges and solutions to

Outcome Monitoring

63

Final Questions? Please complete

your course evaluations

Thank you for your participation!

64

Contact information Jeff Wagers, TxDSHS

Jeffrey.wagers@dshs.state.tx.us 512-533-3022

Ken Ripperger-Suhler, University of Texas Ken.ripperger-suhler@dshs.state.tx.us 254-624-1028