Use of Fidelity Measurement Use of Fidelity Measurement in Statewide Technical in Statewide Technical Assistance ActivitiesAssistance Activities

Gary Bond Festschrift ConferenceGary Bond Festschrift ConferenceIndianapolis, INIndianapolis, IN

September 24, 2009September 24, 2009

Angela L. Rollins, Ph.D.Angela L. Rollins, Ph.D.ACT Center of IndianaACT Center of Indiana

IUPUI Psychology DepartmentIUPUI Psychology Department

OverviewOverview

Background on fidelity use in Background on fidelity use in generalgeneral

Use for fidelity in technical Use for fidelity in technical assistance (TA) efforts across assistance (TA) efforts across various EBPsvarious EBPs

Some new directionsSome new directions Further questionsFurther questions

Uses of Fidelity: Uses of Fidelity: ResearchResearch

Internal validity: examine model Internal validity: examine model adherence/driftadherence/drift

Facilitating communication in literatureFacilitating communication in literature Synthesizing a body of researchSynthesizing a body of research Identifying critical ingredients of Identifying critical ingredients of

program modelsprogram models Defining model adaptationsDefining model adaptations

Uses of Fidelity: TAUses of Fidelity: TA Benchmarks to add perspective

Over timeOver time Comparison to other programsComparison to other programs Comparison to normsComparison to norms

Coaching Written report of strengths, weaknesses,

recommendations for improvement Teaching moments within the

assessment day itself

Uses of Fidelity: TAUses of Fidelity: TA

Defining program standards/critical ingredients

Monitoring adherence to program Monitoring adherence to program standards – tricky new areastandards – tricky new area

Springboard for other TA activitiesSpringboard for other TA activities Data gathering (Indiana)Data gathering (Indiana) Outcomes data reporting (Kansas)Outcomes data reporting (Kansas)

Indiana ACT Fidelity Indiana ACT Fidelity ApproachApproach

2 assessors 2 assessors 1 1 Semi-annual Semi-annual annual after first annual after first

yearyear Even with these reductions, as # Even with these reductions, as #

teams grow and # consultants teams grow and # consultants remains constant, fidelity visits take remains constant, fidelity visits take a bigger piece of the overall a bigger piece of the overall consulting pieconsulting pie

Indiana ACT Fidelity Indiana ACT Fidelity ApproachApproach

Testing phone fidelity measures Testing phone fidelity measures (see (see McGrew talk tomorrow)McGrew talk tomorrow)

Level of burden has reduced a little over Level of burden has reduced a little over time, even for onsite assessmenttime, even for onsite assessment Team leaders more familiar with requestsTeam leaders more familiar with requests Agencies create reports over time (trust?)Agencies create reports over time (trust?) Assessors more adept as well - completing faster Assessors more adept as well - completing faster

than in initial yearsthan in initial years

Still need ongoing support - reduce rater Still need ongoing support - reduce rater drift, feedback on scoring decisionsdrift, feedback on scoring decisions

Multiple Uses of the VisitMultiple Uses of the Visit Data gatheringData gathering

Admission and discharge data for trackingAdmission and discharge data for tracking Tracking raw data used to compute fidelity Tracking raw data used to compute fidelity

scoresscores raw turnover ratesraw turnover rates actual FTE and caseload sizeactual FTE and caseload size actual frequency and intensityactual frequency and intensity

Teaching moments: most extensive Teaching moments: most extensive contact with site for the year in most contact with site for the year in most casescases

Multiple Uses of the VisitMultiple Uses of the Visit Report team’s scores over time in each Report team’s scores over time in each

report (examine annual progress)report (examine annual progress) Additional monitoring of state standards Additional monitoring of state standards

compliancecompliance Additional information added to report where Additional information added to report where

DACTS and state standards differDACTS and state standards differ Noncompliance reported to team initially, to Noncompliance reported to team initially, to

state if not remediated within ~90 daysstate if not remediated within ~90 days Research on real-world data, but problems Research on real-world data, but problems

with restriction in range (state standards with restriction in range (state standards dictate high fidelity)dictate high fidelity)

Indiana DACTS dataIndiana DACTS dataHuman Res Organizati

onServices DACTS Total

Baseline n=34

4.05 (.50)

4.14 (.56)

3.33 (.60)

3.81 (.43)

12 mos n=33

4.28 (.34)

4.56 (.42)

3.70 (.52)

4.14 (.34)

24 mos n=30

4.24 (.27)

4.68 (.23)

3.80 (.44)

4.20 (.26)

36 mos n=27

4.28 (.30)

4.69 (.19)

3.80 (.35)

4.21 (.20)

48 mos n=19

4.24 (.34)

4.65 (.18)

3.97 (.29)

4.25 (.20)

60 mos n=13

4.33 (.34)

4.63 (.26)

3.82 (.58)

4.22 (.34)

72 mos n=13

4.24 (.34)

4.67 (.11)

3.73 (.29)

4.17 (.17)

84 mos n=2

4.41 (.32)

4.86 (.00)

3.95 (.21)

4.36 (.20)

Fidelity Monitoring: IMRFidelity Monitoring: IMR Semi-annual visitsSemi-annual visits ““Baseline” after 3-6 months is more meaningful for Baseline” after 3-6 months is more meaningful for

program developmentprogram development Emphasis on qualitative feedback rather than Emphasis on qualitative feedback rather than

numeric scorenumeric score Data collection via observation of IMR sessions is Data collection via observation of IMR sessions is

vitalvital Immediate feedback at the end of visit important Immediate feedback at the end of visit important

for QI function of fidelityfor QI function of fidelity In progress: fidelity via audiotaped sessions – In progress: fidelity via audiotaped sessions –

based on Mueser’s workbased on Mueser’s work

Across EBPsAcross EBPs

Role of fidelity may vary in emphasis Role of fidelity may vary in emphasis across EBPsacross EBPs Example: Does fidelity measurement of Example: Does fidelity measurement of

IMR play as large a role in IMR play as large a role in implementation as it does for ACT?implementation as it does for ACT? Clinical competency needs pronounced for Clinical competency needs pronounced for

IMR, IDDT (see Mueser talk tomorrow)IMR, IDDT (see Mueser talk tomorrow) Program structural needs for ACT very Program structural needs for ACT very

pronouncedpronounced

Future DirectionsFuture Directions Increasing calls to decrease the burden Increasing calls to decrease the burden

of fidelity measurement (e.g., We want of fidelity measurement (e.g., We want to use one FTE or less for monitoring to use one FTE or less for monitoring our whole state)our whole state) Phone/remote, less intensive version (see Phone/remote, less intensive version (see

McGrew)McGrew) Lower frequency of assessmentsLower frequency of assessments Screening approachScreening approach

key itemskey items follow-up with full visit if flagged during screenfollow-up with full visit if flagged during screen

Future DirectionsFuture Directions Ohio developing The Evaluation Database: Ohio developing The Evaluation Database:

“TED”“TED” Online data entry by consultant after visitOnline data entry by consultant after visit

Perhaps during onsite visit with wireless cardPerhaps during onsite visit with wireless card More automated consensus and report functionMore automated consensus and report function Track when consultants enter data and when reports Track when consultants enter data and when reports

go out: QA for the TAgo out: QA for the TA Raw data elements as well for fuller data gatheringRaw data elements as well for fuller data gathering IDDT and (SE?) so far, may collaborate to tailor to IDDT and (SE?) so far, may collaborate to tailor to

ACT ACT Seeking funding to tie in consumer outcomes as wellSeeking funding to tie in consumer outcomes as well Contact: debra.hrouda@case.eduContact: debra.hrouda@case.edu

Questions to PonderQuestions to Ponder Are statewide fidelity scores a good Are statewide fidelity scores a good

measure of the TA center’s effectiveness?measure of the TA center’s effectiveness? Would outside fidelity assessors (outside Would outside fidelity assessors (outside

the state/entity) produce similar results?the state/entity) produce similar results? Beyond basic requirements for Beyond basic requirements for

certification, what motivates teams to score certification, what motivates teams to score high on fidelity scale?high on fidelity scale?

Can we marry research and practice when Can we marry research and practice when practice yields a restriction in range on practice yields a restriction in range on fidelity scores?fidelity scores?

Thank youThank you

Charles BoyleCharles Boyle Michelle SalyersMichelle Salyers Lia HicksLia Hicks Deb HroudaDeb Hrouda