Effectively Utilizing Data Collection: A Case Study of a Functioning System

Effectively Utilizing Effectively Utilizing Data Collection: A Case Data Collection: A Case Study of a Functioning Study of a Functioning

SystemSystem

Robert BarteltRobert Bartelt&&

Kristen GayKristen Gay

Silver Springs – Martin Luther Silver Springs – Martin Luther SchoolSchool

Where to StartWhere to Start

Agency ValuesAgency Values Frame QuestionsFrame Questions Work with OthersWork with Others

LeadersLeaders Direct Care StaffDirect Care Staff Families and ChildrenFamilies and Children

PilotPilot Improve

Assess

Measure

Design

Data CollectionData Collection

What you get out of yourWhat you get out of your

data is only asdata is only asgood as what you put good as what you put

in….in….

Data Collection ProcessData Collection Process

Choosing What Data to CollectChoosing What Data to Collect

Clinically MeaningfulClinically Meaningful

Does the data add to your understanding of Does the data add to your understanding of

how clients are doing? how clients are doing?

Will your findings help you effect change in Will your findings help you effect change in

your program?your program?



User Friendly for StaffUser Friendly for Staff

Who is going to be collecting the data?Who is going to be collecting the data?

Are measures easy to understand?Are measures easy to understand?

How much training is required?How much training is required?

How many measures do you need?How many measures do you need?

How much time is available for data collection?How much time is available for data collection?


““Crunchable”Crunchable”

Does the measure provide information that is Does the measure provide information that is meaningful for individual analysis?meaningful for individual analysis?

Does the measure provide information that is Does the measure provide information that is meaningful for aggregate analysis?meaningful for aggregate analysis?

Do you have/how will you get the tools to manage Do you have/how will you get the tools to manage the data?the data?



Information Required from Third PartiesInformation Required from Third Parties

What are the requirements from third What are the requirements from third parties?parties?

Can your data collection system be set up so Can your data collection system be set up so as to satisfy these requirements?as to satisfy these requirements?

How will the data represent your agency and How will the data represent your agency and services?services?



Is the data collection system flexible?Is the data collection system flexible?

When a managed care company or a county When a managed care company or a county calls to request data on only “their” children, calls to request data on only “their” children, can you do that easily?can you do that easily?

Can you respond to new requirements without Can you respond to new requirements without expending a lot of additional resources? expending a lot of additional resources?




Do you want to engage in benchmarking?Do you want to engage in benchmarking?

Other similar organizationsOther similar organizations MCO expectationsMCO expectations County expectationsCounty expectations National normsNational norms

Examples of Data to CollectExamples of Data to Collect

Behavioral Rating ScalesBehavioral Rating Scales

Frequencies/Counts (i.e., Lengths of Stay)Frequencies/Counts (i.e., Lengths of Stay)

Patterns of Diagnostic Groups/Presenting ProblemsPatterns of Diagnostic Groups/Presenting Problems

Demographics of PopulationDemographics of Population

Contacts with FamiliesContacts with Families

Critical IncidentsCritical Incidents

Satisfaction SurveysSatisfaction Surveys

Aftercare DataAftercare Data

County, MCO, etc. RequestsCounty, MCO, etc. Requests

Evolution of Data CollectionEvolution of Data Collection As the agency and/or service provided As the agency and/or service provided

becomes more sophisticated, you want to becomes more sophisticated, you want to know more information.know more information.

Start collecting dataStart collecting data

Ask new questionsAsk new questions

What other data will What other data will you want to collect?you want to collect?

Do you have the flexibility to ask more Do you have the flexibility to ask more questions and gather more data?questions and gather more data?

Improve

Assess

Measure

Design

A Look at a Specific A Look at a Specific OutcomeOutcome

RestraintsRestraints A high-risk activityA high-risk activity Upfront goal to reduce the use of restraintsUpfront goal to reduce the use of restraints

What data is needed by internal committees:What data is needed by internal committees: How do you provide the most meaningful data to How do you provide the most meaningful data to

internal committees?internal committees?

How frequently will the data be required?How frequently will the data be required?

What variables are important?What variables are important?

Restraint Reduction Restraint Reduction

Collection of DataCollection of Data What do you want to know?What do you want to know?

Frequency of RestraintsFrequency of Restraints ChildChild Staff Who InitiatesStaff Who Initiates Time of DayTime of Day Length of RestraintLength of Restraint Result (injury of child, injury of staff)Result (injury of child, injury of staff) Children with multiple restraintsChildren with multiple restraints Type of restraint usedType of restraint used

Restraint ReductionRestraint Reduction

Flowchart: Developing a data collection Flowchart: Developing a data collection systemsystem

Data InputData Input


Data InputData InputList Manager







Data Input Data Input

Data OutputData Output

Data OutputData Output

Restraint Data

Restraint ReductionRestraint Reduction Different Types of ReportsDifferent Types of Reports

Trends Over TimeTrends Over Time

Restraints - January 2009 through November 2009

0

5

10

15

20

25

30

Jan-09 Feb-09 Mar-09 Apr-09 May-09 Jun-09 Jul-09 Aug-09 Sep-09 Oct-09 Nov-09

Nu

mb

er o

f R

estr

ain

ts

Unit A Unit B Unit C Unit D


Different Types of ReportsDifferent Types of Reports By UnitBy Unit

Unit A Unit B Unit C Unit D

January 2 10 2 1

February 9 8 8 2

March 27 15 8 1

April 28 7 7 2

May 16 6 18 1

June 13 10 11 1

July 4 9 11 1

August 14 9 7 0

September 15 9 9 1

October 9 8 6 1

November 6 3 7 1

Different Types of ReportsDifferent Types of Reports Restraints by Type of RestraintRestraints by Type of Restraint


Number of Restraints - All Types

0

10

20

30

40

50

60

Jan-

09

Feb

-09

Mar

-09

Apr

-09

May

-09

Jun-

09

Jul-0

9

Aug

-09

Sep

-09

Oct

-09

Nov

-09

Nu

mb

er

of

Re

str

ain

ts

Total Number Physical Restraints Prone Restraints Supine Restraints

Seated Restraints Standing Restraints Moving


Restraints by ChildRestraints by Child

Pareto Chart of Restraints

11%

66%

21%

30%

37%45%

63%60%56%

51%

0

50

100

150

200

250

Dav

id

Gle

nn

Am

ir

Des

tiny

And

reis

Ant

oine

Sha

rif

Kha

leem

Jahl

eel

Cas

sidy

Nu

mb

er o

f R

estr

ain

ts

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Per

cen

tag

e o

f T

ota

l R

estr

ain

ts


Restraints and InjuryRestraints and Injury

Number of Restraints and Child Injury from Restraints

0

10

20

30

40

50

60

Jul-

08

Au

g-0

8

Sep

-08

Oct

-08

No

v-08

Dec

-08

Jan

-09

Feb

-09

Mar

-09

Ap

r-09

May

-09

Jun

-09

Jul-

09

Au

g-0

9

Sep

-09

Restraints Child Injury


Children with Frequent RestraintsChildren with Frequent Restraints

Restraints D. S. - Unit A

0

2

4

6

8

10

12

Sep

-08

Oct

-08

Nov

-08

Dec

-08

Jan-

09

Feb

-09

Mar

-09

Apr

-09

May

-09

Jun-

09

Jul-0

9

Aug

-09

Sep

-09

Oct

-09

Nov

-09

RT MLS

Reporting to Third PartiesReporting to Third Parties HCSISHCSIS reporting reporting

Reporting to Third PartiesReporting to Third Parties•Reports to MCOsReports to MCOs

Reporting to Third PartiesReporting to Third Parties Reports to MCOsReports to MCOs



Responding to Requests by Responding to Requests by FundersFunders

Dilemma:Dilemma: Multiple expectations for reportingMultiple expectations for reporting Various requirements re data structureVarious requirements re data structure

Example:Example: Specific request from DHS to produce Specific request from DHS to produce

structured case notesstructured case notes

Structured Structured Case Note:Case Note:A Request A Request

by DHSby DHS

Measuring Symptom Reduction: An Measuring Symptom Reduction: An Example of the Use of Individual vs. Example of the Use of Individual vs. Aggregate DataAggregate Data

Youth Outcomes Questionnaire Youth Outcomes Questionnaire (YOQ)(YOQ)

64-item standardized measure64-item standardized measure

Total Score and Six SubscalesTotal Score and Six Subscales Intrapersonal DistressIntrapersonal Distress SomaticSomatic Interpersonal RelationsInterpersonal Relations Social ProblemsSocial Problems Behavioral DysfunctionBehavioral Dysfunction Critical ItemsCritical Items

Measuring Measuring Symptom Symptom ReductionReduction

Youth Outcomes

Questionnaire

(YOQ)

Individual Results

Measuring Symptom Measuring Symptom ReductionReduction

Y-OQ Total Score Over Time NM

65

91

106

86

67

65

0

20

40

60

80

100

120

Mar-08

Jun-08

Jul-08

Oct-08

Feb-09

Jun-09

To

tal

Sco

re

Inpatient

Outpatient

Community

Individual Results

Measuring Symptom Measuring Symptom Reduction: Reduction:

A Look at One Child - Two A Look at One Child - Two MeasuresMeasures

YOQ Scores and Number of Restraints for A.C.

0

20

40

60

80

100

120

May

-08

Jun-

08

Jul-0

8

Aug

-08

Sep

-08

Oct

-08

Nov

-08

Dec

-08

Jan-

08

Feb

-08

Mar

-08

Apr

-09

May

-09

Jun-

09

Jul-0

9

Aug

-09

Sep

-09

Oct

-09

Nov

-09

Dec

-09

Month

YO

Q S

core

s

0

1

2

3

4

5

Nu

mb

er of R

estraints

YOQ Restraints

YOQ Community Range

YOQ Outpatient Range

YOQ Inpatient Range

Measuring Symptom ReductionMeasuring Symptom Reduction

YOQ – Aggregate ResultsYOQ – Aggregate Results

Of the children discharged in 2008-2009, 42% Of the children discharged in 2008-2009, 42% showed a decrease in YOQ scores from the showed a decrease in YOQ scores from the second administration of the YOQ until the second administration of the YOQ until the time they were discharged. time they were discharged.

Children with higher YOQ Discharge Scores Children with higher YOQ Discharge Scores (suggesting more clinical symptoms) have a (suggesting more clinical symptoms) have a higher numbers of restraints.higher numbers of restraints.

Development of a New Development of a New Measure: Family WorkMeasure: Family Work

What do we want to know?What do we want to know? How often do families have therapy sessions?How often do families have therapy sessions?

Do families see therapy sessions as helpful?Do families see therapy sessions as helpful?

Do children participate in the sessions?Do children participate in the sessions?

Do meetings address issues that are important Do meetings address issues that are important to the family?to the family?

How can we measure it?How can we measure it?


Design Design MeasureMeasure AssessAssess ImproveImprove

Improve

Assess

Measure

Design


Discharge and Aftercare Discharge and Aftercare DataData

At DischargeAt Discharge Restrictiveness of Living Scale (ROLES)Restrictiveness of Living Scale (ROLES)

Aftercare SurveyAftercare Survey Interviews occur at four points in time:Interviews occur at four points in time:

3-6 months post discharge3-6 months post discharge 1 year post discharge1 year post discharge 2 years post discharge2 years post discharge 3 years post discharge3 years post discharge

At DischargeAt Discharge The Restrictiveness of Living Scale The Restrictiveness of Living Scale

(ROLES)(ROLES)Discharge Location for FY 2008-2009 Using the

Restrictiveness of Living Scale (ROLES)

Family Member (including adoption)

3255%

Foster Family Care10

18%

Therapeutic Foster Care

712%

Group Home2

4%

Residential Treatment

59%Juvenile Detention

12%

Therapeutic Foster Care Foster Family Care

Family Member (including adoption) Juvenile Detention

Residential Treatment Group Home

85% were discharged to a home setting

AftercareAftercare The intent of the Aftercare Study is The intent of the Aftercare Study is

to look at whether children are:to look at whether children are: At HomeAt Home

Still in less restrictive settingStill in less restrictive setting Disruptions in placementDisruptions in placement

In SchoolIn School SuspensionsSuspensions ExpulsionsExpulsions

Out of TroubleOut of Trouble Reported Drug/Alcohol UseReported Drug/Alcohol Use Involvement with Legal SystemInvolvement with Legal System

HealthyHealthy Have regular medical and dental appointmentsHave regular medical and dental appointments

Aftercare Data CollectionAftercare Data Collection

3 Month

6 Month

12 Month

24 Month

36 Month

80%

20%

79%

21%

83%

17%

85%

15%

94%

6%0%

20%

40%

60%

80%

100%

Follow-Up on Children Who Were Discharged to a Less Restrictive Setting -

Where Children Are Up to 3 Years Post-Discharge

Less Restrictive

More Restrictive

At HomeAt Home


In School

At the first interview, 84% of families report being satisfied with their child’s school placement.

Two years after discharge, 74% of families report no disruptions in their child’s education.


Out of Trouble

• A large percentage of the children were reported not to be using either drugs or alcohol. 96% of the children at the 24-month interval reportedly were not using either drugs or alcohol.


Percentage of Children with No Legal Problems

0%

20%

40%

60%

80%

100%

3-6 Months 12 Months 24 Months 36 Months

Month of Interview

No Legal Problems Involvement with LawIn Juvenile Detention Facility On Probation

Out of Trouble

Aftercare Data:Aftercare Data:Representation of NeedsRepresentation of Needs

Leads to a referral to Family Resource Services

More Sophisticated Data More Sophisticated Data AnalysisAnalysis

Use of Excel or SPSS to explore Use of Excel or SPSS to explore trends in data:trends in data: Run ChartsRun Charts Pareto ChartsPareto Charts CorrelationsCorrelations RegressionsRegressions T-testsT-tests

Effectively Utilizing Effectively Utilizing Data CollectionData Collection

Developing an effective data Developing an effective data collection system is a critical part collection system is a critical part of providing quality services of providing quality services today.today.

Performance-based service Performance-based service delivery is essential to our delivery is essential to our practice. practice.

Questions?Questions?

Documents

Effectively Utilizing Data Collection: A Case Study of a Functioning System