Upload
diane-watts
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Building a Data-Driven Culture in Nevada
June 10, 2013
The Performance Indicators Project is a collaboration of the California Department of Social Services and UC Berkeley, and is supported by CDSS , and the Stuart Foundation.
Summit Content Areas• Leading with Data: A Focus on Outcomes
– Review of basic terminology– Avoiding pitfalls and data abuse– Key concepts in performance measurement– Telling the NV Story: State and National Context
• Connecting Data to Practice: Defining the problems
• From Data to Action: Strategy Development and Implementation
• CQI Structure and Function in NV
BASIC TERMINOLOGY AND FORMULAS
Data Analytics 101
Basic Terminology
Descriptive Data• Point-in-time
• Trends
• Comparisons
data source: AFCARS
Basic TerminologyProcess Measures - familiar to staff, relevant at a
caseworker level, current
Outcome Measures - the “big picture” measure of system performance, especially when looked at longitudinally
Measures of Central TendencyMean: the average value for a range of data
Median: the value of the middle item when the data are arranged from smallest to largest
Mode: the value that occurs most frequently within the data
12 4 15 63 7 9 4 17 4 4 7 9 12 15 17 63
4.168
631715129744 Mean
5.102
129 Median
4 Mode
7= 9.7
= 9
Measures of Variability
Minimum: the smallest value within the data
Maximum: the largest value within the data
Range: the overall span of the data
4 Minimum
63 Maximum
59463 Range
4 4 7 9 12 15 17 63
Disaggregation
• One of the most powerful ways to work with data…• Disaggregation involves dismantling or separating out
groups within a population to better understand the dynamics and plan strategies for improvement
• Useful for identifying critical issues that were previously undetected
Aggregate Permanency OutcomesRace/Ethnicity
Age
Region/Circuit
Placement Type
Measuring Change
100
yr baseline
yr) baseline-yr(latest netchange
•How much has this measure changed over time?•What will our performance be next quarter if we increase or decrease by 10%
10% increase = baseline x 1.1
10% decrease = baseline x .90
COMMON DATA PITFALLSData Analytics 101
Common Pitfalls
• Small N – impact on rates and trends• Seasonal variation• Faulty comparisons – failing to consider
demographic and policy differences• Outlier impact on central tendency• Data integrity/Data entry (over or under
emphasized)• Missing or incomplete definitions• Data overload: lack of focus on and connection
to key outcomes
Common Pitfalls: Seasonal variation
Period 1 to 2:38.5% reduction
Period 2 to 5:41.7% increase
Period 1 to 7:10.2% reduction
Common Pitfalls:Small n (impact on rates and trends)
100% reduction! But…from 2 children in
care to 0 children in care
57% increase!But from 7 to 11 children in
care
Common Pitfalls and Graph Interpretation
Data source: UNC at Chapel Hill Jordan Institute for Families website. URL: http://ssw.unc.edu/ma/
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
# initial place Group Home 25 25 17 23 11 5 9 4 2 1 7
Total Entries 269 215 236 253 236 194 190 197 148 156 150
% Initial place Group Home 9% 12% 7% 9% 5% 3% 5% 2% 1% 1% 5%
0%
5%
10%
15%
20%
0
50
100
150
200
250
300
% o
f all
entr
ies
# of
chi
ldre
n
Guilford County: First Entries by Initial Placement Type
Avoiding data overload and Managing with data
One thing the modern computer age has given everyone is data. Lots and lots of data. There is a large leap, however, between having data and learning from it.
W. Gregory Mankiw
Professor of Economics, Harvard
New York Times, Sunday Business Section (Sept. 5 2010) p. 5
Manage with Data
Provides us the ability to:• Compare metrics with agency mission and
practice model• Connect to evidence-based practice and link
processes to desired outcomes• Strategize on what work needs to be done• Focus on end outcomes• Identify what needs attention• Tell the story
Manage with Data
• Pick the right measures for the job• Prioritize reports and measures in line with
agency values, mission, vision• Connect process measures to outcomes/practice
model• Move beyond compliance and “gotcha”• Make it fun!• Celebrate success and tell the story• Use your data to engage the community, create
urgency for action, maintain support
Pick the Right Measures for the Job
• Federal Accountability• Overall Performance Monitoring• Outside Auditing (i.e. consent decrees,
monitors)• Contractor or Provider Performance• Office, Supervisor, Worker Level
accountability
Compliance Performance Outcomes
Continuum
Prioritize Measures in line with Agency Values
• To keep children safe and at home• To improve a child or youth’s well-being• To facilitate a child or youth’s move to swift
& certain permanency
Performance is Guided by Your Values as an Agency: Missouri Key Outcomes
Key Data Reports: How are they all connected?
CFSR and PIP
SCRT
ROMAFCARSNCANDS
SEE ResultsCOA
Connecting the Dots
Management Report: Frequency
of Visits with Caretakers
Case Review Measure: Caregiver involvement in case
planning
Case Review Measure:
Individualized Services
Outcome Measure:
Timely Reunifications
Process Data: AccountabilityRelevant to workers and
supervisors
Intermediate OutcomesRelevant to workers,
supervisors, managers
Outcomes: “So What?”Reflect Key Priorities of
Leadership
CFSR Findings: Relationship of Well-Being to Permanency
Positive ratings on
• Services to children, parents, foster parents
• Involvement of parents in case planning
• Caseworker visits with children
• Caseworker visits with parents
Substantial
achievement on
• Timely achievement of permanency
• Preserving children’s connections while in foster care
supports . . .
Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm
Factors Associated with Timely Reunification, Guardianship, and Permanent Relative Placement
The strongest associations with timely permanency included:
Caseworker Visits with Parents
Child’s Visits with Parents and Siblings in Foster Care
Services to Children, Parents, & Foster Parents
Family/Child Involvement in Case Planning
ASFA Requirements Regarding Termination of Parental Rights
Placement Stability
Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm
Strongest Associations Between Visits and Other Indicators
Both Caseworker Visits with Parents and Caseworker Visits with Children were strongly associated with:
Risk of harm to children
Needs & Services for children, parents, foster parents
Child and parent involvement in case planning
Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm
Other Significant Associations Between Visits and Indicators
Caseworker Visits with Parents and Caseworker Visits with Children were also strongly associated with:
Services to protect children at home Safety Outcome 1 Safety Outcome 2 Timely permanency goals Timely reunification Child’s visits with parents and siblings Relative placements Meeting educational needs Meeting physical health needs Meeting mental health needs
Administration for Children and Families, U.S Department of Health and Human Services, Findings From the Initial Child and Family Services Reviews, 2001–2004. Available at http://www.acf.hhs.gov/programs/cb/cwmonitoring/results/index.htm
Connecting process to outcomes
Use Data to Create Urgency for Action –Target improvements based on your own baseline
Regional variation should generate productive discussion about differences in: • Service array• Community differences in reporting and tolerance for “risk”• Differences across partner agencies, courts, juvenile justice, behavioral health etc…• Demographic risk factors and “case mix”• Case loads, turnover (staff and leadership), and training• A variety of other policy/practice differences
Grounded in good case practice
model principles
Develop presentation
skills
Understand & demystify data
Master qualitative &
quantitative tools
Recognize challenges
Celebrate good practice
Support positive change
Act as a local resource
Grow as managers &
leaders
Your Role as a Data Leader
Knowing when you’ve got it right
• No more “the data are wrong”• Folks own data, know it, act on it• Practice people know the data, data people know
the practice• Field pulls data, asks for reports, initiates actions
tied to the data• Constantly talking about data in a positive way
After a really busy day, the data manager comments:
“I think I liked it better when no one paid attention to the data”
Telling the Story: Key Child Welfare Indicators
So What’s the Story?Describe the issue with as much detail as possible, variation is key to hypothesis development.
• What’s happening right now for all kids?• Has it always been this way?• Is it true in all places, for all ages, for all
racial/ethnic groups?• Is this indicator correlated with any others?• Does it look the same for all types of cases, or in
places where practice is different?
interdependence between measures…
CounterbalancedIndicators ofSystemPerformance
PermanencyThroughReunification,Adoption, orGuardianship
ShorterLengthsOf Stay
StabilityOf Care
Rate of Referrals/Substantiated Referrals Home-Based
Services vs.Out-of-HomeCare
Maintain Positive AttachmentsTo Family, Friends, andNeighbors
Use of LeastRestrictiveForm of Care
Reentry to Care
the current placement system*(highly simplified)
*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference
CHILD INa bunch of
stuff happens CHILD OUT
the foster care system
Trends in Out of Home Care
Data source: AFCARS
Nationwide, the number of children in out of home care is declining. In NV, both the entry rate (per 1,000 children in the population) and the in-care rate are higher than the national average.
FY07 FY08 FY09 FY10 FY11 FY120
1000
2000
3000
4000
5000
6000
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
NV Trends
# in care on 9/30 <18
# entries in FY
# exits in FY
NV In care rate <18
National In care rate <18
NV Entry rate
National entry rate
Safety – The Absence of Repeat Maltreatment
New Y
ork
Iowa
Mich
igan
Nebra
ska
Flor
ida
Ohio
Califo
rnia
Illino
is
Natio
nal A
vg
Mai
ne
Distric
t of C
olum
bia
New Je
rsey
Nevad
a
Kentu
cky
Loui
siana
Wes
t Virg
inia
Mon
tana
Arizon
a
South
Car
olin
a
Delaw
are
New H
amps
hire
Kansa
s
Puerto
Rico
North
Car
olin
a
Virgin
ia
Verm
ont
82
84
86
88
90
92
94
96
98
100
Of all children who were victims of substantiated or indicated abuse or neglect during the first 6 months of the reporting year, what percent did not experience another incident of substantiated or
indicated abuse or neglect within a 6-month period? (FY10
CHILDREN ENTERING CAREManaging with Data in Child Welfare
Children Entering Care: Nevada
Key Questions: Entries
• What is the entry rate – by age/race?• Are entries increasing/decreasing? for all
groups?• What strategies are in place/planned to
reduce entries (and re-entries) into care?
Possible reasons for county differences in entry rates:
• Service array – preventive and in home • Standard of evidence• Law enforcement removals• Demographic risk factors• A variety of other policy/practice differences
Substantial variation year to year is also common in counties with few removals/small populations
CHILDREN IN CAREPOINT IN TIME
Managing with Data in Child Welfare
Key Questions: Children in Care
• What groups of children are in care NOW• What types of placements?• How long have they been in care?• What is needed to move them to
permanency?
Placement Type (Ages 0-17)
Was
hingt
on
New M
exico
Kansa
s
Distric
t of C
olum
bia
New J
erse
yId
aho
Louis
iana
Illino
is
Califo
rnia
(AFCARS)
Miss
ouri
Puerto
Rico
Florida
New H
amps
hire
Arizon
a
New Y
ork
Delawar
e
Virgini
a
Tenne
ssee
Mas
sach
uset
ts
Kentu
cky
Arkan
sas
Iowa
South
Dak
ota
North
Dak
ota
Wes
t Virg
inia
Rhode
Islan
d
Colora
do0%
5%
10%
15%
20%
25%
30%
35%
40%
Of all the children (age 0-17yrs) in care on the last day of the FY, what percent were placed in a congregate care setting? (Group home, shelter care, or residential facility:
excludes detention, and hospitalization)
Placement Type (ages 0-10)
Kansa
s
Louis
iana
Mich
igan
Was
hingt
on
India
naUta
h
Wisc
onsin
Califo
rnia
Mar
yland
Wes
t Virg
inia
Idah
o
Georg
ia
New J
erse
y
Hawaii
Delawar
e
Colora
do
Wyo
ming
Miss
issipp
i
Nation
al
North
Dak
ota
Mas
sach
uset
ts
Florida
Arizon
a
Mon
tana
Texas
South
Dak
ota
South
Car
olina
0%
2%
4%
6%
8%
10%
12%
Of all the children (age 0-10yrs) in care on the last day of the FY, what percent were placed in a congregate care setting? (Group home, shelter care, or residential facil-
ity: excludes detention, and hospitalization)
OUTCOMES: EXITS AND LENGTH OF STAY
National and State Level
Key Questions: Permanency Outcomes
•What proportion of children entering care will eventually reunify?•How does this differ by age at removal?•What percent of children remain in care after 3 years?•Are there differences by age/race?•Is this trend changing over time?
entry cohort
s
exit cohort
s
pointin time
data
Know which view to use
the view matters…
January 1, 2012 December 31, 2012July 1, 2012
Source: Aron Shlonsky, University of Toronto (formerly at CSSR)
the view matters…
0 5 10 15 20 25 30 35 40
q3(75% exited)
q2(50% exited)
q1(25% exited)
Nevada: Length of Stay in Months 2011
(children in care 5 days or more)
2011 entriesN=2103
Jan 1, 2011(point-in-time)N=4135
2011 exitsN=3138
Months in Care
insufficient time elapsed to determine this estimate
entries, point in time and exits views…
22
39
23
8 7
0
5
10
15
20
25
30
35
40
45
<1 yr 1-5 yrs 6-11 yrs 12-14 yrs 15-17 yrs
%
Nevada: Age of Children in Foster Care, 2012
Entries
entries, point in time and exits views…
22
39
23
8 78
37
29
1214
0
5
10
15
20
25
30
35
40
45
<1 yr 1-5 yrs 6-11 yrs 12-14 yrs 15-17 yrs
%
Nevada: Age of Children in Foster Care, 2012
Entries
Point in Time
entries, point in time and exits views…
22
39
23
8 78
37
29
1214
5
43
27
1113
0
5
10
15
20
25
30
35
40
45
<1 yr 1-5 yrs 6-11 yrs 12-14 yrs 15-17 yrs
%
Nevada: Age of Children in Foster Care, 2012
Entries
Point in Time
Exits
Exit Cohort View… but what about those that remain in care?
2007 2008 2009 2010 2011 20120%
10%20%30%40%50%60%70%80%90%
100%
Proportion of Exits by Type: Statewide
Death of child
Transfer
Runaway
No response
Live with relatives
Guardianship
Emancipation
Adoption
Reunification
Timely Reunification (entry cohort)
Puerto
Rico
Delawar
e
Verm
ont
Alaska
Arizon
a
Conne
cticu
t
Main
e
Virgini
a
Kansa
s
Florida
Califo
rnia
Nation
al
North
Dak
ota
New Y
ork
South
Dak
ota
Penns
ylvan
ia
New H
amps
hire
Wes
t Virg
inia
Rhode
Islan
d
Louis
iana
Wisc
onsin
Mas
sach
uset
ts
Tenne
ssee
Hawaii
Colora
do
Minn
esot
a
Arkan
sas
0
10
20
30
40
50
60
70
80
Timely Reunification (FY11): Measure C1.3 Of all first entries who remain in care at least 8 days, what % reunify within 12 months?
%
Nationally, there has been almost no improvement in timely reunification
FY05 FY06 FY07 FY08 FY09 FY100%
10%
20%
30%
40%
50%
60%
39%41% 42% 41% 41%
41%
37%39%
37%
41% 42% 44%
Timely Reunification (C1.3 Entry Cohort)
National Median NV
Re-Entry after Reunification
Puerto
Rico
Virgini
a
Main
e
Mich
igan
Delawar
e
Oklaho
ma
Kansa
s
South
Car
olina
Georg
ia
Hawaii
New M
exico
Arkan
sas
Utah
Distric
t of C
olum
bia
Nation
al
Califo
rnia
Kentu
cky
Mon
tana
Wes
t Virg
inia
New Y
ork
Wyo
ming
Florida
South
Dak
ota
Iowa
Wisc
onsin
New H
amps
hire
Penns
ylvan
ia0
5
10
15
20
25
30
Re-Entry (FY11) Measure C1.4 of all the children reunified, what % re-enter care within 12 months?
Permanency for Longer Stayers
FY05 FY06 FY07 FY08 FY09 FY100%
10%
20%
30%
40%
50%
60%
26% 26% 26%28% 29% 31%
31% 33% 32% 33% 34%
40%
Nationally, exits to permanency among children already in care two years or more has been improving (C3.1)
Nat'l Median NV
Permanency for Longer Stayers
Puerto
Rico
Delawar
e
Illino
is
Califo
rnia
Alabam
a
South
Dak
ota
Virgini
a
Mas
sach
uset
ts
Texas
South
Car
olina
Rhode
Islan
d
Verm
ont
Wisc
onsin
Nation
al
Kentu
cky
North
Car
olina
Iowa
Main
e
India
na
Mich
igan
Hawaii
Penns
ylvan
ia
Tenne
ssee
Was
hingt
on
Alaska
Arizon
a
Wes
t Virg
inia
0%
10%
20%
30%
40%
50%
60%
Achieving Permanency for Longer Stayers (FY11) Measure C3.1: Of all children in care at least two years, what % achieve permanency within the
following year?
Connecting Data to Practice: Using the CQI Framework
Continuous Quality Improvement (CQI)…an ongoing process of identifying, describing, and analyzing strengths
and problems and then testing, implementing, learning from, and
revising solutions.
CQI Relies on…
• An organizational culture that is proactive and supports continuous learning.
• A strong foundation – the mission, vision, and values of the agency.
• The active inclusion and participation of staff at all levels of the agency, children, youth, families, and stakeholders throughout the process.
Key Principles
• Use data and information from multiple sources, qualitative and quantitative
• Data have a purpose: Identify trends and anomalies; find areas for improvement; tell stories about what is happening in practice and policy
• CQI must support staff to improve outcomes for families
Key Principles
• If it ain’t “broke”, it can probably still be “fixed”
• CQI goes beyond “compliance” to “quality”• Meaningful and active engagement of staff
at all levels, children, youth, families, and stakeholders
• CQI requires training, preparation, and consistent ongoing support
DEMYSTIFYING THE LOGIC MODEL
CQI Group Exercise 2
Observe• We’ve noted that:• Children are not exiting to permanency
quickly enough
Explain• And we believe it is because:• Case management and case consultation
has not been consistent
Strategy• So we plan to:• Improve training and supervision; ensure
practice is aligned with policy
Outcome
• Which will result in ENVISIONED OUTCOME:
• An increase in children exiting to permanency within three years
Developed by NY OCFS
…if he had one hour to save the world he would spend 55 minutes defining the problem and only 5 minutes finding the solution.
Before jumping right into solving a problem• Step back• Invest time and effort• Improve understanding
Source: http://litemind.com/problem-definition/ (accessed 6/3/11)
We have noted that:We believe it is because:
So we plan to:
Which will result in:
Observe Explain Prescribe Outcome
HYPOTHESIS STATEMENT: A HIGH LEVEL CAUSE AND EFFECT STATEMENT
Needs and Strengths Assessment Activities
Initial andIntermediate
Observe and Explain Strategies Outcome
LOGIC MODEL: DIGGING DEEPER – MORE DETAIL
OutputsKey End
Outcomes
Where are we now? Observe performance on key measures: review trends and patterns Establish priorities by considering: mandates, greatest areas of
need/opportunity for impact etc… Explain/Explore key underlying factors: both internal and external Consider subpopulations: is performance different by age? Race?
Maltreatment type? Define strengths & areas needing improvement
Administrative Data is only one part of the assessment.
We have noted that:We believe it is because:
So we plan to:
Which will result in:
Observe Explain Prescribe Outcome
HYPOTHESIS STATEMENT: A HIGH LEVEL CAUSE AND EFFECT STATEMENT
Needs and Strengths Assessment Activities
Short Term Outcomes
Observe and Explain Strategies Outcome
LOGIC MODEL: DIGGING DEEPER – MORE DETAIL
OutputsLong Term Outcomes
Where do we want to be? What are the ultimate outcomes that we hope to achieve?
Reduce entries into care Improve likelihood and timeliness of a permanent exit Reduce re-entry Improve health, mental health and education indicators
Group Exercise!!
Observe Explain Strategy Outcome
Observe
• We examined the data and noted that:
• How are you doing on key outcomes? Are they going in the right direction? Is this true everywhere, and for all children? What other indicators are related to this outcome?
Explain• And we believe it is because:• Why? Start with brainstorming, then look to
a variety of existing data – where is there variation? What more do you need to know? How will you find out?
Developed by NY OCFS
Table Discussion 1 – Trends in Timely Permanency
Review regional data packets and CQI handout:
Focus on WHAT and WHY• Describe the trends in timely permanency and related
measures• Are these indicators
– Increasing?– Decreasing?– Staying about the same?
• What does this tell us?• What more do we need to know?
MOVING FROM DATA TO ACTION
CQI Group Exercise 2
We have noted that:We believe it is because:
So we plan to:
Which will result in:
Observe Explain Prescribe Outcome
HYPOTHESIS STATEMENT: A HIGH LEVEL CAUSE AND EFFECT STATEMENT
Needs and Strengths Assessment Activities
Short term outcomes
Observe and Explain Strategies Outcome
LOGIC MODEL: DIGGING DEEPER – MORE DETAIL
OutputsLong Term Outcomes
What will we do to address the issue? Strategies should align with the strengths and needs. What activities are supporting good performance? What are the barriers?
Consider Strategies: What do you control? Where do you need to partner or advocate? Training Programs/Services Policies/practices
We have noted that:We believe it is because:
So we plan to:
Which will result in:
Observe Explain Prescribe Outcome
HYPOTHESIS STATEMENT: A HIGH LEVEL CAUSE AND EFFECT STATEMENT
Needs and Strengths Assessment Activities
Short Term Outcomes
Observe and Explain Strategies Outcome
LOGIC MODEL: DIGGING DEEPER – MORE DETAIL
OutputsLong Term Outcomes
How do we know that the strategy was implemented as planned? What are our timeframes?Examples of outputs: Often a count (and percent) # of people trained # of clients served # of referrals # of meetings held
We have noted that:We believe it is because:
So we plan to:
Which will result in:
Observe Explain Prescribe Outcome
HYPOTHESIS STATEMENT: A HIGH LEVEL CAUSE AND EFFECT STATEMENT
Needs and Strengths Assessment Activities
Short Term Outcomes
Observe and Explain Strategies Outcome
LOGIC MODEL: DIGGING DEEPER – MORE DETAIL
OutputsLong Term Outcomes
How will we know we are heading in the right direction?Short Term outcomes can be expected to change quickly.Examples of measurable improvements: Improve diligent search and engagement Reduce time to adjudication and disposition Increase timely permanency hearings Improved family engagement in case planning