Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Using the Ohio Scales for Assessment and Outcome Measurement in a
Statewide System of Care
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Oklahoma Systems of Care (OSOC) Expansion
Implications for Evaluation
From 2002-2011, using a combination of state funds and funds from Phase IV and Phase VI SAMHSA grants, Oklahoma’s wraparound program expanded from an initial 6 counties to 56.
This rapid expansion necessitated a state evaluation separate from the national evaluation and longitudinal study.
Not all sites would be included in the national evaluation’s longitudinal study.
Data from the national evaluation would cover a small percentage of youths in OSOC wraparound.
Quality improvement and decision support required 100% data collection from enrolled families.
A scalable solution would incorporate line staff as data collectors, so burden – in time and training – had to be limited.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Lincoln
Cimarron Texas Beaver Harper
Ellis
Woods
Woodward
Major
Alfa
lfa
Dewey
Grant
Garfield
Kingfisher
Logan
Roger Mills
Beckham
Greer Kiowa
Custer
Washita
Blaine
Caddo
Canadian
Grady
Har
mon
Jackson
Tillman Cotton
Comanche
Stephens
Jefferson
Cle
vela
nd
McClain
Oklahoma
Potta
wat
omie
Garvin
Carter
Love
Murray
Marshall
Johnston
Bryan
Pontotoc
Sem
inol
e
Choctaw
McCurtain
Atoka
Pushmataha
Hughes
Coal
Pittsburg Leflore Latimer
Haskell
McIntosh
Muskogee
Okm
ulge
e
Creek
Okfuskee Sequoyah
Wagoner
Che
roke
e
Adair
Kay
Noble
Payne
Pawnee
Osage Was
hing
ton
Now
ata
Rogers
Craig Ottawa
Mayes Del
awar
e
Tuls
a
OSOC Sites 2000-2011
2011
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2000
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
OSOC 2001-2011 Families Served, Sites and Counties by Fiscal Year
38
156
304
483
648 762
852
1088
1246
1405
1667
3 6 7 9
13
20
26
31 34
40
50
3 7
10
13 20
31
44
49
56
53
56
0
10
20
30
40
50
60
70
0
200
400
600
800
1000
1200
1400
1600
1800
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
# of
Sit
es a
nd C
ount
ies
Fam
ilies
Ser
ved
Fiscal Year
Clients Served Sites in Operation Counties Served
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Development of State Evaluation
Psychometric tool originally in use would not scale.
Costs and logistics related to credentialing requirements were prohibitive.
Evaluation committee of evaluators, staff, families and youth developed the state evaluation tool set – based on extracts from longitudinal study public domain instruments – in spring and summer of 2004.
2003/2004 national conferences exposed evaluation team members to the Ohio Scales, in use by other Systems of Care communities.
State evaluation committee chose to build OSOC Assessments around the Ohio Scales in July 2004.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Why the Ohio Scales?
Originally designed as a statewide instrument
Qualities appropriate to statewide implementations Self-administration Parent—Administered at baseline
with follow up at 6 month intervals
Youth—Administered to youth aged 12 and older at baseline with follow up at 6 month intervals
Worker—Administered at baseline with follow up at 3 months and then 6 month intervals from baseline
No credentialing requirement
Nominal cost
Brevity
Simple language and item content
Easily understood scoring, interpretation of results
Good and growing research base
Articles on flash drive.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Instructions:
Please rate the degree to which your child has experienced the following problems in the past 30 days N
ot a
t All
Onc
e or
Tw
ice
Seve
ral
Tim
es
Ofte
n
Mos
t of t
he
Tim
e Al
l of t
he
Tim
e
1. Arguing with others 0 1 2 3 4 5 2. Getting into fights 0 1 2 3 4 5 3. Yelling, swearing, or screaming at others 0 1 2 3 4 5 4. Fits of anger 0 1 2 3 4 5 5. Refusing to do things teachers or parents ask 0 1 2 3 4 5 6. Causing trouble for no reason 0 1 2 3 4 5 7. Using drugs or alcohol 0 1 2 3 4 5 8. Breaking rules or breaking the law (out past curfew, stealing) 0 1 2 3 4 5 9. Skipping school or classes 0 1 2 3 4 5
10. Lying 0 1 2 3 4 5 11. Can’t seem to sit still, having too much energy 0 1 2 3 4 5 12. Hurting self (cutting or scratching self, taking pills) 0 1 2 3 4 5 13. Talking or thinking about death 0 1 2 3 4 5 14. Feeling worthless or useless 0 1 2 3 4 5 15. Feeling lonely and having no friends 0 1 2 3 4 5 16. Feeling anxious or fearful 0 1 2 3 4 5 17. Worrying that something bad is going to happen 0 1 2 3 4 5 18. Feeling sad or depressed 0 1 2 3 4 5 19. Nightmares 0 1 2 3 4 5 20. Eating problems 0 1 2 3 4 5
Ohio Scales: Youth Problem Scale Youth Problem Scale (Copyright © January 2000, Benjamin M. Ogles & Southern Consortium for Children)
Produces a score of 0 - 100.
Externalizing Subscale
Internalizing Subscale
Delinquency Subscale
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Ohio Scales: Youth Functioning Scale Youth Functioning Scale (Copyright © January 2000, Benjamin M. Ogles & Southern Consortium for Children)
Instructions:
Please rate the degree to which your child’s problems affect his or her current ability in everyday activities. Consider your child’s current level of functioning. Ex
trem
e Tr
oubl
es
Qui
te a
Few
Tr
oubl
es
Som
e Tr
oubl
es
OK
Doi
ng V
ery
Wel
l
1. Getting along with friends 0 1 2 3 4 2. Getting along with family 0 1 2 3 4 3. Dating or developing relationships with boyfriends or girlfriends 0 1 2 3 4 4. Getting along with adults outside the family (teachers, principal) 0 1 2 3 4 5. Keeping neat and clean, looking good 0 1 2 3 4
6. Caring for health needs and keeping good health habits (taking medicines or brushing teeth) 0 1 2 3 4
7. Controlling emotions and staying out of trouble 0 1 2 3 4 8. Being motivated and finishing projects 0 1 2 3 4 9. Participating in hobbies (baseball cards, coins, stamps, art) 0 1 2 3 4
10. Participating in recreational activities (sports, swimming, bike riding) 0 1 2 3 4 11. Completing household chores (cleaning room, other chores) 0 1 2 3 4 12. Attending school and getting passing grades in school 0 1 2 3 4 13. Learning skills that will be useful for future jobs 0 1 2 3 4 14. Feeling good about self 0 1 2 3 4 15. Thinking clearly and making good decisions 0 1 2 3 4 16. Concentrating, paying attention, and completing tasks 0 1 2 3 4 17. Earning money and learning how to use money wisely 0 1 2 3 4 18. Doing things without supervision or restrictions 0 1 2 3 4 19. Accepting responsibility for actions 0 1 2 3 4 20. Ability to express feelings 0 1 2 3 4
Produces a score of 0 - 80.
Resiliency Subscale
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Ohio Scales Guidelines from the Texas Study
2003 study by the Texas Department of Mental Health and Mental Retardation used comparisons with CBCL and SDQ to validate the scales and to establish classification criteria for the 20-item Ohio Problems and Functioning scales:
Problems Scale
Score of 25 and above = critical impairment
Score of 17 – 24 = borderline impairment
Decrease of 11 or more points = clinically significant improvement
Functioning Scale
Score of 44 and below = critical impairment
Score of 45 – 52 = borderline impairment
Increase of 8 or more points = clinically significant improvement
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Significant Improvement in Ohio Scales Problems and Functioning Scores
Benchmark Value = 70-80% of 'Impaired' Youths Should Show Significant Improvement at 6 Months
(-11 points for Problems, +8 for Functioning)
Site
# Youths with Baseline and
6-Month Assessments
# Youths with Impaired
Baseline Scale Scores
% Youth with Baseline
Impairment
# Showing Significant
Improvement
% of Impaired Showing
Significant Improvement
County 1 13 9 69% 7 78% County 2 47 36 77% 26 72% County 3 38 30 79% 20 67% County 4 11 9 82% 6 67% County 5 12 9 75% 6 67% County 6 16 14 88% 9 64% County 7 25 21 84% 13 62% County 8 73 62 85% 35 56% County 9 10 9 90% 5 56% County 10 32 28 88% 15 54% County 11 11 10 91% 5 50% County 12 16 12 75% 6 50%
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Not Impaired, 154, 22%
Not Improved, 179, 32%
Improved, 381, 68%
Impaired, 554, 79%
FY 2012 Ohio Scales Clinically Significant Improvement
by Impairment at Baseline
6 Months Baseline
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
OSOC Youth Data 2007 – 2011 Referrals to Wraparound from Primary Sources
Ohio Scales Problems – Baseline
56% 51%
73% 71%
68% 71%
18% 14% 16% 17% 18%
12%
26%
34%
11% 12% 14%
17%
0%
20%
40%
60%
80%
Source of Referral (# referrals)
Impaired Borderline Normal
Children and youth referred by Child Welfare and Juvenile Justice are more likely to score in the normal range than those referred by other sources.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
47% 49%
61% 62% 61%
50%
24%
19% 23% 23% 22%
28% 30% 33%
17% 16% 17%
22%
0%
20%
40%
60%
80%
Source of Referral (# referrals)
Impaired Borderline Normal
OSOC Youth Data 2007 – 2011 Referrals to Wraparound from Primary Sources
Ohio Scales Functioning - Baseline
Children and youth referred by Child Welfare and Juvenile Justice are more likely to score not impaired than those referred by other sources.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
OSOC Youth Data 2007 – 2011 Total Enrollments by Impairment at Baseline
Normal, 408, 11%
Borderline, 516, 15%
Impaired - Function Only,
284, 8%
Impaired - Problems Only,
618, 17%
Impaired - Both Function and
Problems, 1761, 49%
Only 11% of enrolled wraparound youths score in the normal range on both Problems and Functioning scales.
Just under half of enrolled youths are critically impaired on both scales.
These data provide evidence that our referral networks are functioning well and that OSOC is providing services to appropriate families.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
OSOC Youth Data 2007 – 2011 6-Month Improvement in Problems / Functioning
by Gender, Caregiver Form
59%
53% 56%
52%
0%
10%
20%
30%
40%
50%
60%
70%
Female (P=407, F=333) Male (P=943, F=800)
% o
f Im
pair
ed Y
outh
s w
/Sig
nifi
cant
Impr
ovem
ent
Gender (P = Problems count; F = Functioning count)
Improved, Problems Improved, Functioning
Females who are rated impaired at baseline show clinically significant improvement at a higher rate at six months than males on both scales.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
37% 39%
37%
63% 61%
63%
0%
10%
20%
30%
40%
50%
60%
70%
6 - 11 12 - 14 15 - 18
Age Group
No improvement Improved
OSOC Youth Data 2007 – 2011 Significant Improvement, Baseline to 6 Months
by Age – Caregiver Form
229
394 279
179
194
112
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Problems Scale Caregiver vs. Youth Responses, by Item (N = 1983)
2.9
1.3
2.4
2.3
2.6
1.6
0.5
1.2
0.6
2.2
2.2
0.6 0.
7
1.4
1.6
1.6
1.5
2.0
0.8 1.
0
2.5
1.2
2.1
1.9
1.9
1.1
0.5
0.9
0.6
1.8
2.3
0.5 0.
6
1.2
1.1 1.
3
1.7
1.7
1.0
1.0
0.0
1.0
2.0
3.0
Resp
onse
Val
ue (
0-5
)
Problems Scale Item
Caregiver YouthOnly five items were rated more or equally problematic by youths as by their caregivers.
Scale values: 0 = Not at All 1 = Once or Twice 2 = Several Times 3 = Often 4 = Most of the Time 5 = All of the Time
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Functioning Scale Caregiver vs. Youth Responses, by Item (N = 1766)
2.5
1.9
2.3
2.4 2.
6
2.5
1.7 1.8
2.2 2.
3
1.7
2.0 2.
1
2.0
1.6
1.7 1.
8
1.9
1.5
1.8
3.2
2.5
2.7
2.7
3.1
3.1
2.3
2.6 2.
7
2.8
2.4 2.5
2.8
2.8
2.5
2.3 2.4 2.
7
2.6
2.5
0
1
2
3
4
Resp
onse
Val
ue (
0-4)
Functioning Scale Item
Caregiver YouthScale values:
0 = Extreme Troubles 1 = Quite a few Troubles 2 = Some Troubles 3 = OK 4 = Doing Very Well
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
OSOC Youth Data 2007 – 2011 High Externalizing/High Internalizing and Improved
95% 88% 88%
84%
70% 72% 70%
45%
26% 29% 28%
37%
44% 45% 61%
65% 68% 61%
65%
68% 58%
67% 72% 73%
86%
59%
74%
71%
0%
20%
40%
60%
80%
100%
High Externalizing High Internalizing Improved Externalizing Improved Internalizing
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
13%
39%
20%
87%
61%
80%
0%
20%
40%
60%
80%
100%
6 to 11 (83) 12 to 14 (69) 15 to 18 (50)
Significant Improvement
OSOC Youth Data 2007 – 2011 Impaired Youths with More than 6 School Days Missed at Baseline
26%
41% 38%
74%
59% 63%
0%
20%
40%
60%
80%
100%
6 to 11 (50) 12 to 14 (49) 15 to 18 (24)
Age Group (# Youths)
No Clinically Significant Improvement Baseline to 6 Months
Over 6 School Days Missed Under 7 School Days Missed
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
OSOC WFI 2008 Pilot Study Ohio Scales Change by Caregiver WFI Score
92.4 90.1 88.7 88.3 83.8
79.5
73.1
50.7 11
10
13
8
2
8
3
6 7
15
13
8
6
4
5 4
0
4
8
12
16
20
0
20
40
60
80
100
Poin
t Ch
ange
in S
cale
(A
bs V
alue
)
WFI
Sco
re
Caregiver WFI Avg. Problem Change Avg. Functioning Change
o In 2008-09, OSOC piloted the WFI 4.0 with 4 sites and 8 wraparound facilitators. The results were compared to the improvements in the Ohio Scales.
o Generally, as WFI ratings fell, so too did Ohio Scales improvement.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Future Plans for the Ohio Scales
Add borderline cases to the outcomes and appropriateness analyses.
Explore the use of the Ohio Scales as a pre-referral team screener, to provide additional information for the team that staffs each referral.
In response to research, recommend that the age range for the Youth Form of the Ohio Scales be extended down to 9-year-olds.
Add usability features to the on-line data system to improve the ability of families and staff to use and interpret Ohio Scales results.
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
Resources Ash, S., & Weis, R. (2009). Recovery Among Youths Referred to Outpatient Psychotherapy: Reliable Change, Clinical
Significance, and Predictors of Outcome. Child and Adolescent Social Work Journal 26, 399-413.
Davis, T., Gavazzi, S., Scheer, S., Kwon, I., Lammers, A., Fristad, M., & Uppal, R. (2011): Measuring Family Caregiver Perceptions of Support in Caring for Children and Youth With Mental Health Concerns, Journal of Social Service Research, 37:5, 500-515.
Ogles, B. M., Carlson, B. Hatfield, D. R., & Karpenkov, V. V. (2008). Models of case mix adjustment for Ohio mental health consumer outcomes among children and adolescents. Administration and Policy in Mental Health and Mental Health Services Research, 35, 295-304.
Dowell, K. A. & Ogles, B. M. (2008). The Ohio Scales Youth Form: Expansion and Validation of a Self-Report Outcome Measure for Young Children. Journal of Child and Family Studies, 17, 291-305.
Turchik, J., Karpenkov, V.V., & Ogles, B. M. (2007). Further Evidence of the Utility and Validity of a Measure of Outcome for Children and Adolescents. Journal of Emotional and Behavioral Disorders, 15, 119-128.
Carlston, D. L. & Ogles, B. M. (2006). The impact of items and anchors on parent-child reports of problem behavior. The Child & Adolescent Social Work Journal, 27, 24-37.
Ogles, B. M., Carlston, D. L., Hatfield, D.R., Melendez, G., Dowell, K., and Fields, S. A. (2006). The role of fidelity and feedback in the wraparound approach. Journal of Child and Family Studies, 15, 114-128.
Ogles, B. M., Dowell, K., Hatfield, D, Melendez, G., & Carlston, D. (2004). The Ohio Scales. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (3rd ed., Vol. 2) (pp. 275-304). Hillsdale, New Jersey: Lawrence Earlbaum.
Ogles, B. M., Melendez, G., Davis, D. C., & Lunnen, K. M. (2001). The Ohio Scales: Practical Outcome Assessment. Journal of Child and Family Studies, 10, 199-212.
Texas Department of Mental Health Mental Retardation. (2003). Validation and norms for the Ohio Scales among children served by the Texas Department of Mental Health and Mental Retardation. http://www.dshs.state.tx.us/mhprograms/RDMCAtrag.shtm
OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION
E-TEAM
The Educational Training, Evaluation, Assessment, and Measurement (E-TEAM) Department of the University of Oklahoma is a full service department with senior researchers, data analysts, technical writers, and data base developers and managers representing decades of experience in all phases of research data processing. E-TEAM also has several years’ experience as evaluation manager for the State of Oklahoma Systems of Care (OSOC) grant community, implementing its national and local evaluations as well as special OSOC collaborative studies with child welfare and juvenile justice. Belinda Biscoe, Ph.D., Assistant Vice President, Public and Community Services, at the University of Oklahoma, College of Continuing Education, is the Director of E-TEAM. E-TEAM has staff with extensive experience in tribal projects in conjunction with the American Indian Institute (also located at the University of Oklahoma). E-TEAM also has bi-lingual staff to facilitate communication with Spanish-speaking customers and subjects.
The Oklahoma Systems of Care E-TEAM evaluation staff:
Melinda Anthony, Family Interviewer
José Diaz, Data Analyst
Kelly Phillips, Research Assistant
Sharon Strait, Research Assistant
Geneva Strech, Senior Research Associate
Stephen Strech, Youth Interviewer
John Vetter, Senior Research Associate
Lisa White, Research Associate