Upload
stacey
View
47
Download
0
Embed Size (px)
DESCRIPTION
CESA 6 Improving the Quality of Child Outcomes Data. Ruth Chvojicek – Statewide Part B Indicator 7 Child Outcomes Coordinator. Objectives. To discuss the significance of and strategies for improving child outcomes data quality - PowerPoint PPT Presentation
Citation preview
CESA 6IMPROVING THE QUALITY
OFCHILD OUTCOMES DATA
Ruth Chvojicek – Statewide Part B Indicator 7
Child Outcomes Coordinator
OBJECTIVES
• To discuss the significance of and strategies for improving child outcomes data quality
• To look at state, CESA and District data patterns as one mechanism for checking data quality
2
QUALITY ASSURANCE: LOOKING FOR QUALITY DATA
I know it is in here somewhere
ONGOING CHECKS FOR DATA QUALITY
Before Good training and assessmentEfficient data systemsTimely and accurate data entry
During
Ongoing supervision of implementationFeedback to implementersRefresher training
AfterReview sample of completed COSFsPattern checking analysis
4
PROMOTING QUALITY DATA - • Through data systems and verification, such as
• Monthly data system error checks e.g. missing & inaccurate data
• Monthly email data reminders• Indicator Training data reports• Good data entry procedures
5
LOOKING AT DATA
6
USING DATA FOR PROGRAM IMPROVEMENT = EIA
EvidenceInferenceAction
7
EVIDENCE Evidence refers to
the numbers, such as“45% of
children in category b”
The numbers are not debatable
8
INFERENCE How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News
we can’t interpret? To reach an inference, sometimes we analyze data in
other ways (ask for more evidence) “Drill Down”
9
INFERENCE Inference is debatable -- even
reasonable people can reach different conclusions
Stakeholders (district personal) can help with putting meaning on the numbers
Early on, the inference may be more a question of the quality of the data
10
ACTION• Given the inference from the numbers,
what should be done?• Recommendations or action steps• Action can be debatable – and often is• Another role for stakeholders• Again, early on the action might have
to do with improving the quality of the data
11
PROMOTING QUALITY DATATHROUGH DATA ANALYSIS
12
THE THREE OUTCOMES
Positive Social Emotional Skills
Acquisition & Use of Knowledge & Skills
Takes Appropriate Action to Meet Needs
Percent of preschool children with IEPs who demonstrate improved:
7-POINT RATING SCALEPLEASE REFER TO HANDOUT – “THE BUCKET LIST”
1• Child does not yet
show functioning expected of a child his or her age in any situation.
• Child’s functioning does not yet include immediate foundational skills upon which to build age-appropriate functioning.
• Child functioning reflects skills that developmentally come before immediate foundational skills.
• Child’s functioning might be described as like that of a much younger child.
2• Child occasionally
uses immediate foundational skills across settings and situation. More functioning reflects skills that are not immediate foundational than are immediate foundational.
3• Child does not yet
show functioning expected for a child of his or her age in any situation.
• Child uses immediate foundational skills, most or all of the time, skills upon which to build age-appropriate functioning.
• Functioning might be described as like that of a younger child.
4• Child shows
occasional age-appropriate functioning across settings and situations. More functioning is not age-appropriate than age-appropriate.
5• Child shows
functioning expected for his or her age some of the time and/or in some settings and situations. Child’s functioning is a mix of age-appropriate and not age-appropriate behaviors and skills.
• Childs functioning might be described as like that of a slightly younger child.
6• Child’s functioning
generally is considered appropriate for his or her age but there are some significant concerns about the child’s functioning in this outcome area. These concerns are substantial enough to suggest monitoring or possible additional support.
• Although age-appropriate, the child’s functioning may border on not keeping pace with age expectations.
7• Child shows
functioning expected for his or her age in all or almost all everyday situations that are part of the child’s life. Functioning is considered appropriate for his or her age.
• No one has any concerns about the child’s functioning in this outcome area.
PATTERN CHECKING - CHECKING TO SEE IF RATINGS ACCURATELY REFLECT CHILD STATUS• We have expectations about how child outcomes data
should look• Compared to what we expect• Compared to other data in the state• Compared to similar states/regions/school districts
• When the data are different than expected ask follow up questions
15
16
QUESTIONS TO ASK Do the data make sense?
Am I surprised? Do I believe the data? Believe some of the data? All of the data?
If the data are reasonable (or when they become reasonable), what might they tell us?
PATTERNS WE WILL BE CHECKING TODAY
1. Entry Rating Distribution
2. Entry Rating Distribution by Eligibility Determination
3. Comparison of Entry Ratings Across Outcomes
4. Entry/Exit Comparison by CESA
5. State Entry Rating Distribution by Race/Ethnicity
6. State Exit Rating Distribution
7. Progress Categories by State/CESA
8. Summary Statements by State/CESA
17
SMALL GROUP DISCUSSION QUESTIONS:
18
1. What do you notice about your local data?
2. What stands out as a possible ‘red flag’?
3. What might you infer about the data?
4. What additional questions does it raise?
5. What next steps might you take?
PREDICTED PATTERN #1
Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle).
Rationale:
Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range).
19
1 2 3 4 5 6 70.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
State 11-12 Entry Rating Distribution
21
PREDICTED PATTERN #2
Groups of children with more severe disabilities should have lower entry numbers than groups of children with less severe disabilities.
STATE 11-12 ENTRY RATING ELIGIBILITY PERCENTAGES
Autism 3.7%Cognitive Disability 0.9%Deaf-Blind 0.0%Emotional Behavioral Disability 0.9%Hearing Impairment 0.8%Orthopedic Impairment 0.9%Other Health Impairment 3.4%Significant Developmental Delay 21.2%Specific Learning Disability 0.2%Speech or Language Impairment 67.6%Traumatic Brain Injury 0.1%Visual Impairment 0.3%
Outcome 1 1 2 3 4 5 6 7Autism 20.4% 38.9% 14.3% 17.9% 7.1% 1.4% 0.0%Cognitive Disability 24.6% 33.3% 24.6% 10.1% 5.8% 1.4% 0.0%Deaf-Blind 0.0% 0.0% 50.0% 50.0% 0.0% 0.0% 0.0%Emotional Behavioral Disability 4.5% 21.2% 12.1% 39.4% 18.2% 4.5% 0.0%Hearing Impairment 3.4% 8.5% 5.1% 5.1% 40.7% 18.6% 18.6%Orthopedic Impairment 13.8% 10.8% 6.2% 6.2% 24.6% 23.1% 15.4%Other Health Impairment 17.3% 21.7% 14.2% 23.6% 15.4% 6.3% 1.6%Significant Developmental Delay 7.3% 24.5% 18.3% 24.7% 17.5% 5.6% 2.0%Specific Learning Disability 6.3% 0.0% 12.5% 12.5% 25.0% 31.3% 12.5%Speech or Language Impairment 0.4% 4.0% 5.8% 9.6% 23.8% 27.9% 28.6%Traumatic Brain Injury 36.4% 36.4% 9.1% 9.1% 9.1% 0.0% 0.0%Visual Impairment 0.0% 19.0% 14.3% 19.0% 4.8% 33.3% 9.5%
Outcome 2 1 2 3 4 5 6 7Autism 16.1% 32.5% 13.9% 20.7% 9.6% 6.4% 0.7%Cognitive Disability 37.7% 42.0% 14.5% 2.9% 2.9% 0.0% 0.0%Deaf-Blind 0.0% 0.0% 50.0% 50.0% 0.0% 0.0% 0.0%Emotional Behavioral Disability 0.0% 4.5% 4.5% 12.1% 19.7% 39.4% 19.7%Hearing Impairment 5.1% 13.6% 16.9% 18.6% 27.1% 13.6% 5.1%Orthopedic Impairment 16.9% 12.3% 9.2% 6.2% 21.5% 23.1% 10.8%Other Health Impairment 19.7% 20.5% 17.3% 10.6% 18.1% 8.3% 5.5%Significant Developmental Delay 8.6% 26.7% 23.6% 21.2% 14.2% 4.6% 1.1%Specific Learning Disability 6.3% 6.3% 25.0% 31.3% 12.5% 12.5% 6.3%Speech or Language Impairment 1.4% 8.7% 14.4% 20.2% 34.1% 17.6% 3.5%Traumatic Brain Injury 45.5% 36.4% 0.0% 9.1% 9.1% 0.0% 0.0%Visual Impairment 14.3% 23.8% 9.5% 0.0% 28.6% 23.8% 0.0%
Outcome 3 1 2 3 4 5 6 7Autism 11.8% 29.3% 18.9% 19.3% 12.5% 5.4% 2.9%Cognitive Disability 27.5% 49.3% 13.0% 4.3% 4.3% 1.4% 0.0%Deaf-Blind 0.0% 0.0% 0.0% 50.0% 50.0% 0.0% 0.0%Emotional Behavioral Disability 1.5% 4.5% 7.6% 18.2% 37.9% 15.2% 15.2%Hearing Impairment 5.1% 6.8% 6.8% 10.2% 16.9% 25.4% 28.8%Orthopedic Impairment 21.5% 16.9% 20.0% 16.9% 16.9% 7.7% 0.0%Other Health Impairment 20.5% 15.4% 17.7% 16.5% 18.5% 9.1% 2.4%Significant Developmental Delay 6.1% 16.6% 17.4% 23.2% 21.4% 12.0% 3.3%Specific Learning Disability 6.3% 6.3% 6.3% 0.0% 25.0% 37.5% 18.8%Speech or Language Impairment 0.5% 2.4% 4.4% 7.1% 16.1% 26.0% 43.5%Traumatic Brain Injury 45.5% 9.1% 18.2% 27.3% 0.0% 0.0% 0.0%Visual Impairment 4.8% 14.3% 14.3% 14.3% 19.0% 33.3% 0.0%
29
PREDICTED PATTERN #3
Functioning at entry in one outcome is related to functioning at entry in the other outcomes.
For cross tabulations we should expect most cases to be in the diagonal and the other to be clustered on either side of the diagonal.
1 2 3 4 5 6 71 13 62 4 47 17 6 2 3 13 20 33 13 2 2 14 9 21 51 25 145 1 9 27 52 70 13 16 5 13 22 91 48 27 1 7 18 84 123 10
Outco
me 1
Outcome 2
1 2 3 4 5 6 71 12 4 1 12 4 31 26 15 9 9 33 10 34 20 31 15 84 5 42 49 41 255 3 5 61 84 1216 1 2 22 43 1357 3 12
Outcome 3Out
come 2
1 2 3 4 5 6 71 13 4 1 12 2 30 21 14 9 3 13 1 6 29 19 10 5 14 3 10 32 48 22 55 1 6 13 77 57 196 1 1 5 22 71 817 2 1 6 37 197
Out
come 1
Outcome 3
PREDICTED PATTERN #4
Large changes in status relative to same age peers between entry and exit from the program are possible but rare.
When looking at the Entry/Exit Rating comparison for individual children we would expect very few children to increase more than 3 points.
33
Outcome 1 Exit
Outcome 1 Entry
1 2 3 4 5 6 7
1 1 2 1 1 1 1
2 4 3 6 6 2 3
3 1 1 1 4 6 2 3
4 4 6 5
5 1 2 4 11 20
6 1 4 18
7 2 19
Outcome 2 Exit
Outcome 2 Entry
1 2 3 4 5 6 7
1 2 1 1 1
2 1 3 6 4 12 2 2
3 1 4 5 9 10
4 1 6 10 4
5 10 10 15
6 1 9 15
7 1
Outcome 3 Exit
Outcome 3 Entry
1 2 3 4 5 6 7
1 3 3 1 1 1 1
2 3 2 4 1 2
3 1 2 3 5 6
4 2 3 9 6
5 1 5 7 13
6 1 1 2 21
7 4 32
PREDICTED PATTERN #5
If children across race/ethnicity categories are expected to achieve similar outcomes, there should be no difference in distributions across race/ethnicity.
Note:
Wisconsin began gathering race/ethnicity data for Indicator 7 on July 1, 2011. This impacts the data on the graphs being reviewed today.
37
Outcome 1 1 2 3 4 5 6 7Asian 4.3% 18.4% 9.9% 9.2% 23.4% 15.6% 19.1%Black 6.4% 15.3% 11.3% 16.7% 20.1% 15.2% 15.2%Hispanic 4.9% 11.9% 12.1% 13.5% 21.5% 21.7% 14.5%American Indian Alaskan 2.7% 10.0% 15.5% 19.1% 24.5% 17.3% 10.9%Hawaiian Other Pacific Islander 0.0% 31.6% 21.1% 10.5% 31.6% 0.0% 5.3%Two or More Races 4.5% 7.2% 11.7% 18.9% 26.1% 18.9% 12.6%White 2.8% 9.5% 8.1% 13.2% 21.4% 22.2% 22.8%
Outcome 2 1 2 3 4 5 6 7Asian 5.7% 14.9% 24.1% 20.6% 22.0% 12.8% 0.0%Black 7.1% 20.0% 23.6% 19.7% 20.4% 7.6% 1.6%Hispanic 7.6% 19.3% 22.8% 20.7% 19.8% 8.0% 1.7%American Indian Alaskan 2.7% 21.8% 17.3% 27.3% 21.8% 9.1% 0.0%Hawaiian Other Pacific Islander 10.5% 15.8% 15.8% 42.1% 10.5% 0.0% 5.3%Two or More Races 4.5% 12.6% 12.6% 23.4% 30.6% 10.8% 5.4%White 3.6% 11.8% 13.3% 19.1% 31.2% 17.1% 3.9%
Outcome 3 1 2 3 4 5 6 7Asian 4.3% 7.8% 5.7% 11.3% 15.6% 22.0% 33.3%Black 6.0% 10.5% 11.5% 13.5% 18.8% 20.3% 19.5%Hispanic 3.6% 9.9% 8.6% 13.0% 16.8% 25.6% 22.5%American Indian Alaskan 2.7% 5.5% 13.6% 15.5% 22.7% 20.0% 20.0%Hawaiian Other Pacific Islander 15.8% 5.3% 21.1% 15.8% 15.8% 5.3% 21.1%Two or More Races 2.7% 4.5% 7.2% 13.5% 20.7% 22.5% 28.8%White 2.7% 6.6% 7.8% 10.6% 17.0% 20.5% 34.9%
41
PREDICTED PATTERN #6
Children will differ from one another in their exit scores in reasonable ways. (At exit there will be a few children with very high or very low numbers.
42
OSEP PROGRESS CATEGORIES
43
PROGRESS CATEGORIESPLEASE REFER TO HANDOUT “CHILD OUTCOMES DATA CONVERSION”
Percentage of children who:
a. Did not improve functioningb. Improved functioning, but not sufficient to move
nearer to functioning comparable to same-aged peers
c. Improved functioning to a level nearer to same-aged peers but did not reach it
d. Improved functioning to reach a level comparable to same-aged peers
e. Maintained functioning at a level comparable to same-aged peers
44
PREDICTED PATTERN #7
Children will differ from one another in their OSEP progress categories in reasonable ways.
Note – A graph of this predicted pattern should have a similar distribution expected in entry & exit ratings (bell curve).
46
A B C D E0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
2010-2011 Outcome 1 Progress Category Percentages
Axis Title
A B C D E0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
2010-2011 Outcome 2Progress Category Percentages
Axis Title
A B C D E0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
2010-2011 Outcome 3Progress Category Percentages
Axis Title
SUMMARY STATEMENT #1 PLEASE SEE HANDOUT – “SUMMARY STATEMENTS FOR TARGET SETTING
1. Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.
c + d___ a + b + c + d
51
SUMMARY STATEMENT #2
2. The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.
d + e__ a + b + c + d + e
52
SS1: Greater than Expected Growth SS2: Exited Within Age Expectations0
10
20
30
40
50
60
70
80
90
100
2010-2011 Outcome 1 Social Emotional
Axis Title
SS1: Greater than Expected Growth SS2: Exited Within Age Expectations0
10
20
30
40
50
60
70
80
90
100
2010-2011 Outcome 2 Knowledge & Skills
Axis Title
SS1: Greater than Expected Growth SS2: Exited Within Age Expectations0
10
20
30
40
50
60
70
80
90
100
2010-2011 Outcome 3 Action to Meet Needs
Axis Title
Wrap Up &
Next Steps
58
TAKE HOME MESSAGE
If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.
What do you if the data are not as good as they should be?
Answer: Continue to improve data collection through ongoing quality assurance
SO
60
KEY TO GOOD DATA
Have a good outcome
measurement
QUALITY CHECKS DURING DATA COLLECTION
During
Ongoing supervision of implementationFeedback to implementersRefresher training
61
QUALITY REVIEW OF COSF TEAM DISCUSSION:• Team discussion• Parent input• Documentation of rating
discussion• Multiple assessment
sources• Child’s everyday
functioning – Not test scores
• Child’s full range of functioning (AE/IF/F)
Ongoing Supervision:• Feedback to teams
critical• Refresher training• Beware of “auto pilot”
and “drift”
Methods:• Observation• Videos
62
QUALITY REVIEW OF COMPLETED COSFS
• Complete?• Adequate evidence?• Match the outcome area?• Based on functional behaviors?• Across settings and situations?• Ratings consistent with evidence?
63
NEXT STEPS?
1. What steps might you take after today’s discussion?
2. What other data patterns would be interesting / helpful to look at and interpret?
3. What types of TA/support/follow up might be appropriate next steps?
64
Thank You for Participating Today!
For more information:www.collaboratingpartners.com
http://sped.dpi.wi.gov/sped_spp-preout
http://projects.fpg.unc.edu/~eco/index.cfm