Child Outcomes: Understanding the
Requirements in order to Set Targets
Presentation to the Virginia Interagency Coordination
Council09-09-09
Infant & Toddler Connection of Virginia
1
Topics to be Covered Today
The three child outcomesOSEP reporting requirementsHow the data is gatheredOSEP Progress CategoriesSummary StatementsConsiderations when reviewing baseline data for setting targets
OSEP
Infant & Toddler Connection of Virginia 2
OSEP Reporting Requirements: Child
Outcomes
Positive social emotional skills (including positive social relationships)Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])Use of appropriate behaviors to meet their needs
Infant & Toddler Connection of Virginia 3
OSEP Reporting Categories
Percentage of children who: a. Did not improve functioningb. Improved functioning, but not sufficient to
move nearer to functioning comparable to same-aged peers
c. Improved functioning to a level nearer to same-aged peers but did not reach it
d. Improved functioning to reach a level comparable to same-aged peers
e. Maintained functioning at a level comparable to same-aged peers
Infant & Toddler Connection of Virginia 4
OSEP Reporting Categories
3 indicators X 5 Reporting Categories
15 numbers to Report to OSEP
Infant & Toddler Connection of Virginia 5
Reporting Schedule
2008 and 2009Reporting Schedule
Data in reporting categories at exit for all children who have been in the program for at least 6 months
First submission was due February 2008 for the year July 1, 2006- June 30, 2007.
Infant & Toddler Connection of Virginia 6
Reporting Schedule2010
• in SPP format: – Baseline– Targets for 2 reporting years– Improvement activities for 2 reporting years
2011 and 2012• in APR format:
– Actual data, progress and slippage, etc.– Local reporting of [summary statement %s]
Infant & Toddler Connection of Virginia 7
Understanding the Three Child Outcomes
In Virginia, we refer to the OSEP child outcomes as child indicators to distinguish between the individual child outcomes and the OSEP outcomes.
Infant & Toddler Connection of Virginia 8
The Child Indicator Summary Form (CISF)
7 point rating scaleTeam summarizes multiple data sources (NOT an assessment)Rating the status of child’s functioning at entry and again at exitComparing child’s functioning to what is expected at his/her age
Infant & Toddler Connection of Virginia 9
The two CISF questions
a. To what extent does this child show age-appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7)
b. b. Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No)
Infant & Toddler Connection of Virginia 10
Essential Knowledge for Completing the
CISF
Between them, team members must: Know about the child’s functioning
across settings and situations Understand age-expected child
development Understand the content of the three
child outcomes Know how to use the rating scale Understand age expectations for child
functioning within the child’s culture
Infant & Toddler Connection of Virginia 11
7 Point Rating Scale
See handout
Infant & Toddler Connection of Virginia 12
Infant & Toddler Connection of Virginia 13
Understanding theOSEP Progress
Categories
Helping Children Move Toward
Age-expected Functioning
Assumption: Children can be described with regard to how close they are to age expected functioning for each of the 3 outcomesBy definition, most children in the general population demonstrate the outcome in an age-expected wayBy providing services and supports, EI is trying to move children closer to age expected behavior
Infant & Toddler Connection of Virginia 14
Key PointsThe OSEP categories describe types of progress children can make between entry and exitTwo scores or ratings (entry and exit) are needed to calculate what OSEP category describes a child progressService providers are not responsible for calculating progress (this happens at the state level)
Infant & Toddler Connection of Virginia 15
The “a” category a. Percent of infants and toddlers
who did not improve functioning
Children who acquired no new skills or regressed during their time in the programDidn’t gain or use even one new skillChildren with degenerative conditions/ significant disabilities
Infant & Toddler Connection of Virginia 16
CISF ratings and the “a” category
Rated lower at exit than entry; ORRated 1 at both entry and exit;
AND Scored “No” on the progress question (b)
Infant & Toddler Connection of Virginia 17
Early Childhood Outcomes Center18
Entry Exit
Early Childhood Outcomes Center19
Entry Exit
The “b” categoryb. Percent of infants and toddlers who
improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers
Children who acquired new skills but continued to grow at the same rate throughout their time in the program Gained and used new skills but did not increase their rate of growth or change their growth trajectories while in services
Infant & Toddler Connection of Virginia 20
CISF ratings and the “b” category
Rated 5 or lower at entry; ANDRated the same or lower at exit;
AND
“Yes” on the progress question (b)
Infant & Toddler Connection of Virginia 21
Early Childhood Outcomes Center22
Entry Exit
Early Childhood Outcomes Center23
Entry Exit
Early Childhood Outcomes Center24
Entry Exit
The “c” category
c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it
Children who acquired new skills but accelerated their rate of growth during their time in the programMade progress toward catching up with same aged peers but were still functioning below age expectations when they left the programChanged their growth trajectories --“narrowed the gap”
Infant & Toddler Connection of Virginia 25
CISF ratings and the “c” category
Rated higher at exit than entry;
AND
Rated 5 or below at exit
Infant & Toddler Connection of Virginia 26
Early Childhood Outcomes Center27
Entry Exit
The “d” categoryd. Percent of infants and toddlers who
improved functioning to reach a level comparable to same-aged peers
Children who were functioning below age expectations when they entered the program but were functioning at age expectations when they leftStarted out below age expectations, but caught up while in services
Infant & Toddler Connection of Virginia 28
CISF ratings and the “d” category
Rated 5 or lower at entry;
AND
Rated 6 or 7 at exit
Infant & Toddler Connection of Virginia 29
Early Childhood Outcomes Center30
Entry Exit
The “e” category
e. Percent infants and toddlers who maintained functioning at a level comparable to same-aged peers
Children who were functioning at age expectations when they entered the program and were functioning at age expectations at exit
Infant & Toddler Connection of Virginia 31
CISF ratings and the “e” category
Rated 6 or 7 at entry; AND
Rated 6 or 7 at exit
Infant & Toddler Connection of Virginia 32
Early Childhood Outcomes Center33
Entry Exit
Early Childhood Outcomes Center34
Entry Exit
The Challenge….
Progress data included5 progress categories For each of 3 outcomesTotal of 15 numbers reported each year
Too many interrelated targets to understandOSEP asked for a recommendation
Infant & Toddler Connection of Virginia 35
Development of the Summary Statements
ECO presented options to states and ECO work groups via conference callsTwo sessions at December, 2008 EC Conference Posted on the ECO web site for commentsECO made recommendation to OSEP
Infant & Toddler Connection of Virginia 36
Final DeliberationOSEP put the summary statements out for public commentComments came in that were thoughtful, but not necessarily consistent with one anotherAdvantages and disadvantages to all optionsPaper documenting the process on the ECO websitehttp://www.fpg.unc.edu/~eco/assets/pdfs/summary_of_target_setting-2.pdf
Infant & Toddler Connection of Virginia 37
The Summary Statements
Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.
The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.
Infant & Toddler Connection of Virginia 38
Understanding the Data
Sample data will be used in the next several slides to demonstrate the process for calculating the summary statements from the data in the 5 OSEP categories
Infant & Toddler Connection of Virginia 39
Example of State Progress Data for 2008-2009
Positive social-emotional skills (including social relationships):
Number of
children
% of children
a. Percent of infants and toddlers who did not improve functioning
40 4
b. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers
150 15
c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach
270 27
d. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers
300 30
e. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers
240 24
Total N=1000
100%
Summary Statement Data
Required Summary Statement 1: Of those children who entered the program
below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program= 75%
Required Summary Statement 2: The percent of children who were functioning
within age expectations in each Outcome by the time they exited the program= 54%
Infant & Toddler Connection of Virginia 41
Where do the #s come from?
42
Prog cat
# %
a 40 4b 150 15c 270 27d 300 30e 240 24
760 (a, b, c, and d) or 76% of the children entered the program functioning below age expectations
240 (e) or 24% of the children entered and exited functioning at age expectations
Where do the #s come from?
Measurement for Summary Statement 1: Percent = # of infants and toddlers reported in
progress category (c) plus # of infants and toddlers reported in category (d) divided by [# of infants and toddlers reported in progress category (a) plus # of infants and toddlers reported in progress category (b) plus # of infants and toddlers reported in progress category (c) plus # of infants and toddlers reported in progress category (d)] times 100.
Infant & Toddler Connection of Virginia 43
Where do the #s come from?
44
Prog cat
# %
a 40 4
b 150 15
c 270
27
d 300
30
e 240 24
570 (c and d) of the 760 (a, b, c, and d) changed their growth trajectories (made greater than expected progress)
270 +300= 570 760
= 75%
45Summary Statements Calculator -April 14, 2009
Where do the #s come from?
Measurement for Summary Statement 2: Percent = # of infants and toddlers reported in progress category (d) plus [# of infants and toddlers reported in progress category (e) divided by the total # of infants and toddlers reported in progress categories (a) + (b) + (c) + (d) + (e)] times 100.
Infant & Toddler Connection of Virginia 46
Where do the #s come from?
47
Prog cat
# %
a 40 4
b 150 15
c 270 27
d 300 30e 240 24
300+240= 540 1000
= 54%
30% of the children reached age expectations by exit and 24% of the children entered and exited at age expectations
What can we say about the children’s
progress?Part C Outcome 1: successful social relationships with peers and adults, following rules for social interactions96% of children participating in Part C made progress in their social relationships while they were enrolled.The 4% of children who did not make progress included children with the most severe disabilities and/or degenerative conditions. Can you describe them?
Infant & Toddler Connection of Virginia 48
Positive Social Relationships
24% of the children participating in Part C were functioning at age expectations at entry and at exit in this outcome area. 54% of the children were functioning at age expectations in this outcome area when they exited the program. (summary statement 2)
30% started out behind and caught up24% entered and exited at age expectations
Infant & Toddler Connection of Virginia 49
Positive Social Relationship
75% of the children who entered the program below age expectations made greater than expected gains; they made substantial increases in their rates of growth. i.e. changed their growth trajectories (summary statement 1)
Infant & Toddler Connection of Virginia 50
Setting Targets
Quality Data is essential for setting realistic targetsTwo strategies for examining data
Data qualityPotential for program improvement
Parameters, guidance for target setting
Infant & Toddler Connection of Virginia 51
Can you trust the data?Mechanisms that impact quality of data:
ITOTS mechanisms to prevent inaccurate data entryPersonnel Training
Initial Statewide trainingTraining resources, including online training modules and Implementation ManualsLocal training and oversight
Review of Data – TA Guidance at Regional Meetings
Infant & Toddler Connection of Virginia 52
Can you trust the data?
Look at the data across all systems in order to analyze data quality:
Begin by identifying outliers Examples: look at the percentages reported for certain categories across local programs
Infant & Toddler Connection of Virginia 53
Percentages reported in category “a” across 30
(sample) local programs
54
Look at the Impact of the Outliers on the
Data
State percentage for “e” with all data= 32.1%
Revised percentage for “e” with outliers removed= 27.7%
55
Example of data with outliers removed
Progress Category
Original % Clean %
a 4 2b 15 17c 27 30d 30 31e 24 20
Sum St 1 75 76Sum St 2 54 51
Clean data (without the outliers) may be a more accurate picture of where you are starting
56
Suggested Strategy
Analyze your data with your local system outliers included and excluded so you can gauge the impact they are having on your state level data.
Infant & Toddler Connection of Virginia 57
Important Note about Reporting Data to
OSEP
Consider clean data when deciding about reasonable targets, BUTTurn in the original data to OSEP in the SPP report!You can discuss the clean data in the rationale for your targets.
Infant & Toddler Connection of Virginia 58
Which local systems can be targeted for program improvement?
Compare the summary statement data by local programs to identify which programs have the most potential for improvement.
Infant & Toddler Connection of Virginia 59
Summary Statement Percentages
by Local Program
60
Data Quality Considerations
What do you know about the local systems with the least and the most progress in the summary statements? i.e. the programs with:
the lowest and highest percentages of children at age expectation at exitthe lowest and highest percentage of children making greater than expected gains
Infant & Toddler Connection of Virginia 61
Examples of Key Questions
Are the children similar at entry?Are the higher performing local systems participating in special projects or have they instituted training programs?Are there systems issues in lower performing local systems that would explain differences in outcomes? e.g. personnel shortages
Infant & Toddler Connection of Virginia 62
Bottom Line Question
Could either system or practice focused improvement activities targeted toward the lowest performing local systems improve the child outcomes?
Infant & Toddler Connection of Virginia 63
The Math of Target Setting
How much would the data change if the lowest local programs moved toward the mean?Improvements in the lowest programs will result in improvement in your statewide dataExperiment with your data to determine what targets are reasonable in your state
Infant & Toddler Connection of Virginia 64
Next Steps
VICC Subgroup meeting (1/2 day) to review Virginia’s data in detail and develop recommendations for targets.Presentation of recommendations from the VICC subgroup to the full VICC at the December VICC meeting
Infant & Toddler Connection of Virginia 65
Additional Information
Options and ECO Recommendations for Summary Statements for Target Setting
http://www.fpg.unc.edu/~eco/assets/pdfs/summary_of_target_setting-2.pdf
Summary Statement SlidesTarget Setting Slides
http://www.fpg.unc.edu/~eco/pages/events.cfm
Infant & Toddler Connection of Virginia 66