Child Outcomes: Understanding the Requirements in order to Set Targets Presentation to the Virginia Interagency Coordination Council 09-09-09 Infant &

Child Outcomes: Understanding the

Requirements in order to Set Targets

Presentation to the Virginia Interagency Coordination

Council09-09-09

Infant & Toddler Connection of Virginia

1

Topics to be Covered Today

The three child outcomesOSEP reporting requirementsHow the data is gatheredOSEP Progress CategoriesSummary StatementsConsiderations when reviewing baseline data for setting targets

OSEP

Infant & Toddler Connection of Virginia 2

OSEP Reporting Requirements: Child

Outcomes

Positive social emotional skills (including positive social relationships)Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])Use of appropriate behaviors to meet their needs


OSEP Reporting Categories

Percentage of children who: a. Did not improve functioningb. Improved functioning, but not sufficient to

move nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers


OSEP Reporting Categories

3 indicators X 5 Reporting Categories

15 numbers to Report to OSEP


Reporting Schedule

2008 and 2009Reporting Schedule

Data in reporting categories at exit for all children who have been in the program for at least 6 months

First submission was due February 2008 for the year July 1, 2006- June 30, 2007.


Reporting Schedule2010

• in SPP format: – Baseline– Targets for 2 reporting years– Improvement activities for 2 reporting years

2011 and 2012• in APR format:

– Actual data, progress and slippage, etc.– Local reporting of [summary statement %s]


Understanding the Three Child Outcomes

In Virginia, we refer to the OSEP child outcomes as child indicators to distinguish between the individual child outcomes and the OSEP outcomes.


The Child Indicator Summary Form (CISF)

7 point rating scaleTeam summarizes multiple data sources (NOT an assessment)Rating the status of child’s functioning at entry and again at exitComparing child’s functioning to what is expected at his/her age


The two CISF questions

a. To what extent does this child show age-appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7)

b. b. Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No)


Essential Knowledge for Completing the

CISF

Between them, team members must: Know about the child’s functioning

across settings and situations Understand age-expected child

development Understand the content of the three

child outcomes Know how to use the rating scale Understand age expectations for child

functioning within the child’s culture


7 Point Rating Scale

See handout



Understanding theOSEP Progress

Categories

Helping Children Move Toward

Age-expected Functioning

Assumption: Children can be described with regard to how close they are to age expected functioning for each of the 3 outcomesBy definition, most children in the general population demonstrate the outcome in an age-expected wayBy providing services and supports, EI is trying to move children closer to age expected behavior


Key PointsThe OSEP categories describe types of progress children can make between entry and exitTwo scores or ratings (entry and exit) are needed to calculate what OSEP category describes a child progressService providers are not responsible for calculating progress (this happens at the state level)


The “a” category a. Percent of infants and toddlers

who did not improve functioning

Children who acquired no new skills or regressed during their time in the programDidn’t gain or use even one new skillChildren with degenerative conditions/ significant disabilities


CISF ratings and the “a” category

Rated lower at exit than entry; ORRated 1 at both entry and exit;

AND Scored “No” on the progress question (b)


Early Childhood Outcomes Center18

Entry Exit


Entry Exit

The “b” categoryb. Percent of infants and toddlers who

improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers

Children who acquired new skills but continued to grow at the same rate throughout their time in the program Gained and used new skills but did not increase their rate of growth or change their growth trajectories while in services


CISF ratings and the “b” category

Rated 5 or lower at entry; ANDRated the same or lower at exit;

AND

“Yes” on the progress question (b)



Entry Exit


Entry Exit


Entry Exit

The “c” category

c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it

Children who acquired new skills but accelerated their rate of growth during their time in the programMade progress toward catching up with same aged peers but were still functioning below age expectations when they left the programChanged their growth trajectories --“narrowed the gap”


CISF ratings and the “c” category

Rated higher at exit than entry;

AND

Rated 5 or below at exit



Entry Exit

The “d” categoryd. Percent of infants and toddlers who

improved functioning to reach a level comparable to same-aged peers

Children who were functioning below age expectations when they entered the program but were functioning at age expectations when they leftStarted out below age expectations, but caught up while in services


CISF ratings and the “d” category

Rated 5 or lower at entry;

AND

Rated 6 or 7 at exit



Entry Exit

The “e” category

e. Percent infants and toddlers who maintained functioning at a level comparable to same-aged peers

Children who were functioning at age expectations when they entered the program and were functioning at age expectations at exit


CISF ratings and the “e” category

Rated 6 or 7 at entry; AND

Rated 6 or 7 at exit



Entry Exit


Entry Exit

The Challenge….

Progress data included5 progress categories For each of 3 outcomesTotal of 15 numbers reported each year

Too many interrelated targets to understandOSEP asked for a recommendation


Development of the Summary Statements

ECO presented options to states and ECO work groups via conference callsTwo sessions at December, 2008 EC Conference Posted on the ECO web site for commentsECO made recommendation to OSEP


Final DeliberationOSEP put the summary statements out for public commentComments came in that were thoughtful, but not necessarily consistent with one anotherAdvantages and disadvantages to all optionsPaper documenting the process on the ECO websitehttp://www.fpg.unc.edu/~eco/assets/pdfs/summary_of_target_setting-2.pdf


The Summary Statements

Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.

The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.


Understanding the Data

Sample data will be used in the next several slides to demonstrate the process for calculating the summary statements from the data in the 5 OSEP categories


Example of State Progress Data for 2008-2009

Positive social-emotional skills (including social relationships):

Number of

children

% of children

a. Percent of infants and toddlers who did not improve functioning

40 4

b. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers

150 15

c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach

270 27

d. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers

300 30

e. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers

240 24

Total N=1000

100%

Summary Statement Data

Required Summary Statement 1: Of those children who entered the program

below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program= 75%

Required Summary Statement 2: The percent of children who were functioning

within age expectations in each Outcome by the time they exited the program= 54%


Where do the #s come from?

42

Prog cat

# %

a 40 4b 150 15c 270 27d 300 30e 240 24

760 (a, b, c, and d) or 76% of the children entered the program functioning below age expectations

240 (e) or 24% of the children entered and exited functioning at age expectations


Measurement for Summary Statement 1: Percent = # of infants and toddlers reported in

progress category (c) plus # of infants and toddlers reported in category (d) divided by [# of infants and toddlers reported in progress category (a) plus # of infants and toddlers reported in progress category (b) plus # of infants and toddlers reported in progress category (c) plus # of infants and toddlers reported in progress category (d)] times 100.



44

Prog cat

# %

a 40 4

b 150 15

c 270

27

d 300

30

e 240 24

570 (c and d) of the 760 (a, b, c, and d) changed their growth trajectories (made greater than expected progress)

270 +300= 570 760

= 75%

45Summary Statements Calculator -April 14, 2009


Measurement for Summary Statement 2: Percent = # of infants and toddlers reported in progress category (d) plus [# of infants and toddlers reported in progress category (e) divided by the total # of infants and toddlers reported in progress categories (a) + (b) + (c) + (d) + (e)] times 100.



47

Prog cat

# %

a 40 4

b 150 15

c 270 27

d 300 30e 240 24

300+240= 540 1000

= 54%

30% of the children reached age expectations by exit and 24% of the children entered and exited at age expectations

What can we say about the children’s

progress?Part C Outcome 1: successful social relationships with peers and adults, following rules for social interactions96% of children participating in Part C made progress in their social relationships while they were enrolled.The 4% of children who did not make progress included children with the most severe disabilities and/or degenerative conditions. Can you describe them?


Positive Social Relationships

24% of the children participating in Part C were functioning at age expectations at entry and at exit in this outcome area. 54% of the children were functioning at age expectations in this outcome area when they exited the program. (summary statement 2)

30% started out behind and caught up24% entered and exited at age expectations


Positive Social Relationship

75% of the children who entered the program below age expectations made greater than expected gains; they made substantial increases in their rates of growth. i.e. changed their growth trajectories (summary statement 1)


Setting Targets

Quality Data is essential for setting realistic targetsTwo strategies for examining data

Data qualityPotential for program improvement

Parameters, guidance for target setting


Can you trust the data?Mechanisms that impact quality of data:

ITOTS mechanisms to prevent inaccurate data entryPersonnel Training

Initial Statewide trainingTraining resources, including online training modules and Implementation ManualsLocal training and oversight

Review of Data – TA Guidance at Regional Meetings


Can you trust the data?

Look at the data across all systems in order to analyze data quality:

Begin by identifying outliers Examples: look at the percentages reported for certain categories across local programs


Percentages reported in category “a” across 30

(sample) local programs

54

Look at the Impact of the Outliers on the

Data

State percentage for “e” with all data= 32.1%

Revised percentage for “e” with outliers removed= 27.7%

55

Example of data with outliers removed

Progress Category

Original % Clean %

a 4 2b 15 17c 27 30d 30 31e 24 20

Sum St 1 75 76Sum St 2 54 51

Clean data (without the outliers) may be a more accurate picture of where you are starting

56

Suggested Strategy

Analyze your data with your local system outliers included and excluded so you can gauge the impact they are having on your state level data.


Important Note about Reporting Data to

OSEP

Consider clean data when deciding about reasonable targets, BUTTurn in the original data to OSEP in the SPP report!You can discuss the clean data in the rationale for your targets.


Which local systems can be targeted for program improvement?

Compare the summary statement data by local programs to identify which programs have the most potential for improvement.


Summary Statement Percentages

by Local Program

60

Data Quality Considerations

What do you know about the local systems with the least and the most progress in the summary statements? i.e. the programs with:

the lowest and highest percentages of children at age expectation at exitthe lowest and highest percentage of children making greater than expected gains


Examples of Key Questions

Are the children similar at entry?Are the higher performing local systems participating in special projects or have they instituted training programs?Are there systems issues in lower performing local systems that would explain differences in outcomes? e.g. personnel shortages


Bottom Line Question

Could either system or practice focused improvement activities targeted toward the lowest performing local systems improve the child outcomes?


The Math of Target Setting

How much would the data change if the lowest local programs moved toward the mean?Improvements in the lowest programs will result in improvement in your statewide dataExperiment with your data to determine what targets are reasonable in your state


Next Steps

VICC Subgroup meeting (1/2 day) to review Virginia’s data in detail and develop recommendations for targets.Presentation of recommendations from the VICC subgroup to the full VICC at the December VICC meeting


Additional Information

Options and ECO Recommendations for Summary Statements for Target Setting

http://www.fpg.unc.edu/~eco/assets/pdfs/summary_of_target_setting-2.pdf

Summary Statement SlidesTarget Setting Slides

http://www.fpg.unc.edu/~eco/pages/events.cfm


Documents

Child Outcomes: Understanding the Requirements in order to Set Targets Presentation to the Virginia Interagency Coordination Council 09-09-09 Infant &