National Picture – Child Outcomes for Early Intervention and Preschool Special Education

National Picture – Child Outcomes for Early Intervention and Preschool

Special Education

Kathleen HebbelerAbby Winer

Cornelia Taylor

August 26, 2014

On Today’s Call

• Brief review of the national data

• Data quality– Completeness of data– State-to-state variation– Change over time

2

State Approaches to Measuring Child Outcomes – 2012-13

3

Approach Part C (N=56)

Preschool(N=59)

COS 7 pt. scale 42/56 (75%) 37/59 (63%)

One tool statewide 8/56 (14%) 9/59 (15%)

Publishers’ online analysis

1/56 (2%) 6/59 (10%)

Other 5/56 (9%) 7/59 (12%)

Method for Calculating National Estimates

• Weighted average of states that met minimum quality criteria

• Minimum Quality Criteria:– Reporting data on enough children• Part C – 28% or more of exiters• Part B Preschool – 12% or more of

child count– Within expected patterns in the data• category ‘a’ not greater than 10% • category ‘e’ not greater than 65%

4

Number of States that Met Criteria for Inclusion in the National Analysis

5

08-09 09-10 10-11 11-12 12-13

Part C 19 29 39 33 41

Part B Preschool 15 33 36 39 41

Part C: Reason States Were Excluded from Analyses (out of 51)

6

Reason Part C state was excluded 2010-11 2011-12 2012-13State is sampling 3 2 1No outcomes data reported 0 0 0Reported outcomes data on less than 28% of reported exiters 3 6 4

Had at least one outcome with category a greater than 10% or category e greater than 65% 4 5 3

Reported outcomes data on less than 28% of reported exitersANDHad at least one outcome with category a greater than 10% or category e greater than 65%

1 4 2

Questionable data quality based on review of SPP/APR and knowledge gained through TA 1 0 0

States included in the analysis 39 33 41

Part B Preschool: Reason States Were Excluded from Analyses (out of 51)

7

Reason Part B state was excluded 2010-11 2011-12 2012-13State is sampling 4 2 3No progress category data reported 1 2 1No outcomes data reported 1 0 0Reported outcomes data on less than 12% of child count 2 4 1

Had at least one outcome with category a greater than 10% or category e greater than 65% 4 3 4

Reported outcomes data on less than 12% of child countANDHad at least one outcome with category a greater than 10% or category e greater than 65% 0 0 0

Questionable data quality based on review of SPP/APR and knowledge gained through TA 2 0 0

No child count data available 1 0 1States included in the analysis 36 39 41

8

9

10

11

What We See

• Continuing to see consistency over time• Increasing number of states who meet minimum quality criteria

for national analysis• Increasing number of children in the child outcomes data

12

Current Emphasis of State Requests

• Data Quality– Increasing the number of children/families in the data– Pattern checking to identify data quality issues– Training, guidance, supervision, etc.

• Using Data for Program Improvement– SSIP data analysis• Identifying trends in the data• Identifying areas of low and high performance• Identifying meaningful differences

13

Part C: Percent of States by Completeness of Child Outcomes Data*

* Completeness = (total with outcomes data/total exiters)14

Part B Preschool: Percent of States by Completeness of Child Outcomes Data*

* Completeness = (total with outcomes data/child count)15

State Level Variation and Patterns

16

Part C: Positive Social Emotional Skills Progress Category ‘b’ – All States

17

National: 21%

18

National: 66%

Part C: State Variation: Greater than Expected Growth – Social Emotional, 2012-2013, All States

Part B Preschool: Knowledge and Skills Progress Category ‘b’ – All States

19

National: 15%

20

National: 53%

Part B Preschool: State Variation: Exited within Age Expectations – Knowledge and Skills, 2012-2013, All States

Part C: Exited within Age Expectations by State Percent of Exiters Not Eligible for Part B*

2012-13 - All States

21

Part C: Average Percentage Who Exited within Age Expectations by State Percent Served*,

2012-13 – All States

22*http://therightidea.tadnet.org/assets/2514

Part C: Average Percentage Who Exited within Age Expectations by ITCA Eligibility

Category, 2012-13, All States

23

Part B Preschool: Average Percentage Who Exited within Age Expectations by State

Percent Served, 2012-13, All States

24

Variation Over Time

25

What Types of Change are Important?

26

Types of Change 2008-09 2012-13

Small variations from year to year are expectedLarge consistent increases are good news particularly when linked to programmatic changes

Large consistent decreases require explanation (e.g. changing population)

Large up and down changes are an indicator of questionable data quality and require explanation

Part C: Longitudinal Patterns All StatesOutcome 1 Summary Statement 1

27

Part C: Longitudinal Patterns All States, Last 3 Years, Outcome 1 Summary Statement 1

28

Part C: Longitudinal Patterns States Included in National Estimate, Last 3

Years Outcome 1 Summary Statement 1 (30 states)

29

Part B 619: Longitudinal Patterns All StatesOutcome 1 Summary Statement 1

30

Part B 619: Longitudinal Patterns All States, Last 3 Years, Outcome 1 Summary Statement 1

31

Part B 619: Longitudinal Patterns States Included in National Estimate, Last 3 Years

Outcome 1 Summary Statement 1 (25 states)

32

Part C: Statistically Significant* Change between 2011-12 and 2012-13 : 41 States That

Met the Criteria

Statistically Significant Change OC1-SS1 OC2-SS1 OC3-SS1 OC1-SS2 OC2-SS2 OC3-SS2

Negative4 3 3 8 6 7

None32 32 33 28 27 27

Positive5 6 5 5 8 7

33

*p<.01

Part B Preschool: Statistically Significant* Change between 2011-12 and 2012-13: 39

States** That Met the CriteriaStatistically Significant Change OC1-SS1 OC2-SS1 OC3-SS1 OC1-SS2 OC2-SS2 OC3-SS2

Negative6 4 4 4 2 6

None26 24 30 29 32 28

Positive7 11 5 6 5 5

34

** Two states excluded because there was insufficient data from 2011-12 to make the comparison

*p<.01

Conclusions

• The data continue to be used by the federal government to justify funding.

• Results Driven Accountability is shining a spotlight on each state’s child outcomes data.

• States can expect more scrutiny around data quality.– The criteria used for the national analyses do not set a high bar

for data quality.

35

How We Can Help!

• State data quality profiles for FFY 2012-13 were sent out– Email Abby Winer with questions: abby.winer@sri.com

• Contact us for help with data quality analysis and quality assurance activities

• Contact us for help with program improvement planning and data analysis

36

State Child Outcomes Data Quality Profiles FFY 2012-13

38

Updated National Graphing Template

38

http://www.ectacenter.org/eco/pages/summary.asp#summarygraph

Data Quality: Pattern Checking

http://www.ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf Training materials on looking at data: http://www.ectacenter.org/eco/assets/ppt/LookingAtData_revised.ppt

40

Other Resources

• Additional data quality resources– http://www.ectacenter.org/eco/pages/quality_assurance.asp

• Data analysis for program improvement http://www.ectacenter.org/eco/pages/usingdata.asp

• SSIP-related Resources– http://www.ectacenter.org/~calls/2014/ssip/ssip.asp

40

Reminders & Announcements

• We hope to see you at the Improving Data, Improving Outcomes Conference!

• The Outcomes for Children and Family Outcomes flyers will be available at the conference

• Stay tuned for an upcoming presentation of Family Data: Indicator C4 Highlights

41

42

Find More Resources At:

http://www.ectacenter.org/eco