Evaluation of NDPC-SD

AKA CIPP Evaluators: Elaine Carlson and Tom Munk

Assist in summative evaluation of the center Helped develop standardized logic model Helped develop summative evaluation plan Helped develop evaluation instruments Will conduct data analysis of summative

evaluation data

Center to Improve Project Performance

NDPC-SD Logic Model

Inputs• Funding• Expertise• Lessons learned • Evidence-based

practices• Research base• Time• Materials and

technology• OSEP priorities• SEA/LEA needs• State readiness

Goals

• 1. Increase the awareness of policymakers, administrators, and practitioners about dropout prevention, reentry, and school completion.

• 2. Increase the number of states that set and meet reasonable and rigorous performance targets for Indicators 1 and 2.

Other Federal activities, OSEP policy environment, Clemson’s experience with the previous NDPC-SD, general education reforms, and economic trends that influence labor markets.

External Factors/Context

Activities Outputs Direct Intermediate Long-term

Outcomes

• A. Provide knowledge development.

Increase the awareness of policymakers,

administrators, and practitioners about school completion.

• Number and quality of comprehensive reviews completed on schedule

• Number of contacts with general TA providers on incorporating school completion models and practices

Decrease the number of students with disabilities who drop out of school.

• Number of training activities developed and delivered

• Number of forums held• Number/type/content/timing of TA activities• Type/number/audience of TA&D activities• Type/number/content of materials produced• Number of APR support materials developed• Amount of TA delivered and outcomes reported• Number of high-quality data systems developed

Increase the number of providers incorporating school completion in the school improvement context.

Improve and increase SEA/LEA use of data-based improvement planning, including developing and improving data systems to track students at risk of dropping out.

Implement and evaluate effective, comprehensive school-completion models, practices, and systems.

Increase the number of students with disabilities who re-enter high school or enter other credentialing or training programs.

• 3. Help SEAs and LEAs develop and improve data systems to track students at risk of dropping out.

• 4. Help SEAs and LEAs implement and evaluate effective, comprehensive school-completion models, practices, and systems for students with disabilities.

• C. Provide technical assistance & dissemination.

Improve and increase SEA/LEA use of SPP/APR indicators to drive school completion activities.

Improve SEA/LEA school-completion policies and procedures.

Increase the number of students with disabilities who graduate.

Increase the number of states that propose

SPP/APR activities for IDEA Part B Indicator1 and Indicator 2 that are aligned with their performance targets for

those indicators.

• Number of SPR, APR, and performance data analyses completed

• Number/attendance of advisory committee meetings

• Number/type/intensity of collaboration activities with OSEP TA & D network and Comprehensive Centers

• Extent of participation in Exiting CoP

• B. Provide leadership and coordination.

1. Increase awareness of policymakers, administrators and practitioners about dropout, school completion and reentry

2. Increase the number of states that set and meet reasonable and rigorous performance targets for Indicators 1 and 2

3. Help SEAs and LEAs develop and improve data systems to track students at risk of dropping out

4. Help SEAs and LEAs implement and evaluate effective, comprehensive school-completion models, practices, and systems for students with disabilities

NDPC-SD Goals

Knowledge Development

Leadership and Coordination

Technical Assistance and

Dissemination

NDPC-SD Activities

Number & quality of comprehensive reviews completed on schedule

Number of contacts with general TA providers on incorporating school completion models and practices

Knowledge Development Outputs

Number of SPR, APR, and performance data analyses completed

Number/attendance of advisory committee meetings

Number/type/intensity of collaboration activities with OSEP TA & D network and Comprehensive Centers

Extent of participation in Exiting CoP

Leadership & Coordination Outputs

Number of training activities developed and delivered

Number of forums held Number/type/content/timing of TA activities Type/number/audience of TA&D activities Type/number/content of materials produced Number of APR support materials developed Amount of TA delivered and outcomes

reported Number of high-quality data systems

developed

TA & Dissemination Outputs

Increase the number of providers incorporating school completion in the school improvement context

Increase the awareness of policymakers, administrators, and practitioners about school completion

Improve and increase SEA/LEA use of data-based improvement planning, including developing and improving data systems to track students at risk of dropping out

Direct Outcomes

Improve SEA/LEA school-completion policies and procedures

Improve and increase SEA/LEA use of SPP/APR indicators to drive school completion activities

Increase the number of states that propose SPP/APR activities for IDEA Part B Indicator1 and Indicator 2 that are aligned with their performance targets for those indicators

Intermediate Outcomes

Decrease the number of students with disabilities who drop out of school

Increase the number of students with disabilities who re-enter high school or enter other credentialing or training programs

Increase the number of students with disabilities who graduate

Long-Term Outcomes

User surveys for Big IDEAs, website, teleseminars

TA/training surveys TA log (who, what, where, when…) Counts of activities for monthly reports School Implementation Survey SEA and LEA Impact Surveys

Data Collection Instruments

Purpose

1. Determine where a school is in the development and implementation of its school-completion initiative. The first administration will establish a baseline. Subsequent administrations should show the school progressing through the stages of implementation.

2. Assesses school practices supporting school completion

3. Determine how much knowledge of the school-completion initiative individual respondents have

School Implementation Survey

Completed by individuals in schools implementing a school-completion initiative based on the NDPC-SD framework

Collects school-level information about: Training School team Planning process School data collection and analysis Selection of evidence-based practices Professional development Evaluation of the initiative Implementation and progress of interventions

School Implementation Survey

Survey was first administered in Georgia & Maryland in fall 2009 to establish baselines

Will be administered in all new training sites in the future

Will be re-administered annually

School Implementation Survey Data Collection

1. School performance data2. Demographic data3. Information about policies &

procedures4. Academic engagement data 5. Information about instructional

systems6. Behavioral engagement data7. Affective engagement data8. Information about professional

development

School Implementation Survey Data: The Most Commonly Used Data in Planning

School Implementation Survey Data: School Has Identified Suitable EBPs

No Don't know In progress Yes Not applicable No response0

25

50

75

100

125

150

175

200

225

250

275

300

17

46

187

134

07

Num

ber

of

Responses

(N = 391 respondents)

School Implementation Survey Data: School Has Determined Level of Implementation

(N = 391 respondents)

No Don't know In progress Yes Not applicable No response0

25

50

75

100

125

150

175

200

225

250

275

300

21

46

202

116

0 6

Nu

mb

er

of

Resp

on

ses

School Implementation Survey Data: School Has Set Timelines for Implementation

(N = 391 respondents)

No Don't know In progress Yes Not applicable No response0

25

50

75

100

125

150

175

200

225

250

275

300

30

54

197

102

08

Nu

mb

er

of

Resp

on

ses

School Implementation Survey Data: Implementation: Interventions in Use

0

50

100

150

200

250

300

350 340320

233247 255 257

225 220

147

202

268

167

253

(N = 391 respondents)

Num

ber

of

responses

Should we split our sample into 2 groups: School Team Leader

Everyone else who is surveyed in the school

Does the School Implementation Survey appear to be measuring what we need?

How could we improve its effectiveness?

Questions for the Committee:School Implementation Survey

Under development! We need to assess the impact of NDPC-SD’s technical assistance on SEA and LEA practices and policies around school completion

State and Local Impact Survey(s)

One survey or two?Who should complete it/them?Draft survey – what to add, delete or revise?

Questions for the Committee: SEA and LEA Impact Survey(s)