The New York State School The New York State School Improvement Grant InitiativeImprovement Grant Initiative

Scientific and Evidence Based Evaluation of Scientific and Evidence Based Evaluation of SIG/SPDG Initiatives: One State’s ResponseSIG/SPDG Initiatives: One State’s Response

Office of Professional Research & Office of Professional Research & Development, Syracuse University, NYDevelopment, Syracuse University, NY

REVISITING NYSIG REVISITING NYSIG AS DESIGNED:AS DESIGNED:PROGRAM & PROGRAM & EVALUATIONEVALUATION

The NYS SIG Initiative is The NYS SIG Initiative is designed to: designed to:

Reduce achievement gap between Reduce achievement gap between special and general education special and general education students in high and low need students in high and low need schools.schools.

Reduce or eliminated the Reduce or eliminated the disproportionality of language and disproportionality of language and ethnic minority students in ethnic minority students in classification and placement classification and placement practices.practices.

LEA

REGIONAL SCHOOL SUPPORT CENTERS (RSSCs)

SPECIAL EDUCATION RESOURCE AND TRAINING CENTERS (SETRCs)

INSTITUTIONS OF HIGHER EDUCATION (IHEs)

SPECIAL EDUCATION QUALITY ASSURANCE (SEQA)

STATE IMPROVEMENT GRANT

HIGHER EDUCATION SUPPORT CENTER

VESID: RESOURCES, TA, OVERSIGHT

Data Collection: Student Outcomes & Performance Indicators

Root Cause Analysis, Strategic Planning & Goal Setting

Implementation of Activities

The Evaluation Logic Model

NYSED (VESID)

Two ‘Strands’Two ‘Strands’

SIG Resources and SIG Resources and partnerships applied to two partnerships applied to two strands:strands:

Inservice TeachersInservice Teachers Pre-Service TeachersPre-Service Teachers

Evaluation GoalsEvaluation Goals

Tracking implementationTracking implementation

Identifying areas/strategies for Identifying areas/strategies for program improvementprogram improvement

Capturing outcomes (according to Capturing outcomes (according to stage of development of the stage of development of the program) program)

Evaluation GoalsEvaluation Goals Implementation Implementation

to dateto date Degree of match Degree of match

between over between over arching goals of arching goals of SIG and program SIG and program activities activities

Degree of match Degree of match between district between district stated goals and stated goals and activitiesactivities

Strengths of the SIG Strengths of the SIG Challenges to the Challenges to the

program and program and strategies to meet strategies to meet these challengesthese challenges

Lessons learned and Lessons learned and emerging themesemerging themes

Intermediate Intermediate outcomes & outcomes & performance performance measure attainmentmeasure attainment

Targets for Targets for EvaluationEvaluation

District readiness District readiness Partnership development with IHEsPartnership development with IHEs Utility/effectiveness of technical Utility/effectiveness of technical

assistance entities (VESID funded, assistance entities (VESID funded, HESC in particular)HESC in particular)

Outcomes (particularly student Outcomes (particularly student improvement)improvement)

Multi-Method Evaluation Multi-Method Evaluation ApproachApproach

Formative assistanceFormative assistance Surveys (IHE faculty, recent teacher Surveys (IHE faculty, recent teacher

preparation graduates, school preparation graduates, school administrators, training participants).administrators, training participants).

Interviews and focus groupsInterviews and focus groups Document analysisDocument analysis Site studies (inc interviews with SIG Site studies (inc interviews with SIG

District grant recipients: admin, staff, TA District grant recipients: admin, staff, TA providers and parents)providers and parents)

Quantitative analysis of student Quantitative analysis of student performance dataperformance data

Evolution of a Systems ChangeEvolution of a Systems Change Sustained changeSustained change

Initial changes Initial changes (perceptions/data)(perceptions/data)

Implementation of planned Implementation of planned activities to address goalsactivities to address goals

Desire for change and setting Desire for change and setting of goals (buy in)of goals (buy in)

Relationship buildingRelationship building

AwarenessAwareness

RReporting: Using the Logic eporting: Using the Logic Model FrameworkModel Framework

Faithfulness of implementationFaithfulness of implementation Effectiveness of activities Effectiveness of activities Lessons learned/Ingredients for Lessons learned/Ingredients for

successsuccess Challenges noted through qualitative Challenges noted through qualitative

data collectiondata collection Outcome data (according to Outcome data (according to

developmental stage of the Initiative)developmental stage of the Initiative) Emerging policy issuesEmerging policy issues RecommendationsRecommendations

Participating IHEs with Participating IHEs with assistance from the HESC assistance from the HESC are expected to:are expected to:

Develop, implement and/or sustain inclusive Develop, implement and/or sustain inclusive teacher preparation programsteacher preparation programs

Link with identified districts in their area to Link with identified districts in their area to provide professional development and research provide professional development and research assistanceassistance

Provide for professional development among Provide for professional development among other teacher educators at their university or other teacher educators at their university or collegecollege

Document analysisDocument analysis

Grant applicationsGrant applications IHE agreementsIHE agreements Initiative and School Initiative and School

Grant reportsGrant reports Meeting minutes:Meeting minutes:

Management Management meetings meetings

Statewide Task Force Statewide Task Force IHE meetingsIHE meetings

Regional Task Force Regional Task Force meetingsmeetings

Surveys, focus groups, and interviewsSurveys, focus groups, and interviewsWITH THE IHEWITH THE IHE Perceptions of status of Perceptions of status of

partnerships, their role partnerships, their role and change in districtsand change in districts

Changes in their Changes in their teacher preparation teacher preparation programs programs

Usefulness of Usefulness of assistance from the assistance from the HESCHESC

Promising practices Promising practices related to partnerships related to partnerships and teacher and teacher preparation programspreparation programs

WITH RECENT WITH RECENT GRADUATESGRADUATES

Perceptions of Perceptions of preparedness preparedness

Identification of high Identification of high impact activities and impact activities and strategies in teacher strategies in teacher preparation preparation programsprograms

Participating School Districts with Participating School Districts with assistance from the SIG Teams are assistance from the SIG Teams are expected to:expected to: Undertake a root cause analysis process Undertake a root cause analysis process

supported by SETRC Professional supported by SETRC Professional Development Specialists/RSSC Special Development Specialists/RSSC Special Education Specialists.  Education Specialists. 

Develop plan for professional Develop plan for professional development and submit SIG District development and submit SIG District grant application.  grant application. 

Work with RSSC, SETRC, & SIG Teams to Work with RSSC, SETRC, & SIG Teams to determine the most effective use of SIG determine the most effective use of SIG Team resources.  Team resources. 

SIG Teams then provide job-embedded SIG Teams then provide job-embedded (sequential, on-going, and specifically (sequential, on-going, and specifically job-relevant) professional development job-relevant) professional development and tracking in the field. and tracking in the field. 

Site Visit ComponentSite Visit Component

Annual selection of SIG Schools for Annual selection of SIG Schools for site visitssite visits

Based on Variety of VariablesBased on Variety of Variables– Geographical LocationGeographical Location– Need Need – Urban, Suburban, RuralUrban, Suburban, Rural– Promising PracticesPromising Practices

Logic Model for Logic Model for Instrument Development Instrument Development and Data Collectionand Data Collection

Interview Protocol for Site Visits Interview Protocol for Site Visits Developed Using Logic ModelDeveloped Using Logic Model

Items Tapping into Documenting:Items Tapping into Documenting:– ActivitiesActivities– Outputs of ActivitiesOutputs of Activities– Intermediate Outcomes Intermediate Outcomes – End OutcomesEnd Outcomes

Outputs of ActivitiesOutputs of Activities

Changes in Teacher, Parent, Changes in Teacher, Parent, Admin. Perceptions or BeliefsAdmin. Perceptions or Beliefs

Recognizing a Need to be Recognizing a Need to be AddressedAddressed

Recognizing New Method to Recognizing New Method to Address NeedAddress Need

Recognizing Need for New Recognizing Need for New Instructional PracticesInstructional Practices

Intermediate Intermediate Outcomes Outcomes

Changes in Actual PracticesChanges in Actual Practices– E.g. Instructional Practices Across E.g. Instructional Practices Across

Buildings and at the Individual Buildings and at the Individual Classroom LevelClassroom Level

Changes in Stakeholders Working Changes in Stakeholders Working Together Together – E.g. Increased Parent InvolvementE.g. Increased Parent Involvement

Student Classroom OutcomesStudent Classroom Outcomes– E.g. Daily Behavior, Homework, E.g. Daily Behavior, Homework,

AttitudeAttitude

End OutcomesEnd Outcomes

Changes in classification, placement, Changes in classification, placement, and declassification practicesand declassification practices– Compared with state averagesCompared with state averages– DisproportionalityDisproportionality

Changes in Performance on State ELA Changes in Performance on State ELA and Math Assessmentsand Math Assessments– General Education StudentsGeneral Education Students– For Students with Special NeedsFor Students with Special Needs– Minority StudentsMinority Students– ESL ESL

BenefitsBenefits

Able to document patterns Able to document patterns between district and partnersbetween district and partners

Able to clearly delineate Able to clearly delineate improvement by professional improvement by professional development methoddevelopment method– Focus efforts on one building or Focus efforts on one building or

across the entire districtacross the entire district Able to “tell story” over a period of Able to “tell story” over a period of

time and see changestime and see changes

Just the numbers….Just the numbers….

Scope of the effort:Scope of the effort:

Total SIG School Districts Year One Total SIG School Districts Year One – Five: 51 (not counting Big 5)– Five: 51 (not counting Big 5)

Total Schools in ‘Big 5’: 19Total Schools in ‘Big 5’: 19 Total IHEs Partnering With Total IHEs Partnering With

Schools: 44Schools: 44 Total IHEs Participating in SIG: 65Total IHEs Participating in SIG: 65

RESPONSIVE RESPONSIVE EVALUATION EVALUATION

DESIGNDESIGN

Other Evaluation Other Evaluation MethodsMethods

Regional Think TanksRegional Think Tanks SIG Team InterviewsSIG Team Interviews School Outcome Data TrackingSchool Outcome Data Tracking Participant Training SurveyParticipant Training Survey SIG Team Document ReviewSIG Team Document Review Observation & ParticipationObservation & Participation

Review of MethodsReview of Methods

Revisiting SIG Evaluation Original DeRevisiting SIG Evaluation Original Design: Review of Methods Year One-Fisign: Review of Methods Year One-Fiveve

Responsive Evaluation Responsive Evaluation ModelModel

During the five years of the evaluation the During the five years of the evaluation the design and methodologies needed to design and methodologies needed to respond to:respond to:

External shifts/expectations/needsExternal shifts/expectations/needs Concurrent Internal programmatic Concurrent Internal programmatic

changes/shiftschanges/shifts

Some of these shifts were anticipated and Some of these shifts were anticipated and were worked into the original design, were worked into the original design, and some were not….and some were not….

The NY SIG Responsive EvalThe NY SIG Responsive Evaluation Modeluation Model

Internal Shifts: Program Internal Shifts: Program Changes & Evaluation Changes & Evaluation ResponsesResponses

Partnerships: Challenges engaging Partnerships: Challenges engaging parentsparents Regional Think Tanks. Regional Think Tanks.

Roles: Changes to the way NY thought Roles: Changes to the way NY thought about roles and responsibilities of SIG about roles and responsibilities of SIG Teams and RSSC/SETRC partnersTeams and RSSC/SETRC partners SIG SIG Team Interviews, Training Surveys.Team Interviews, Training Surveys.

Practice: Challenges connecting with Practice: Challenges connecting with schools across vast geographical schools across vast geographical areasareas SIG Team Interviews. SIG Team Interviews.

Programming: Introduction of new Programming: Introduction of new program componentsprogram components Observation. Observation.

More Internal Shifts & More Internal Shifts & Responses…..Responses…..

Start-up: Time needed for start-up Start-up: Time needed for start-up led to school grants for 3+ yearsled to school grants for 3+ years Document Review.Document Review.

Roll-out: Changes to NYCBOE Roll-out: Changes to NYCBOE structure and school functioningstructure and school functioning Document Review & Observation.Document Review & Observation.

Reporting: Institution of school Reporting: Institution of school reporting mechanismsreporting mechanisms Document Document Review.Review.

A special note…A special note…

Reality Check: Many of the internal Reality Check: Many of the internal shifts noted were linked to external shifts noted were linked to external shifts and needs of ‘external’ shifts and needs of ‘external’ stakeholders….stakeholders….

Stakeholder Needs: Stakeholder Needs: Understanding Impact & Understanding Impact & OutcomesOutcomes

Stakeholders: NYSED, USDOE, WESTATStakeholders: NYSED, USDOE, WESTAT Challenges: Challenges:

– Identifying what the treatment is.Identifying what the treatment is.– Identifying what people are doing and why.Identifying what people are doing and why.– Identifying who is receiving services, who Identifying who is receiving services, who

isn’t, and how much.isn’t, and how much.– Identifying the evidence base of the said Identifying the evidence base of the said

activities.activities.– Identifying impact on schools and students Identifying impact on schools and students

without use of experimental design.without use of experimental design.– Identifying changes in practice tied to SIG Identifying changes in practice tied to SIG

initiatives.initiatives.– Availability (or lack) or data.Availability (or lack) or data.– Identifying responsiveness and alignment to Identifying responsiveness and alignment to

State needs.State needs.

Responses of Stakeholders to Responses of Stakeholders to ChallengesChallenges External Shifts External Shifts

National Evaluation.National Evaluation. Encourage Alignment with State Encourage Alignment with State

Performance Plan.Performance Plan. Development of Federal Development of Federal

Performance Measures.Performance Measures. ‘‘Collective call’ to utilize scientific Collective call’ to utilize scientific

and evidence based practice.and evidence based practice.

National EvaluationNational Evaluation

Requirement: Understand Requirement: Understand outcomes/change in: child performance, outcomes/change in: child performance, teacher or administrator behavior, teacher or administrator behavior, systems functioning, and scaling up of systems functioning, and scaling up of successful practices. successful practices.

Evaluation Responses:Evaluation Responses:– Provision of summary and full reports.Provision of summary and full reports.– Phone Conference Interviews.Phone Conference Interviews.– Attendance at 3 day multi-state workshop.Attendance at 3 day multi-state workshop.– Responding to written information & Responding to written information &

clarification requests.clarification requests.– Move from sample data analysis to cohort Move from sample data analysis to cohort

data analysis.data analysis.

State Performance PlanState Performance Plan

Requirement: Respond to indicators including: Requirement: Respond to indicators including: graduation, drop-out, assessment participation, graduation, drop-out, assessment participation, suspension/expulsion, LRE, dispoprotionality, suspension/expulsion, LRE, dispoprotionality, parent involvement etc.parent involvement etc.

Evaluation Responses:Evaluation Responses:– ‘‘Retrofit’ of the evaluation plan to align with new and Retrofit’ of the evaluation plan to align with new and

developing program components.developing program components.– Regional think tanks on: parent involvement, Regional think tanks on: parent involvement,

disproportionality and culturally responsive practicedisproportionality and culturally responsive practice– Participation in Disproportionality Learning CommunityParticipation in Disproportionality Learning Community– Tracking new initiatives in response to SPP (i.e. Tracking new initiatives in response to SPP (i.e.

Learning Communities)Learning Communities) Observation & Participation, Observation & Participation, Document ReviewDocument Review

– Informed student and school outcome data points.Informed student and school outcome data points.– Move from sample data analysis to cohort data Move from sample data analysis to cohort data

analysis.analysis.

Performance MeasuresPerformance Measures

Requirement: Collect evidence in the areas of: Requirement: Collect evidence in the areas of: scientific and evidence based opportunities scientific and evidence based opportunities offered and personnel trained, sustainability of offered and personnel trained, sustainability of efforts, teacher retention, alignment with State efforts, teacher retention, alignment with State Performance Plan.Performance Plan.

Evaluation Responses:Evaluation Responses:– SIG Team Interviews to further understand SIG Team Interviews to further understand

implementation and outcomes.implementation and outcomes.– Development of SIG Team Effort To Date Rubric.Development of SIG Team Effort To Date Rubric.– Regional Think Tanks on sustainability.Regional Think Tanks on sustainability.– Strengthening of IHE and School Survey instruments Strengthening of IHE and School Survey instruments

in the areas of documenting change in pre-service in the areas of documenting change in pre-service and in-service teacher practice & retention.and in-service teacher practice & retention.

Scientific & Evidence Scientific & Evidence Based PracticeBased Practice

Requirement: Personnel trained under Requirement: Personnel trained under programs supported by SPDG will have programs supported by SPDG will have the knowledge and skills to deliver the knowledge and skills to deliver scientifically or evidence-based scientifically or evidence-based practices to children with disabilities.practices to children with disabilities.

Evaluation Responses:Evaluation Responses:– Development of SIG Team Effort To Date Development of SIG Team Effort To Date

Rubric. Rubric tracks: session topics, Rubric. Rubric tracks: session topics, evidence base of content, # trainings, # evidence base of content, # trainings, # personnel.personnel.

– Documentation of practice through SIG Documentation of practice through SIG Coordinator and SIG Team interviews. Coordinator and SIG Team interviews.

Where did we meet Where did we meet stakeholder needs?stakeholder needs?

– Identifying what the treatment is.Identifying what the treatment is.– Identifying what people are doing Identifying what people are doing

and why.and why.– Identifying who is receiving Identifying who is receiving

services, who isn’t, and how much.services, who isn’t, and how much.– Identifying changes in practice tied Identifying changes in practice tied

to SIG initiatives.to SIG initiatives.– Identifying responsiveness and Identifying responsiveness and

alignment to State needs.alignment to State needs.

Where do we still Where do we still struggle?struggle?

– Identifying the evidence base of the Identifying the evidence base of the said activities.said activities.

– Identifying impact on schools and Identifying impact on schools and students without use of experimental students without use of experimental design.design.

– Availability (or lack) or data. i.e. Availability (or lack) or data. i.e. ‘Mining’ data down to the individual ‘Mining’ data down to the individual level to respond to SPP indicators level to respond to SPP indicators involving IEPs and student outcomes, involving IEPs and student outcomes, transition etctransition etc

Beyond SIGBeyond SIG

– Identifying the evidence base of the Identifying the evidence base of the said activitiessaid activities incorporation of incorporation of requirement into grant applications requirement into grant applications and site selection processes.and site selection processes.

– Identifying impact on schools and Identifying impact on schools and students without use of experimental students without use of experimental designdesign strengthening district strengthening district reporting requirements. reporting requirements.

– Availability (or lack) or dataAvailability (or lack) or data changes to cohort size, data collection changes to cohort size, data collection direct from schools and/or regions.direct from schools and/or regions.