Measuring Implementation: School-Wide Instructional Staff Perspective Amy Gaumer Erickson, Ph.D. University of Kansas Evaluator: Kansas & Missouri SPDGs

Measuring Implementation:

School-Wide Instructional Staff Perspective

Amy Gaumer Erickson, Ph.D.University of KansasEvaluator: Kansas & Missouri SPDGs

SIG Evaluators WebinarMarch 17, 2011

Gaumer Erickson (2011)

What types of measures are implemented? Self-Report Interviews/Focus Groups Observations Retrospective/Reflective Learning Goal Attainment Ratings of Fidelity Outcome Data


What perspectives do we need to get a complete picture of implementation in schools?

Measures/observations completed by: Implementation Coaches School Leadership Teams All School Instructional Staff Students Families/Communities


Measures completed by Implementation Coaches SW-PBS Benchmarks of Quality (BoQ)

MiBLSI: http://miblsi.cenmi.org/MiBLSiModel/Evaluation/Measures/BenchmarksofQuality.aspx

SW-PBS School-wide Evaluation Tool (SET) OSEP Technical Assistance Center on

PBIS: http://www.pbis.org/evaluation/evaluation_tools.aspx Gaumer Erickson (2011)

Measures completed by School Leadership Teams SW-PBS Team Implementation Checklist (TIC)

PBIS Surveys: http://www.pbssurveys.org/pages/Home.aspx

Effective Behavior Support Team Implementation Checklist (EBS-SAS) PBIS Surveys:

http://www.pbssurveys.org/pages/Home.aspx Planning and Evaluation Tool for Effective Reading

Supports – Revised (PET-R) DIBELS Data System:

http://dibels.uoregon.edu/resources/ Implementation Rubrics

Illinois PBIS Network: http://www.pbisillinois.org/


Student Measures DIBELS

DIBELS Data System: https://dibels.uoregon.edu/resources.php

AIMSweb AIMSweb: http://www.aimsweb.com/

School-Wide Information System (SWIS) SWIS: http://www.swis.org/

State Performance Plan APR Data (Indicators 1, 2, 3, 4, 5, 13, 14)

Student Perceptions


Family/Community Measures Parent Satisfaction

Missouri Improvement Program Advanced Questionnaire http://dese.mo.gov/divimprove/sia/dar/advance_questionnaire_surveys.html

Family as a Teacher Inventory Parent-Teacher Involvement Questionnaire

Fast Track Project http://www.fasttrackproject.org/data-instruments.php

Interviews/Focus Groups


Measures completed by All School Instructional Staff Self-Assessment Survey (SAS)

PBIS Surveys: http://www.pbssurveys.org/pages/Home.aspx

Standards Assessment Inventory National Staff Development Council

http://www.learningforward.org/standards/sai.cfm

Classroom/Teacher ObservationsGaumer Erickson (2011)

Which perspective do you think your grants measure most effectively?

A: Implementation Coaches B: School Leadership Teams C: All School Instructional Staff D: Stakeholders:

Student/Parents/Communities


School Staff Survey Design Identified essential features of school reform

initiatives (e.g., RTI, SW-PBS, PLCs, High Schools that Work, Reading First)

Analyzed other measures completed by school instructional staff (e.g. PBIS Self-Assessment Survey, RTI Implementation Tool, National Staff Development Council Standards Assessment Inventory, Missouri School Improvement Process Faculty Advanced Questionnaire)

Wrote over 100 items and then narrowed it down to 33 through pilot testing with schools and analysis by a state evaluation team

Implemented survey in 14 schoolsGaumer Erickson (2011)

School Staff Survey Reliability Overall Alpha: 0.98 Three Factors

School Implementation (17 items): 0.95 Classroom Implementation (10 items): 0.85 Individual Student Implementation (5 items):

0.90 Four Groups

Teachers (N=294): 0.96 Administrators (N=8): 0.90 Other Certified Staff (N=20): 0.97 Noncertified Staff (N=6): 0.85


School Implementation1) I can summarize the school's shared vision/mission.2) I have a clear understanding of the phrase: “tiered levels of academic & behavior

support.”3) I feel that my administrators are committed to implementing tiered levels of

academic & behavior supports. 4) I receive coaching/mentoring to implement evidence-based instructional

practices.5) I think my school does a good job of addressing the academic & behavior needs

of students at tier 1 (universal).6) I think my school does a good job of addressing the academic & behavior needs

of students at tier 2 (small group).7) I think my school does a good job of addressing the academic & behavior needs

of students at tier 3 (intensive).8) I regularly see students move between tiers of support as their academic needs

change.9) I regularly see students move between tiers of support as their behavior needs

change.10) I am involved in meetings where data results are discussed & problem solving

occurs.11) I receive school-wide academic & behavior data in usable & understandable

formats.12) I participate in professional development where I learn how to monitor students'

progress & use progress monitoring data.13) I have the time necessary to analyze student data & problem solve with my

colleagues.14) I think my school is a good place to work.15) I think my school has an effective process in place to identify available resources

(e.g., materials, technology, people).16) I evaluate the effectiveness of core instruction based on progress monitoring

data.17) I think that the current behavior/academic programs in my school are improving

education for students in my school.Gaumer Erickson (2011)

Classroom Implementation1) I use assessment data at least three times a year to monitor

students' progress.2) I adapt the environment, curriculum, & instruction based on

each student’s academic & behavior data.3) I participate in professional development where I learn ways

to improve my instructional practices.4) I receive coaching/mentoring to help me implement tiered

levels of academic & behavior support.5) I have the technology & resources that I need to provide

effective instruction.6) I am able to meet the students' diverse needs.7) I consider my students' background when I teach and/or

interact with students.8) I regularly communicate with families regarding student

academic & behavior goals/progress.9) I make informed decisions based on feedback from families.10) I collaborate with my colleagues on a regular basis.


Individual Student Implementation1) I am involved in action planning tiered supports with the

other staff & administrators at my school.2) When I am concerned about a student’s academic

success, I collaborate with a team to identify intervention.

3) I feel that the team that addresses academic needs provides valuable feedback & makes informed decisions.

4) When I am concerned about a student’s behavior success, I collaborate with a team to identify intervention.

5) I feel that the team that addresses behavioral needs provides valuable feedback & makes informed decisions.

6) I think my school does a good job in including parents as team members in data-based decision-making.


Reporting Results School Summaries (to support data-

based decision-making and action planning)

School Comparisons Across Years Dissemination & Evaluation of Program

Effectiveness SPDG APR






SPDG APR ReportingPercent of school personnel involved in the MIM who report the use of data-driven decision making; [correlated with SPP Indicator 3; Program Performance Measure 1.2].To identify the level of data-driven decision making, school staff were asked to rank the statement, I adapt the curriculum, instruction, & environment based on each student’s academic & behavior data, on the MIM School Staff Survey. The results show that 79% of MIM school staff report using data to make adaptations for students. In addition, school staff was asked to rank the following statements related to data-driven decision making:I use assessment data at least three times a year to monitor students' progress. Out of 429 school staff who responded, 351 (82%) ranked this statement as a 4 or 5.I am involved in meetings where data results are discussed & problem solving occurs. Out of 444 school staff who responded, 302 (68%) ranked this statement as a 4 or 5.My students move between tiers of support as their academic needs change. Out of 424 school staff who responded, 247 (58%) ranked this statement as a 4 or 5.My students move between tiers of support as their behavior needs change. Out of 410 school staff who responded, 204 (50%) ranked this statement as a 4 or 5.


Discussion What other methods/tools are you using

to obtain data from all school instructional staff?

Is a measure like the School Staff Survey something that would be helpful on your projects?


For More Information:Amy Gaumer Erickson, Ph.D.University of [email protected]


Documents

Measuring Implementation: School-Wide Instructional Staff Perspective Amy Gaumer Erickson, Ph.D. University of Kansas Evaluator: Kansas & Missouri SPDGs