Washington State
Using Data to Drive Statewide Improvement
EffortsGreg Roberts, Evaluation Research Services (ERS)
Leslie Pyper, SPDG Director,
Learning Improvement Coordinator, Special Education, OSPI SPDG Director’s Webinar
November 17, 20101
Part 1 (Greg)Evaluation Approach
Application of Evaluation Approach
Early and Preliminary Findings
Part 2 (Leslie)
Thinking about SystemsLeveraging Evaluation EffortsCreating Systemic Change in WA
2
Evaluation Approach• Theory-driven evaluation• Program theory - what must be done to achieve desired
goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated
• Normative theory and causative theory• Treatment refers to the services, materials, and activities
thought to be essential to generating desired changes.
3
Evaluation Approach• Program model as a vehicle for talking about exceedingly
complex social phenomena• Basis for initially capturing important information• Basis for clearly and usefully reporting evaluation activities
and findings • Means of identifying important research questions• Means of identifying program elements and outcomes for
measurement• Mechanism for Answers for identifying important features of
an implementation environment(s)
4
Evaluation Approach• Causative theory (see figure above) represents empirical
and substantive knowledge of the relationships that link a program’s treatments, its implementation processes, and its intended outcomes.
• Impact theory and impact evaluation are perhaps the most well advertised elements of this framework.
5
Washington State Application
• Program theory/model• Evaluation questions• Evaluation design• Measures, indicators• Procedures
– Survey– Site Visit– Extant Data
6
Evaluation Questions• To what extent are the School-wide Activities in Figure 1
implemented in funded districts?• To what extent do Local Circumstances in Figure 1 inhibit or
facilitate implementation of School-wide Activities?• To what extent are School-wide Activities related to the
Change Mechanisms in Figure 1?• To what extent are School-wide Activities related to
Outcomes in Figure 1?• To what extent are Change Mechanisms related to
Outcomes in Figure 1?
8
Evaluation Questions• To what extent are the School-wide Activities in Figure 1
implemented in funded districts?• What Local Circumstances do districts and schools find
challenging?• What are the 2008-2009 levels of student outcomes in the
funded and matched districts?
9
Early and Preliminary Findings
• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded
districts? • Level of Implementation
11
Early and Preliminary Findings
• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded
districts? • Assessment Knowledge
12
Early and Preliminary Findings
• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded
districts? • Frequency of Assessment
13
Early and Preliminary Findings
• Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded
districts? • Frequency of
Decision-making
14
Early and Preliminary Findings
• Question 2: What Local Circumstances do districts and schools find challenging?– Teachers and others tend to endorse practices that
are RTI-aligned, but mostly when they are not described in terms of RTI
– Relatively limited understanding on what RTI is and how it works, and a number of site visit participants had questions about the degree to which their “model” aligns with research or best practice
– Implementation theory
15
Early and Preliminary Findings• What are the 2008-2009 levels of student outcomes in the
funded and matched districts?– Percentage of students passing the 2008 WASL in Reading and Math, respectively– Baseline for ongoing comparison of changes in student outcomes
16
Ongoing and Next Steps
• Small-scale study in one demo site (Walla Walla)
• Analysis of Year 2 data (2009-2010)• Collection of Year 3 data (2010-2011)• Refocusing program model and evaluation
questions
19
1. Are you working with an evaluator in developing and/or monitoring implementation of RTI in your state?
Feedback Pod
Yes/No
20
“Cheshire,” Alice began rather timidly, “Would you tell me please, which way I ought to go from here?”
“That depends a good deal on where you want to get to,” said the Cat.
“I don’t much care-” said Alice.
“Then it doesn’t matter which way you go,” said the Cat.
Alice’s Adventures in Wonderland by Lewis Carrol
Michigan Dept Ed. ppt 2-16-0921
295 School Districts
Educational Service DistrictsESDs
OSPI (SEA)
Local Control
Legislated as separate entities
Special Ed
SPDG
Demo Districts
State Leadership
Team
All special ed folks
22
Feedback Pod
A. directly by the SEA.
B. by other entities with direct guidance/training from the SEA to those entities. C. by other entities without coordinated influence from the SEA.
2. In my state, regional technical assistance and training is provided …
23
295 districts divided among 9 regional ESDs Leslie = ESD liaison for special ed CSA
Spokane*Spokane**Seattle*Seattle
Improvement planning - by region, using data profiles Root causes (linking activities to root causes)Quarterly updatesEvaluation of activities
Olympia*Olympia*
* May choose to focus some efforts on RTI and/or PBIS…
24
Federal
State
Local Education Agencies
TA System
Teachers/Staff
Effective Practices
System Alignment
Fixsen, 2008 25
Federal
State (OSPI)
LEAs
TA System (ESDs)
Teachers/Staff
Effective Practices
WA System Alignment
Adapted from Fixsen, 2008
IHEsTeacher Prep/
Tech Asst
Special Ed State Needs
Projects
26
Other TA efforts --- State Needs Projects
Needed a “system” to manage these projects and needed a system to provide TA to them.
to identify a “state need” to ensure consistency/alignment to provide training/support to the project staff
Projects are supposed to address a “state need” --
Realized that there wasn’t a process --
27
Contracted with UW-Tacoma to evaluate the State Needs Projects
Are they serving statewide? Are they using resources/materials aligned with State policies and current research? Are they providing on-going support? Are they collecting reliable data? Are they making adjustments based on evaluation of their efforts? Are they having an impact on practice? Are they having an impact on indicator performance? 28
Engage SEA departments/ESDs in strategic planning
Multiple depts + ESDs = Implementation Team
NCRTI – NWRCC – WRRC Need to develop common language Need to align messaging/PD/use of coaches… Need to develop review/vetting process for RTI Develop strategic plan for support
Build a sustainable State structure for RTI
Build capacity of SEA, ESDs to support districts
29
295 School Districts
Coordinated Services Agreement (CSA) w/ ESDs
(regional TA providers)
SEA/ESD Implementation Team
Special EdSPDG
30
OSPI Implementation Team- really should be called the Transformation Team
Teaching & Learning (Reading, Math)Classroom Assessment District & School ImprovementSpecial EducationMigrant/bilingual/ELLTitle I/LAPHighly CapableEquity & Civil RightsSecondary Ed & Dropout PreventionStudent Achievement (Achievement Gap)Ed TechnologyFinancial Resources & Governmental Relations (Data Governance)
Educational Service Districts (ESDs)
31
This would begin to align many efforts at the SEA – which would also impact the ESDs…
Dropout preventionDistrict & School Improvement & Accountability
K12 Reading Model
New Math framework
Comprehensive Assessment System
SPDG32
Implementation Team developed an assessment tool: (OSPI Efforts Inventory)
Assessing activities across SEA (pilot…11/12/10…two departments)
In order to develop a strategic plan for the state, we needed to understand WHAT everyone was doing, and HOW IT CONNECTED to the RTI structure.
33
Focus Area
Essential Components (RTI Framework) OSPI Efforts
Multi-Level System
ScreeningProgress
Monitoring
Data-Based
Decision Making Documents
Profes-sional
Develop-ment
Fund-ing
SourceData
Collected
Core Instruction/
Support
Interven-
tionsReading
Writing
Mathematics
Language
Behavior
Alcohol, Tobacco, or Other Drug UseMental/
Emotional Health
OSPI Efforts InventoryDepartment Contact Person(s) Date
34
3. We have developed common language across the SEA and our resources are aligned and collectively support the RTI framework.
Feedback Pod
True/False
36
engaged IHEs in IRIS Center training recruited IHE reps on State LT (gen & sped) engaged multiple IHEs in regional training meeting with several IHEs re: RTI efforts
Build capacity of IHEs
Build a sustainable State structure for RTI
UW-T 325T grant – participating on Advisory Board Project RTI: Restructuring, Transforming, Implementing a Dual-track RTI Teacher Preparation Program
37
4. We have developed common language across the state (IHEs, SEA, regional TA providers) and our resources are aligned and collectively support the RTI framework.
Feedback Pod True/False
38
Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990)
• Plan – Decide what to do• Do – Do it (be sure)• Study – Look at the results• Act – Make adjustments • Cycle – Do over and over again until the
intended benefits are realized
PDSA Cycles: Trial & Learning
Dean L. Fixsen & Karen A. Blasé, 2009 39
“Knew”:
Lack of understanding of pilot site requirements Lack of consistent PD opportunities Lack of a state structure for PDLack of involvement of SEA beyond sped
o lack of understanding about connectionso lack of alignment o no vetting of materials
Lack of common understanding of RTI framework Lack of data system to manage RTI efforts Lack of preparation for leadership of RTI efforts Likely a lack of fidelity
40
Evaluation results –
1-2 PD opportunities (avg) More than half gave incorrect response or no response regarding defining screening & progress monitoring Collecting data but teachers struggling to use data Basic knowledge of RTI lacking Lack of buy-in Difficulty moving students through tiers Difficulty in using staff to provide interventions Is anyone monitoring fidelity?
41
So, need to develop a system to support demo sites and all others who want to implement RTI:
Develop a state structure for support of RTI develop common language develop a strategic (state-level) plan align efforts (messaging, resources, training,…)
Purchase/develop a data system for demo sitesProvide ‘hands on’ support to demo sites
42
Screening /progress monitoring Selecting EBPs Adapting on-site technical assistance Developing monthly TA calls (in addition to quarterly mtgs) Developing a Guide for Selection of EBPs “ “ Guide for Selection of Trainers/materials Regional training – IHEs, ESD, multiple SEA staff Using integrity rubric
Training for RTI demo sites
43
Build a data system
WA received $17.3 million for longitudinal data system -- requested meeting with data folks (SEA) -- have been meeting with Student Information Director and Data Governance Coordinator (SEA) -- also invited DSIA, Classroom Assessment
Spectrum K12Demo for Implementation teamReviewing training plans Meeting to draw up a “proof of concept” system
44
Supports for Implementation
RTI Coordinator mtgs (quarterly)Site visits (technical assistance)Will begin monthly contacts to provide additional supports
Developing a series of trainings for all(will consist of face-to-face, webinar, and recorded sessions)
45
How did you use your data this past year?
Student-level dataBuilding-level dataDistrict-level data
Protocols in place?
How is your data system driving implementation in your district???
Questions for RTI Coordinators
46
5. My SEA used data this past year to:
Feedback Pod
A) improve content, delivery, format, or target audience of PDB) revise guidance on RTI frameworkC) influence policy-makers, TA providers, SEA divisionsD) improve data collectionE)determine funding efforts F) revise evaluation effortsG)obtain technical assistance/support H) other: _____________________
Check all that apply
47
Intensive Technical Assistance w/ the National Center on RTI (MOU)
•Common language •Alignment across efforts •Build capacity at the SEA, IHEs, ESDs, & districts to support implementation of evidence-based practices (EBPs)•Data system for demonstration districts (expandable)•Information dissemination system to expand RTI efforts across universities, districts and professional organizations
48
Effective State Support Structures for RTI Include…
• Leadership, consensus, and aligned vision • Consistent messaging across departments• Capacity-building supports for systems that
support LEAs/schools (IHEs, ESDs), and for LEAs and schools directly
• Collection and use of data to inform decisions at all levels (student, classroom, school, LEA, SEA)
• Regulatory language, in general and/or special education, as appropriate, to support RTI
From Innovations conference, 201049
Invited to participate in the SISEP Community of Practice
Putting together a team of folks from the SEA & ESDs --- - from Implementation team - from CSA group
Our CSA group is now looking at “scaling up evidence-based practices”
50
Presenting to Cabinet 11/22/10 Putting the puzzle together for them:
info on efforts of multiple depts (including pilot data - OSPI Efforts Inventory) connecting RTI to the Superintendent’s Five Strategic Priorities # districts implementing across the state CSA work
>50% report implementing RTI
51
152 (51.5%) districts report implementing RTI
142 Reading – Elem (48.1%) 79 Reading – Middle (26.8%) 42 Reading – High School (14.2%) 98 Math – Elem (33.2%) 67 Math – Middle (22.7%) 37 Math – High School (12.5%) 69 Behavior – Elem (23.4%) 44 Behavior – Middle (14.9%) 26 Behavior – High School (8.8%)
2009-10 school year
52