Slide 1

Response to Intervention: Using Data to Enhance Outcomes for all Students Amanda VanDerHeyden Education Research and Consulting, Inc. Slide 2 16 x 3 = 48 hours Slide 3 Data allow us to Provide faster, more effective services for ALL children Work smarter not harder, better utilize the talents of the school psychologist and school- based assessment and intervention teams. Make implementation SIMPLE and EASY for teachers (low cost, few errors) Prevent diagnosis Slide 4 RTI The systematic use of assessment data to most efficiently allocate resources in order to enhance learning for all students (Burns & VanDerHeyden, 2006).The systematic use of assessment data to most efficiently allocate resources in order to enhance learning for all students (Burns & VanDerHeyden, 2006). Slide 5 Essential Elements Screening Intervention intensity matched to child need Progress monitoring Outcomes of intervention efforts linked to service allocation decisions and program evaluation Slide 6 Why RTI in EC/EI? Adults embrace the idea that early intervention is meant to repair and prevent future learning and behavior deficits and excesses Less learning history available Technical adequacy of measures is problematic (Neiworth & Bagnato, 1992) Slide 7 Ideas that have Failed us Children will outgrow early skill deficits Waiting/delaying intervention Retention Transition classes Slide 8 Ideas that have Failed us Providing an enriched environment is sufficient to meet the needs of all typically-developing children Slide 9 Measures of Cognitive Ability Not Useful at K and 1st Children who can detect/manipulate rhymes, phonemes, or syllables learn more quickly to read irrespective of IQ, vocab, memory, and SES (Wagner et al., 1994). Measures of cognitive ability were not useful for reaching screening decisions or allocating instructional resources (Vellutino, 1996) Slide 10 Direct Msrs of Early Literacy were Useful Letter-sound association and use of sounds to read words strong predictor of reading achvmt at end of first grade (Smith, Simmons, & Kameenui, 1998). Direct msrs of early reading skills were useful (Vellutino, 1996) Slide 11 Science of Prevention Entering phonological ability, SES, and attention/behavior strongly predict reading growth (Torgesen et al., 1999). Torgesen et al. (2001)- 40% of children exposed to intensive intervention were exited from special education within one year of completion (60 8-10 year old children identified with LD) More than half of children provided with intervention at 1st grade performed in the average range following intervention (Vellutino, 1998). Slide 12 Early Math Findings Children are not attaining minimal standards for competency in math by end of formal schooling. Children in poverty disproportionately represented (Griffin & Case, 1997; Starkey, Klein, & Wakeley, 2004). Early intervention repairs and prevents (Fuchs, Fuchs, & Karns, 2001; Griffin & Case, 1997; Phillips, Fuchs, Fuchs, & Hamlett, 1996). Slide 13 Consensus to Prevent most reading problems by reducing the # of children who enter school with poor emergent literacy skills (oral language, print knowledge, phonological processing skills)- NRP Prevent early mathematics deficits by screening, providing intervention in early numeracy Permit school success by proactive and early training in ready to learn behaviors These are not new ideas to EC/EI Slide 14 Lets abandon the ideas that have failed us Emphasis on Maturation Lets intervene early Emphasis on Enriched Environment Lets ensure sufficient opportunities to gain important skills Emphasis on Teacher and Parent self-report and checklists Lets obtain direct measures of child performance at regular intervals and in response to intervention trials Slide 15 Identification Accuracy CBA + RTI CriterionITBSWJ-R STEEP Sensitivity.761.58 Specificity.89.99.77 Positive Predictive Power.59.67.44 Negative Predictive Power.951.86 Teacher Referral Sensitivity.46.33.42 Specificity.69.94.85 Positive Predictive Power.19.17.45 Negative Predictive Power.89.97.83 VanDerHeyden, et al., 2003 Slide 16 Team Decision-Making VanDerHeyden, Witt, & Gilbertson, 2007 Slide 17 System Outcomes Referrals reduced greater than half % who qualify improved at 4 of 5 schools SLD down from 6% of children in district in 2001-2002 (with baseline upward trend) to 3.5% in 2003-2004 school year Corresponding gains on high-stakes tests (VanDerHeyden & Burns, 2005) Intervention successful for about 95 to 98% of children screened VanDerHeyden, Witt, & Gilbertson, 2007 Slide 18 Cost Reduction VanDerHeyden, Witt, & Gilbertson, 2007 Slide 19 What Proportion of Ethnicity Represented Before and After Intervention in Risk Category? VanDerHeyden & Witt, 2005 Slide 20 Great Implementers Follow the aimline and attend to implementation integrity Understand the variables of effective instruction and engage in contextualized assessment that is technically valid for the purposes needed AND has treatment utility Minimize meeting time and avoid the science of strange behavior Provide adequate resources and space for principals to be effective instructional leaders and hold them accountable for results Evaluate quality of all programs locally and make decisions about continued use based on DATA. Slide 21 RTI in Early Childhood Amanda VanDerHeyden Education Research & Consulting, Inc. Slide 22 CBA or Mastery Model Slide 23 CBM or General Outcome Slide 24 CBA versus CBM Target: Drinking from cup at snack time Hypothesis: grasping problem Intervention: alter handle to make grasping easier, provide practice using adapted cup, reinforce correct cup use Alternative hypotheses are plausible that can be tested directly Skill related-- child cannot bring cup to mouth fluently Performance related-- child is not thirsty, does not like the drink offered, finds spilled liquid aversive Slide 25 Was intervention effective? Baseline Adapted Cup Slide 26 Did you solve the problem? BaselineAdapted Cup Slide 27 How about now? BaselineAdapted Cup Slide 28 BaselineAdapted CupPreferred Liquid Slide 29 We should Measure the behaviors we are targeting and measure whether those effects contribute to improved general outcomes Assessment has too narrowly focused on individual sub-skills generally in the form of a developmental checklist Slide 30 Response to Intervention: Using Data to Enhance Outcomes for all Students Amanda VanDerHeyden Education Research and Consulting, Inc. Slide 31 Grade level corresponding to age 1 2 3 4 Reading grade level 4 3 2 1 5 2.5 5.2 At Risk on Early Screening Early Screening Identifies Children At Risk of Reading Difficulty Low Risk on Early Screening This Slide from Reading First Experts J From Reading First Slide 32 Grade level corresponding to age 1 2 3 4 Reading grade level 4 3 2 1 5 2.5 5.2 Early Intervention Changes Reading Outcomes At Risk on Early Screening Low Risk on Early Screening 3.2 Control With research- based core but without extra instructional intervention 4.9 Intervention With substantial instructional intervention This Slide from Reading First Experts J From Reading First Slide 33 Evolution Wait to Fail Lets provide services early! Costly sp ed programs not improving learning Lets shift resources to provide services in less restrictive setting! Increasing numbers of children struggling in general ed Lets provide help in general education! Traditional measures are de-contextualized and the constructs are problematic Lets help children who struggle academically by measuring performance in response to certain intervention strategies and then deliver what works! Slide 34 Rationale for System Change 200-300% increase in SLD Level and Rate of Performance Return to General Education Lack of Certified Teachers No demonstrated instructional techniques that differentially benefit SLD Drop-out Disproportionate Representation by Ethnicity Slide 35 Impetus Faster, more effective services for ALL children Work smarter not harder, better utilize the talents of the school psychologist and school-based assessment and intervention teams. Make implementation SIMPLE and EASY for teachers (low cost, few errors) Slide 36 Improved Identification Accuracy for LD Under RTI models intervention becomes a specified, operationalized variable, thus false positive identification errors should be reduced dramatically. Removing the current reliance on teacher identification and requiring direct measures of child performance in context will enhance identification accuracy. Slide 37 More Effective Intervention RTI is likely to facilitate less restrictive interventions and placements for children. RTI allows school psychologists to bring their expertise to bear on assessment strategies at the classroom level and assist teachers to use data formatively to enhance their instructional programming. Slide 38 Possible Challenges of RTI Slide 39 Decision-making Criteria Must be operationalized and validated through research The purpose of RTI will be critical to determining how implementation should proceed Slide 40 New Challenges for Teams Effective intervention delivery will depend on relevant intervention variables To be effective the intervention must have been: Properly identified Implemented with integrity and with sufficient frequency, intensity, and duration Slide 41 STEEP Model Screening to Enhance Educational Progress Slide 42 Screening tells you How is the core instruction working? What problems might exist that could be addressed? Most bang-for-the-buck activity Next most high-yield activity is classwide intervention at Tier 2. Slide 43 Tier 1: Screening Screening Math Screening 2 minutes. Scored for Digits Correct Writing Screening 3 Minutes. Scored for Words Written Correctly Reading Screening 1 Minute. Scored for Words Read Correctly Slide 44 Tier 1: Math Screening Math Probe: Group administered. Materials: Worksheet consisting of a series of problems sampling the target skill(s) (e.g., sums to 5, double digit multiplication with regrouping). Timing: 2 minutes Information obtained: digits correct in two minutes. Slide 45 Math Probe Example Total Digits: 38 Errors: 5 Digits Correct: 33 Slide 46 Writing Probe: Group administered. Materials: story starter (e.g., If I had a million dollars) printed at the top of a blank page. Timing: 1 minute to think, 3 minutes to write. Scoring: words written or correct word sequences in three minutes. Tier 1: Writing Screening Slide 47 Writing Example Slide 48 Tier 1: Reading Screening Reading Probe: Individually administered Materials: A content-controlled reading passage. Procedure: The student reads aloud as the teacher listens and records errors. Timing: 1 minute Information obtained: words read correctly in one minute. Slide 49 CBM Reading: Sample Scoring TRW=63 Errors=6 CRW=58 Slide 50 Screening Guidelines Efforts at Tier 1 pay off with fewer children needing individual intervention 3 times per year, single probe Use small team of trained coaches Prepare all needed materials in a packet for each teacher Score and return within 1 week on graph Use data to generate aimlines, can be used to set benchmarks Slide 51 Class-wide Screening Slide 52 Feedback to Teachers Slide 53 Tier 2: Class-wide Intervention Slide 54 No Class-wide Problem Detected Slide 55 Tier 2: Cant Do/Wont Do Assessment Cant Do/Wont Do Individually-administered Materials Academic material that student performed poorly during class assessment. Treasure chest: plastic box filled with tangible items. 3-7 minutes per child Slide 56 Cant Do/Wont Do Assessment Slide 57 Decision Rule Following Cant Do/Wont Do Assessment Slide 58 Tier 3: Individual Intervention Slide 59 Response to Intervention Before Intervention During Intervention Avg. for his Class Intervention in Reading #Correct Intervention Sessions Each Dot is one Day of Intervention Slide 60 Before Intervention During Intervention #Correct Avg. for his Class Response to Intervention Slide 61 Instructional range Frustrational range Vehicle for System Change: System-wide Math Problem Each bar is a students performance Slide 62 Re-screening Indicates No Systemic Problem Fourth Grade Slide 63 Rest of Grade at Standard ABCDEFClassroom Slide 64 Spring 2003 Classroom F F Slide 65 Teacher moved to lower grade in Fall 2003 Slide 66 Class-wide Intervention Teacher F Mult 0-12 0 20 40 60 80 100 120 10/24/200310/31/2003 11/7/2003 11/14/200311/18/2003 Weeks Digits Correct Two Minutes Slide 67 Increased Difficulty- Intervention Continues Slide 68 Mixed Mult/Div/Fractions Probe Classroom F Slide 69 Academic SystemsBehavioral Systems 1-5% 5-10% 80-90% Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Universal Interventions All students Preventive, proactive Universal Interventions All settings, all students Preventive, proactive Any Curriculum Area Students Dave Tilly, 2005 Slide 70 Class-wide Intervention Use pair-peered practice (classwide peer tutoring, PALS) Model, Guided Practice, Independent timed practice with delayed error correction, reward contingency Slide 71 Acquisition Fluency Generalization Instructional Hierarchy To gain the steepest growth, introduction of new skills should happen here Core Instruction- Not manipulated But fluency building should happen here with an instructional level skill Intervention Focus was here Finally, problem-solving/ application practice should occur here with a mastery level skill Core Instruction- Not Manipulated but could be Slide 72 Slide 73 Reading classwide intervention Slide 74 Select a Few Good Interventions to Keep it Simple ClasswideIndividual MathFlash card Practice Cover copy compare Cue Cards Highlighted errors ReadingListening PreviewRepeated Readings Error Correction Key Words Slide 75 Slide 76 Slide 77 Intervention Plan- 15 Min per Day Protocol-based classwide peer tutoring, randomized integrity checks by direct observation Model, Guide Practice, Independent Timed Practice with delayed error correction Group performance contingency Teachers encouraged to Scan papers for high error rates Do 5-min re-teach for those with high-error rates Provide applied practice using mastery-level computational skill Slide 78 Usually the higher-level reader, reads (models) first. Rotating high level readers helps maintain motivation Slide 79 Intervention Plan Class Median reaches mastery range for skill, next skill is introduced Following promising results at one site in 2002-2003, lead to implementation district- wide grades 1-8 for all children by 2004- 2005. Slide 80 Class-wide Math Intervention Slide 81 With teacher support Consider time, resources, materials Remove skill barriers with classroom training for students classroom coaching for teachers Remove implementation barriers after use new steps follow-up supportive meetings to problem solve. frequent acknowledgment of a teachers efforts Slide 82 Unsupported means. Team recommends intervention. All materials have to be created from scratch Teachers sent to organize class and train No one may be an expert to help the teacher No or not enough frequent objective data to collect or interpret Slide 83 Tell Rational Step by step protocol Show Model Do Train students Implement with guided practice Implement independently with support Training Package Slide 84 Time consuming Complex not yet fluent Cant Do Poor management No child change Wont do No reinforcement for teacher behaviors No teacher change prevents Address Common Reasons for Resistance Lack of materials prevents Too much work avoidance prevents Slide 85 88% of interventions are not used without support Decisions do not always correspond to data (someone must check) Slide 86 Tier 2 Intervention Effects Slide 87 Class 1 at Screening Slide 88 Class 1: Following 10 Days Intervention Slide 89 Class 1: Following 15 Days Intervention Slide 90 Class 2 at Screening Slide 91 Class 2: Following 5 Days Intervention Slide 92 Class 2: Following 10 days Intervention Slide 93 Class 3 at Screening Slide 94 Class 3: Following 5 days Intervention Slide 95 Following 10 Days Intervention Slide 96 Contextually-Relevant Comparisons and Use of Trend Data Slide 97 5 th Grade Math Intervention Slide 98 Instructional range Frustrational range Pre-post changes to performance detected by CBM Each bar is a students performance Slide 99 Fourth Grade Slide 100 Questions Is there a classwide problem? Is there a gradewide problem? Whats the most efficient way to deliver intervention? Slide 101 Academic SystemsBehavioral Systems 1-5% 5-10% 80-90% Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Universal Interventions All students Preventive, proactive Universal Interventions All settings, all students Preventive, proactive Any Curriculum Area Students Dave Tilly, 2005 Slide 102 Tier 3 Assessment Data Instructional level performance Error analysis (high errors, low errors, pattern) Effect of incentives, practice, easier task Verify intervention effect Same implementation support as Tier 2 Instructional-level materials; Criterion-level materials Slide 103 Tier 3 Implement for 5-15 consecutive sessions with 100% integrity Link to referral decision Weekly graphs to teacher and weekly generalization probes outside of classroom, supply new materials Troubleshoot implementation weekly Slide 104 Lessons Learned Most individual interventions for reading Standard protocols with slight modifications are best Most interventions are successful (should be successful) Generalization must be attended to Team will not follow data without support and training to do so Coordinate intervention start times with principal and stagger start dates (10-15 at a time plus Tier 2s). Organize master schedule for data collection and Tier times Slide 105 Tier 3 Intervention >5% of children screened (total population) IF solid Tier 1 Possibly as low as 2% IF solid Tier 1 and Tier 2 About 1-2% failed RTI; 10% of most at-risk Slide 106 Successful RTI Slide 107 Slide 108 Successful Writing Slide 109 Successful RTI Slide 110 Successful Math Slide 111 Unsuccessful Math Slide 112 Digits Correct Two Minutes Weeks 121 Pass the AIMS Slide 113 Here is 3 rd grade at Cottonwood. Each circle corresponds to a students score on a reading CBM probe in March and the AIMS reading score the same month. So the circle near the blue lines is a child who read 158 wc/min and scored 486 on the AIMS Reading. The diagonal line represents the best fit line or the line closest to all the circles. This shows there is a strong positive correlation between CBM and AIMS reading scores with this group. Slide 114 How to Set a Benchmark 431 = pass for AIMS. This line does not move This is words read correctly per minute. You move this line up and down to catch as many of those who will not pass as possible. Slide 115 Setting 95 wc/min as the pass standard for CBM These are children who were predicted to pass AIMS based on CBM and did pass. Hits These are the children who were predicted to fail AIMS based on CBM who did fail. Hits These are the children predicted to fail AIMS who actually passed. False Positive Errors. These are the children predicted to pass AIMS who actually failed. False Negative Errors. The worst kind of error. Slide 116 Moving the horizontal line up will catch two more cases who failed the AIMS, but will result in many more false positive errors. Thus, 95 wc/min at 3 rd grade at Cottonwood is a good standard that will tell you which children are likely to fail the AIMS reading section. Slide 117 ROC printout Slide 118 Mastery Model Measurement (CBA) Letter naming fluency Isolated sound fluency Beginning sound fluency Ending sound fluency Slide 119 General Outcome Measurement (CBM) Words read correctly per minute Letter naming fluency Isolated sound fluency Beginning sound fluency Ending sound fluency Slide 120 Tracking Year-Long Growth aimline mastery instructional Slide 121 Tips for Effective Implementation Slide 122 Our Recipe for Intervention Success PREPARE Identify and Use standard protocols for intervention Develop all needed materials Develop packets or put on a central web site Determine graphing program Slide 123 Our Recipe for Intervention Success TRAIN Explain Watch the teacher do it with the actual child before you leave Call or meet teacher after first day to problem solve Slide 124 COLLECT DATA AND SUPPORT Each week, graph intervention performance and do a generalization check with the child. Graphed feedback to teachers with generalization checks for individual intervention once per week Response-dependent performance feedback to sustain implementation accuracy Monthly CBM to track growth and enhance existing Tier 1 Programs or advise new Tier 1 Data to principal weekly. Summarize effects and integrity of procedures. Our Recipe for Intervention Success Slide 125 Tracking Title Progress Slide 126 Our Recipe for Intervention Success USE DATA TO MAKE DECISIONS RTI successful if child performs criterion-level probe (from screening) in the instructional range. RTI unsuccessful if 15 consecutive intervention sessions and criterion probe is not in the instructional range. Increase task difficulty for intervention if child scores at mastery on task during intervention sessions Slide 127 Infrastructure for Implementation Grade-level planning periods can be utilized Special education team at school can be utilized School Psych must be on-site 1 day/week Developing master schedule for Tier 1, 2, and 3 intervention times is useful Integrate efforts with evaluation referral team efforts (consider major reduction in meeting time and shift to intervention efforts!) Slide 128 Materials needed Computer and software to organize data Student data imported. Clerical person to enter data on-site for tier 1 screen only. Color printer to print graphs + extra color cartridges Probe materials, digital count-down timers Intervention protocols, intervention materials (e.g., flashcard sets, reading materials) Access to copier and some assistance with copying Reinforcers for treasure chest (no more than $500 per school) Slide 129 Guidelines for Implementers Use single trial scores for screening Following screening, grade-wide graphs to principal Return data to teachers within 48 hours with personal interpretation at grade-level team meeting Include principal in critical meetings Involve teachers at all stages Slide 130 Guidelines for Implementers Learn about curriculum and instruction. Integrate RTI with ongoing school and system reform efforts Thoughtfully merge to subtract duplicate activities and to enhance more comprehensive supplemental and core instructional support activities that may be in place Use RTI data to evaluate the value of ALL instructional programs or resource allocation decisions. Quantify bang for the buck using student performance data. Slide 131 Lessons Learned from Vail USD Infrastructure for education being a results- based enterprise Accountability Principal is the Instructional Leader of a school Principal as change agent School psychologist as change agent Replace resources/substitute dont add Minimize meetings Track outcomes that matter Slide 132 Great Instructional Leaders Have a filter Allocate time and resources according to their filter Use an AIMLINE Have a framework for making data-driven decisions (know how to access the data they need to reach timely decisions) Hold teachers, staff, students accountable Research findings on effective schooling Slide 133 Great School Psychologists Hand the principal the data the principal did not know to ask for but cant live without Follow the aimline and attend to implementation integrity Understand the variables of effective instruction and engage in contextualized assessment that is technically valid for the purposes needed AND has treatment utility Minimizes meeting time and avoids the science of strange behavior Slide 134 Great Districts Minimize time away from school, but use time together to review school improvement implementation efforts and ongoing results Have the will to proactively chart the course of a district Provide adequate resources and space for principals to be effective instructional leaders and hold them accountable for results Respect the role of parents and actively engage them Have a framework for evaluating results (know how to access data for decision making) Evaluate quality of all programs locally and make decisions about continued use based on DATA. Slide 135 Great Teachers Use data to identify where more/different/less instruction is needed Have as a goal to accelerate all learning of all children Proactively address barriers to learning Take responsibility for learning that occurs in the classroom Are confident and ready to collaborate in the classroom Appreciate childhood and children (a little humor, lots of patience, enthusiasm) Slide 136 How To Do Classwide Intervention within RTI Amanda VanDerHeyden Education Research and Consulting, Inc. Slide 137 Slide 138 Slide 139 In Low-Achieving Classrooms (more than 50% of class scored in the frustrational range on probes) STEEPTeacher Referral Sensitivity.75.55 Specificity.88.68 Positive Predictive Power.69.35 Negative Predictive Power.91.82 VanDerHeyden & Witt, 2005 Slide 140 In Average to High-Achieving Classrooms (less than 20% of class scored in frustrational range on probes) STEEPTeacher Referral Sensitivity.670 Specificity1.0.67 Positive Predictive Power 1.00 Negative Predictive Power.97.95 VanDerHeyden & Witt, 2005 Slide 141 Identification Accuracy High-achieving classrooms (50%) Procedures paired with RTI criterion were more accurate than other commonly used screening devices VanDerHeyden & Witt, 2005 Slide 142 Research Finding VanDerHeyden, Broussard, et al. (2004). Prior to single instructional session, children receiving special ed services were significantly lower performing on math probes. Following single instructional session, no significant difference was observed. Slide 143 Mixed Mult/Div/Fractions Probe Classroom F Slide 144 Sore Thumb Test Slide 145 Response to Classwide Intervention Student A Slide 146 Individual Math Intervention Cant Do Problem Slide 147 Successful Math Intervention BaselineIntervention Novel, grade-level probe Slide 148 Why do Classwide Intervention Efficiency Accuracy Efficacy Slide 149 Rest of Grade at Standard ABCDEFClassroom Slide 150 Using Screening Data to Identify Class-wide and System-wide Instructional Problems Slide 151 Consider The Task Integrity of Administration Reliability of Scoring Use software to organize the data Slide 152 Mult 0-9 4 th Grade Fall Screening Slide 153 Mult/Div/Fractions 4 th Grade Winter Slide 154 Grade-wide Data Slide 155 Slide 156 Slide 157 3 rd Grade Mult 0-9 Spring Slide 158 Guided Practice Slide 159 Fourth Grade Reading Level: Math Skill 1: Math Skill 2: Slide 160 Questions Is there a classwide problem? Is there a gradewide problem? Whats the most efficient way to deliver intervention? Slide 161 Slide 162 Slide 163 Slide 164 Slide 165 What Data do you Want for Principal? Slide 166 Slide 167 Slide 168 Slide 169 Questions Is there a classwide problem? Is there a gradewide problem? Whats the most efficient way to deliver intervention? Slide 170 Independent Practice Slide 171 First Grade Reading What do you want to know? Is there a class-wide problem? Is there a grade-wide or systemic problem? Whats the most efficient way to deliver intervention? (whole class, small group, individual) What is the next step for Class 1, 2, 3, 4? Slide 172 Class 1 Slide 173 Class 2 Slide 174 Class 3 Slide 175 Class 4 Slide 176 Grade-wide Data Slide 177 Slide 178 Class 4 Slide 179 Prepare for Training Day Locate probes (e.g., worksheet factory, intervention central, basic skill builders) Identify Graphing Program (excel) Locate or develop scripts (gosbr.net; interventioncentral.org) Develop quick access to materials for teachers Determine integrity monitoring plan Identify common time for intervention Set start date Slide 180 Set a daily routine. Time, location of materials, process for weekly assessment. Set a date and time for 30-min training Set a date for a later 15-minute first practice time with teacher Slide 181 Materials needed Computer and software to organize data Student data imported. Clerical person to enter data on-site for tier 1 screen only. Color printer to print graphs + extra color cartridges Probe materials, digital count-down timers Intervention protocols, intervention materials (e.g., flashcard sets, reading materials) Access to copier and some assistance with copying Reinforcers for treasure chest (no more than $500 per school) Slide 182 Measurement Plan Weekly probe of Intervention skill Weekly probe of Retention of previously mastered computational skills Monthly probe using GOM approach to monitor progress toward year-end computational goals To this you might add an application measure Slide 183 Intervention Plan Class Median reaches mastery range for skill, next skill is introduced Following promising results at one site in 2002-2003, lead to implementation district- wide grades 1-8 for all children by 2004- 2005. Slide 184 Provided teacher/student a script that tells. what the student has to do and when what the teacher should do to support student how the student will know how he/she is doing Treatment considerations for integrity issues All steps are clearly needed Includes lots of student response opportunities Disrupts class as little as possible Requires little teacher time ( < 15 min/day) Considers resources to decrease teacher effort Used simple language All the materials are available Rationale Slide 185 Trainer 1.Observe the teacher using the steps on the intervention script 2. Check off steps used. 3. Prompt the teacher to do any missed step. 4. Problem Solve any noted blockers 5. Continue until accurately implemented without prompts Slide 186 Lack of practice with feedback Non-specific steps In adequate classroom management Not enough child assistance for bx change Lack of reinforcement For teacher behaviors Low implementation Low frequencies interferes prevents Why verbal and modeling training alone do not work: Inadequate materials prevents No instructions when problems arise Slide 187 Math Partners Progress Chart Count every digit that is not circled. This is your score! Write your score on your math sheet. Find todays date on this page and write your score on the line. Put a Star on the graph to mark todays score. Slide 188 Weekly Progress Monitoring Administer classwide math worksheet Target skill once per week Criterion skills periodically to monitor growth Use incentives to maximize performance Apply decision rules Slide 189 Progress Review Review folders to ensure that intervention was used correctly for at least 4 days that week If this is not the case, conduct another in- class training day. Graph weekly progress monitoring assessment data Slide 190 Decision making Review data to make decisions: DATA OUTCOME 1: Class median is below mastery range and most students gaining digits correct per week. ACTION: Consider implementing intervention for an additional week and then review progress again. Slide 191 Decision making DATA OUTCOME 2: Class median is below mastery range and most students are not gaining digits correct per week: ACTION: Check Integrity first and address with training if needed. Consider implementing intervention for an additional week with incentives or easier task and then review progress again. Slide 192 Decision making DATA OUTCOME 3: If the class median is above mastery range then consider: ACTION: Increasing task difficulty and continuing classwide intervention. ACTION: For students performing in the frustration range, consider Tier 3 assessment and intervention. Slide 193 Troubleshoot Intervention SupportYesNo Was the intervention developed to ensure that it required minimal classroom time and resources and fit within daily classroom routines? Are materials readily available to the teacher? Was a step-by-step coach card provided? Was the teacher shown how to implement the intervention by a coach? Did the coach observe implementation of the intervention to ensure that the teacher could use the intervention correctly and had all needed materials? Was weekly follow-up support provided to the teacher after initial training? Are integrity data graphed to show used correctly? Is an administrator involved? Slide 194 Results Slide 195 Tier 1 Screening Indicates Class- wide Problem Slide 196 Tier 2: Class-wide Intervention Teacher F Mult 0-12 0 20 40 60 80 100 120 10/24/200310/31/2003 11/7/2003 11/14/200311/18/2003 Weeks Digits Correct Two Minutes Slide 197 Contextually-Relevant Comparisons and Use of Trend Data Slide 198 Slide 199 Instructional range Frustrational range Pre-post changes to performance detected by CBM Each bar is a students performance Slide 200 Fourth Grade Slide 201 Additional Research Questions What level of performance predicted strongest subsequent growth given intervention? What level of performance predicted skill would be retained about 3 months after it was taught? Did mastery of foundation skills shorten the number of trials required to master more complex related skills? Slide 202 What level of performance predicted strongest subsequent growth given intervention? Across 4 weeks of intervention (4 datapoints) OLS used to estimate slope Children achieving slopes equal to or greater than the 66 th percentile were identified as strong responders Starting fluency (prior to intervention) was identified for the group of strong responders and range was estimated as average starting fluency +/- 1 standard deviation Tested new criterion on second set of scores Burns & VanDerHeyden, 2006 Slide 203 General Findings Growth rates and trials to criterion varied dramatically across skills Retention probe was strongest predictor of year- end SAT-9 performance Mastery level performance on early skills predicted fewer trials to criterion on future related complex skills Fluency scores higher than mastery predicted retention of skill over time (about +20 dc/min) VanDerHeyden & Burns, 2008; VanDerHeyden & Burns, in submission Slide 204 Identification Accuracy CBA + RTI CriterionITBSWJ-R STEEP Sensitivity.761.58 Specificity.89.99.77 Positive Predictive Power.59.67.44 Negative Predictive Power.951.86 Teacher Referral Sensitivity.46.33.42 Specificity.69.94.85 Positive Predictive Power.19.17.45 Negative Predictive Power.89.97.83 VanDerHeyden, et al., 2003 Slide 205 Percent Identified at each Tier Identified CBM (Classwide Assessment) 55 (15%) CBM + Reward (Performance/skill Deficit Assessment) 40 (11%) CBM + Reward + Instruction (STEEP +) 22 (6%) Teacher Referral 32 (19%) CIBS-R 64 (18%) DRA 17 (9%) RTI Criterion Assessment 17 (5%) WJ-R 12 ITBS deficit 3 (4%) VanDerHeyden, et al., 2003 Slide 206 Academic SystemsBehavioral Systems 1-5% 5-10% 80-90% Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Universal Interventions All students Preventive, proactive Universal Interventions All settings, all students Preventive, proactive Any Curriculum Area Students Dave Tilly, 2005 Slide 207 To work smart, we must ask What is the purpose of our assessment? How do we know it serves our purpose? Is this the cheapest way to do it? Slide 208 Our Goal Collect the best information in the shortest possible period of time Slide 209 Strategy in a Nutshell Identify the goal (DVs) Behavior to increase (fluency, comprehension) Behavior to decrease (errors) Match the strategy to the goal (Daly et al., 1996) Monitor the DVs and the IVs (intervention variables) Slide 210 Find instructional level (sampling back) Identify the root of the problem (e.g., division is difficult because subtraction is not fluent or multiplication is not fluent; poor decoding skills v. dolce words; production v. accuracy in writing) This is the most important part of the process Slide 211 Accurate Letter Naming Fluent Letter Naming Association of Letters with phonemes Accurate Letter Sound Production Fluent Letter Sound Production Pronounce beginning word sounds Define the Behaviors/skills Slide 212 Identify Reinforcers and Logical Consequences Use a treasure chest Use an activity survey or reinforcer checklist Use incidental teaching strategy Use logical or natural consequences Slide 213 Measure Baseline Performance and Set Goals Slide 214 Functional Assessment What is an effective intervention? Slide 215 Functional Assessment Slide 216 BLInterventionPerformance Feedback Slide 217 Troubleshooting Intervention Effects at Tier 3 Slide 218 Teachers must weigh the following What outcomes does not completing work produce? Escape from assignment, from classroom setting Peer attention Adult attention (even if it is negative). Some students are so motivated to obtain adult attention that it does not matter if the attention is negative or positive. What outcomes does completing work produce? Positive feedback from the teacher Positive attention from peers, status Access to fun activities or reinforcement Avoidance of punitive consequences Slide 219 Antecedent Variables Task Difficulty, Sequencing of Skills Time actively engaged in learning (AET) Opportunities to respond Other lesson variables (pacing, exemplars) Behaviors interfering with instruction (teacher and child) Slide 220 Task Difficulty, Sequencing Sample back measuring fluency of performance on basic skills The idea is to identify the weak point in the chain Define the target skill for intervention and the criterion skill (goal) Slide 221 Academic Engaged Time Impacts opportunities to respond Robust predictor of achievement Average 2 nd grader (Rosenshine) spent less than 1 hour AET per day. Check transitions, classroom management, time allocated for independent practice, active monitoring/scanning Slide 222 Other Lesson Variables Presentation of materials and Sequencing of Lesson Organized Clear, redundant examples Exemplars sufficient S+ and S- Checking for student understanding Pacing of lesson Slide 223 Behaviors Interfering with Instruction/Intervention Teacher behaviors Implementation accuracy and consistency Fuchs & Fuchs, 1987; Gresham, 1991; Happe, 1982; Wickstrom, Jones, LaFleur, & Witt, 1998 Teacher understanding/adequately trained Train to fluency criterion (Chandler, Lubeck, & Fowler, 1992) Teacher acceptability of intervention (prospective, ongoing, link to changes) Adequate resources to conduct intervention Slide 224 Behaviors Interfering with Instruction Child behaviors Disruptive or inattentive behaviors Concurrent options available (access to reinforcing outcomes by not completing intervention) Consider cant do/wont do (although programming for motivation is important anyway) Slide 225 Consequences Reinforcing consequences (for correct and incorrect performance) Escaping task Extra attention (staying in at recess may be reinforcing) Feedback Frequency Immediacy Accuracy Correct error immediately, have student repeat response correctly, match response to instructional situation & learner (Heubusch & Lloyd, 1998) Slide 226 Tier 3 Intervention >5% of children screened (total population) IF solid Tier 1 Possibly as low as 2% IF solid Tier 1 and Tier 2 About 1-2% failed RTI; 10% of most at-risk VanDerHeyden et al., 2007 Slide 227 Tier 3 Findings Most interventions for reading Math is next Math is at least two-dimensional--- computational and operational fluency plus application or conceptual understanding Most interventions are not implemented well and thats why they fail Tier 3 interventions are likely to occur on below grade level tasks AND require acquisition-type instruction (discrimination training to establish accurate responding) Slide 228 Successful Math Intervention Slide 229 Unsuccessful Math Intervention Slide 230 Integrity Matters 59% Integ96% Integrity Slide 231 Integrity Matters Slide 232 Slide 233 Integrity Untreated integrity problems become student learning deficits, schoolwide learning problems, and false positive decision errors Integ problems affect dose and quality of the treatment (an intervention implemented with fidelity is a functionally different intervention than one implemented inconsistently Integ positively correlated with student learning gains, amount of intervention covered Even veteran sites require monitoring and follow-up Slide 234 Tips for Effective Implementation Slide 235 Our Recipe for Intervention Success PREPARATION Identify and Use standard protocols for intervention Develop all needed materials Develop packets or put on a central web site Determine graphing program Slide 236 Our Recipe for Intervention Success TRAIN Explain Watch the teacher do it with the actual child before you leave Call or meet teacher after first day to problem solve Slide 237 DATA COLLECTION AND SUPPORT Each week, graph intervention performance and do a generalization check with the child. Graphed feedback to teachers with generalization checks for individual intervention once per week Response-dependent performance feedback to sustain implementation accuracy Monthly CBM to track growth and enhance existing Tier 1 Programs or advise new Tier 1 Data to principal weekly. Summarize effects and integrity of procedures. Our Recipe for Intervention Success Slide 238 DATA DECISION -MAKING RTI successful if child performs criterion-level probe (from screening) in the instructional range. RTI unsuccessful if 15 consecutive intervention sessions and criterion probe is not in the instructional range. Increase task difficulty for intervention if child scores at mastery on task during intervention sessions Slide 239 Guidelines for Implementers Use single trial scores for screening Following screening, grade-wide graphs to principal Return data to teachers within 48 hours with personal interpretation at grade-level team meeting Include principal in critical meetings Involve teachers at all stages Slide 240 Guidelines for Implementers Learn about curriculum and instruction. Integrate RTI with ongoing school and system reform efforts Thoughtfully merge to subtract duplicate activities and to enhance more comprehensive supplemental and core instructional support activities that may be in place Use RTI data to evaluate the value of ALL instructional programs or resource allocation decisions. Quantify bang for the buck using student performance data. Slide 241 Infrastructure for Implementation Grade-level planning periods can be utilized Special education team at school can be utilized School Psych must be on-site 1 day/week Developing master schedule for Tier 1, 2, and 3 intervention times is useful Integrate efforts with evaluation referral team efforts (consider major reduction in meeting time and shift to intervention efforts!) Use existing instructional periods to target student needs more effectively See NASDSE blueprint for implementation Brown-Chidsey book coming from Guilford Slide 242 Resources www.naspweb.org- early childhood position statementswww.naspweb.org- Best Practices Big Math for Little Kids Number Worlds Headsprout Kansas Center www.rtinetwork.org Slide 243 For More Information amandavande@gmail.com www.isteep.com Thank you to the US Dept of Education for providing all film clips shown in this presentation