Developing and Implementing a Quality RtI Process Edward Daly, Ph.D. University of Nebraska-Lincoln

Developing and Developing and Implementing a Implementing a Quality RtI ProcessQuality RtI Process

Edward Daly, Ph.D.Edward Daly, Ph.D.

University of Nebraska-LincolnUniversity of Nebraska-Lincoln

Taking RtI “to Taking RtI “to Scale”…Scale”…

And then I looked out my And then I looked out my window…window…

= 1

Is This Really Something Is This Really Something New?New? No!No! Previous versions of IDEA have required…Previous versions of IDEA have required…

““Prereferral” intervention Prereferral” intervention The need to rule out lack of instruction in reading or math or The need to rule out lack of instruction in reading or math or

limited English proficiency limited English proficiency Gathering relevant functional and developmental information Gathering relevant functional and developmental information

and information related to enabling the child to be involved in and information related to enabling the child to be involved in and progress in the general curriculum…and progress in the general curriculum…

The conceptual basis for intervention design has existed for The conceptual basis for intervention design has existed for more than 30 years (Bijou, 1970; Bijou, Peterson, & Ault, 1968).more than 30 years (Bijou, 1970; Bijou, Peterson, & Ault, 1968).

National Academy of Sciences 1982 report on National Academy of Sciences 1982 report on overrepresentation of minorities (Heller, Holtzman, & Messick, overrepresentation of minorities (Heller, Holtzman, & Messick, 1982)1982)

So, What is New?So, What is New?

Intervention data can now be used to Intervention data can now be used to make eligibility decisionsmake eligibility decisions Includes child study assessment informationIncludes child study assessment information Functional assessmentsFunctional assessments Direct assessment and behavioral Direct assessment and behavioral

observationsobservations

Why is RtI Important to Why is RtI Important to BothBoth General and Special Education?General and Special Education?

The Paradox of Special EducationThe Paradox of Special Education Skrtic (1991)Skrtic (1991)

““Structurally, special education is a non-rational Structurally, special education is a non-rational system, an organizational artifact that functions as system, an organizational artifact that functions as a legitimizing device. Culturally, it distorts the a legitimizing device. Culturally, it distorts the anomaly of school failure and thus preserves the anomaly of school failure and thus preserves the prevailing paradigm of school organization, which prevailing paradigm of school organization, which ultimately reinforces the presuppositions of ultimately reinforces the presuppositions of organizational rationality and human pathology in organizational rationality and human pathology in the profession and institution of education in the profession and institution of education in society.” (p. 182)society.” (p. 182)

Why is RtI Important to Why is RtI Important to BothBoth General and Special Education?General and Special Education?

RtI presents a unique opportunity for RtI presents a unique opportunity for educators to get it right…educators to get it right… Transdisciplinary integration of services for Transdisciplinary integration of services for

students across a continuum that meets all students across a continuum that meets all students’ needsstudents’ needs

Implementing RtI: Implementing RtI: Technical AdequacyTechnical Adequacy

Focus on Reliability and Validity of DecisionsFocus on Reliability and Validity of Decisions Standards for Educational and Psychological Testing: Standards for Educational and Psychological Testing: “It is the “It is the

user who must take responsibility for determining whether or user who must take responsibility for determining whether or not scores are sufficiently trustworthy to justify anticipated not scores are sufficiently trustworthy to justify anticipated uses and interpretations.” (p.31)uses and interpretations.” (p.31)

Barnett, Lentz, & Macmann (2000): “Although psychometric Barnett, Lentz, & Macmann (2000): “Although psychometric variables such as reliability and validity have been traditionally variables such as reliability and validity have been traditionally seen as quality of assessment instruments… practitioners seen as quality of assessment instruments… practitioners must consider the psychometric qualities of their must consider the psychometric qualities of their decisionsdecisions across the sequence of problem solving.” (p. 356)across the sequence of problem solving.” (p. 356)

Problem Solving as a Continuous Evaluation CycleProblem Solving as a Continuous Evaluation Cycle

Problem Solving as a Problem Solving as a Continuous Evaluation CycleContinuous Evaluation Cycle

“Is there a discrepancy

between current and expected

performance?”

Why is there a problem?

“By how much should the student grow?”

“What will be done to resolve the problem?”

“Did it work? What do we do

next?”


Reliability and Validity of DecisionsReliability and Validity of Decisions Problem Solving as a Continuous Problem Solving as a Continuous

Evaluation CycleEvaluation Cycle Local Validation!Local Validation!


Validating Practice…Easier said than Validating Practice…Easier said than done…done… Reading research has not been readily Reading research has not been readily

translated into practice.translated into practice. Generalizability to actual classroom conditionsGeneralizability to actual classroom conditions Weak generalization of treatment effectsWeak generalization of treatment effects

(Lyon & Moats, 1997)(Lyon & Moats, 1997)

Practitioners need methods that guide them in Practitioners need methods that guide them in investigatinginvestigating interventions directly for their interventions directly for their students.students.

Scientifically supported interventions are not Scientifically supported interventions are not enough!enough!

NASDSE: “Selection and implementation of NASDSE: “Selection and implementation of scientifically based instruction/intervention scientifically based instruction/intervention markedly increases the probability of, but does markedly increases the probability of, but does not guarantee, positive individual response. not guarantee, positive individual response. Therefore, individual response is assessed in RtI Therefore, individual response is assessed in RtI and the modifications to instruction/intervention and the modifications to instruction/intervention or goals are made depending on results with or goals are made depending on results with individual students.”individual students.”

Batsche, Elliot, Graden, Grimes, Kovaleski, Prasse, Reschly, Batsche, Elliot, Graden, Grimes, Kovaleski, Prasse, Reschly, Schrag, & Tilly (2005)Schrag, & Tilly (2005)

Empirical Validation Versus Empirical Validation Versus LocalLocal Validation Validation

May Have Great Scientific May Have Great Scientific Support, But Not A Valid Support, But Not A Valid Intervention for This Student!Intervention for This Student!

010203040506070

1 3 5 7 9 11 13 15 17 19

Correct Read Words and Errors Per Minute Ongoing

Progress Monitoring

Time

Baseline Planned Intervention

This One is Valid for This This One is Valid for This Student!Student!

010203040506070

1 3 5 7 9 11 13 15 17 19

Correct Read Words and Errors Per

Minute

Time

Baseline Planned Intervention 1 Intervention 2

Technical Adequacy of RtI:Technical Adequacy of RtI:Local ValidationLocal Validation

We have to We have to proveprove it works for students. it works for students. If it does not work, we have not finished If it does not work, we have not finished

our job.our job. We are actually testing the treatments We are actually testing the treatments

and not the student!and not the student!

Developing a Valid RtI Developing a Valid RtI Process for StudentsProcess for Students

Quality IndicatorsQuality Indicators Developmental Progression of Developmental Progression of

ImplementationImplementation Strong Operational Model for Continuum Strong Operational Model for Continuum

of RtI Service Deliveryof RtI Service Delivery Evaluation Plan: Validating the ModelEvaluation Plan: Validating the Model

Procedural IntegrityProcedural Integrity Child OutcomesChild Outcomes Refinements and Ongoing EvaluationRefinements and Ongoing Evaluation

Quality Indicators in Quality Indicators in Nebraska: Implementing RtI Nebraska: Implementing RtI in Schoolsin Schools

Parent InvolvementParent Involvement Universal Screening and AssessmentUniversal Screening and Assessment Individual Progress MonitoringIndividual Progress Monitoring Planned Service Delivery RulesPlanned Service Delivery Rules Scientifically Supported InstructionScientifically Supported Instruction Intervention DeliveryIntervention Delivery SLD VerificationSLD Verification

Quality Indicators in Quality Indicators in Nebraska: Organizational Nebraska: Organizational CapacityCapacity

School-wide buy-in and implementation School-wide buy-in and implementation planplan

Team leadershipTeam leadership Integration of ServicesIntegration of Services Implementation InfrastructureImplementation Infrastructure

Developmental Progression Toward Full Developmental Progression Toward Full Scale Implementation of RTIScale Implementation of RTI

Problem Solving Team Engages inData Based Problem Solving

Universal Screening to Identify Students At-Risk for

Academic Difficulties

Coordinated Intervention Delivery

Application of Problem Solving

Data to LD Eligibility

Integrating RtI With Integrating RtI With Existing PracticesExisting Practices

School Improvement PlanSchool Improvement Plan Pre-referral Intervention teamsPre-referral Intervention teams Title I ServicesTitle I Services Professional Learning CommunitiesProfessional Learning Communities

Implementing RtI: Implementing RtI: Procedural IntegrityProcedural Integrity

PROCEDURAL CHECKLIST as an PROCEDURAL CHECKLIST as an operational definition of RtIoperational definition of RtI AccountabilityAccountability Formative evaluation and program Formative evaluation and program

improvementimprovement

Implementing RtI: Implementing RtI: Procedural IntegrityProcedural Integrity

PROCEDURAL CHECKLIST as an PROCEDURAL CHECKLIST as an operational definition of RtIoperational definition of RtI AccountabilityAccountability Formative evaluation and program Formative evaluation and program

improvementimprovement

Procedural Checklist Procedural Checklist Outcome ExampleOutcome Example

Procedural ChecklistProcedural Checklist

04-0504-05 05-0605-06 ChangeChange

NN 6464 229229 +165+165

MeanMean(% steps (% steps

completed)completed)

80%80% 93%93% +13%+13%

SDSD(% steps (% steps

completed)completed)

19%19% 9%9%


Differences across 04-05 and 05-06 Differences across 04-05 and 05-06 school years:school years:

Approximately 250% more students served in Approximately 250% more students served in the RtI processthe RtI process

A 13% increase in the percentage of A 13% increase in the percentage of procedural steps completed procedural steps completed

Less variability in percentage of steps Less variability in percentage of steps completed among casescompleted among cases


Despite vast increase in number of students Despite vast increase in number of students served in RtI process in 2served in RtI process in 2ndnd year, higher year, higher procedural fidelity was obtained!!procedural fidelity was obtained!!

Possible influences:Possible influences: Increased organization, comfort and familiarity Increased organization, comfort and familiarity

with processwith process More training for pilot school psychologistsMore training for pilot school psychologists Increased streamlining of services within Increased streamlining of services within

buildingsbuildings

Descriptive Evaluation of Descriptive Evaluation of Individual Child Study Cases Individual Child Study Cases

Produces a quantitative synthesis across Produces a quantitative synthesis across outcome metricsoutcome metrics

Procedural Integrity (Procedural Checklist)Procedural Integrity (Procedural Checklist) Effect Sizes & Percentage of Nonoverlapping Effect Sizes & Percentage of Nonoverlapping

DataData Graphic Analysis WorksheetGraphic Analysis Worksheet Goal Attainment ScalingGoal Attainment Scaling Significance According to “Structured Criteria Significance According to “Structured Criteria

for Visual Inspection of Data”for Visual Inspection of Data” Treatment integrityTreatment integrity Social validitySocial validity

““The significant problems The significant problems we face cannot be solved by we face cannot be solved by the same level of thinking the same level of thinking that created them.”that created them.”

Albert EinsteinAlbert Einstein

Real Change in SchoolsReal Change in Schools

Fullan (1991)Fullan (1991) ““innovations can be adopted for symbolic political or innovations can be adopted for symbolic political or

personal reasons: to appease community pressure, personal reasons: to appease community pressure, to appear innovative, to gain more resources. All of to appear innovative, to gain more resources. All of these forms represent these forms represent symbolicsymbolic rather than rather than realreal change.” (p. 28)change.” (p. 28)

““Even good ideas may represent poor investments Even good ideas may represent poor investments on a large scale if the ideas have not been well on a large scale if the ideas have not been well developed or if the resources to support developed or if the resources to support implementation are unavailable.” (p. 28)implementation are unavailable.” (p. 28)

ConclusionsConclusions

Building a Reliable and Valid Model for Building a Reliable and Valid Model for RtIRtI Quality IndicatorsQuality Indicators Procedural IntegrityProcedural Integrity Local Validation Through Databased Local Validation Through Databased

Problem SolvingProblem Solving Strategic and Sequential Development and Strategic and Sequential Development and

RefinementRefinement

ReferencesReferences American Educational Research Association, American Psychological American Educational Research Association, American Psychological

Association, & the National Council on Measurement in Education Association, & the National Council on Measurement in Education (1999). (1999). Standards for educational and psychological testing.Standards for educational and psychological testing. Washington, DC: American Psychological Association. Washington, DC: American Psychological Association.

Barnett, D. W., Lentz, F. E., Jr., & Macmann, G. (2000). Psychometric Barnett, D. W., Lentz, F. E., Jr., & Macmann, G. (2000). Psychometric qualities of professional practice. In E. S. Shapiro & T. R. Kratochwill qualities of professional practice. In E. S. Shapiro & T. R. Kratochwill (Eds.), (Eds.), Behavioral assessment in schools: Theory, research, and clinical Behavioral assessment in schools: Theory, research, and clinical foundations.foundations. (2nd ed., pp. 355-386). New York, NY: Guilford (2nd ed., pp. 355-386). New York, NY: Guilford Publications. Publications.

Bijou, S. W. (1970). What psychology has to offer education--now. Bijou, S. W. (1970). What psychology has to offer education--now. Journal of Applied Behavior Analysis, 3,Journal of Applied Behavior Analysis, 3, 65-71. 65-71.

Bijou, S. W., Peterson, R. F., & Ault, M. H. (1968). A method to integrate Bijou, S. W., Peterson, R. F., & Ault, M. H. (1968). A method to integrate descriptive and experimental field studies at the level of data and descriptive and experimental field studies at the level of data and empirical concepts. empirical concepts. Journal of Applied Behavior Analysis, 1,Journal of Applied Behavior Analysis, 1, 175-191. 175-191.

Fuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept Fuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept for reconceptualizing the identification of learning disabilities. for reconceptualizing the identification of learning disabilities. Learning Learning Disabilities Research and Practice, 13,Disabilities Research and Practice, 13, 204-219. 204-219.

References ContinuedReferences Continued Fisher, W. W., Kelley, M. E., & Lomas, J. E. (2003). Visual aids and Fisher, W. W., Kelley, M. E., & Lomas, J. E. (2003). Visual aids and

structured criteria for improving visual inspection and interpretation of structured criteria for improving visual inspection and interpretation of single-case designs. single-case designs. Journal of Applied Behavior Analysis, 36, Journal of Applied Behavior Analysis, 36, 387-406. 387-406.

Fullan, M. G. (1991). Fullan, M. G. (1991). The new meaning of educational changeThe new meaning of educational change (2nd ed.). (2nd ed.). New York, NY: Teachers College Press. New York, NY: Teachers College Press.

Heller, K. A., Holtzman, W. H., & Messick, S. (Eds.). (1982). Heller, K. A., Holtzman, W. H., & Messick, S. (Eds.). (1982). Placing Placing children in special education: A strategy for equity.children in special education: A strategy for equity. Washington, DC: Washington, DC: National Academy Press. National Academy Press.

Lyon, G. R., & Moats, L. C. (1997). Critical conceptual and methodological Lyon, G. R., & Moats, L. C. (1997). Critical conceptual and methodological considerations in reading intervention research. considerations in reading intervention research. Journal of Learning Journal of Learning Disabilities, 30,Disabilities, 30, 578-588. 578-588.

President’s Commission on Excellence in Special Education. (2002). President’s Commission on Excellence in Special Education. (2002). A A new era: Revitalizing special education for children and their familiesnew era: Revitalizing special education for children and their families. . Available on-line: Available on-line: http://http://www.ed.gov/inits/commissionsboards/whspecialeducationwww.ed.gov/inits/commissionsboards/whspecialeducation

Skrtic, T. M. (1991). Skrtic, T. M. (1991). Behind special education: A critical analysis of Behind special education: A critical analysis of professional culture and school organization.professional culture and school organization. Denver, CO: Love Publishing Denver, CO: Love Publishing Co.Co.

Documents

Developing and Implementing a Quality RtI Process Edward Daly, Ph.D. University of Nebraska-Lincoln