Upload
lorin-pearson
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Measures that Support Implementation of High Quality
Inclusive Practices
Presentation at National Early Childhood Inclusion InstituteChapel Hill, NC
May 21, 2014
Patricia Snyder, PhDProfessor and David Lawrence Jr. Endowed Chair in Early Childhood Studies University of Florida
Mary Louise Hemmeter, PhDProfessor Department of Special EducationVanderbilt University
Elena Soukakou, PhDSenior LecturerUniversity of Roehampton, London, UK
Moderator: Pamela J. Winton, PhDSenior Scientist & Director of Outreach FPG Child Development Institute
Agenda
Welcome and introductionOverview of three measuresQuestion & answer after each measure
Facilitated discussion
Teaching Pyramid Observation Tool (TPOT) (Fox, Hemmeter, & Snyder, 2013) Available from Paul Brookes Publishing
Embedded Instruction Observation System --Teacher Version (EIOS-T) (Crowe, Snyder, Crow, Mullin, & Embedded Instruction Project, 2011) Available from the authors
Inclusive Classroom Profile (ICP) (Soukakou, 2012).Available from the author
Overview of the Inclusive Classroom
Profile (ICP) Dr. Elena Soukakou,
Author
Copyright © 2013 by Elena P. Soukakou
Measures inclusive, classroom-level practices that support the individualized needs of children with disabilities
ICP Content
Structured observation measure1–7 point rating scale12 items
ICP Structure
Young children with
disabilities can experience low
quality in classes that
are otherwise rated as being
of high quality
Wolery, et al., 2000
National Professional Development Center on Inclusion
1. Adaptation of space and materials2. Adult involvement in peer interactions3. Adult guidance of children’s activities and play4. Conflict resolution5. Membership6. Relationships between adults and children7. Support for social communication8. Adaptation of group activities9. Transitions between activities10. Feedback11. Family-professional partnerships12. Monitoring children’s learning
ICP Items
As a research toolAs a classroom evaluation tool
As a professional development tool
National Professional Development Center on Inclusion
How Can the ICP Be Used?
Children with identified disabilities in the context of classroom activities and social interactions with adults and peers
Teachers, teacher assistants, specialists
National Professional Development Center on Inclusion
Who Is Being Observed?
Observation Teacher
interview Document
review
National Professional Development Center on Inclusion
Administration
1.Exploratory research2.Conceptualization and domain delineation3. Item generation4.Expert review5.Pilot study in the UK
See Soukakou, E. P. (2012). Measuring quality in inclusive preschool classrooms: Development and validation of the Inclusive Classroom Profile (ICP). Early Childhood Research Quarterly, 27(3), 478-488)
ICP Development: Five Phases
1st pilot study in the UK showed promising results on reliability & validity (Soukakou, 2012)
2nd pilot study in the US in collaboration with:
National Professional Development Center on Inclusion
Pilot Studies on the ICP
NC Department of Instruction,Exceptional Children
Did assessors learn to use the ICP with accuracy?
What is the evidence for reliability and validity?
Did assessors find the ICP useful and acceptable for program evaluation?
National Professional Development Center on Inclusion
ICP Pilot Study (US): Research Questions
51 inclusive classrooms in one state Public Pre-K (5),
Head Start (13), Developmental Day programs (13), Other child care centers (20)
150 children with disabilities Mean age of children= 4.43 years
National Professional Development Center on Inclusion
Sample: Classrooms
51 ICP assessments 50 ECERS-R assessments Assessor survey for
gathering data on ICP acceptability
National Professional Development Center on Inclusion
Procedures
ITEM ICC ICP 1 Adaptation of Space, Materials and Equipment
.62 ICP 2 Adult Involvement in Peer Interactions
.78 ICP 3 Adult Guidance of Children’s Play
.11 ICP 4 Conflict Resolution
.70 ICP 5 Membership
.84 ICP 6 Relationships between Adults and Children
.75 ICP 7 Support for Communication
.51 ICP 8 Adaptations of Group Activities
.72 ICP 9 Transitions between Activities
.95ICP 10 Feedback
.60ICP 11 Family-Professional Partnerships
.99ICP 12 Monitoring Children’s Learning
.99
Results: Inter-Rater Reliability
ECERS-R Scale ICP Total Score
Space and Furnishings 0.48***Personal Care 0.21**Language and Reasoning 0.47***Program Structure 0.29*Activities 0.30*Interactions 0.38**Parent and Staff 0.38**ECERS Total Score 0.48***
Results: Rank-Order Correlations Between ICP and ECERS
Note: *p<.05, **p<.01, ***p<.001
Results: Discriminant Validity
Mean(SE)/B(SE)
Child Care 3.67 (0.15)a
Developmental Day
5.12 (0.19)b
Head Start 4.64 (0.19)b
Public Pre-K 4.76 (0.30)b
Note: Means not sharing superscripts are significantly different.
On a 1–5 point scale, 4 assessors:
Rated the importance of the ICP constructs measured very highly (m= 5)
Would highly recommend the ICP measure to others (m=5)
Found the measure easy to administer (m= 4)
Felt well prepared after the reliability training observations (m=4)
National Professional Development Center on Inclusion
Results: Social Validity
Assessors established adequate administration and reliability proficiency upon training.
Evidence for construct validity.
Differences in quality across types of programs
Assessors found the ICP easy to use and useful for program evaluation
National Professional Development Center on Inclusion
Summary of Findings
Training program for users. Online overview materials at http://npdci.fpg.unc.edu/measuring-quality-inclusion-inclusive-classroom-profile
Professional development curriculum for PD providers/consultants
National Professional Development Center on Inclusion
Next Steps
http://pdc.fpg.unc.edu
PDC@FPG
Putting Knowledge to Work
ICP training available at PDC@FPG
Questions?
Teaching Pyramid Observation Tool
(TPOT)Mary Louise Hemmeter
Lise FoxPatricia Snyder
Tertiary Intervention: Few Children
Secondary Prevention:
Some Children
Universal Promotion: All Children
The Pyramid Model: Promoting Social and Emotional Competence and Addressing
Challenging Behavior
27
TPOT Materials
28
TPOT Score Sheet & TPOT Manual
TPOT Sample Items
http://products.brookespublishing.com/Teaching-Pyramid-Observation-Tool-TPOT-for-Preschool-Classrooms-Set-Research-Edition-P727.aspx
Teaching Pyramid Observation Tool (TPOT)
The TPOT was developed to measure the fidelity with which teachers implement Pyramid Model practices
Provides information that can be used to:Describe “quality” of implementation of TPOT practicesCompare implementation within and across
teachers/classroomsSupport program-wide implementation and improvement
activities Identify needs of teachers for training and implementation
support
Using the TPOT Observations
Conducted for a minimum of 2 hours Must observe centers or free play, at least one
teacher-directed activity, and the transitions between activities
Focus of observation is primarily lead teacher’s implementation of practices, but consider all adults
InterviewsFor those practices that might not or cannot
be observed during the 2-hour observation
Organization of the TPOT: Three Subscales Key Practice Items: Multiple indicators
associated with each item Each indicator rated yes, no, or N/O (only when noted)
Red Flags Each item rated yes or no
Using Effective Strategies for Responding to Challenging Behavior Item only scored when challenging behavior observed Includes three indicators for responses to each incident of
challenging behavior Each of these three Indicators rated as yes or no for each
incident
14 Key Practice Items
Observation items1. Schedules, routines, and
activities (SR)2. Transitions between activities
(TR)3. Supportive conversations (SC)4. Promoting engagement (ENG)5. Providing directions (PD)6. Collaborative teaming (CT)7. Teaching behavior
expectations (TBE)8. Teaching social skills and
emotional competencies (TSC)
Observation and interview items9. Teaching friendship skills (FR)10. Teaching children to express
emotions (TEE)11. Teaching problem-solving
(TPS)
Interview items12. Interventions for children
with persistent challenging behavior (PCB)
13. Connecting with families (COM)
14. Supporting families in using Pyramid Model practices (INF)
Example Key Practice Item
33
Item
Indicator
Score
Columns
No Opportunity
Key Practice Item: Schedules, Routines, & ActivitiesObservation Only
34
Key Practice Item: Teaching Friendship Skills Observation AND Interview
35
Interview Questions for Teaching Friendship Skills
Red Flags Subscale
37
Challenging Behavior Subscale
38
Defining the Behavior – What to Note
40
Studies to Examine Psychometric Integrity of TPOT™ Scores
Primarily Head Start classrooms TPOT observations
50 classrooms 2 raters 3 occasions
300 total TPOTs Generalizability study to look at
dependability of scores across items, raters, occasions of measurement
CLASS (Pianta, LaParo, & Hamre, 2008)
50 classrooms Administered between 2nd and 3rd TPOT
observation Inter-observer agreement for 34% (n = 17) Convergent score validity
Work reported was supported, in part, by Institute of Education Sciences grant (R324A07212) to Vanderbilt University. The information and opinions expressed are those of the authors, not the funding agency.
Generalizability Study Findings Less than .01% of
variance on key practice indicators due to raters
<1% of variance on key practice indicators due to occasion
6.12% of variance on key practice indicators due to classroom/teacher
Phi coefficient = absolute decisions
G coefficient = relative decision Phi key practice indicators = .89 G key practice indicators = .95 Phi red flags = .76 G red flags = . 84
Good news! Dependability in rank ordering of
classrooms and dependability in scores across raters, occasions
41
Snyder, P., Hemmeter, M.L., Fox, L., Bishop, C., & Miller, M.D. (2013). Developing and gathering psychometric evidence for a fidelity instrument. Journal of Early Intervention, 35, 150-172.
42
TPOT™ and CLASS™ Correlations
N = 50TPOT Key Practices
Emotional Support (ES)
.70
Classroom Organization (CO)
.73
Instructional Support (IS)
.76
Snyder, P., Hemmeter, M.L., Fox, L., Bishop, C., & Miller, M.D. (2013). Developing and gathering psychometric evidence for a fidelity instrument. Journal of Early Intervention, 35, 150-172.
Noteworthy Correlations:TPOT™ and CLASS™
TPOT Key Practices subscale and every CLASS dimension and domain
TPOT Red Flags subscale and every CLASS dimension and domain (negative relationships)
General teaching items on TPOT (SR, TR, SC, ENG, PD) and each CLASS dimension and domain
Most targeted teaching items on TPOT (TBE, TSC, TEE, TPS, FR) and Instructional Support CLASS domain
TPOT Connecting with Families with each dimension and domain on CLASS
Additional detail in Chapter 7 in TPOT manual43
Another Study: TPOT™ and ECERS-R
Noteworthy correlationsTPOT Key Practices subscale and overall ECERS-R10 of the 14 TPOT key practice items and overall
ECERS-RTPOT Red Flags subscale and overall ECERS-R
(negative relationships)
Additional detail in Chapter 7 in TPOT manual
44
45
Another Study: Pre-SET and Select TPOT-Pilot Version Key Practice Items(N = 31 Classrooms)
Note. Adapted from Steed and Pomerleau (2012). N = 31 classrooms. a = Seven environmental items included on pilot version of TPOT. * = p < .05 ** = p < .01
Additional detail in Chapter 7 in TPOT manual
46
1 2 3 40
10
20
30
40
50
60
70
80
90
100
Use of TPOT in Potential Efficacy Study (N = 40)Control
InterventionT
ota
l T
PO
T s
co
re
Training
Sept. Nov. Feb. Apr.
Comparison
Interven-tion
Coaching
Figure 1. Mean TPOT scores across 4 waves. Total TPOT indicators = 108. Wave 4 [t(40.03)=6.80, p<.001, Cohen’s d=2.6) Hemmeter, M.L., Fox, L., Snyder, P., & Algina, J. (2011, April). Efficacy of a classroom-wide model for promoting social-emotional development and preventing challenging behavior. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Additional Results from Potential Efficacy Study: SSIS Child Outcomes
Adjusted Means Cohen’s d Effect Size
Intervention Control
Target Children
Social 88.6 84 .41
Problem Behavior 108.7 115.5 -.52*
Non Target Children
Social 103.8 96.4 .46*
Problem Behavior 95.2 99 -.29
Mean frequency positive social interactions for 60 min observation session across 4 waves. Average frequency of positive social interactions for target children in each classroom used to derive means for each group.Mean experimental Wave 4 = 19.96 (SD = 10.7); control Wave 4 = 15.65 (SD = 8.5)
Additional Results from Potential Efficacy Study: Target Child Social Interaction Behaviors
Using the TPOT to Inform Decision Making Using the TPOT in coaching
Running TPOTFormal TPOTGoal setting/action planning
Using the TPOT program wideMonitor implementation of PW implementationPlan professional development
Using the TPOT in monitoring/evaluation
Sched
ules, ro
utines, an
d activiti
es
Transiti
ons betw
een ac
tivities
Supporti
ve co
nversa
tions
Promoting c
hildren
’s enga
gemen
t
Provid
ing dire
ctions
Collaborati
ve te
aming
Teach
ing beh
avior e
xpect
ations
Teach
ing socia
l skills
and em
otional co
mpetencie
s
Teach
ing frie
ndship sk
ills
Teach
ing child
ren to
expres
s emotions
Teach
ing pro
blem so
lving
Interve
ntions for c
hildren
with
persist
ent c
hallen
ging b
ehav
ior
Connecting w
ith fa
milies
Supporti
ng Fam
ily use
of the P
yramid M
odel prac
tices
Key Prac
tices Su
bscale
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of Indicators Observed for Key Practice ItemsInitial - 8 Teachers November - 8 Teachers April - 8 Teachers
Questions?
Embedded Instruction Observation System - Teacher Version
(Crowe, Snyder, Crow, Mullin, & Embedded Instruction Project, 2011)
Work reported was supported, in part, by Institute of Education Sciences grant (R324A070008) to the University of Florida. The information and opinions expressed are those of the authors, not the funding agency.
Manual
Scoring Sheet
Description of EIOS-T
Continuous event observational coding system
Used by teachers to record the frequency and accuracy of complete learning trials embedded within ongoing classroom activities, routines, and transitionsAdult or environmentally arranged antecedents to
elicit a targeted child behaviorWhether the target child behavior occurredConsequences and additional help (if appropriate)
provided following child behavior
Development of EIOS-T
Adapted from Embedded Instruction Observation System (EIOS) - Research Version 2.0 (Snyder et al., 2010)
EIOS was primary dependent measure in a Goal 2 study funded by IES*
Adapted from the the Teacher Performance Rate and Accuracy Scale (Ross, Singer-Dudek, & Greer, 2005) Designed to quantify interlocking three-term
contingencies or “learn units”*Institute of Education Sciences Project No. R324A070008: Impact of Professional Development on Preschool Teachers’ Use of Embedded Instruction Practices
Interlocking Three-Term Contingencies
Three-Term Contingencies for
Teacher
Instructional Components
Three-Term Contingencies for
Child
Teacher Antecedent 1 Child [reaches for ball]
Teacher Behavior 1 Teacher: “Say ball.” Child Antecedent
Teacher Consequence 1/Teacher Antecedent 2
Child: “Ball.” Child Behavior
Teacher Behavior 2 Teacher: [gives child a balI]
Child Consequence
Teacher Consequence 2 Child [plays with ball]
Note. Table adapted from Ross et al., 2005
From: Barton, E., Bishop, C., & Snyder, P. (in press). High quality instruction through complete learning trials: Blending intentional teaching with embedded instruction. Young Exceptional Children Monograph.
Targeted Child Behavior
Priority Learning Targets
Universal
Individualized
IEP Content
General Curriculum Commercial
GeneralCurriculum
LocallyDeveloped
Targeted Curricula(e.g., early literacy, social emotional)
Early Learning Foundations
EIOS-R
Focused on documenting Occurrence of learning
trials Categories of
antecedents, consequences and error corrections Adult-delivered Peer-delivered Environmentally
arranged Procedural accuracy of
learning trial components
EIOS-T
Focused on documenting Occurrence of learning
trials Category of
antecedent delivered Teacher-delivered Environmentally
arranged
Presence or absence of complete learning trial components
EIOS-T Development Simplified and reduced EIOS-R codes Developed coding manual and scoring sheet Developed and piloted training
Piloted with preschool teacher who participated in Goal 2 study
Revised coding manual based on teacher feedback
Piloted with three preschool teachers in a multiple baseline across teachers single-subject experimental study (Bishop, Snyder, & Crow, 2014)
EIOS-T Codes
Following EIOS-T Coding
Pilot Data Using EIOS-T Multiple baseline across three teachers single-
subject experimental study (Bishop et al., 2014) When provided feedback about their coding accuracy,
2 of 3 teachers were able to record accurately the occurrence of learning trials
When provided with feedback about their coding accuracy, all 3 teachers were able to record accurately the fidelity with which they implemented complete learning trial components
When teachers began to accurately record the occurrence and accuracy with which they delivered CLT components, the fidelity with which they implemented CLTs increased
Questions?
How are these measures similar?
Could these measures be used in an integrated fashion? How would that integration occur?
What is the role of measures such as these in terms of supporting inclusion within the broader early childhood quality movement?
Discussion
THANK YOU
Data Slides to Show if Needed
National Professional Development Center on Inclusion
ICP Sample: Lead Teacher Characteristics*
Head
StartChild Care
Public
Pre-KDev Day Total
Mean Mean Mean Mean Mean
Course hours in special education
9.25 1.18 1.5016.0
87.52
Number of years of teaching child(ren) with a disability
10.77
5.95 4.40 5.48 6.91
Number of years of teaching in EC
13.62
12.85
5.60 8.1011.1
2* Based on teacher report
National Professional Development Center on Inclusion
ICP Sample: Classroom and Child Characteristics
Head Start
Child Care
PublicPre-K
Dev. Day
Total
Mean Mean Mean Mean Mean
Number of children in classroom
17.15 16.55 15.00 16.38 16.51
Number of children with IEP in classroom
2.62 2.15 2.40 4.69 2.94
Age of youngest child (Yrs) 3.54 3.20 4.00 3.69 3.49
Age of oldest child (Yrs) 4.54 4.30 4.60 4.62 4.47
Number of adults in classroom 2.38 1.70 2.20 3.46 2.37
Children with IEP/adults ratio 1.13 1.22 1.07 1.43 1.23
All children/adults ratio 7.66 10.91 7.10 5.02 8.21
ECERS-R score 4.95 4.58 5.14 5.31 4.92
ICP score 4.64 3.67 4.76 5.12 4.39
Speech and language (38%) Developmental delay (37%) Autism (12%) Other health impairment (5%) Sensory impairment (4%) Multiple disabilities (1%) Orthopedic impairment (1%) Don’t know (2%) * Based on teacher report
ICP Sample: Children’s Primary Diagnoses*
88% of classrooms had at least one child with a moderate or severe level of disability in at least one area. *
* Based on teacher responses using an adaptation of the ABILITIES Index (Simeonsson & Bailey, 1991).
National Professional Development Center on Inclusion
ICP Sample: Children with Disabilities
Head Start
Child Care
Public
Pre-K
Dev.
Day TotalChildren receiving majority of services in classroom
17.65%
37.21%
8.33%67.21
%57%
National Professional Development Center on Inclusion
ICP Sample: Percent of Children Receiving the Majority of Specialized Services in the Classroom
In North Carolina, the percentage of children receiving the majority of specialized services in the classroom is 50.97%*
Nationally, the percentage of children receiving the majority of specialized services in the classroom is 41.67%*
*OSEP Report to Congress, 2011
EIOS-T Teacher Coding Accuracy
Average Percent Occurrence Agreement
No Training EIOS-T Training
EIOS-T Feedback
Natalie 12 (0 – 20) 62 (38 – 75) 90 (75 – 100)
Rhonda 31 (9 – 50) 36 (9 – 50) 47 (18 – 76)
Brenda 33 (0 – 66) 41 (20 – 63) 90 (78 – 100)
Average Percent Component Agreement
No Training EIOS-T Training
EIOS-T Feedback
Natalie N/A 66 (40 – 83) 88 (80 – 100)
Rhonda N/A 66 (33-83) 83 (69-100)
Brenda N/A 67 (39-89) 82 (61-83)
Teacher Implementation of Complete Learning Trials
Perc
enta
ge o
f Pro
ced
ura
lly C
orr
ect
Learn
ing T
rials