Upload
gabriel-rankin
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
What Works Regarding Social Skills Interventions Using Single Subject
Design
Jeffrey Chenier, M.A., Aaron J. Fischer, Katherine Hunter, Emily Patty, Lisa Libster, M.A., Kristen O’Leary, Haley York, Natalie Robichaux
and Frank Gresham, Ph.D.
Introduction
• Scientifically Based Research – Section 9101(37) NCLB:
Research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs
(No Child Left Behind Act, 2001)
Introduction
• What Works Clearinghouse (2006): Group Design– Meet Evidence Standards:
• “well designed and implemented randomized controlled trials”
– Meet Evidence Standards with reservations: • “quasi-experiments with equating and no severe
design or implementation problems or randomized clinical trials with severe design or implementation problems”
Introduction
• What Works Clearinghouse (2010): Single Case (SC)– Meet Evidence Standards
• IV must be systematically manipulated, with the researcher determining when and how the IV conditions change
• Each outcome variable must be measured systematically over time by more than one assessor, and the study needs to have IOA calculated 20% of the time in each condition, and IOA percentage must meet minimum thresholds
– 0.80 IOA or 0.60 Cohen’s Kappa• Study must include at least three attempts to demonstrate an
intervention effect at three different points in time or with three different phase repetitions
– Phase must have a minimum of three data points
– Effect size estimation follows if a study has either Strong Evidence or Moderate Evidence
Introduction
• Meta-Analysis– Strube & Hartmann (1982)
• Objective method for summarizing a body of empirical findings
– Emphasizes the direction and magnitude of effects across studies for a particular intervention
What Works Clearinghouse (2010)• No agreed upon method or gold
standard to calculate effect sizes from single-case design research – Problems
• How to quantify the effect?– How accurate is the effect?
• How comparable are the effects across other SC designs?
• How comparable are the effects compared to group design effect sizes?
Current Effect Size Estimators (WWC, 2010)• Nonparametric Methods
– Percentage of Nonoverlapping Data (PND), Percentage of All Nonoverlapping Data (PAND), Percent Exceeding the Median (PEM)
• “Distributional properties of these measures are unknown, so standard errors and statistical tests are not formally justified.”
– Additionally, trend is not addressed • Because of the lack of statistical justification, only use if an
approximate size of the effect is desired.• Wolery et al. (2010) compared four overlapping methods to
visual inspection of effect and each method had its own host of issues, so much that they called for their abandonment
– Visual analysis only agreed 121/160 on whether the treatment was effective or not
Current Effect Size Estimators (WWC, 2010)
• Parametric Methods– Regression Estimates
• Advantages– Familiarity– Ability to model trends– Ability to attain an Effect Size from a single case
• Disadvantages– Inability to deal with complex structures present in
single case design
Current Effect Size Estimators (WWC, 2010)
• Parametric Methods– Multilevel Modeling
• Advantages– Ability to account for complexity of design
• Disadvantages– Unfamiliarity– Technically challenging and time consuming– Different metric from group design Effect Sizes,
therefore the estimate is not comparable
Current Effect Size Estimators (WWC, 2010)• Quantitative Methods
– Differing methods to calculate a Standardized Mean Difference statistic (current study)
• Advantages– Encourages inclusion of SC designs in evaluating effects of
interventions– Potentially gives another method in which to rank order
interventions• Disadvantages
– Not completely comparable to group design research» Pooled within-group variance not comparable to pooled
within phase variance– Small n leads to imprecise estimates– Trend is not assessed
Summary of Effect Size Estimators for SC Design (WWC, 2010)
• Simply put, science is not there yet• Nonparametric estimators should be reported
with a parametric estimator (regression)– Multilevel methods are not ready
• Quantitative methods are not as statistically sound as they should be, but the base from which to build is present
Social Skills
• Learned behaviors that enable positive interactions and allow for escape/avoidance of negative interactions
• Academic Enablers (DiPerna & Elliott, 2002)– Better predictor of academic achievement in 8th
grade than 3rd grade academic achievement (Caprara et. al, 2000)
• Myriad of problems co-occurring with social skills deficits– Both externalizing and internalizing
• Does social skills training work?– Gresham, Cook, Crews, and Kern, 2004
Meta-Analysis n studies
ES g ES rBESD Control
BESD Treatment
Ang & Hughes (2001) 38 .62 .30 35% 65%
Beelman et al. (1994) 49 .47 .23 38% 62%
Losel & Beelmn (2003) 84 .38 .19 40% 60%
Schneider (1992) 79 .89 .40 30% 70%
Schneider & Byrne (1985) 51 .65 .31 34% 66%
Quinn et al. (1999) 35 .20 .10 45% 55%
Means M = .60 M = .29 M = 35% M = 65%
Introduction
Introduction
Introduction
• Godbold et. al, 2010– Contrast Analyses
Introduction
• Research question– Evidence is there for Primary Programs; is
there evidence for Secondary Programs?
Other Meta Analyses Since 2000Meta Analysis Disability Type n studies Intervention Type ES statistic ES Degree of Effect
Bellini & Akullian, 2007 Autism Spectrum 15 Video Modeling PND 81% Effective
Bellini & Akullian, 2008 Autism Spectrum 7 Video Self Monitoring PND 77% Questionable/Effective
Bellini et al., 2007 Autism 15 Child Specific PND 71% Questionable
Bellini et al., 2007 Autism 7 Collateral Skills PND 75% Questionable
Bellini et al., 2007 Autism 20 Comprehensive PND 72% Questionable
Bellini et al., 2007 Autism 10 Peer Mediated PND 62% Questionable
Bellini et al., 2007 Autism 55 Total PND 70% Questionable
Kokina & Kern, 2010 Autism Spectrum 18 Social Stories PND 60% Questionable
Wang & Spillane, 2009 Autism 2 CBT d .47-1.24, .24-.59 Medium - Large, Small - Medium
Wang & Spillane, 2009 Autism 1 CBT PND 100% Very Promising
Wang & Spillane, 2009 Autism 9 Others PND 80.77% Effective
Wang & Spillane, 2009 Autism 9 Peer Mediated PND 60.69% Questionable
Wang & Spillane, 2009 Autism 6 Social Stories PND 67.21% Questionable
Wang & Spillane, 2009 Autism 11 Video Modeling PND 84.25% Effective
Durlak & Weissberg, 2010 Typical 68
After School Programs (Targeting
Positive Social Behaviors)
g 0.19 Small (Significantly different from zero)
Schneider et al, 2008 Autism 19 Social Behavior phi 0.72 Large
Summary
• 7 total studies– 6 with Autism Spectrum, 1 with Typically
Developing
• Multiple interventions available• Effectiveness
– 3 very effective– 3 moderately effective– 1 not as effective (but still statistically significant)
Method
• Literature Search, 2000-2009– Keyword Search in PsycINFO
– 5940 Articles Total
Social Skills +
Competence
Intervention
Training
Method
• Coding 1 Primary Inclusionary Criteria– No Books, Reviews, Meta-Analyses, Group Designs,
Dissertations• Coding 1 Secondary Inclusionary Criteria
– “Is study a social skills intervention or does it target a social skill?” (YES)
– Does study focus on ages 3-21, or through high school? (YES)
– Does study target drugs, alcohol, or sexual offenders? (NO)
• 296 studies remained, 100% IOA in coding 1 (approximately 22% of articles)
Method• Coding 2 Primary Inclusionary Criteria
– Is the full article in English?– Does article include single subject graphs?
• No AB design
• Coding 2 Secondary Inclusionary Criteria– Does study fit our Social Skills definition?
• Gresham, Van, and Cook, 2006:– Facilitates initiating and maintain positive social relationships– Contributes to peer acceptance and friendship development– Results in satisfactory school adjustment– Allows individuals to cope with and adapt to the demands of the social
environment
– Is the study not part of a larger treatment package?• Coded 190 unique studies
– 64 studies on to Coding 3 (IOA = 92% for 38% of studies)
Method
• Coding 3 Primary Inclusion Criteria– If one participant, more than 1 replication
across setting or behavior– Presence of variability in baseline and
treatment conditions across at least 2 participants, settings, or behaviors
– Graphs in which UnGraph was able to score – 40 studies eligible for analysis (IOA 100% on
30% of studies)
Method
– Design Type and Subtype
– Research Question– Main Unit of
Comparison– Participant Info– Phase Info– Dataset Info
• Measurement Strategy• DV information• IOA• Treatment Efficacy
– Study Quality• Three replications
across or within?• Treatment Integrity?• IV Operationally
Defined?• DV Operationally
Defined?• IOA
• Coding 3
Method
• Data Extraction– UnGraph (Biosoft, 2004)
• Extracts numerical data from graphs and puts it into Microsoft Excel
– High reliability and validity in collecting data from single subject graphs (Shadish et al., 2009)
Method
• Effect Size Calculation (Shadish, 2007)– G = (Mt – Mb) / sp
• yields a standardized mean difference statistic
– Currently the best quantitative method available, but not absolutely accurate
Method1) Calculate Mean of Baseline and Tx Panels One, Two, and Three
2) Calculate BL and Tx Mean of Means across the three panels
3) Calculate Standard Deviation of BL and Tx from the Mean of Means
4) Calculate Effect Size for Positive Social Interactions
Results
Participants # Disability Type # Ethnicity #
Total # 148 ADHD 24 % Reported of n 41.10%
Age 2-5 28 ODD 9 Caucasian 28
Age 6-8 8 Asperger's Disorder 9 African American 21
Age 9-12 50 Autism 37 Hispanic 6
Age 13-18 18 BD, EBD, ED 19 Asian 1
Unspecified 8 Language Delay 2 Bi-Racial 1
Male 123 Intellectual Disability 9 Vietnamese 1
Female 23 Typically Developing 44 Latino 1
Classrooms 2 PDD 1 Native American 1
Williams 1
JournalsJournal Name # of Articles
Education & Treatment of Children 5Behavioral Disorders 5Focus on Autism and Other Developmental Disabilities 5Journal of Positive Behavior Interventions 4Psychology in the Schools 3Behavior Modification 2Therapeutic Recreation Journal 2Journal of Autism and Developmental Disorders 2Education and Training 2Journal of Applied Behavior Analysis 2Behavior Therapy 1School Psychology Quarterly 1Remedial and Special Education 1Topics in Early Childhood Special Education 1Journal of Early Intervention 1Journal of the Association for Persons with Severe Handicaps 1International Journal of Play Therapy 1Education Psychology 1
Educational Psychology in Practice 1
Results
• Are social skills interventions evaluated by single subject methods effective?
• Yes• g=3.06
Results
• Independent Variable
IV n ESDirect Instruction 17 2.79Computer-Based 3 2.02*Peer Mediated 5 2.67
Self Management 6 3.53Social Stories 4 3.94
SODA 2 5.54*Reinforcement Based 3 2.17
*p<.05
Results
• Independent Variable– Largest effects occurred with interventions
that target self-awareness / self-monitoring of behavior as the primary independent variable
Results
• Dependent VariableDV n ES
AET/On Task Bx 5 1.57Total Destructive Behaviors (TBD)/ Problem
Behaviors / Negativistic Bx 7 1.30Prosocial Behaviors
Cooperative Play 2 5.54*Positive Statements, Terminations 4 2.12
Prosocial Behavior 13 2.80Sportsmanship 3 2.01
Social Play Skills - Recess 3 3.18SILA 2 1.21*
Seeking Help when lost 1 6.92*
*p<.05
Results
• Dependent Variable– Larger effects are seen when targeting
prosocial behavior over problem behavior
Results
Recipient Director
Recipient n ES
Student 37 3.15*
Classroom 1 2.68
Teacher 1 2.7
Peer 1 2.34
Director n ES
Experimenter 14 3.41
Teacher 10 3.02
Student 4 2.48
Computer 3 2.02*
Coach/Caregiver 3 2.04*
Interventionist 4 3.62*
Peer 2 3.81*
*p<.05 *p<.05
Results
• Recipient and Director– Largest effects are seen when the student
is the target of the intervention and when either an independent experimenter/interventionist or peer is the director of the intervention
Results
• Disability
Disability n ESAutism 13 4.04*
Asperger's 3 4.00*ADHD/ODD 4 2.47ED/BD/EBD 4 2.31
Language Impaired 1 2.68Intellectual Disability 2 1.82*Typically Developing 13 2.48
*p<.05
Results
• Disability*– Interventions are most effective with
children who have disabilities along the Autism spectrum
Results
• Setting
Setting n ES
School / Classroom 23 3.29*
School settings not classroom
6 2.70
Recess 5 2.43*
College Campus 2 2.60
Other 4 3.25*
Results
• Setting– The largest effects were seen when
interventions were implemented in schools
Results
• Number of Components
# Components n ES
2 4 4.02*
3 9 2.66
4 8 3.38*
5 3 2.31
6 10 3.02
7 4 3.34
8 1 1.84*
9 1 2.68
Results
• Components**– Increasing the number of components in
an intervention did not increase the magnitude of effect
• Size may not matter, quality matters
Results
• Treatment Integrity
Integrity n ES
Monitored 3 2.55
No 19 2.71
Yes 18 3.46*
Results
• Treatment Integrity*– Studies that report percentage of integrity
had largest effects
Results
• Reinforcement
Reinforcement Provided n ES
Yes 23 2.88
No 17 3.29
Results• Reinforcement**
– No difference in these studies in regards to reinforcement
• Could be a definitional issue. Coded articles that specified reinforcement given, not necessarily lack of reinforcement
– Reinforcement should be provided if necessary to acquire behavior change
• Quality of intervention may be more important for some
Results
• Study Quality
Quality n ES
3-3.5 5 3.49
4-4.5 22 2.30
5 13 4.16
Results
• Study Quality*– Highest effects were found in studies that
had highest quality ratings, although not statistically significant
– Studies/interventions should be implemented with highest quality possible
Summary of ResultsContrasting Significantly Higher Significantly Lower
IV Self Awareness(1) Computer-Based IVsDV Prosocial Behavior Problem Behavior
Recipient Student
Director Peer, Interventionist Computer, Coach/CaregiverDisability(2) Autism Spectrum Intellectual Disability
Setting School RecessComponents(3) Less More
Integrity Taken(2) Yes NoReinforcement(3) Neither NeitherStudy Quality Higher(4) Lower
1) Mixture of Ivs, all having self awareness qualities2) Matches effect of group design study3) Opposite of effect of group design study4) Not significantly higher, but higher, and similar to our group design finding
Limitations
• Effect size estimator not entirely accurate
• No correction for small sample size
• Stringent selection criteria / data analytic method responsible for abandonment of nearly 40% of single subject studies– Both parametric and nonparametric methods
would have had a larger n
Discussion and Future Directions• All interventions were effective
– If assessment leads to a social skills deficit, with almost any kind of student, there are interventions that work
• Single subject meta-analyses are not yet as informative/definitive as they could be, but current best practice is still to aggregate magnitude and direction of effects across studies– Calculate results using other available methods
and comparing effects (NASP in Philly 2012????)
Questions and Comments?
• For additional copies of this presentation, either check the NASP website or contact Jeffrey Chenier at [email protected]
Selected Citations
• Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single case designs technical documentation. Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.
• Wolery, M., Busick, M., Reichow, B., & Barton, E.E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. The Journal of Special Education, 44, 18-28.