Class 11 -12
Chapters 5 & Elkins (1989)
Elkin et al: Purpose
Test feasibility of the collaborative clinical trial model
Examine relative efficacy of CBT, IPT, and Medication for Depression
2
NIMH Treatment of Depression Collaborative Research Program
U. of Pittsburg George Washington U. U. of Oklahoma 250 Patients: Major depressive disorder 28 therapists: years experience 2 -27; 71% male
10 psychologists 18 psychiatrists
3
4
Experimental Between-Group Designs
1. Post-Test Only Control2. Pre-Test -- Post-Test Control3. Solomon Four Group (combination of 1 and 2 above)
Factorial Design more than one independent variable; interactions treatment X therapist
or patient characteristic
Dependent Sample Design (Matching)
5
Experimental Between-Group Designs
1. Post-Test Only Control2. Pre-Test -- Post-Test Control3. Solomon Four Group (combination of 1 and 2 above)
Factorial Design - Post Hoc more than one independent variable; interactions
treatment X patient characteristic (depression level at intake)
Dependent Sample Design (Matching)
6
IVs: Experimental Groups:
Cognitive Behavioral Therapy Interpersonal Therapy
16 individual sessions/ 50 min.Medication + Clinical Management* Pill-Placebo + Clinical Management*
1st session 55 min.; then 20 to 25 min.
* Minimal supportive therapy condition
Dependent Variables
Clinical Evaluator
Self Report
7
Dependent Variables
Clinical Evaluator
• Hamilton Rating Scale Depression (HRSD)
• Global Assessment Scale (GAS)
Self Report
• Beck Depression Inventory (BDI)
• Hopkins Symptom Checklist (HSCL-90)
8
Outcome Research Strategies
Primary Analyses
Secondary Analyses (Post-Hoc)
9
Outcome Research Strategies
Primary AnalysesTreatment packageComparative
Secondary AnalysesClient Variation -moderation effect?
10
Outcome Research Strategies
Secondary AnalysesClient Variation -moderation effect
depression level at intake as moderator of relation of treatment groups to outcome
Were outcomes across treatment types different for patients with higher versus lower levels of depression at pre-test?
11
Control Groups
CBT IPTMedication + Clinical Management* Pill- Placebo + Clinical Management*
* Minimal supportive therapy condition
12
Treatments & Therapists
Cognitive Behavioral Therapy
Interpersonal Therapy
Different group of experienced therapists
Medication + Clinical Mngmnt
Pill-Placebo + Clinical Mngmnt
Same therapists - psychiatrists
13
Ensure Valid Treatments
Specify the treatment(s)
Therapist training/monitoring
Fidelity Checks
14
Ensure Valid Treatments
Specify the treatment(s)Manuals
Therapist training/monitoringFidelity Checks- therapy tapes
Collaborative Study Psychotherapy Rating Scale (CSPRS)
Treatments could be discriminated 95% of the time
15
Attrition (>15 sessions or 12 weeks)
Total: 77/239 32%CBT 32%IPT 23%Meds/CM 33%Placebo/CM 40%
16
Which group to use in outcome analysis??
Total N = 239 CompletersN = 155
15 weeks or12 sessions
End-Point N = 204
At least 3.5 weeks or4 sessions
End PointN = 239
Intent to Treat Group(last assessment or pre-test)
17
Assessment Times
Pre treatment
Post Treatment4, 8, 12 weeksTermination – 15 weeksFollow up: 6, 12, 18 months
18
19
Analyses of Pre-test/Post-test (1)
Paired T-Test to examine differences between pre-test and post-test scores (p. 974)
How Many ??
Table 1 Completer Group: At least 12 sessions; n=155 (page 975)
20
21
Analyses of Pre-test/Post-test (1)
Paired T-Test to examine differences between pre-test and post-test scores (p. 974)
How Many ??
4 Treatment groups X 4 Outcome measures CBT HRSD IPT GAS IMI-CM BDI Pla-CM HSCL-90 X 3 Samples – Completers; End Point 204; 239
Findings – T-Tests
22
Analyses of Post-test scores Use pre-test as a covariate in analyses of
co-variance to compare mean post-test scores across the 4 treatment groups
Calculate a residualized change scores – amount of variability in the post-test that is not associated with the pre-test score
Used a P<.10 in ANCOVAS and p = .10/.06 = .017 for pair-wise comparisons (p.974)
23
24
ANCOVAS: Post test scores
Statistically significant differences between groups in scales at post-test Four 3 X 4 ANCOVAS: differences across
treatments in Post-treatment scores in: HRSD, GAS --- BDI, HSCL90
3 (sites) X 4 (treatment groups) Analyses reported only for treatment groups
combining them across sites
Co-VariatesPre-test scores Marriage Status (1,2)
Why not MANCOVAS? P.973
25
Table 1 Completer Group: At least 12 sessions; n=155 (page 975) p<.10
26
Table 1 End Point 239 Group CBT IPT IMI-CM PLA-CM p<.10
27
Findings Pair-wise ComparisonsSample Clinical Evaluator Self-Report
Completer N = 155
BDI Pairwise NSHSCL-90-T p=.006
IMI-CM<PLA-CM
EP-204GASIMI-CM<PLA-CM (trend p=.020)
EP-239HRSDep IPT, IMI-CM<PLA-CMGAS p =.010 IMI-CM<PLA-CM
(trend p=.017,.018)
28
29
Measuring Change Elkin et al. 1989
Statistical significance
Clinical significance
30
Measuring Change Elkin et al. 1989
Statistical significance Differences between groups in scales at
post-test controlling for pre-test scores
Clinical significance Percentage of participants that changed
from dysfunctional to functional level (using cut-off scores)
Clinical Significance Recovery Analysis
Proportion of patients who improved vs. not improved
Cut Off Scores Not Depressed HRSD < 6 and BDI < 9 Depressed HRSD > 6 or BDI > 9
Statistical Analyses Chi square: Proportion of depressed and non-
depressed patients across treatment groups at termination.
32
33
34
End Point 239 HRSD p = .04 CBT IPT IMI-CM P-CM
Chi Square (Χ2) tests to what extent the proportion in each group is what may be expected by chance or if it is larger or smaller than expected…….
IPT = IMI-CM>Placebo-CM CBT - % comparison was not sig. for any group
Proportion of cases that met recovery criteria
36%(ns)
Proportion of cases that met recovery criteria
43%
Proportion of cases that met recovery criteria
42%
Proportion of cases that met recovery criteria
21%
Secondary Analyses
To examine effect of pre-treatment severity (HRSD/GAS) on outcome by treatment groupDVs: Post-treatment scores Severity Criteria
HRSD>20 44% of sample GAS<50 41%
Covariate Marital Status
35
2X4 ANCOVA (severity x treatment) DVs- Post Test HRSD, GAS, BDI, HSCL-90
Main Effect for
Main Effect for
(Interaction term)***
36
2X4 ANCOVA (severity x treatment) DVs- Post Test HRSD, GAS, BDI, HSCL-90
Main Effect for Severity More Severe Pre-Test HRSD>20; GAS<50 Less Severe Pre-Test
Main Effect for Treatment CBT IPT IMI-CM P-CM
Severity X Treatment (interaction term)*******37
38
Interaction Effect HRSD Severity x TG Dependent Variables: HRSD* GAS, BDI, HSCL-90 (p.976)
Completer S BDI IPT IMI-CM P-CMHigh Depression
Low Depression
Completer S BDI IPT IMI-CM P-CMHigh Depression
Low Depression
Completer* CBT IPT IMI-CM P-CMHigh HRSD Low HRSD
End Point 239^ CBT IPT IMI-CM P-CMHigh HRSDLow HRSD
4 sets of 3 2X4 Ancovas: 4DVs, 3 sample subgroups *p<.10; ^p<.11
End Point 204* CBT IPT IMI-CM P-CMHigh HRSD
Low HRSD
39
Interaction Effect GAS Severity x TG: Dependent Variables: HRSD GAS, BDI, HSCL-90
Completer S BDI IPT IMI-CM P-CMHigh Depression
Low Depression
Completer S BDI IPT IMI-CM P-CMHigh Depression
Low Depression
Completer** CBT IPT IMI-CM P-CMHigh GASLow GAS
End Point 239* CBT IPT IMI-CM P-CMHigh GASLow GAS
End Point 204**** CBT IPT IMI-CM P-CMHigh GAS
Low GAS
40
Treatment by Severity Interaction/end-point 204 sampleHigher score Negative Outcome Higher Score Positive Outcome
Summary All Pairwise analyses following interaction effects p.976
Less severe groups: no differences across treatment groups
More severe groups IPT more effective than PLA-CM in 3
instances all in the HRSD measure in the END Point Sample 204 (3 out of 4 comparisons)
IMI-CM more effective than PLA-CM across a number of measures (8 out 10 comparisons)
41
42
Figure 2Recovery Rates (%) endpoint /204 sample
Figure 2 Recovery Rates (%) endpoint /204 sample for severity groups (p.977)
Less severe subgroups: NS differences among treatments for all samples with HRSD or GAS.
More severe subgroups for HRSD and GAS: Consistent findings across the three samples IPT>PLA-CM 5/6 and IMI-CM>PLA-CM 6/6
43
Threats to Statistical Conclusion ValidityAre the observed relations among variables accurate?
Power
Unreliability of Measures
Unreliability of Treatment Implementation
Extraneous Variance in the Experimental Setting
Heterogeneity of Participants
44
Threats to Statistical Conclusion ValidityAre the observed relations among variables accurate?
Power • Large N by group range 34-62 +• Outcome measures are well-known +• Power analyses 81-95% for medium effects +• p<.10 for Mancovas and .10/6 for pairwise comp
Unreliability of Measures
Unreliability of Treatment Implementation
• Experienced Therapists – 2-27yrs Mean = 11 +• Manuals, training per treatment group +• Closely monitored +• Taped sessions – 95% correctly classified +
Extraneous Variance in the Experimental Setting
• Not known for the most part -• 28 therapists from 3 – 11 patients each -• no way to control for therapist effects P. 980 -• one site CBT another site IPT similar to
Meds/CM
Heterogeneity of Participants
• Random assignment to groups +• Only included 45% of those screened. + • Mostly women 70% female +• 89% white participants + 45
Threats to Internal ValidityCan we conclude that there is a causal relation between the IV and the DV? Did treatment cause differences in DV across groups? Selection Who gets assigned to which group?
History
Attrition What do we know about drop-outs?
Repeated Testing Effects
Reaction to Control Group Assignment
46
Threats to Internal ValidityCan we conclude that there is a causal relation between the IV and the DV?Selection Used Randomization-
See factors under Heterogeneity of Participants
History Time frame of study not reportedDid therapy happen at about the same time for everyone?
Attrition Relatively high attrition rates - 32% -- about 25% was for negative reasons related to treatment- (-)Early terminators were more depressed at intake (-)
Repeated Testing Effects
Tested at frequent intervals –’ pre-test, 4, 8, 12, weeks, termination 6 12 and 18 months follow-up
Reaction to Control Group Assignment
Not known – but could be the case. Placebo/CM experienced the highest attrition – 32% CBT—23% IPT – 33% Meds/CM -- 40% Placebo/CM
47
Threats to Construct Validity
To what extent variables capture desired constructs
Mono-Operation Bias(Instruments)
Mono-Method BiasExperimenter Expectancies
48
Threats to Construct Validity
To what extent variables capture desired constructs
Mono-Operation Bias(Instruments)
• Used 4 different outcome measures HIRSD, BDI, GAS, HCSL-90 +
• Measures of well-known psychometric properties +
Mono-Method Bias• Used both patient self report and clinician
completed measures +• Measures of well-known psychometric
properties +
Experimenter Expectancies
• Clinicians not blind to therapy modality-• Psychiatrist blind to Med condition +
49
Threats to External ValidityCan we generalize observed relations across persons, settings and times
Person-Units
Outcome Measures
Settings
50
Threats to External ValidityCan we generalize observed relations across persons, settings and times
Person-Units• Highly selected sample (-)• Only 45% screened were selected (-)• Generalizable to white (89%) women (70%)
highly educated (75% coll degree or some coll) who were less severely depressed (p.974)
Outcome Measures• Interview and self –report measures +• Clinical significance recovery rates +• Statistically significant findings were not
consistent across measures – HRSD detected more differences in depression that BDI -
Settings Empirical Question ????
51
52
Results: Summary 1/3
Paired T test showed stat. sig. differences (p<.001) in Pre- Post scores in all measures for all three groups of participants (even placebo pill/CM) Intent-to treat Completers Minimum
3.5<Sessions Completers of all or most sessions
At least 12<sessions > 15 (n=155)
Results: Summary 2/3
ANCOVAS showed no stat sig differences in pre-test scores in
any measure for any treatment group
Stat sig differences in post-test BDI/HSCL90 Completers HSRD/GAS Total Group (239)
54
Results: Summary 3/3
Pairwise Follow-up ANCOVA HSCL-90 IMI-CM> PLA-CM (Completer) GAS -- IMI-CM>PLA-CM (Total 239 group) HRSD IPT, IMI-CM>trend PLA-CM (Total 239)
Recovery Findings (Clinical Significance) IPT, IMI-CM > PLA-CM ( End-Point 239) 43% 42% 21% Post-test HRSD<6 CBT = 36% NS