This article was downloaded by: [Curtin University Library]On: 10 October 2014, At: 09:15Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Psychotherapy ResearchPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/tpsr20

Supervisor variance in psychotherapy outcome in

routine practice

Tony G. Rousmanierea, Joshua K. Swiftb, Robbie Babins-Wagnerc, Jason L. Whippled & SandyBerzinsc

a Student Health and Counseling, University of Alaska Fairbanks, Fairbanks, AK, USAb Department of Psychology, University of Alaska Anchorage, Anchorage, AK, USAc Calgary Counseling Center, Calgary, AB, Canadad Department of Psychology, University of Alaska Fairbanks, Fairbanks, AK, USAPublished online: 02 Oct 2014.

To cite this article: Tony G. Rousmaniere, Joshua K. Swift, Robbie Babins-Wagner, Jason L. Whipple & SandyBerzins (2014): Supervisor variance in psychotherapy outcome in routine practice, Psychotherapy Research, DOI:10.1080/10503307.2014.963730

To link to this article: http://dx.doi.org/10.1080/10503307.2014.963730

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

EMPIRICAL PAPER

Supervisor variance in psychotherapy outcome in routine practice

TONY G. ROUSMANIERE1, JOSHUA K. SWIFT2, ROBBIE BABINS-WAGNER3,JASON L. WHIPPLE4, & SANDY BERZINS3

1Student Health and Counseling, University of Alaska Fairbanks, Fairbanks, AK, USA; 2Department of Psychology,University of Alaska Anchorage, Anchorage, AK, USA; 3Calgary Counseling Center, Calgary, AB, Canada & 4Departmentof Psychology, University of Alaska Fairbanks, Fairbanks, AK, USA

(Received 19 December 2013; revised 22 August 2014; accepted 1 September 2014)

AbstractObjective: Although supervision has long been considered as a means for helping trainees develop competencies in theirclinical work, little empirical research has been conducted examining the influence of supervision on client treatmentoutcomes. Specifically, one might ask whether differences in supervisors can predict/explain whether clients will make apositive or negative change through psychotherapy. Method: In this naturalistic study, we used a large (6521 clients seen by175 trainee therapists who were supervised by 23 supervisors) 5-year archival data-set of psychotherapy outcomes from aprivate nonprofit mental health center to test whether client treatment outcomes (as measured by the OQ-45.2) differeddepending on who was providing the supervision. Hierarchical linear modeling was used with clients (Level 1) nested withintherapists (Level 2) who were nested within supervisors (Level 3). Results: In the main analysis, supervisors explained lessthan 1% of the variance in client psychotherapy outcomes. Conclusions: Possible reasons for the lack of variability betweensupervisors are discussed.

Keywords: supervision; training; psychotherapy outcome

Clinical supervision is widely regarded as an essentialcomponent of psychotherapy training (Bernard &Goodyear, 2014). The primary goals of supervisioncover two broad domains: facilitating the professionaldevelopment of supervisees and enhancing/protectingthe welfare of psychotherapy clients (Falender &Shafranske, 2004). The impact of supervision on thefirst domain (i.e., supervisees) has been well docu-mented. For example, effective supervision hasbeen associated with increased supervisee self-efficacy(Gibson, Grey, & Hastings, 2009), decreased super-visee anxiety (Inman et al., 2014), skill acquisition(e.g., Lambert & Arnold, 1987), encouraged super-visee autonomy and increased supervisee openness(Ladany, Mori, & Mehr, 2012), and reduced confu-sion about professional roles (Ladany & Friedlander,1995). Likewise, the effects of inadequate or harmfulsupervision include increased supervisee anxiety(Gray, Ladany, Walker, & Ancis, 2001), decreased

supervisee self-disclosure (Mehr, Ladany, & Caskie,2010), increased multicultural misunderstandings(Ladany & Inman, 2012), and even causing super-visees to drop out of the field entirely (Ellis, 2001; seealso Ellis et al., 2013). Thus, it can be said with someconfidence that supervisors can have both positiveand negative effects on supervisees.

The impact supervisors have on clients is lessclear and has been the subject of some debate (e.g.,Holloway & Neufeldt, 1995; Ladany & Inman, 2012;Watkins, 2011). In their examination of the theoret-ical connection between supervisors and client out-come, Wampold and Holloway (1997) suggested thatthe wide range of therapist characteristics that caninfluence psychotherapy outcomes may also apply tosupervisors (e.g., therapeutic style and emotionalwell-being; see Beutler, Machado, & Neufeldt,1994). Additionally, characteristics specific to super-visors may affect client outcome, such as experience

Correspondence concerning this article should be addressed to Tony G. Rousmaniere, Student Health and Counseling, University of AlaskaFairbanks, 612 N Chandelar Dr, Fairbanks, AK 99775, USA. Email: trousmaniere@gmail.com

Psychotherapy Research, 2014http://dx.doi.org/10.1080/10503307.2014.963730

© 2014 Society for Psychotherapy Research

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

level as a supervisor and supervisory competence.However, Wampold and Holloway (1997) also raisedsignificant doubts about the impact supervisors mayhave on client outcome due to the many potentiallymediating variables at the supervisor, therapist, andclient levels that separate supervisors from clientoutcome, “Detection of a relation between supervi-sion process and the patient’s rating of patient change(the most distal outcome) would be expected to beextremely small” (Wampold & Holloway, 1997,p. 23).

Despite this debate, improving client welfare iscommonly viewed as the raison d’etre for clinicaltraining, and thus, there have been numerous callsfor examining this topic (e.g., Holloway & Carroll,1996; Lambert & Arnold, 1987). Indeed, the impactof supervision on client psychotherapy outcome hasbeen called the “acid test” of good supervision(Ellis & Ladany, 1997, p. 485). Three reviews ofthe supervision literature have examined this topic,with mixed results. In a review of 10 studies, Freitas(2002) found some relationship between supervi-sion and client treatment outcome. However, thosestudies were all also found to have methodologicalproblems significant enough to raise questionsabout the validity of their findings (e.g., use ofoutcome measures with poor psychometric proper-ties and nonrandom assignment of participants). Ina review of 11 studies, Milne, Sheikh, Pattison, andWilkinson (2011) found that “the blend of train‐ing and supervisory methods … were effective infacilitating supervisor and supervisee (therapist)development, which … was associated with patientbenefits” (pp. 61–62). However, Milne et al. (2011)noted that only 2 of the 11 studies in the re-view examined the direct effects of supervision onclient psychotherapy outcome, so the “clinicaloutcome estimate should be treated with greatcaution” (p. 62). Likewise, in his review of theliterature, Watkins (2011) noted that most studiesthat explored the impact of supervision on clientpsychotherapy outcome relied on supervisors’ orsupervisees’ perceptions, rather than client self-reports and psychometrically sound measures.Regarding the supervisor-to-client treatment out-come connection, Watkins concludes, “After acentury of psychotherapy supervision and over halfa century of supervision research, we still cannotempirically answer that question” (p. 252).

Perhaps the most frequently cited study on thedirect effects of supervision on treatment outcome isBambling, King, Raue, Schweitzer, and Lambert(2006). This study had a randomized that of experi-mental design, wherein 127 clients with a primarydiagnosis of major depressive disorder were ran-domly assigned to 127 licensed therapists for eight

sessions of problem-solving treatment. Half of thetherapists were split into two groups who receivedweekly supervision with either a process-focus orworking alliance skill-focus, based on a supervisionmanual developed for this study; the other half of thetherapists received no supervision. After eight weeksof treatment, clients receiving therapy from thera-pists in the supervision group had significantly higherscores on both the Working Alliance Inventory(WAI; Horvath & Greenberg, 1989) and BeckDepression Inventory (BDI; Beck, Steer, & Garbin,1987), than clients in the no-supervision group.Notably, clients in the two supervision groups hadmuch lower rates of noncompletion at eight sessions(3.0% and 6.1%) than clients in the no-supervisiongroup (35%). Data from this study thus suggest thatsupervision may contribute to client outcome, atleast in controlled experimental conditions, whencompared to licensed clinicians not receiving super-vision. However, trainees cannot practice withoutsupervision, so the implications of these findings topsychotherapy training (at least at the prelicensurestage) are unclear. Additionally, the authors acknow-ledge the possibility that therapist allegiance effectsor working alliance effects (WAI scores were tightlycorrelated with BDI scores) may account for some ofthe findings (Bambling et al., 2006).

Reese et al. (2009) performed a controlled studyin which the outcomes of trainees receiving supervi-sion that included regular outcome feedback (n = 9)were compared to the outcomes of trainees receivingsupervision without regular outcome feedback(n = 10). Data included 115 psychotherapy casescollected over the course of year. Trainees in thesupervision-with-feedback condition had signifi-cantly better outcomes than trainees receiving super-vision without feedback. Notably, no significantdifferences were found between supervisors withinthe treatment conditions (Reese et al., 2009).

Another approach to this question is to examinewhether clients vary in their psychotherapy out-comes, depending on who is assigned to supervisethe case. Analogous to how psychotherapy researchhas been able to identify “supershrinks” (Okiishi,Lambert, Nielsen, & Ogles, 2003; Miller, Hubble, &Duncan, 2008), it may be possible to identify“super-supervisors,” based on their supervisees’ out-comes. Callahan, Almstrom, Swift, Borja, and Heath(2009) took this approach to examining the effects ofsupervision on client outcome using naturalisticarchival data from a psychology department trainingclinic that tracked client outcome as part of routinepractice with two measures: the Symptom Checklist-90, Revised (Derogatis, 1992) and the Beck Depres-sion Inventory-II (BDI-II; Beck, Steer, & Brown,1996). Participants in their study were 76 adult

2 T. G. Rousmaniere et al.

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

psychotherapy clients who were randomly assignedto 40 trainee therapists who were in supervision withnine supervisors. Supervisors were found to havea significant effect on client treatment outcome,accounting for 16% of the variance. However, thestudy’s small sample size (the mean number of casesper supervisor was eight, and six supervisors had fiveor fewer cases) makes these findings difficult toreliably generalize.

Thus, there is preliminary evidence from bothexperimental and naturalistic data that supervisorsmay influence or at least predict client outcome.However, the existing research is limited by consist-ing of only a few studies with relatively small samplesizes. Thus, this topic warrants further study. Thepurpose of our current study was to further explorethe variance of impact of supervisors on clientpsychotherapy outcome in a naturalistic trainingsetting by using a large data-set and more sensitivestatistical procedures (hierarchical linear modeling,HLM) than have been used in previous studies.Specifically, our goal was to examine the amount ofvariance in client psychotherapy outcome accountedfor by supervisors, in a manner similar to recentmeta-analytic research examining the factors con-tributing to psychotherapy outcome (e.g., Wampold,2010). If supervisors were found to explain a signi-ficant amount of variance, we then planned examin-ing whether a set of supervisor level variables(supervisor demographics, experience, and type)could predict the amount of change that clientsmade through psychotherapy.

Method

Participants

Archival data from a large private nonprofit,community-based counseling center in WesternCanada were used for this study. The counselingcenter tracks the clinical outcome of all clients, on asession-by-session basis. A total of 7929 adult clientswere seen in the counseling center by supervisedtrainee therapists over a 5-year period. However, 886of those clients had no OQ data for any of theirsessions and 392 were missing an intake OQ score,and thus were removed from the data file. For thosewho were missing a last session OQ score (n = 759),we used a last observation carried forward method(the last OQ score that was recorded was used astheir end score). In order to ensure that therapist/supervisor case averages were not based on a singleclient/supervisee, only therapists who saw at leasttwo clients and supervisors who provided supervi-sion for at least two therapists were retained in ourdata-set. These minimum criteria resulted in the

removal of 20 of the 195 therapists and 3 of the 26supervisors. The final sample included 23 super-visors, 175 therapists, and 6562 clients.

Clients. Over half (57.0%) of the client partici-pants were females, 42.5% were males, and 0.5%did not declare their gender. Of the client partici-pants, 33% were married or in a common-lawrelationship, 15.9% were separated or divorced,43.4% were single, and 7.6% were other or did notdeclare their relationship status. For education,32.8% of the client participants had some highschool or less, 34.2% had a college or technicalschool degree, 30.4% had a university degree, and2.5% had other or did not declare their level ofeducation. For employment, 70.6% of client partici-pants who indicated an employment status workedfull-time, 11.6% worked part-time, 6.8% were stu-dents, 1.6% were retired, 3.7% were on disabilityleave, and 5.8% were other. The average age forparticipants was 37.72, SD = 11.24, ranging from18 to 92 years. Clients presented with a range ofconcerns typical for community mental health cen-ters, including family/marital problems (45.9%) andpersonal functioning problems, including stress,depression, eating disorders, and anxiety disorders(51.7%), occupational/vocational issues, includingjob satisfaction, work conflict, and career pathchoices (1.1%), and social/community concerns,including social isolation, lifestyle choices, financial,or legal problems (1.3%) as their primary concern.Most (83.0%) of the clients were provided individualcounseling, and 17% were provided couples’ coun-seling. The average intake OQ score for these clientswas 71.33, SD = 26.32, and they attended anaverage of 4.81 sessions (SD = 5.36), ranging from1 to 92.

Therapists. There were two categories of super-visees (n = 175) in this data: practicum students andresidents. Practicum supervisees were students whowere still completing their master’s level courses insocial work, psychology, marriage and family therapy,and pastoral counseling. Resident supervisees hadgraduated with a master’s degree in psychology, socialwork, or marriage and family therapy and wereobtaining hours toward their registration as a licensedpractitioner. Supervisees were ages 23–mid-50s. Prac-ticum supervisees were randomly matched with asupervisor for a single 8-month assignment. Residentsupervisees were randomly matched with a supervisorfor a single 1-year assignment. Supervisor degree levelwas not a factor in trainee assignment: both practicumand resident supervisees could be assigned to masters-or doctoral-level supervisors. A few (<5%) residentscontinued working at the center for additional years.

Psychotherapy Research 3

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

Both practicum and resident supervisees received onehour of individual and two hours of group supervisionper week, from the same supervisor. All therapists hadonly one supervisor at a time. The included therapiststreated an average of 37.50 (SD = 43.48) clients.Attempts were made to obtain additional demographicdata on therapists. However, the data were drawnfrom archival records, and the supervisees in the datahave since left the center, so additional demographicdata were not available.

Supervisors. The included supervisors (n = 23)supervised an average of 11.96, SD = 9.18, therapistsand an average of 285.30, SD = 349.10, clients. Most(69.57%) of the supervisors were female, and 30.43%weremale. About half (47.48%) of the supervisors hadMS Psych degrees, 30.43% had MSW degrees, and21.74% had Ph.D. degrees. Supervisors were of ages28–64 years and 60.87% had 1–5 years of supervisoryexperience, 17.39% had 6–10 years, 8.70% had 11–15years, 8.70% had 16–20 years, and 4.35% had over 20years of supervisory experience. Supervisors at thecenter are required to work from a therapeutic mod-ality, but the choice of modality is left to the super-visor. The supervisors in this data-set practiced awide range of modalities (e.g., cognitive-behavioral,psychodynamic, solution-focused, family systems,Ericksonian, and strategic). Therapeutic modalitieswere roughly evenly distributed among supervisors,with no single modality used by more than 20% ofsupervisors. All supervisors had taken at least oneacademic course in supervision, and a few had takenmultiple courses. Attempts were made to obtainadditional demographic data on supervisors. How-ever, the data were drawn from archival records, andapproximately 65% of the supervisors have since leftthe center, so additional demographic data were notavailable.

Measure

The self-report OQ–45.2 (Lambert et al., 1996) wasused as the outcome measure in this study. The OQ-45.2 was designed to help therapists monitor clients’progress in therapy via weekly administrations overthe course of treatment. The OQ-45.2 is comprisedof 45 items. Each item is rated on a 5-point Likertscale; high scores indicate more disturbance. Theitems on the OQ-45.2 can also be divided into threesubscale scores: Subjective Discomfort (intrapsychicfunctioning, e.g., “I feel blue”), Interpersonal Rela-tionships (e.g., “I feel lonely”), and Social RolePerformance (e.g., “I feel stressed at work/school”).The score of all items can be combined for a totalscore of 0–180. For this study, only the OQ-45.2total score was used. The OQ-45.2 is regarded to

have adequate psychometric properties: The manualreports an internal consistency of .93, a test–retestreliability of .87, and high concurrent validity with anumber of other measures, including the SymptomChecklist-90-Revised, Beck Depression Inventory,State-Trait Anxiety Inventory, Inventory of Interper-sonal Problems, and Social Adjustment Scale(Lambert et al., 2004). The OQ-45.2 has demon-strated sensitivity to change for clients in treatment,while remaining stable for people not in treatment(Vermeersch, Lambert, & Burlingame, 2000), andthe manual reports no differences based on gender(Lambert et al., 2004).

Procedure

Therapists and clients conducted therapy as usualthroughout the period that archival data were col-lected. Most (>95%) of clients were randomlyassigned to therapists (supervisees). Less than 5%of clients were assigned to specific therapists due tospecific requests by the client (e.g., pastoral counsel-ing). Clients paid $1 to over $160 per session forpsychotherapy. Some (<15%) clients paid for coun-seling with private insurance. All adult clients at thecenter complete the OQ-45.2 prior to every psycho-therapy session, as part of routine treatment. Thera-pists and supervisors were able to review clients’OQ-45.2 scores throughout the course of therapy.All clients and their resultant data were treated inaccordance with the Ethical Principles of Psycholo-gists and Code of Conduct.

Data Analysis Plan

In this study, we were interested in testing whetheror not client outcomes differed between the includedsupervisors. Given that clients were nested withintherapists, and therapists were nested within super-visors, HLM with maximum likelihood estimationwas utilized to test for variance at both of theselevels. The baseline model predicting client OQchange scores (defined as the difference betweenstart OQ and end OQ scores) included therapistsat Level 2, and supervisors at Level 3, with theintercepts modeled as random effects. The Waldstatistic was used to test the variance at the therapistand supervisor levels. Additionally, interclass corre-lations (ICC) were calculated in order to identify theamount of variance in client OQ change scores thatwas explained by therapists and supervisors. Onemight hypothesize that the impact of the supervisorpartially depends on the experience level of thesupervisee (i.e., supervisors may play a biggerrole with less-experienced supervisees and thusmore supervisor variability would be seen with a

4 T. G. Rousmaniere et al.

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

sample of less-experienced therapists). Thus, wenext tested this interaction hypothesis by enteringthe therapist type (practicum students vs. residents)at Level 2 and comparing a model where supervisorvariance was held constant across therapist type (theoriginal baseline model) to a model where supervisorvariance was allowed to vary depending on thetherapist type. Last, we were interested in examiningwhether supervisor characteristics could predict cli-ent outcomes. Supervisor field (social work vs.psychology), level of education (MS vs. Ph.D.),and years of experience providing supervision wereadded to the baseline model as fixed effects. Com-parison of the −2LL values between models wasused to determine if this model was a better fit forthe data compared to the baseline model. Althoughother variables could have been added at Levels 1and 2 that may have explained a significant amountof variance in client OQ change scores, given thefocus of this manuscript, we chose to only test Level3 (supervisor) predictors. In running these analyses,we removed 41 cases (14 who demonstrated extremenegative change and 27 who demonstrated extremepositive change) that were determined to be outliers(z value >3.5) on the main outcome variable (OQchange scores).

Results

On average, the 6521 client participants improved byM = 8.81, SD = 17.30, points on the OQ-45.2 overthe course of their treatment, ranging from animprovement of 73 points to a deterioration of 55points. At the therapist level, therapists on averagesaw a M = 8.26, SD = 5.40, point change for theirclients, ranging from an average improvement of27.5 points for one therapist to an average deteri-oration of 11.33 points for another therapist. At thesupervisor level, supervisors on average saw aM = 8.72, SD = 2.52, point change for their clients,ranging from an average improvement of 14.00 forone supervisor to an average improvement of 2.21points for another supervisor. Figure 1 plots thesupervisors’ means for their therapists’ and clients’average change. Overall, 4.9% of the sample dete-riorated (OQ score showed a 14 point or greaterincrease from treatment from start to end), 64.9%displayed no change on the OQ (less than 14 pointincrease or decrease from start to end), 8.4% made areliable improvement (14 point or greater decreasefrom start to end) but were still in the clinical range(64 or greater) upon terminating therapy, and 21.9%made a clinically significant change by the end oftreatment (14 point or greater decrease and an endOQ score in the nonclinical range). Percentages ofclient deterioration, no change, reliable improvement,

and clinically significant change are reported sepa-rately for each supervisor in Table I.

Parameter estimates and −2LL values for thedifferent models that we tested can be found inTable II. The first baseline model was used todetermine whether therapists (Level 2) and super-visors (Level 3) differ among themselves in averageclient outcomes. In this baseline model, the esti-mated variance between therapists was significant,estimated variance = 2.51, 95% CI [1.13, 5.57], WaldZ = 2.46, p < .05; however, the estimated variancebetween supervisors was not, estimated variance =0.12, 95% CI [0.00, 19.64], Wald Z = 0.38, p > .05.Calculation of ICCs indicated that only a smallamount of variance (0.84%) in client OQ changescores was accounted for by differences betweentherapists, and virtually no variance (0.04%) in clientOQ change scores was accounted for by differencesbetween supervisors.

In order to test for an interaction between super-visor variance and therapist type (residents vs.interns), in this second model, we allowed thesupervisor variance to vary between the two therapisttypes. A comparison of the −2LL values between themodels where supervisor variance was held constantacross therapist type (−2LL = 55,670.90) and themodel where supervisor variance was allowed todiffer depending on the therapist type (−2LL =55,671.09) indicated that the second model did notfit the data any better than the first, χ2 = 0.19, p >.05. The lack of significant findings indicates that inthis sample client outcomes did not vary betweensupervisors any differently depending on whether thetreating clinician was an intern or a resident.

Although supervisors were found not to differ in theamount of change that was made by the clients whowere seen by the therapists that they supervised, it isstill possible that some supervisor characteristics could

–10

–5

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Aver

age

clie

nt O

Q c

hang

e

Supervisor

Figure 1. Average client change on the OQ-45.2 with 95% CIerror bars plotted by supervisors.

Psychotherapy Research 5

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

predict the change that clients made. In our finalmodel, we added supervisor experience, supervisorfield (social work vs. psychology), and supervisor level(MS vs. Ph.D.) to the original baseline model withboth types of trainee therapists (practicum interns andresidents). A comparison of the −2LL values indicatedthat this model was not significantly better than thebaseline model, χ2(3) = 2.43, p > .05, thus indicatingthat taken together these supervisor variables did notexplain a significant amount of variance in clientoutcomes above what was explained by the baseline

model. Additionally, given the full model, there wasno evidence that client outcomes differed dependingon the supervisor’s level of experience, t(553.19) =0.77, p > .05, whether the supervisor’s field was socialwork or psychology, t(701.13) = 0.48, p > .05, andwhether the supervisor had obtained a MS or a Ph.D.,t(277.76) = 1.63, p > .05.

In addition to these main tests, we also conducted aseries of five analyses to test the sensitivity of the resultsto the sample assumptions. First, we conducted allanalyses without inputting a last observation carried

Table II. Parameter estimates for the models examining client OQ change scores with therapists modeled at Level 2 and supervisors atLevel 3.

Baseline model withboth types oftherapists

Model wheresupervisor varianceis allowed to vary by

therapist typeFull model with bothtypes of therapists

Fixed effectsIntercept 8.67* 8.70* 7.70*Supervisor experience 0.22Supervisor field (social work vs. psychology) –0.28Supervisor level (MS vs. Ph.D.) 1.14

Random effectsTherapists 2.51* 2.63* 2.40*Supervisors 0.12 0.00 0.00Residual 296.63* 296.47* 296.67*

Deviance (−2LL) 55,670.90 55,671.09 55,668.47

*p < .01.

Table I. Percentages of client change reported separately for each supervisor.

SupervisorNo. ofcases

Avg. OQ-45change

Percentage ofdeterioration

Percentage of nochange

Percentage of reliableimprovement

Percentage of clinicallysignificant change

1 24 2.21 12.5 75.0 12.5 0.02 42 4.95 4.8 78.6 4.8 11.93 166 6.20 2.4 75.9 7.8 13.94 454 7.01 8.4 62.8 7.0 21.85 51 7.65 5.9 74.5 3.9 15.76 111 7.89 5.4 65.8 10.8 18.07 821 7.91 5.6 65.9 9.0 19.58 221 7.94 7.2 63.8 9.0 19.99 452 8.02 4.0 67.9 7.5 20.610 52 8.02 3.8 65.4 21.2 9.611 265 8.31 2.3 68.3 9.8 19.612 259 8.51 5.4 64.5 8.9 21.213 365 8.97 4.4 64.4 7.7 23.614 177 9.05 4.0 66.7 8.5 20.915 59 9.19 5.1 64.4 8.5 22.016 163 9.39 4.9 62.6 9.8 22.717 1583 9.64 4.4 63.7 7.4 24.518 453 9.96 4.2 64.0 9.1 22.719 369 9.98 4.1 61.2 6.5 28.220 362 10.02 5.0 62.2 10.5 22.421 10 12.60 10.0 70.0 0.0 20.022 46 13.24 6.5 56.5 15.2 21.723 16 14.00 0.0 56.3 12.5 31.3

6 T. G. Rousmaniere et al.

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

forward score for those clients who were missing a lastsession OQ score. The only difference in resultsbetween this sample and the original sample reportedabove was that in the baseline model, with both typesof trainees (residents and interns) the variance betweentherapists was not significant. Second, as a moreinclusive approach, we conducted all analyses withoutdeleting any therapist or supervisors (even if they onlyworked with one client/supervisee). No differenceswere found between the results using this sample andthe original sample. Third, as a more conservativeapproach, we conducted all analyses after removingany therapist if they treated less than five clients (27additional therapists were removed) and any supervisorif they supervised less five therapists (five additionalsupervisors were removed). The only difference inresults between this sample and the original samplewas that in the final model cases supervised by MS-level supervisors on average displayed a significantlygreater change (1.47 points greater) than cases super-vised by Ph.D.-level supervisors, t(235.32) = 2.02, p =.05. However, it should be noted that although thissupervisor variable was significant, the overall modelwith this and other supervisor variables entered did notexplain variance in client outcomes any better than theempty model, χ2(3) = 4.04, p > .05. Fourth, weconducted all analyses without removing any outliers.Two differences with the original sample wereobserved: (i) with both types of trainees (residentsand interns) the variance between therapists was notsignificant, and (ii) cases supervised by MS-levelsupervisors on average displayed a significantly greaterchange (1.62 points greater) than cases supervised byPh.D.-level supervisors, t(266.5) = 2.19, p = .03. Forthe last sensitivity analysis, we examined whether thepresence of couples counseling cases in the dataaffected the results. Given the nature of couplescounseling, data from these clients violate the assump-tion of independence of observations. Rather thanincluding another level of nesting for these clients, wealso conducted all analyses with the couples casesremoved. No differences between the results using thissample and the original sample were found.

Discussion

The purpose of this study was to explore the amountof variance in client psychotherapy outcomeaccounted for by supervisors, in a naturalistic train-ing setting. All available outcome data from thera-pists and supervisees working at a community-basedcounseling center over a five-year period wereincluded. The data thus present a comprehensivepicture of how supervision is routinely being prac-ticed at a large counseling center.

The main finding of this study was that, for a five-year period at this mental health center, super-visors accounted for .04% of the variance in clientpsychotherapy outcome. The lack of variabilitybetween supervisors in client outcomes was demon-strated across variables at the supervisor level—supervisor experience level, field (social work vs.psychology), and degree (MS vs. Ph.D.)— as wellas the trainee level (residents vs. interns). The reasonfor the difference in findings is not immediately clear.One possibility may be that this study included amuch larger sample size at all three levels (supervisor,therapist, and client), and utilized a different statist-ical test (HLM) than either of the previous studies.Alternately, the difference in results may be due tothe different samples in each study: all of the super-visors in this sample may have achieved a similar levelof competence in supervision, and thus had similarlypositive outcomes, in contrast to the no-supervisioncondition in Bambling et al. (2006), and the super-visors in Callahan et al. (2009), who may have haddifferent levels of supervision competency, and thusbetter or worse outcomes. Another possible explana-tion for the difference in results is that all supervisorsand trainees in this study were receiving session-by-session outcome feedback, unlike the participantsin Bambling et al. (2006) and Callahan et al. (2009).Given that the use of regular feedback in supervisionhas been shown to significantly improve outcomes(e.g., Lambert & Shimokawa, 2011; Reese et al.,2009), it is possible that the ubiquitous use ofoutcome feedback itself may have been powerfulenough to obscure the outcome variance due toother supervision variables, and thus made the effectsof all supervisors similar.

One possible reason for the lack of difference inclient outcomes between supervisors may be con-founding variables within the chain of supervisionwhich we were unable to control for in this study.Supervision theorists have proposed that many vari-ables at the supervisor, therapist, and client levels maymoderate supervisor effects on client outcome, andthus reduce the amount of variance in outcome thatmay be attributed to supervisors (e.g., Wampold &Holloway, 1997). To affect client outcome, super-visors’ interventions have to, in effect, travel throughthree layers of mediating variables: client variables,therapist variables, and supervisor variables. Recentresearch suggests that the mediating effects of vari-ables at all three levels may be quite strong. At theclient level, multiple variables have been shownto greatly moderate the effects of psychotherapy(e.g., the therapeutic working alliance, Norcross &Lambert, 2011; client expectations for change, Swift,Greenberg, Whipple, & Kominiak, 2012). Indeed, ithas been suggested that client variables themselves

Psychotherapy Research 7

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

may account for most of the variance in psychother-apy outcome (e.g., symptom severity and treatmenthistory; Bohart & Tallman, 2010). If this is accurate,then these variables may serve to minimize variance insupervisor effects on outcome.

Likewise, research has identified multiple variablesat the supervisee level that may moderate super-visors’ effects on client outcome. The most promin-ent variable is the supervisory working alliance,which has been demonstrated to vary widely acrosssupervisory dyads (e.g., Inman et al., 2014). Otherpotentially mediating supervisee variables include theextent of supervisee nondisclosure, which has beenshown to frequently be quite high (Mehr et al., 2010),and supervisees’ perceptions of the extent of collab-oration in supervision, which may often be low(Rousmaniere & Ellis, 2013). Additionally, variablesat the supervisor level may moderate supervisors’effects on outcome, such as the method or modelfor supervision (Bernard & Goodyear, 2014). Forexample, live one-way-mirror supervision mayaddress the possibility that supervisees may forgetlessons from supervision (e.g., Rousmaniere & Fre-derickson, 2013).

A significant limitation of this study was the lack ofa control group of trainees not receiving supervision(due to legal and ethical concerns), which limited thestudy to exploring the variation of treatment out-come associated with supervisors, rather than ifsupervisors had an effect on treatment outcome atall (unlike Bambling et al., 2006). Thus, the findingsdo not imply supervisors in this study did notenhance or protect client welfare. Rather, the findingsuggests that client psychotherapy outcome wasextremely similar when aggregated across super-visors. Given that the mean change for all clientswas 8.81 points on the OQ-45.2, the findings maysuggest that, on average, competent client treatmentwas provided, under the care of these supervisors.

Another limitation is the small sample of super-visors, which could have contributed to the mixedfindings. Additionally, all data came from one mentalhealth center, and it is unclear if the findings of thisstudy generalize to other locations. For example, it ispossible that this counseling center may just have verysimilar supervisors (e.g., most are females), thusexplaining the findings of minimal difference betweensupervisors. However, this is unlikely, given that the23 supervisors who participated practice a range ofclinical modalities. Thus, this study should be repli-cated at other locations. Another limitation is that thetwo hours per week of group supervision that practi-cum and resident supervisees received (from the samesupervisor) may have influenced the results. Forexample, good advice from peers may have moder-ated the impact of bad supervision, and vice-versa.

Another limitation is that all supervisors, therapists,and patients were provided with OQ-45.2 feedback,so it was impossible to assess whether feedback insupervision had a moderating effect. Other limitationswere that data were only obtained via client self-reportfrom one outcome measure (the OQ-45.2), and thatlimited demographic information was obtained onsupervisors and supervisees (due to confidentialityconcerns and availability of records).

Implications for Theory and Research

Supervision literature has widely conceptualizedsupervisors as one of, if not the, primary gatekeeperstasked with ensuring and protecting client welfare(Bernard & Goodyear, 2014). However, there hasbeen theoretical debate about the extent to whichsupervisors can affect client outcome (e.g., Ladany &Inman, 2012; Wampold & Holloway, 1997), andthis debate has been informed by little empirical data(e.g., Watkins, 2011). The findings from this studyallow us to explore one perspective on this question:the degree of variance in supervisor effects on clientoutcome, from the actual practice of 23 supervisorsand 175 supervisees, with 6521 psychotherapy cases.One possible positive interpretation of the minimalvariance is that the overall average welfare of 6521clients was ensured and protected because the meanclient outcome improved by 8.81 points. This couldbe taken as an indication that the supervisors in thestudy were overall successful at protecting clientwelfare. Future research on this topic would benefitfrom controlling for the many variables that maymoderate supervisors’ effects on client outcome atthe supervisor, supervisee, and client levels (e.g., thesupervisory working alliance, and the influence ofpeers in group supervision). It is our hope that futureresearch will further explore these importantquestions.

Acknowledgment

The authors would like to thank Michael Ellis forhelp with this manuscript.

ReferencesBambling, M., King, R., Raue, P., Schweitzer, R., & Lambert, W.

(2006). Clinical supervision: Its influence on client-ratedworking alliance and client symptom reduction in the brieftreatment of major depression. Psychotherapy Research, 16, 317–331. doi:10.1080/10503300500268524

Beck, A. T., Steer, R. A., & Brown, G. K. (1996). The BDI-IImanual. London: Harcourt Brace.

Beck, A. T., Steer, R. A., & Carbin, M. G. (1987). Psychometricproperties of the beck depression inventory: Twenty-five years of

8 T. G. Rousmaniere et al.

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

evaluation. Clinical Psychology Review, 8(1), 77–100. doi:10.1016/0272-7358(88)90050-5

Bernard, J. M., & Goodyear, R. K. (2014). Fundamentals of clinicalsupervision (5th ed.). Needham Heights, MA: Allyn & Bacon.

Beutler, L. E., Machado, P. P., & Neufeldt, S. (1994). Therapistvariables. In A. E. Bergin & S. Garfield (Eds.), Handbook ofpsychotherapy and behavior change (4th ed. pp. 229–269).Oxford: John Wiley & Sons.

Bohart, A. C., & Tallman, K. (2010). Clients: The neglectedcommon factor in psychotherapy. In B. L. Duncan, S. D. Miller,B. E. Wampold, & M. A. Hubble (Eds.), The heart and soul ofchange: Delivering what works in therapy (2nd ed. pp. 83–111).Washington, DC: American Psychological Association.

Callahan, J. L., Almstrom, C. M., Swift, J. K., Borja, S. E., &Heath, C. J. (2009). Exploring the contribution of supervisorsto intervention outcomes. Training and Education in ProfessionalPsychology, 3(2), 72–77. doi:10.1037/a0014294

Derogatis, L. R. (1992). SCL-90-R: Administration, scoring &procedures manual-II (2nd ed.). Towson, MD: Clinical Psycho-metric Research, Inc.

Ellis, M. V. (2001). Harmful supervision, a cause for alarm:Comment on. Journal of Counseling Psychology, 48, 401–406.doi:10.1037/0022-0167.48.4.401

Ellis, M. V., Berger, L., Hanus, A. E., Ayala, E. E., Swords, B. A.,& Siembor, M. (2013). Inadequate and harmful clinicalsupervision: Testing a revised framework and assessing occur-rence. The Counseling Psychologist, 42(4), 434–472.doi:10.1177/0011000013508656

Ellis, M. V., & Ladany, N. (1997). Inferences concerning super-visees and clients in clinical supervision: An integrative review.In C. E. Watkins (Ed.), Handbook of psychotherapy supervision(pp. 447–507). New York, NY: Wiley.

Falender, C. A., & Shafranske, E. P. (2004). Clinical supervision: Acompetency-based approach. Washington, DC: American Psy-chological Association.

Freitas, G. J. (2002). The impact of psychotherapy supervision onclient outcome: A critical examination of 2 decades of research.Psychotherapy: Theory, Research, Practice, Training, 39, 354–367.doi:10.1037/0033-3204.39.4.354

Gibson, J. A., Grey, I. M., & Hastings, R. P. (2009). Supervisorsupport as a predictor of burnout and therapeutic self-efficacyin therapists working in ABA schools. Journal of Autism andDevelopmental Disorders, 39, 1024–1030. doi:10.1007/s10803-009-0709-4

Gray, L. A., Ladany, N., Walker, J. A., & Ancis, J. R. (2001).Psychotherapy trainees’ experience of counterproductive eventsin supervision. Journal of Counseling Psychology, 48, 371–383–383. doi:10.1037/0022-0167.48.4.371

Holloway, E., & Carroll, M. (1996). Reaction to the specialsection on supervision research: Comment on Ellis et al.(1996), Ladany et al. (1996), Neufeldt et al. (1996), andWorthen and McNeill (1996). Journal of Counseling Psychology,43(1), 51–55. doi:10.1037/0022-0167.43.1.51-55

Holloway, E. L., & Neufeldt, S. A. (1995). Supervision: Itscontributions to treatment efficacy. Journal of Consulting andClinical Psychology, 63, 207–213. doi:10.1037/0022-006X.63.2.207

Horvath, A. O., & Greenberg, L. S. (1989). Development andvalidation of the working alliance inventory. Journal of Counsel-ing Psychology, 36, 223–233. doi:10.1037/0022-0167.36.2.223

Inman, A. G., Hutman, H., Pendse, A., Devdas, L., Luu, L., &Ellis, M. V. (2014). Current trends concerning supervisors,supervisees, and clients in clinical supervision. In C. E.Watkins & D. Milne (Eds.), The international handbook ofclinical supervision (pp. 61–102). Oxford: Wiley Press.

Ladany, N., & Friedlander, M. L. (1995). The relationshipbetween the supervisory working alliance and trainees' experi-ence of role conflict and role ambiguity. Counselor Educationand Supervision, 34, 220–231. doi:10.1002/j.1556-6978.1995.tb00244.x

Ladany, N., & Inman, A. G. (2012). Training and supervision. InE. M. Altmaier & J. C. Hansen (Eds.), The Oxford handbook ofcounseling psychology (pp. 179–207). New York, NY: OxfordUniversity Press.

Ladany, N., Mori, Y., & Mehr, K. E. (2012). Effective andineffective supervision. The Counseling Psychologist, 41(1), 28–47. doi:10.1177/0011000012442648

Lambert, M. J., & Arnold, R. C. (1987). Research and thesupervisory process. Professional Psychology: Research and Prac-tice, 18, 217–224. doi:10.1037/0735-7028.18.3.217

Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K.,Okiishi, J., Burlingame, G. M., & Reisinger, C. W. (1996).Administration and scoring manual for the OQ-45.2. Stevenson,MD: American Professional Credentialing Services.

Lambert, M. J., Morton, J. J., Hatfield, D., Harmon, C., Hamilton,S., Reid, R. C., … Burlingame, G. M. (2004). Administration andscoring manual for the outcome questionnaire, 45.2. Salt Lake City,UT: OQ Measures.

Lambert, M. J., & Shimokawa, K. (2011). Collecting clientfeedback. Psychotherapy, 48(1), 72–79. doi:10.1037/a0022238

Mehr, K. E., Ladany, N., & Caskie, G. I. L. (2010). Traineenondisclosure in supervision: What are they not telling you?Counselling and Psychotherapy Research, 10(2), 103–113.doi:10.1080/14733141003712301

Miller, S., Hubble, M., & Duncan, B. (2008). Supershrinks.Therapy Today, 19, 4–9.

Milne, D., Sheikh, A., Pattison, S., & Wilkinson, A. (2011).Evidence-based training for clinical supervisors: A systematicreview of 11 controlled studies. The Clinical Supervisor, 30(1),53–71. doi:10.1080/07325223.2011.564955

Norcross, J. C., & Lambert, M. J. (2011). Evidence-based therapyrelationships. In J. C. Norcross (Ed.), Psychotherapy relation-ships that work (2nd ed., pp. 3–21). New York, NY: OxfordUniversity Press.

Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003).Waiting for supershrink: An empirical analysis of therapisteffects. Clinical Psychology & Psychotherapy, 10, 361–373.doi:10.1002/cpp.383

Reese, R. J., Usher, E. L., Bowman, D. C., Norsworthy, L. A.,Halstead, J. L., Rowlands, S. R., & Chisholm, R. R. (2009).Using client feedback in psychotherapy training: An analysis ofits influence on supervision and counselor self-efficacy. Train-ing and Education in Professional Psychology, 3, 157–168.doi:10.1037/a0015673

Rousmaniere, T. G., & Ellis, M. V. (2013). Developing theconstruct and measure of collaborative clinical supervision:The supervisee’s perspective. Training and Education in Profes-sional Psychology, 7, 300–308. doi:10.137/a0033796

Rousmaniere, T., & Frederickson, J. (2013). Internet-basedone-way-mirror supervision for advanced psychotherapytraining. The Clinical Supervisor, 32(1), 40–55. doi:10.1080/07325223.20130.778683

Swift, J. K., Greenberg, R. P., Whipple, J. L., & Kominiak, N.(2012). Practice recommendations for reducing prematuretermination in therapy. Professional Psychology: Research andPractice, 43, 379–387. doi:10.1037/a0028291

Vermeersch, D. A., Lambert, M. J., & Burlingame, G. M.(2000). Outcome questionnaire: Item sensitivity to change.Journal of Personality Assessment, 74, 242–261. doi:10.1207/S15327752JPA7402_6

Psychotherapy Research 9

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014

Wampold, B. E. (2010). The research evidence for commonfactors models: A historically situated perspective. In B. L.Duncan, S. D. Miller, B. E. Wampold, & M. A. Hubble (Eds.),The heart and soul of change: Delivering what works in therapy(2nd ed. pp. 49–81). Washington, DC: American PsychologicalAssociation.

Wampold, B. E., & Holloway, E. L. (1997). Methodology, design,and evaluation in psychotherapy supervision research. In C. E.

Watkins (Ed.), Handbook of psychotherapy supervision (pp. 11–27).New York, NY: Wiley.

Watkins, C. E. (2011). Does psychotherapy supervision contributeto patient outcomes? Considering thirty years of research. TheClinical Supervisor, 30, 235–256. doi:10.1080/07325223.2011.619417

10 T. G. Rousmaniere et al.

Dow

nloa

ded

by [C

urtin

Uni

vers

ity L

ibra

ry] a

t 09:

15 1

0 O

ctob

er 2

014