This article was downloaded by: [Society for Psychotherapy Research ]On: 07 November 2013, At: 08:12Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Psychotherapy ResearchPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/tpsr20

Psychometric properties of the OutcomeQuestionnaire-45.2: The Norwegian version in aninternational contextIngunn Amblea, Tore Gudeb, Sven Stubdala, Tuva Oktedalenb, Anne Marie Skjortenc, Bror JustAndersend, Ole André Solbakkene, Hanne H. Brorsone, Espen Arnevikef, Michael J. Lambertg

& Bruce E. Wampoldbh

a Outpatient Clinic, Modum Bad Psychiatric Center, Vikersund, Norwayb Research Institute, Modum Bad Psychiatric Center, Vikersund, Norwayc Drammen District Psychiatric Center, Vestre Viken HF, Drammen, Norwayd Baerum DPC, Vestre Viken HF, Drammen, Norwaye Department of Psychology, University of Oslo, Oslo, Norwayf Department of Addiction Treatment, Oslo University Hospital, Oslo, Norwayg Department of Psychology, Brigham Young University, Provo, UT, USAh Department of Counseling Psychology, University of Wisconsin-Madison, Madison, WI, USAPublished online: 05 Nov 2013.

To cite this article: Ingunn Amble, Tore Gude, Sven Stubdal, Tuva Oktedalen, Anne Marie Skjorten, Bror Just Andersen,Ole André Solbakken, Hanne H. Brorson, Espen Arnevik, Michael J. Lambert & Bruce E. Wampold , Psychotherapy Research(2013): Psychometric properties of the Outcome Questionnaire-45.2: The Norwegian version in an international context,Psychotherapy Research, DOI: 10.1080/10503307.2013.849016

To link to this article: http://dx.doi.org/10.1080/10503307.2013.849016

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

EMPIRICAL PAPER

Psychometric properties of the Outcome Questionnaire-45.2:The Norwegian version in an international context

INGUNN AMBLE1*, TORE GUDE2, SVEN STUBDAL1, TUVA OKTEDALEN2,ANNE MARIE SKJORTEN3, BROR JUST ANDERSEN4, OLE ANDRÉ SOLBAKKEN5,HANNE H. BRORSON5, ESPEN ARNEVIK5,6, MICHAEL J. LAMBERT7, &BRUCE E. WAMPOLD2,8

1Outpatient Clinic, Modum Bad Psychiatric Center, Vikersund, Norway; 2Research Institute, Modum Bad PsychiatricCenter, Vikersund, Norway; 3Drammen District Psychiatric Center, Vestre Viken HF, Drammen, Norway; 4Baerum DPC,Vestre Viken HF, Drammen, Norway; 5Department of Psychology, University of Oslo, Oslo, Norway; 6Department ofAddiction Treatment, Oslo University Hospital, Oslo, Norway; 7Department of Psychology, Brigham Young University,Provo, UT, USA & 8Department of Counseling Psychology, University of Wisconsin-Madison, Madison, WI, USA

(Received 6 March 2013; revised 20 August 2013; accepted 16 September 2013)

AbstractObjective: Monitoring of ongoing psychotherapy is of crucial importance in improving the quality of mental health care bydetecting therapies being off track, which requires that the instrument used is psychometrically sound. This studyinvestigates the psychometric properties of the Norwegian version of the Outcome Questionnaire 45.2 (OQ-45) and situatesthe results in an international context. Method: Data from one non-clinical sample (N = 338) and one clinical sample (N =560) were compared to international samples investigating reliability, cut-offs, and factor structure. Results: The resultsshow adequate reliability and concurrent validity. Conclusions: The means, clinical cut-offs, and the reliable change indexvary across countries. However, the means of the OQ-45 for nonclinical samples correlate highly with external values ofnational well-being, indicating that the OQ-45 is a valid instrument internationally. The factor analyses in the present studydo not confirm the hypothesized factor structure of the OQ-45, but are similar to the results internationally.

Keywords: psychotherapy monitoring; feedback; psychometric properties; Outcome Questionnaire-45; OQ-45

Introduction

In general, psychotherapy is effective (Lambert,2013; Lambert & Ogles, 2004). A meta-analysis ofresearch outcome studies has shown that about 75%of the patients improve (Hansen, Lambert, & For-man, 2002), although in routine outpatient psycho-therapy practice less than 50% of the cases improveand about 5–10% actually get worse, being treatmentfailures or non-responders (Hansen & Lambert,2003). This reality entails suffering and reducedquality of life for the patients and imposes seriouseconomic burdens on national costs due to repeatedtreatment, sick leave, and /or disability.

Monitoring of ongoing psychotherapy is consid-ered to be of importance in improving the quality ofmental health care by detecting when therapy isoff track (Harmon et al., 2007; Hawkins, Lambert,Vermeersch, Slade, & Tuttle, 2004; Lambert,Whipple, et al., 2002; Lambert, Whipple, Smart,Vermeersch, & Nielsen, 2001; Whipple et al., 2003).By giving feedback to the therapists, therapists canalter treatment to bring the therapy on track againand thus prevent deterioration (De Jong, van Sluis,Nugter, Heiser, & Spinhoven, 2012; Lambert &Shimokawa, 2011; Lambert, Whipple, et al., 2003).During the last years, systems for monitoring patient

Correspondence concerning this article should be addressed to Ingunn A. Amble, Modum Bad, N-3370 Vikersund, Norway. Email: ingunn.amble@modum-bad.no

Psychotherapy Research, 2013http://dx.doi.org/10.1080/10503307.2013.849016

© 2013 Society for Psychotherapy Research

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

progress and providing feedback have beendeveloped and implemented (e.g., Evans et al.,2000; Lambert et al., 1996; Miller & Duncan,2000; Pinsof et al., 2009).

One of the most widely used systems is the OQAnalyst, which uses the Outcome Questionnaire45.2 (referred to as the OQ-45) as the instrumentto examine patient progress (Hatfield & Ogles, 2004;Lambert, Kahler, Harmon, & Burlingame, 2011).The OQ-45 is a 45-item patient self-report with aTotal Score and three subscales designed to assessexperience of psychological distress, interpersonalfunctioning, and contentment with social role func-tioning (Lambert et al., 1996), and has demon-strated sensitivity to change during the course ofpsychotherapy (Vermeersch et al., 2004).

The psychometric properties of the OQ-45 havebeen studied extensively in the United States(Beretvas, Kearney, & Barón, 2003; Bludworth,Tracey, & Glidden-Tracey, 2010; Chapman, 2003;Kim, Beretvas, & Sherry, 2010; Lambert et al., 1996,2011; Mueller, Lambert, & Burlingame, 1998). Dueto promising results in the USA, the OQ-45 has alsobeen widely used around the world. Mental healthsystems as well as cultural values related to mentalhealth vary from one country and context to another(Bhugra & Bhui, 2007), and it therefore should not beassumed that properties of the instrument might beinvariant across cultures. However, variations do notnecessarily invalidate the instrument; indeed, suchvariations might well inform us about differences inmental health services and cultural contexts.

To date, the OQ-45 has been translated into morethan 20 languages and psychometric analyses havebeen conducted in Germany (Lambert, Hannöver,Nisslmüller, Richard, & Kordy, 2002), the Nether-lands (de Jong et al., 2007), Italy (Chiappelli, Coco,Gullo, Bensi, & Prestano, 2008; Lo Coco et al., 2008),Sweden (Wennberg, Philips & de Jong, 2010), andChina (Qin & Hu, 2008). The findings of these studiesare presented in Table I.

Several conclusions can be drawn from theseinternational efforts. First, the OQ-45, in the USand the translated versions in other countries,appears to be a reliable (stable and internally con-sistent) instrument. Second, the validity of theOQ-45, as evidenced by relatively robust correlationswith other instruments, as predicted, appears to bestrong across the various countries. Third, however,there appears to be significant variability in themeans for the clinical and non-clinical samples.The mean of the OQ-45 Total Score of the non-clinical samples ranges from 38.7 for the Dutchversion to 61.0 for the Chinese version, a differencegreater than the Reliable Change Index of 14 used inthe USA. As well, the means of clinical samples also

vary, ranging from 78.7 for the Chinese version to83.1 for the US version. As a consequence, theclinical cut-off of the OQ-45 differs as well, rangingfrom 55 in the Netherlands to 68 in China. TheReliable Change Index is equal to approximately 14in most countries, except Italy and China, where wehave calculated it to be 18 and 20, respectively.Finally, confirmatory factor analyses reveal that thethree-factor solution hypothesized by Lambert hasnot been consistently confirmed, although the fit ofthe three factors to the data could be characterized as“moderately good.” Other solutions, including oneto 10 factors, as well as bi-level solutions (e.g.,specific factors nested within one general factor),have also been investigated (Beretvas et al., 2003;Bludworth et al., 2010; Chapman, 2003; Chiappelliet al., 2008; de Jong et al., 2007; Kim et al., 2010;Mueller et al., 1998).

The purpose of the present study was to investig-ate the properties of the Norwegian version of theOQ-45 and to situate the results in an internationalcontext. Of particular interest are the differences inclinical and nonclinical means for various countries,as it is not clear why these should demonstrate suchvariability. Of course there are methodological rea-sons for such differences, the most salient of which isthat the samples from which these estimates arederived may not be representative, as the samples arebest characterized as “convenience” samples. How-ever, the variability in means may also reflect truedifferences in well-being among countries. In par-ticular, we are interested in whether the variability innon-clinical means among countries reflects differ-ences in national well-being.

Specifically, in this study, we investigated thefollowing research questions.

(1) Will the Norwegian version of the OQ-45show satisfactory reliability and concurrentvalidity?

(2) Will the norms for clinical and non-clinicalpopulations in Norway, as well as the clin-ical cut-off and reliable change index, besimilar to the international comparators?

(3) Will the original hypothesized three-factorstructure be confirmed in theNorwegian data?

(4) Will the means of the OQ-45 Total Scoreof the non-clinical population correlate withnational well-being?

Methods

Participants and Data Collection Procedure

To ensure heterogeneity of the non-clinical popula-tion, participants were sampled from four

2 I. Amble et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

Table I. Psychometric testing of OQ-45 in different countries

Country USA GermanyThe

Netherlands Italy Sweden China Norway

Authors Lambert, 1996/2011 Lambert,Hannöver,2002

De Jong, 2007 Chiappelli,Lo Coco,2008

Wennberg,2010

Qin, 2008 Present study

Test-retestPearson’s r

.84 .89 .91 .90 .85

Internalconsist. α

.93 .93 .96 .92 .95 .91 .93

Concurrent validity correlations with Total Scale and corresponding Subscale (with Norwegian correlations for DivergentValidity)

Total Subscale Total Subscale Total Subscale Total Subscale

SCL-90 GSI .72 .70(SD) .73 .76(SD) .80 .80(SD) .68 .64(SD).51(IR).60(SR)

IIP .63 .50(IR) .66 .55(IR) .59 .50(IR).52(SD).55(SR)

SAS .60 .41(SR) .58 .45(SR) .67 .70(SR).58(SD).53(IR)

Clinical and non-clinical sample statistics (SD in parentheses)Clinical samples Community health

outpatient clinics(N = 100/342)

Mental healthinstitutions(N = 1920)

Mental healthcareorganizations(N = 301)

Substanceabuse(N = 227)

Students(N = 75)

Communityoutpatients(N = 280)

Students (N = 76/486)Mean age 39.6/29.6/? 37.3 33.9 44.0 23.8 35.8Mean OQ-45 83.1(22) 79.5(25) 80.0(23.1) 80.0(24.9) 78.7(23.9) 90.8(21.1)

Non-clinicalsamples

Community(N = 102/815)

Community,universityassoc.(N = 232)

Community(N = 810)

Community,university(N = 514)

Students(N = 200)

Community,Students,Empl. healthinstitutions,(N = 338)

Students(N = 238/538)

Mean age 42.5/21.1/? 35.6 44.3 26.1 21.6 46.7Mean OQ-45 45.2(18.6) 46.2(18.5) 38.7(16.0) 54.7(20.1) 61.0(14.4) 37.3(18.5)

ClinicalCut off

63 55 66 68 62

Reliable changeindex

14 14 18 20 16

National experienced well-beingWell-being 7.2 6.7 7.5 6.4 4.7 4.7 7.6

Hypothesized three-factor structure: symptom distress, interpersonal relations, and social role performanceCFA* Moderate fit, Mueller

et al., 1998 (CFI =.83), Bludworth et al.,2010 (CFI = .64)

Moderate fit,de Jong et al.,2007 (CFI= .95)

Moderate fit,Lo Cocoet al., 2008(CFI = .78)

Three-factorsolutionmatched“quitewell,”Wennberget al., 2010

Notconducted

Moderate fit(CFI = .92)

Alternates Four-factor bilevel,Bludworth et al., 2010;nine factors, Chapman,2003; one factor,Mueller, 1998

Five factors, deJonget al., 2007

Four-factorbilevel,Lo Cocoet al., 2008

Ten factors Two factors

Note. SCL-90R = The Symptom Checklist 90-Revised; IIP = The Inventory of Interpersonal Problems; SAS = The Social AdjustmentRating Scale; SD = Symptom Distress Scale; IR = Interpersonal Relations Scale; SR = Social Role Scale.* For CFA, only CFI are given, as this statistic was reported in all studies.

OQ in Norway 3

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

different populations: inhabitants in a rural Norwe-gian community, members of a fitness center in asmall city, students, and employees of mental healthinstitutions. The clinical samples were obtained frompsychiatric routine care clinics, three inpatient andthree outpatient clinics. The samples used in thisstudy are presented in Table II.

Non-clinical samples. The total non-clinicalsample (TNCS, N = 338), which was used forcalculating the non-clinical mean, Cronbach’s α,and clinical cut-off, consisted of three subsamples.

Non-clinical sample 1 (NCS1). OQ-45 scoreswere obtained from adult residents in a ruralNorwegian community with a population of approxi-mately 12,000. The sampling was conducted byposting two OQ-45 Questionnaires with stampedenvelopes in 300 random mailboxes with informa-tion emphasizing the inclusion criteria: Age above18 years and not having a psychiatric diagnosis orbeing in current psychiatric treatment. Informationabout the project was also given in the local newspa-per. We received 99 envelopes with a total of 127completed forms, which represents an estimatedresponse rate of approximately 27% (assumingthat national rate of 1.6 adults per household). Inaddition 59 members of a local fitness center com-pleted the OQ-45 in a city of approximately 30,000inhabitants.

Non-clinical sample 2 (NCS2). OQ-45 wascollected from 46 students from different depart-ments at the University of Oslo and a nearby college.

Non-clinical sample 3 (NCS3). A total of 106employees at different psychiatric institutions through-out Norway completed the OQ-45. Of these 57 per-sons filled it in twice, at a 1-week interval, providingdata for calculating a test-retest correlation.

There were no significant differences in OQ-45Total Scores among the (i) rural residents, (ii) fitnesscenter members, (iii) students, and (iv) healthinstitutions employees (F(3, 334) = 2.19, p = .089,η2 = .019) and consequently analyses combinedsubsamples when possible to increase sample size.

Clinical samples. The total clinical sample(TCS) (N = 560) consisted of three subsamples.

Clinical sample 1 (CS1). Data from 280 patientswere collected as initial scores from three outpatientclinics in the southeast of Norway. These data werealso used for calculating means, reliability, clinicalcut-off, and the Reliable Change Index.

Clinical sample 2 (CS2). In this sample 184patients (others than in CS1) from the same out-patient clinics as mentioned above filled in OQ-45once during their course of treatment.

Table II. Non-clinical and clinical sample demographics

Sample Abbreviation AnalysesN total usedin analysis

Percentfemale

Meanage

Deleted due to > 8missing items

Non-clinical samplesa

1. Community andfitness center

NCS1 Mean, clinical cut-off, Cronbach’s α 186 70.5 51.33 5

2. Students NCS2 Mean, clinical cut-off, Cronbach’s α 46 65.1 23.60 23. Employees ofhealthinstitutions

NCS3 Mean, clinical cut-off, Cronbach’s α,test-retestb

106 75.0 49.45 2

Total non-clinicalsample

TNCS Mean, clinical cut-off, Cronbach’s α 338 70.7 46.69 9

Clinical samples1. OutpatientsInitial scores

CS1 Mean, clinical cut-off, Cronbach’s α.Reliable Change index. Factor analyses

280 67.9 35.8 0

2. Outpatients Notinitial scores

CS2 Factor analyses, Cronbach’s α 184 74.3 33.3 3

3. Inpatients CS3 Factor analyses, Cronbach’s α 96 56.1 39.4 2Total clinicalsample

TCS Factor analyses, Cronbach’s α 560 68.2 34.5 5

Clinical samples for validation1. Outpatients CS4 Concurrent validity with SCL-90 & IIP 61 70.5 36.9 02. Outpatients CS5 Concurrent validity with SAS 54 70.4 37.0 0

aNot all of the individuals filled in Age and Gender.bFifty-seven out of 106 filled in OQ-45 twice with 1-week interval.

4 I. Amble et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

Clinical Sample 3 (CS3). In this sample 96patients from three inpatient clinics completed theOQ-45 at the beginning of therapy.

In the outpatient samples used in this study(CS1), about one-third of patients were diagnosedwith anxiety disorders, about one-third with affectivedisorders, about 15% with unspecified diagnoses,about 6% with some type of behavioral disorder, andabout 4% with eating disorders; less than 4% of thepatients were diagnosed with a psychotic disorder,personality disorder, or substance use disorder. Ofcourse, the validity of diagnoses in clinical settingsmust be considered with caution. Because we wantedto collect a naturalistic patient sample, no diagnosticexclusion criteria were used although patients whosetherapists assessed that the patient was unable tounderstand the meaning of OQ-45 were not invitedto participate.

Clinical sample 4 (CS4). This sample consistedof 61 patients (a subsample of CS1) who completedthe Symptom Checklist 90-Revised (SCL-90-R) andThe Inventory of Interpersonal Problems Circum-plex (IIP-64-C) as well as the OQ-45 prior to theirintake session. The SCL-90-R and IIP-64-C werecompleted some days to weeks before the OQ-45;time between the administrations was examined inthe analyses.

Clinical sample 5 (CS5). This sample consistedof 54 patients (another subsample of CS 1) whocompleted the Social Adjustment Scale (SAS) aswell as the OQ-45 at the beginning of therapy.

Instruments

The Outcome Questionnaire (OQ-45). TheOQ-45 is composed of 45 items which are assessedwith a 5-point Likert scale (0 = never, 1 = rarely,2 = sometimes, 3 = frequently, 4 = almost always),with nine of the items reverse scored to limit thelikelihood of biased responses. The range of thescores is 0 to 180, with higher scores being indicativeof greater levels of psychological distress. The OQ-45 is typically given prior to the first and eachsubsequent therapy session, in either paper/pencilor electronic formats, and takes about 5–10 minutesto complete. The sum of all items gives a TotalScore. There are three subscales: Symptom Distress(SD, mainly depression and anxiety, 25 items),Interpersonal Relations (IR, 11 items), and SocialRole functioning (SR, 9 items) (Lambert, 1983).

The translation process involved a group of threenative Norwegian therapists and researchers whoseparately translated the English OQ-45 into Nor-wegian, and then discussed discrepancies until

consensus was obtained in the group. This consen-sual version was translated back to English by anexperienced American researcher and therapist livingand working in Norway, in cooperation with thegroup of Norwegian therapists and the developer oftheOQ-45 Michael Lambert in order to obtain semanticequivalence (Flaherty, Gaviria, Pathak, & Mitchell,1988). The final version was obtained by discussionin the extended group until consensus was reached.

Participants who left more than eight of theOQ-45 questions unanswered were not included.In total, only 14 participants had more than eightmissing items. In the case of missing values, a meanfor the remaining scale items was calculated andreplaced the missing values. However, missingvalues were not replaced in the analyses making useof data on item level (viz. factor analyses andanalyses of internal consistency). Of the 132 partici-pants who had one to eight missing values, 64% hadone missing value, 19% had two missing values, 8%had three missing values, and 8% had more thanthree but fewer than eight missing values.

The Symptom Checklist 90-Revised (SCL-90-R). The Symptom Checklist 90-Revised (SCL-90-R) (Derogatis, 1977; Derogatis & Melisaratos,1983) provides information about the client’s experi-ence of psychological symptoms as measured by ninesymptom areas, and an index for overall symptomburden. The questionnaire contains 90 questionsrated on a Likert scale from 0 = not at all to 4 = verymuch, examining how he or she has experienced thepast week, including the present day. Only the indexfor overall symptom burden (GSI) of the Norwegianedition (Pedersen & Karterud, 2004) was used inthis study. The internal consistency of the originalversion is reported to be .77 to .90 and the test-retestreliability .78 to .90 (Derogatis, 1994). The Norwe-gian version has been found to be internally consist-ent (α = .96), and stable (1-week test-retest r = .83)(Vassend, Lian, & Andersen, 1992). It was expectedthat the SCL-90-R GSI would be correlated highlywith the SD scale of the OQ-45.

The Inventory of Interpersonal Problems(IIP-64-C). The IIP-64-C (Alden, Wiggins, &Pincus, 1990) is a shortened version of the TheInventory of Interpersonal Problems (Horowitz,Rosenberg, Baer, Ureño, & Villaseñor, 1988). TheIIP-64-C contains 64 statements describing commoninterpersonal problems. The patient is instructed toconsider each item regarding whether it has been aproblem with respect to any significant person in thepatients’ life and is assessed with a Likert scalethat ranges from 0 = not at all to 4 = extremely.

OQ in Norway 5

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

The IIP-64 contains eight subscales and a globalscore. The global score of the Norwegian version ofIIP-64 was used in the present study. The originalversion has adequate psychometric properties withCronbach’s α of .72 to .85 and test-retest r of .78(Horowitz, Alden, Wiggins, & Pincus, 2000). TheNorwegian IIP has similar psychometric properties,with α ranging from .82 to .94 for the sub-scales and rof .98 for the test-retest for the Total IIP score(Monsen, Hagtvet, Havik, & Eilertsen, 2006; Peder-sen, 2002). It was expected that the IIP wouldcorrelate highly with the IR subscale of the OQ-45.

The Social Adjustment Rating Scale. TheSocial Adjustment Rating Scale (Weissman &Bothwell, 1976) is a 54-item questionnaire thatmeasures performance over the past 2 weeks in sixareas of functioning: work; domestic or academicresponsibilities; social and leisure activities; relation-ship with extended family; marital role as a spouse;parental role; and membership in the family unit.Each item is rated on a 5-point scale (1 to 5), withthe higher score being indicative of greater distressor impairment. The psychometric properties of theoriginal version are Cronbach’s α of .74 and test-retest r of .80 (Edwards, Yarvis, Mueller, Zingale, &Wagman, 1978). It was expected that the SAS wouldcorrelate highly with the SR subscale of the OQ-45.

National experienced well-being. Experiencedwell-being was measured by the New EconomicsFoundation in 2012. Over 1000 respondents in 151countries in the world were assessed using the“ladder of life” developed by the Gallup WorldPoll, where the bottom rung (scale value zero)represents the worst possible life and the top rungthe best possible life (scale value 10) (see NewEconomics Foundation, 2012). The national experi-enced well-being scores derived in this fashion arethe most extensive worldwide survey of well-beingusing the same instrument in existence (New Eco-nomics Foundation, 2012). The experienced well-being scores for the seven countries are found inTable I.

Results

Reliability

Using the scores of the clinical sample (TCS, N =560, Table II), Cronbach’s α for the Total Score was.93, with .90 for the Symptom Distress subscale, .78for Interpersonal Relation scale, and .77 for SocialRole scale.

For the non-clinical sample (TNCS, N = 338)Cronbach’s α was .93 for the Total Score, .91 for the

Symptom Distress scale, .73 for the InterpersonalRelation scale and .75 for the Social Role scale.

A test-retest with 1-week interval from 57 of thepersons in the non-clinical sample 1 (NCS1) yieldeda Pearson’s r of .85 for the Total Score, .90 for theSymptom Distress scale, .70 for the InterpersonalRelation scale and .74 for the Social Role scale.Generally, the reliabilities were in the range of thevalues found internationally and are satisfactory toclassify the Norwegian OQ-45 as reliable.

Concurrent and Divergent Validity

To examine the concurrent validity of the OQ-45,the Total Score as well as relevant subscales werecorrelated with the referent instrument using theclinical sample 3 (CS3, N = 61) for the SCL-90Rand IIP-64C, and clinical sample 4 (CS4, N = 54)for the SAS (Table II). As shown in Table I,the correlations for the referent instruments and theOQ-45 Total Score ranged from .59 to .68. Thecorrelations for the specific subscales were as fol-lows: .68 for the SCL-90-GSI vs. SD, .50 for IIP vs.IR and .67 for the SAS vs. SR. The correlations withthe SCL-90R were weaker than those found else-where, but the correlations with the IIP-64C aresimilar to the international samples, and the correla-tions of the SAS with SR were higher than expected(see Table I).

Divergent validity was assessed by correlatingthe OQ-45 subscales with non-referent instruments(e.g., SCL-90, a symptom measure, with SR). Ascan be seen in Table I, these divergent validitycoefficients were nearly as large as the conver-gent validity coefficients, and in some instanceslarger.

Clinical Cut-Off

As presented in Table I, the mean for the OQ-45Total Score was 37.3 for the non-clinical sample(TNCS, N = 338, Table II) and of 90.8 for theclinical outpatient sample (CS1, N = 280). Based onthese samples, the cut-off between case/non-case-ness, according to the Jacobson and Truax method(Jacobson & Truax, 1991), was as follows:

Cutoff ¼ MClinical � sdNonClinicalð Þþ MNonClinical � sdClinicalð Þ/ sdClinical þ sdNonClinicalð Þ

¼ ð90.8 � 18.5Þ þ ð37.3 � 21.1Þ/ð21.1þ 18.5Þ¼ 62.4

We checked for gender differences and foundslightly higher means in the OQ-45 Total Score infemales (93.7, SD = 21.2) compared with males(87.7, SD = 20.8) (t = 2.16, p = 0.03) in the clinical

6 I. Amble et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

sample (CS1, N = 280). In the non-clinical sample(TNCS, N = 338), there were no gender differences.The difference in the clinical sample did not changethe clinical cut-off level when tested for each genderseparately.

Reliable Change Index

The reliable change index (RCI), calculated from theOutpatient sample (see Table II), using the formulaprovided by Jacobson and Truax (Jacobson & Truax,1991), was as follows

RCI ¼ 1.96SDffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi2 1� rð Þ

p

¼ 1.96 � 21.1ð Þffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi2 1� 0.926ð Þ

p¼ 15.91

where SD was the standard deviation and r Cron-bach’s α.

Prediction Of Patient Status

Within the total sample of outpatients assessed at thebeginning of treatment (CS1, N = 280), 91.4% hadan OQ-45 Total Score greater than the Norwegiancut-off of 62, thus being classified as clinical cases.On the other hand 90.8% of the non-clinical sample(viz., TNCS, N = 338) had an OQ-45 Total Score of62 or less, and were thus correctly classified as non-cases.

The Correlations between the Subscales

Each subscale is meant to measure a specific domainof functioning and therefore the subscales should berelatively uncorrelated. The correlations among thesubscales ranged from .51 to .66 (SD/IR = .61, SD/SR = .66, IR/SR = .51), perhaps higher than isdesirable but similar to the values produced in otherstudies (cf., Chiappelli et al., 2008; de Jong et al.,2007; Lambert, Hannöver, et al., 2002; Umphress,Lambert, Smart, Barlow, & Clouse, 1997; Qin &Hu, 2008).

Factor Structure

We conducted a confirmatory factor analysis ofLambert’s three-factor structure. The confirmatorymodels were conducted with LISREL (8.8). Accord-ing to Jöreskog and Sörbom (1993) a robust max-imum likelihood estimation procedure suggested bySatorra and Bentler (1990) is preferred for dataobtained from Likert-type format and representativeof an ordinal scale. The subprogram PRELIS 2.8was used to estimate the respective covariance andtheir asymptotic covariance matrices assuming

alternative parameterization for the factor analyticmodels (Jöreskog, 2005; Sörbom & Jöreskog, 1999).General guidelines suggest the following criteria forconfirming a hypothesized factor structure: RMR <.10, RMSEA < .05, CFI and NNFI > .9, a non-significant χ2, and χ2/df < 2 (Browne & Cudeck,1993; Byrne, 2009; Hu & Bentler, 1998; MacCal-lum, MacCallum, Browne, & Sugawara, 1996).

Fitting Lambert’s three-factor model to the datayielded coefficients that did not meet standardcriteria (RMR = .092; RMSEA = .082; CFI = .92;NNFI = .92; Satorra–Bentler Scaled χ2942 =4448.0863, p < 0.01; χ2/df = 4.72), although it couldbe said that the fit was moderately good—that is,some coefficients were in the acceptable range.These fit indices were strikingly similar to the valuesproduces by similar analyses in other countries (seeTable I for a comparison of CFI indices). Thecorrelation among the latent factors of SD, IR, andSR ranged from .57 to .75, demonstrating that thethree factors were not independent.

For a number of reasons, we did not modify thehypothesized factor structure to better fit the data,test other models, or conduct exploratory factoranalyses. First, the subscales are rarely used inresearch or clinical work, and if they are, Lambert’sthree subscales are used. Second, such strategiescapitalize on sample characteristics and are contra-dictory to a confirmatory modeling strategy. Third,the factor structures have varied across countries andadding another “exploratory” structure would notadd to our understanding of the OQ-45 or augmentthe clinical application of the OQ-45.

Correlation of Non-Clinical Means andExperienced Well-being

The correlation between the non-clinical means andnational experienced well-being, which are pre-sented in Table I, was −.942, indicating the lowerthe OQ-45 non-clinical mean for individual coun-tries, the greater the experienced well-being. Clearly,the magnitude of this correlation is remarkably large.

Discussion

Consonant with other international studies, wefound a satisfactory reliability for the NorwegianOQ-45 Total Score as well as for the subscales, forboth internal consistency and test-retest correlationvalues. It appears that translation and administrationof the OQ-45 in Norway does not affect the reliab-ility of the scores obtained from the OQ-45. As well,the Norwegian samples established concurrent va-lidity with other validated instruments, consistentwith international results.

OQ in Norway 7

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

The means of clinical and non-clinical samples, aswell as the clinical cut-off, vary widely across coun-tries that have investigated properties of the OQ-45(see Table I). The means for the non-clinicalsamples were lowest in Norway and the Netherlands(37.3 and 38.7 respectively), considerably higher inthe United States and Germany (45.2 and 46.2,respectively), and much higher in Italy and China(54.7 and 61.0, respectively). The range, which was23.7 OQ-45 points, is larger than the RCI for any ofthe countries. The means for the clinical samplesvaried less, although the range was still relativelylarge (viz., 11.3); the highest and the lowest meanswere for Norway and the Netherlands (90.8 and79.5, respectively). Given the range in means, notsurprisingly the clinical cut-offs and RCI also variedconsiderably across countries.

There might be several possible reasons for thevariability in means for the clinical and non-clinicalsamples across the different countries. First, thesamples used in the various countries are consider-ably different, as shown in Table I. The non-clinicalsamples are difficult to compare, as they containstudent populations, different community samples,university employees, and health employees andprofessionals, with mean ages ranging from 22 to47. The sizes of the samples also vary substantially,ranging from 75 to 1920, which affects the precisionof the estimates. There are also considerable differ-ences between the clinical samples in the variouscountries, varying from pure student samples tosamples from both inpatient and outpatient clinicsin different health institutions, with mean agesranging from 24 to 44. The relatively high level ofdistress found in the Norwegian clinical samplescould be understood in the light of the way theNorwegian health system is designed. Primary carephysicians in Norway and the community health caresystem treat most of the patients with psychiatricdisorders; patients are strictly screened before theyare referred to psychiatric outpatient clinics of thetype utilized in the present study. Clearly differencesin sample composition across studies and countriesmight well affect clinical and non-clinical means.

A second set of variables that might affect theobserved variation in clinical and non-clinical meanmight be related to culture. The manner in whichpeople in various cultures experience and expresspsychological distress varies (Bhugra & Bhui, 2007;Bhugra et al., 2011). Additionally, due to economics(i.e., resources available to residents; see NewEconomics Foundation, 2012) as well as culture,residents in various countries differ in their experi-enced well-being and thus the differences in OQ-45means may importantly reflect true differences inwell-being in the countries in which the OQ-45 has

been studied. This notion was investigated by cor-relating the non-clinical OQ-45 means with nationalexperienced well-being and a remarkable associationwas found. Norway had the lowest OQ-45 non-clinical mean and the highest experienced well-beingindex, whereas China showed the reverse (i.e., thehighest non-clinical OQ-45 mean and the lowestwell-being index). Given this remarkably large cor-relation, it may be that the OQ-45 is a valid indicatorof psychological distress and that the variability innon-clinical scores presented in Table I reflects truedifferences in well-being among countries.

Clearly, norms and clinical cut-off scores need to bedetermined for each country in which the OQ-45 isused. That is to say, these values may not be trans-portable across countries. In developing these statist-ics, careful attention needs to be paid to the samplesused, as they appear to differ considerably among thecountries in which the OQ-45 has been investigated.

The hypothesized three independent factor struc-ture of the OQ-45 has not been verified in any of thecountries, although the fit might be described as“moderately good.” Factor analyses of data fromvarious countries have revealed solutions with differ-ing numbers of factors (one to ten factors), as well asbilevel models. In our study, as well as others, thecorrelations among the measured, as well as latent,factors are quite high. Additionally, although theconvergent validity coefficients in the present studywere satisfactory, the divergent validity coefficientswere almost as large or larger. Clearly, in this andother studies, the evidence for three independentfactors is not convincing. However, because the OQ-45 is intended to be used in clinical settings, thequestion is whether the evidence for a three-factorsolution is good enough to be used clinically acrosscountries or whether alternative structures should beadopted in the various contexts. Typically the OQ-45 Total Score is used both clinically and in researchto assess global psychological functioning. We con-tend that the results of the present study and theother studies suggest that the Total Score of the OQ-45 is a reliable and valid measure for assessingtherapy progress and as an outcome measure inpsychotherapy trials. For clinical use the subscalesmight well be “good enough” to provide the therapistinformation about the patient that could be clinicallyactionable. However, given the issues with the factorstructure of the OQ-45, we contend that using thethree hypothesized factors across countries or usingidiosyncratic factors in various countries for researchpurposes is not warranted at this time.

There are a number of limitations to the presentstudy. Similar to the studies in other countries, thesamples used to investigate the OQ-45 were notrandom samples from the clinical and non-clinical

8 I. Amble et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

populations inNorway. An attemptwasmade to createheterogeneous samples, but nevertheless they clearlydeparted from random samples and the selectionbiases are unknown. Our samples, although relativelylarge, are smaller than the American and Dutch ones.Psychometric analyses generally require large samples.Nevertheless, the convergence of the results with othercountries as well as the remarkable correlation witheconometric measurement of well-being provide com-fort about the results obtained in this study.

The results of the present study suggest that theNorwegian version of the OQ-45 is adequate forclinical and research use in Norway, particularly withregard to the OQ-45 Total Score. As well, the compar-ison among countries has demonstrated that the OQ-45 is a valid indicator of the well-being of a population.

Acknowledgments

The authors would like to thank patients and stafffrom the Modum Bad outpatient clinic and Dram-men and Baerum District Psychiatric Centers whohave delivered clinical data to this study, as well asinhabitants of Modum Municipality, students, mem-bers of Friskis and Svettis Fitness Center in Hoene-foss and employees at health institutions who havedelivered norm data.

Funding

This research was funded by the Norwegian Ministryof Health, Extrastiftelsen (The Norwegian Councilfor Mental Health), and Modum Bad.

Note1 Dr. Lambert is a co-developer of the OQ-45 and OQ-Analystand is co-founder of OQMeasures, a partnership that marketsassessment tools and distributes them for a fee.

References

Alden, L. E., Wiggins, J. S., & Pincus, A. L. (1990). Constructionof circumplex scales for the Inventory of Interpersonal Pro-blems. Journal of Personality Assessment, 55, 521–536.

Beretvas, S., Kearney, L., & Barón, A. (2003). A shortened formof the Outcome Questionnaire: A validation of scores acrossethnic groups. Published on line by the Counseling and MindHealth Centte of Texas University.

Bhugra, D., & Bhui, K. (2007). Textbook of cultural psychiatry:Cambridge: Cambridge University Press.

Bhugra, D., Gupta, S., Bhui, K., Craig, T., Dogra, N., Ingleby,J. D., .… Qureshi, A. (2011). WPA guidance on mental healthand mental health care in migrants. World Psychiatry, 10, 2–10.

Bludworth, J. L., Tracey, T. J. G., & Glidden-Tracey, C. (2010).The bilevel structure of the Outcome Questionnaire–45. Psy-chological Assessment, 22, 350–355. doi:10.1037/a0019187

Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessingmodel fit. In K. A. Bollen & J. Long (Eds.), Testing structuralequation models (pp. 136–162). Newbury Park, CA: Sage.

Byrne, B.M. (2009). Structural equation modeling with AMOS.Basic concepts, applications and programming. Boca Raton, FL:CRC Press.

Chapman, J. E. (2003). Reliability and validity of the progressquestionnaire: an adaptation of the outcome questionnaire. DrexelUniversity.

Chiappelli, M., Lo Coco, G., Gullo, S., Bensi, L., & Prestano, C.(2008). The Outcome Questionnaire 45.2. Italian validation ofan instrument for the assessment of phychological treatments.Epidemiologia e Psichiatria Sociale, 17, 152–161.

De Jong, K., Nugter, M. A., Polak, M. G., Wagenborg, J. E. A.,Spinhoven, P., & Heiser, W. J. (2007). The Outcome Ques-tionnaire (OQ-45) in a Dutch population: A cross-culturalvalidation. Clinical Psychology & Psychotherapy, 14, 288–301.doi:10.1002/cpp.529

De Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J.,& Spinhoven, P. (2012). Understanding the differentialimpact of outcome monitoring: Therapist variables that mod-erate feedback effects in a randomized clinical trial. Psychother-apy Research, 22, 464–474. doi:10.1080/10503307.2012.673023

Derogatis, L. (1994). SCL-90-R Symptom Checklist-90-R adminis-tration, scoring and procedures manual. Minneapolis, MN:National Computer Systems.

Derogatis, L. R. (1977). SCL-90–R: Administration, scoring andprocedures Manual I for the revised version of other instruments ofthe Psychopathology Rating Scale series. Baltimore, MD: JohnsHopkins University.

Derogatis, L. R., & Melisaratos, N. (1983). The Brief SymptomInventory: an introductory report. Psychological Medicine, 3,595–605. doi:10.1017/S0033291700048017

Edwards, D. W., Yarvis, R. M., Mueller, D. P., Zingale, H. C., &Wagman, W. J. (1978). Test-taking and the stability ofadjustment scales: Can we assess patient deterioration? Evalu-ation Review, 2, 275–291. doi:10.1177/0193841X7800200206

Evans, J. M.-C., Margison, F., Barkham, M., Audin, K., Connell,J., & McGrath, G. (2000). CORE: Clinical outcomes in routineevaluation. Journal of Mental Health, 9, 247–255. doi:10.1080/713680250

Flaherty, J. A., Gaviria, F. M., Pathak, D., & Mitchell, T. (1988).Developing instruments for cross-cultural psychiatric research.Journal of Nervous and Mental Disease, 176, 260–322. doi:10.1097/00005053-198805000-00001

Hansen, N. B., & Lambert, M. J. (2003). An evaluation of thedose–response relationship in naturalistic treatment settingsusing survival analysis. Mental Health Services Research, 5, 1–12.doi:10.1023/A:1021751307358

Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). Thepsychotherapy dose-response effect and its implications fortreatment delivery services. Clinical Psychology: Science andPractice, 9, 329–343. doi:10.1093/clipsy.9.3.329

Harmon, S. C., Lambert, M. J., Smart, D. M., Hawkins, E.,Nielsen, S. L., Slade, K., & Lutz, W. (2007). Enhancingoutcome for potential treatment failures: Therapist–clientfeedback and clinical support tools. Psychotherapy Research,17, 379–392. doi:10.1080/10503300600702331

Hatfield, D. R., & Ogles, B. M. (2004). The use of outcomemeasures by psychologists in clinical practice. ProfessionalPsychology: Research and Practice, 35, 485–491. doi:10.1037/0735-7028.35.5.485

Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L.,& Tuttle, K. C. (2004). The therapeutic effects of providingpatient progress information to therapists and patients. Psycho-therapy Research, 14, 308–327. doi:10.1093/ptr/kph027

Horowitz, L., Alden, L., Wiggins, J., & Pincus, A. (2000). IIP-64/IIP-32 professional manual. San Antonio, TX: PsychologicalCorporation.

OQ in Norway 9

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013

Horowitz, L. M., Rosenberg, S. E., Baer, B. A., Ureño, G., &Villaseñor, V. S. (1988). Inventory of interpersonal problems:psychometric properties and clinical applications. Journal ofConsulting and Clinical Psychology, 56, 885–892. doi:10.1037/0022-006X.56.6.885

Hu, L.T., & Bentler, P.M. (1998). Fit indices in covariancestructure modeling: Sensitivity to underparameterized modelmisspecification. Psychological Methods, 3, 424–453. doi:10.1037/1082-989X.3.4.424

Jacobson, N. S., & Truax, P. (1991). Clinical significance: astatistical approach to defining meaningful change in psycho-therapy research. Journal of Consulting and Clinical Psychology,59, 12–19. doi:10.1037/0022-006X.59.1.12

Jöreskog, K. G. (2005). Structural equation modeling with ordinalvariables using LISREL: Technical report. Lincolnwood, IL:Scientific Software International.

Jöreskog, K., & Sörbom, D. (1993). LISREL 8: The SIMPLIScommand language. Chicago, IL: Scientific Software.

Kim, S. H., Beretvas, S. N., & Sherry, A. R. (2010). A validationof the factor structure of OQ-45 scores using factor mixturemodeling. Measurement and Evaluation in Counseling and Devel-opment, 42, 275–295. doi:10.1177/0748175609354616

Lambert, M. J. (1983). The assessment of psychotherapy outcome:New York: John Wiley & Sons.

Lambert, M. J. (2013). Efficacy and effectiveness of psychother-apy. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook ofpsychotherapy and behavior change (6th ed., pp. 169–218).Hoboken, NJ: Wiley.

Lambert, M.J., & Ogles, B. (2004). The efficacy and effectivenessof psychotherapy. In M. J. Lambert (Ed), Bergin and Garfield’shandbook of psychotherapy and behaviour change (pp. 139–193):Chichester: John Wiley & Sons.

Lambert, M. J., Burlingame, G. M., Umphress, V., Hansen, N. B.,Vermeersch, D. A., Clouse, G. C., & Yanchar, S. C. (1996). Thereliability and validity of the Outcome Questionnaire. ClinicalPsychology & Psychotherapy, 3, 249–258. doi:10.1002/(SICI)1099-0879(199612)3:4<249::AID-CPP106>3.0.CO;2-S

Lambert, M. J., Hannöver, W., Nisslmüller, K., Richard, M., &Kordy, H. (2002). Fragebogen zum Ergebnis von Psychother-apie. Zeitschrift für klinische Psychologie und Psychotherapie, 31,40–46. doi:10.1026//1616-3443.31.1.40

Lambert, M.J., Kahler, M., Harmon, C., & Burlingame, G. M.(2011). Administration and Scoring Manual Outcome Question-naire OQ-45.2. OQ Measures L.L.C.

Lambert, M. J., & Shimokawa, K. (2011). Collecting clientfeedback. Psychotherapy, 48, 72–79. doi:10.1037/a0022238

Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A.,Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians toroutinely track patient outcome? A meta-analysis. Clinical Psycho-logy: Science and Practice, 10, 288–301. doi:10.1093/clipsy.bpg025

Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A.,& Nielsen, S. L. (2001). The effects of providing therapists withfeedback on patient progress during psychotherapy: Are out-comes enhanced?Psychotherapy Research, 11, 49–68. doi:10.1080/713663852

Lambert, M. J., Whipple, J., Vermeersch, D., Smart, D.,Hawkins, E., Nielsen, S., & Goates, M. (2002). Providingtherapists with feedback on patient progress as a method ofenhancing psychotherapy outcomes: A replication. ClinicalPsychology and Psychotherapy, 9, 91–103. doi:10.1002/cpp.324

Lo Coco, G., Chiappelli, M., Bensi, L., Gullo, S., Prestano, C., &Lambert, M. (2008). The factorial structure of the outcomequestionnaire-45: A study with an Italian sample. ClinicalPsychology & Psychotherapy, 15, 418–423. doi:10.1002/cpp.601

MacCallum, R.C., Browne, M.W., & Sugawara, H.M. (1996).Power analysis and determination of sample size for

covariance structure modeling. Psychological Methods, 1, 130–149. doi:10.1037/1082-989X.1.2.130

Miller, S. D., & Duncan, B. (2000). The Outcome Rating Scale.Retrieved from http://www.talkingcure.com/measures.htm.

Monsen, J. T., Hagtvet, K. A., Havik, O. E., & Eilertsen, D. E.(2006). Circumplex structure and personality disorder corre-lates of the Interpersonal Problems Model (IIP-C): Constructvalidity and clinical implications. Psychological Assessment, 18,165–173. doi:10.1037/1040-3590.18.2.165

Mueller, R. M., Lambert, M. J., & Burlingame, G. M. (1998).Construct validity of the Outcome Questionnaire: A confirm-atory factor analysis. Journal of Personality Assessment, 70, 248–262. doi:10.1207/s15327752jpa7002_5

New Economics Foundation. (2012). The Happy Planet Index:2012 Report: A global index of sustainable well-being. RetrievedJuly 15, 2013, from http://www.happyplanetindex.org/assets/happy-planet-index-report.pdf.

Pedersen, G. A. (2002). Revised Norwegian version of theInventory of Interpersonal Problems—Circumplex (IIP-C).Tidsskrift for Norsk Psykologforening,39, 25–34.

Pedersen, G., & Karterud, S. (2004). Is SCL-90R helpful for theclinician in assessing DSM-IV symptom disorders?Acta Psychia-trica Scandinavica, 110, 215–224. doi:10.1111/j.1600-0447.2004.00321.x

Pinsof, W. M., Zinbarg, R. E., Lebow, J. L., Knobloch-Fedders,L. M., Durbin, E., Chambers, A., … Friedman, G. (2009).Laying the foundation for progress research in family, couple,and individual therapy: The development and psychometricfeatures of the initial Systemic Therapy Inventory of Change.Psychotherapy Research, 19, 143–156. doi:10.1080/10503300802669973

Qin, Y.-F., & Hu, S.-J. (2008). Usability report of OutcomeQuestionnaire-45 in part of Chinese sample. Chinese Journal ofClinical Psychology, 16, 138–140.

Satorra, A., & Bentler, P. M. (1990). Model conditions forasymptotic robustness in the analysis of linear relations. Compu-tational Statistics & Data Analysis, 10, 235–249. doi:10.1016/0167-9473(90)90004-2

Sörbom, D., & Jöreskog, K. (1999). PRELIS 2: User’s referenceguide. Chicago, IL: Scientific Software International.

Umphress, V. J., Lambert, M. J., Smart, D. W., Barlow, S. H., &Clouse, G. (1997). Concurrent and construct validity of theOutcome Questionnaire. Journal of Psychoeducational Assess-ment, 15, 40–55. doi:10.1177/073428299701500104

Vassend, O., Lian, L., & Andersen, H. T. (1992). Norwegianversions of the NEO-Personality Inventory, Symptom Checklist90 Revised, and Giessen Subjective Complaints List. I.Tidsskrift for Norsk Psykologforening, 29, 1150–1160.

Vermeersch, D. A., Whipple, J. L., Lambert, M. J., Hawkins, E.J., Burchfield, C. M., & Okiishi, J. C. (2004). Outcomequestionnaire: Is it sensitive to changes in counseling centerclients?Journal of Counseling Psychology, 51, 38–49. doi:10.1037/0022-0167.51.1.38

Weissman, M. M., & Bothwell, S. (1976). Assessment of socialadjustment by patient self-report. Archives of General Psychiatry,33(9), 1111–1115. doi:10.1001/archpsyc.1976.01770090101010

Wennberg, P., Philips, B., & de Jong, K. (2010). The Swedishversion of the Outcome Questionnaire (OQ-45): Reliability andfactor structure in a substance abuse sample. Psychology andPsychotherapy: Theory, Research and Practice, 83, 325–329.doi:10.1348/147608309X478715

Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W.,Nielsen, S. L., & Hawkins, E. J. (2003). Improving the effects ofpsychotherapy: The use of early identification of treatment andproblem-solving strategies in routine practice. Journal of Coun-seling Psychology, 50, 59–68. doi:10.1037/0022-0167.50.1.59

10 I. Amble et al.

Dow

nloa

ded

by [

Soci

ety

for

Psyc

hoth

erap

y R

esea

rch

] at

08:

12 0

7 N

ovem

ber

2013