9
Regular article Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders? Sylke Andreas, (Ph.D.) a, , Karin Harries-Hedder, b Wolfgang Schwenk, (Ph.D.) c , Maria Hausberg, a Uwe Koch, (Ph.D., M.D.) a , Holger Schulz, (Ph.D.) a a Department of Medical Psychology, University Medical Center Hamburg-Eppendorf, D-20246 Hamburg, Germany b Therapiehilfe e.V., Hamburg, D-22089 Hamburg, Germany c Rehabilitation Clinic for substance disorders, D-25821 Bredstedt, Germany Received 17 July 2009; received in revised form 9 January 2010; accepted 16 March 2010 Abstract The Health of the Nation Outcome Scales (HoNOS) is an internationally established clinician-rated instrument. The aim of the study was to assess the psychometric properties in inpatients with substance-related disorders. The HoNOS was applied in a multicenter, consecutive sample of 417 inpatients. Interrater reliability coefficients, confirmatory factor analysis, and regression tree analyses were calculated to assess the reliability and validity of the HoNOS. The factor validity of the HoNOS and its total score could not be confirmed. After training, all items of the HoNOS revealed sufficient values of interrater reliabilities. As the results of the regression tree analyses showed, the single items of the HoNOS were one of the most important predictor of service utilization. The HoNOS can be recommended for obtaining detailed ratings of the problems of inpatients with substance-related disorders as a clinical application in routine mental health care at present. Further studies should include comparisons of HoNOS and Addiction Severity Index. © 2010 Elsevier Inc. All rights reserved. Keywords: Substance abuse; Symptom severity; HoNOS; Psychometric; Psychotherapy 1. Introduction The differential assessment of severity in patients with substance-related disorders has gained increasing relevance during recent years (Cacciola, Koppenhaver, Alterman, & McKay, 2009; Drake, Bartels, Teague, Noordsy, & Clark, 1993). The Health of the Nation Outcome Scales (HoNOS; Wing et al., 1998), introduced in 1998, is an instrument that assesses the severity in different problem areas (e.g., pathology or environmental factors) in patients with mental disorders from a clinician-rated perspective. Today, the HoNOS is an internationally well-established clinician-rated instrument. It is routinely applied in many countries (e.g., Denmark, Nova Scotia, Italy, Australia) in the treatment of patients with mental disorders (Bech et al., 2003; Kisely, Campbell, Crossman, Gleich, & Campbell, 2007; Lora et al., 2001; Trauer et al., 1999) and is gaining increasing relevance in the framework of the development of case group systems (Andreas et al., 2009; Buckingham, Burgess, Solomon, Pirkis, & Eagar, 1998; Gaines et al., 2003). In line with international developments and due to the lack of suitable national clinician-rating instruments for the differential assessment of severity, the HoNOS was translated into German with authorization from the test authors. The translation process and first empirical results on the practicability, reliability, and validity have been previously described elsewhere (Andreas et al., 2007; Andreas et al., 2010). The aim of the HoNOS development was to construct an instrument that assesses the severity of psychosocial problems in 12 independent areas irrespective of the patient's diagnosis and can be routinely applied (Wing et al., 1998). The period evaluated by the HoNOS consists of the 14 days prior to completing the questionnaire. It is a Journal of Substance Abuse Treatment 39 (2010) 32 40 Corresponding author. Department of Medical Psychology, Center for Psychosocial Medicine, University Medical Center Hamburg-Eppendorf, Martinistr. 52, Building W26, D-20246 Hamburg, Germany. Tel.: +49 0 40 7410 57705; fax: +49 0 40 7410 54940. E-mail address: [email protected] (S. Andreas). 0740-5472/10/$ see front matter © 2010 Elsevier Inc. All rights reserved. doi:10.1016/j.jsat.2010.03.011

Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

Embed Size (px)

Citation preview

Page 1: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

Journal of Substance Abuse Treatment 39 (2010) 32–40

Regular article

Is the Health of the Nation Outcome Scales appropriate for the assessmentof symptom severity in patients with substance-related disorders?

Sylke Andreas, (Ph.D.)a,⁎, Karin Harries-Hedder,b Wolfgang Schwenk, (Ph.D.)c,Maria Hausberg,a Uwe Koch, (Ph.D., M.D.)a, Holger Schulz, (Ph.D.)a

aDepartment of Medical Psychology, University Medical Center Hamburg-Eppendorf, D-20246 Hamburg, GermanybTherapiehilfe e.V., Hamburg, D-22089 Hamburg, Germany

cRehabilitation Clinic for substance disorders, D-25821 Bredstedt, Germany

Received 17 July 2009; received in revised form 9 January 2010; accepted 16 March 2010

Abstract

The Health of the Nation Outcome Scales (HoNOS) is an internationally established clinician-rated instrument. The aim of the study wasto assess the psychometric properties in inpatients with substance-related disorders. The HoNOS was applied in a multicenter, consecutivesample of 417 inpatients. Interrater reliability coefficients, confirmatory factor analysis, and regression tree analyses were calculated to assessthe reliability and validity of the HoNOS. The factor validity of the HoNOS and its total score could not be confirmed. After training, allitems of the HoNOS revealed sufficient values of interrater reliabilities. As the results of the regression tree analyses showed, the single itemsof the HoNOS were one of the most important predictor of service utilization. The HoNOS can be recommended for obtaining detailedratings of the problems of inpatients with substance-related disorders as a clinical application in routine mental health care at present. Furtherstudies should include comparisons of HoNOS and Addiction Severity Index. © 2010 Elsevier Inc. All rights reserved.

Keywords: Substance abuse; Symptom severity; HoNOS; Psychometric; Psychotherapy

1. Introduction

The differential assessment of severity in patients withsubstance-related disorders has gained increasing relevanceduring recent years (Cacciola, Koppenhaver, Alterman, &McKay, 2009; Drake, Bartels, Teague, Noordsy, & Clark,1993). The Health of the Nation Outcome Scales (HoNOS;Wing et al., 1998), introduced in 1998, is an instrument thatassesses the severity in different problem areas (e.g.,pathology or environmental factors) in patients with mentaldisorders from a clinician-rated perspective.

Today, the HoNOS is an internationally well-establishedclinician-rated instrument. It is routinely applied in manycountries (e.g., Denmark, Nova Scotia, Italy, Australia) in

⁎ Corresponding author. Department of Medical Psychology, Center forPsychosocial Medicine, University Medical Center Hamburg-Eppendorf,Martinistr. 52, Building W26, D-20246 Hamburg, Germany. Tel.: +49 0 407410 57705; fax: +49 0 40 7410 54940.

E-mail address: [email protected] (S. Andreas).

0740-5472/10/$ – see front matter © 2010 Elsevier Inc. All rights reserved.doi:10.1016/j.jsat.2010.03.011

the treatment of patients with mental disorders (Bech et al.,2003; Kisely, Campbell, Crossman, Gleich, & Campbell,2007; Lora et al., 2001; Trauer et al., 1999) and is gainingincreasing relevance in the framework of the development ofcase group systems (Andreas et al., 2009; Buckingham,Burgess, Solomon, Pirkis, & Eagar, 1998; Gaines et al.,2003). In line with international developments and due to thelack of suitable national clinician-rating instruments for thedifferential assessment of severity, the HoNOS wastranslated into German with authorization from the testauthors. The translation process and first empirical results onthe practicability, reliability, and validity have beenpreviously described elsewhere (Andreas et al., 2007;Andreas et al., 2010).

The aim of the HoNOS development was to construct aninstrument that assesses the severity of psychosocialproblems in 12 independent areas irrespective of thepatient's diagnosis and can be routinely applied (Winget al., 1998). The period evaluated by the HoNOS consistsof the 14 days prior to completing the questionnaire. It is a

Page 2: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

33S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

12-item instrument for the assessment of severity in mentalillness. Each of the 12 items describes a specific problemarea, such as “problems with depressed mood,” byindicating various degrees of severity (from no problemto very severe problem). The 12 items can be grouped intofour scales: Behavioral Problems, Symptomatic Problems,Impairment, and Social Problems. It is possible todetermine an overall score for the first 10 items as wellas for all of the items combined. An extensive glossaryaccompanies the instrument, with detailed instructions forthe evaluation. For example, for the item “problems withdepressed mood,” the raters are instructed to indicate a“clinically insignificant problem with no need for treat-ment” if the patient exhibits mild depression or slightchanges in mood. On the other hand, the rater is to indicatea “severe to very severe problem” on the HoNOS when thepatient exhibits severe to very severe depression, forexample, with feelings of guilt. The HoNOS can becompleted after the clinical interview with the patient. Therating takes approximately 5 minutes, depending onexperience (Andreas et al., 2007).

A number of studies have evaluated the psychometricproperties of the HoNOS in patients with mental disorders(Bech et al., 2003; Kisely et al., 2007; Lora et al., 2001;Trauer, 1999; Trauer et al., 1999). One central validity aspectis the factor structure of the instrument. On the basis ofcontent criteria, Wing et al. (1998) assumed the following 4subscales: “behavior” (Item 1, “aggressiveness”; Item 2,“nonaccidental self-injury”; and Item 3, “problem drinking ordrug-taking”), “impairment” (Item 4, “cognitive problems,”and Item 5, “physical illness or disability problems”),“symptoms” (Item 6, “hallucinations and delusions”; Item7, “depressed mood”; and Item 8, “other mental andbehavioral problems”), and “social problems” (Item 9,“problems with relationships”; Item 10, “problems withactivities of daily living”; Item 11, “problems with livingconditions”; and Item 12, “problems with occupation andactivities”). Thus far, studies investigating the original factorstructure revealed no satisfactory results (Trauer, 1999).

Although psychometric assessments of the HoNOS havebeen presented (Bech et al., 2003; Kisely et al., 2007; Loraet al., 2001; Pirkis et al., 2005; Trauer et al., 1999), empiricalstudies have not yet investigated the specific applicability ofthe HoNOS for inpatients with substance-related disorders.

Hence, the aim of this study is to analyze the reliability,the factor structure, and the prognostic validity of theHoNOS in inpatients with substance-related disorders whoare admitted to a clinic.

2. Materials and methods

2.1. Design and setting

The multicenter study was conducted from July 2004 toJanuary 2006 in four rehabilitation clinics with inpatients ofsubstance-related disorders. The treatment concept of these

clinics comprises psychotherapy and sociotherapy, andindicative groups conducted in individual and group settings.The average treatment duration at the four clinics was 10weeks (SD = 4 weeks).

Patients were consecutively included in the study afterproviding written informed consent. Two therapists (16 intotal) assessed the patient's severity with the HoNOS at thebeginning of treatment (Andreas et al., 2007). The ratingswere made after the intake clinical interview with the patient.The interview covers the current situation of the patient (e.g.,symptoms and problems, social situation, and motivation fortreatment). After approximately 60 ratings per clinic, atraining that was similar to that of the test authors wasconducted (Wing et al., 1998). HoNOS ratings prior to andafter the training were used for the analyses. In addition tothe assessment of the HoNOS, sociodemographic (e.g., age,gender, level of education) and clinical parameters (e.g.,diagnoses) were collected. At the end of treatment, theservice utilization was documented using the coding of theclassification of therapeutic procedures (KlassifikationTherapeutischer Leistungen [KTL]; Bundesversicherung-sanstalt für Angestellte [BfA], 2000).

2.2. Sample description

The sample is composed of 417 males and females, all ofwhich fulfilled the criteria for substance-related disordersaccording to the International Statistical Classification ofDiseases, 10 Revision (ICD-10; World Health Organisation,1992). At the time of admission, all inpatients had undergonedetoxification treatment and had been abstinent for at least 2weeks. As displayed in Table 1, more than two-thirds of theinpatients were male (83%, n = 314). On average, patientswere 34 years old (SD = 10 years); 37% of patients wereyounger than 29 years old. Most of the patients were single(56%, n = 234) and had an elementary school and juniorhigh/secondary education (completion of ninth grade; 53%,n = 158). Twenty-one percent of patients were blue-collarworkers, and 20% were unemployed. The most common firstsubstance-related diagnoses were alcohol dependence (31%,n = 106), opioid dependence (21%, n = 74), and tobaccoconsumption (17%, n = 58; see Table 1).

2.3. Categories of therapeutic procedures

Service utilization was operationalized using the catego-ries of the Classification of Therapeutic Procedures (KTL;Bundesversicherungsanstalt für Angestellte [BfA], 2000),which were documented by therapists at the end of treatment.The KTL is a procedures coding system that definestherapeutic units and standards for the entire field ofrehabilitation medicine in Germany (Bundesversicherung-sanstalt für Angestellte [BfA], 2000). Table 2 provides thedefinition of the categories using an exemplary categoryfrom the psychotherapeutic procedures. The total time use(in minutes) for each patient was determined by adding the

Page 3: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

Table 1Clinical and demographic details of the total sample (N = 417)

Characteristics of the sample n %

GenderFemale 63 17Male 314 83Age (M = 34, SD = 10)b 29 years 133 3730–39 years 113 3140–49 years 80 2250–59 years 23 6N 60 years 14 4Marital statusSingle 234 56Married/Registered partnership 73 18Divorced 45 11Separated/Widowed 16 3Highest educational levelElementary school and junior high/secondary education (completion of 9th grade) 158 53GCSEs/ O-Level (completion of 10th grade) 70 23High school/A-Level/ vocational diploma/entrance qualification for university or technical school (completion of 12th grade) 26 9Without school certificate 31 10Special school certificate 15 5In education at present 1 0Occupational statusLabor/Skilled labor 64 21Unemployed (registered at job center) 58 20Employee/Civil servant 31 10Retired 11 4Student school/University 5 2Other (occupational training, house wife/husband, self-employed, no occupation) 129 44First diagnosis (ICD-10)Mental and behavioral disorders through…F10.2: alcohol dependence 106 31F11.2: opioids dependence 74 21F19.2: multiple substance use and consumption of other psychotropic substances 58 17F12.2: cannabinoids dependence 42 12F14.2: cocaine dependence 29 8F15.2: other stimulants, including caffeine 5 1F13.2: sedativa or hypnotica dependence 2 1F33: recurring depressive disorder 1 0F63: abnormal habits and disorders of impulse control 28 8

34 S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

frequencies per category multiplied by the average durationof a category. In addition to psychotherapeutic procedures,ergotherapy, social and occupational counseling, exercise

Table 2Example category of the classification of therapeutic procedures (Bundesversicher

r35 Exp

Quality characteristics: PsyOccupational group DocSubject area PsyIndication AnxTherapy goal ImpLocation TheEquipment VidOther TraType of application PsyDuration 30–Frequency 1–4No. of patients 1 p

and movement therapy, logopedia, and design, art, andmusic therapy were also included in the calculation of thetotal service utilization.

ungsanstalt für Angestellte [BfA], 2000)

osure treatment in vivo

chotherapy r35tor or psychologist (diploma) and otherschiatry/Psychotherapy/Psychosomatic, substance-related disordersiety and obsessive–compulsive disorders (ICD 3000,ICD 3002,ICD 3003)rovement of anxiety coping, reduction of compulsion, learning self-controlrapy room, fieldeo equipment if applicableining in behavioral therapychotherapy180 minutes× per weekatient

Page 4: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

35S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

2.4. Statistical methods of analyses

Intraclass correlation coefficients (ICCs) between twotherapists assessing the HoNOS at the beginning of treatmentwere calculated for the determination of interrater reliabil-ities. Fleiss and Cohen (1973) showed that the ICC isequivalent to weighted kappa for measures of reliability, andLandis and Koch (1977) provided “rules of thumb” for theinterpretation of kappa coefficients. According to these rules,kappa values between 0.21 and 0.40 are “fair,” thosebetween 0.41 and 0.60 are “moderate,” those between 0.61and 0.80 are “substantial,” and those between 0.81 and 1.00are “almost perfect.”

In line with the above-stated results, according to whichthe ad hoc postulated scale structure of the test authors (Winget al., 1998) could not be replicated, the factor structure ofthe German sample of inpatients with substance-relateddisorders was also analyzed using confirmatory factoranalysis (Amos 4.0; Arbuckle, 1999). The model qualitywas determined using the fit indices recommended by Huand Bentler (1999; standardized root mean square residual[SRMR; Bentler, 1995], root mean square error ofapproximation [RMSEA; Steiger, 1990], and comparativefit index [CFI; Bentler, 1990]). Recommended cutoff valuesare the following: RMSEA ≤0.06, SRMR ≤0.08, and CFI≥0.95 (Hu & Bentler, 1999).

Product–Moment correlations were calculated for theanalysis of the relationship between individual items of theHoNOS and the total service utilization of therapeuticbenefits. The interpretation of the correlation values arebased on the recommendations by Cohen (1988; r ≥ .10small, r ≥ .30 medium, and r ≥ .50 large effect size [ES]).

Considering the clinical service reality, we decided to usea regression tree analysis to figure out combinations orinteractions of inpatient characteristics, which are related toservice utilization (Strobl, Malley, & Tutz, 2009). For theanalysis of the prognostic validity of items and the totalscores of the HoNOS, regression tree analyses were applied.It is expected that single items of the HoNOS and the totalscore are significant predictors of service utilization ininpatient treatment. Thereby, the total resource use of alltherapeutic benefits was predicted by taking into accountHoNOS single items and other central predictors of serviceutilization (e.g., substance-related disorders or sociodemo-graphic variables). Considering the specific referrals frompatients with substance-related disorders with specificcharacteristics (e.g., diagnoses, symptom severity) to theclinics, the variable “clinic” is highly confounded withpatient characteristics. Therefore, we excluded the variable“clinic” from further analysis.

The regression tree analysis is a segmentation methodthat divides a population into mutually exclusive, internallyhomogenous subgroups using a selection of predictors(e.g., HoNOS items). These subgroups are expected todiffer from one another with regard to a set criterion(service utilization in minutes) so that maximum homoge-

neity within subgroups and maximum heterogeneitybetween subgroups are simultaneously achieved (Magid-son, 1997). We analyzed the data with the “Exhaustive-CHAID” algorithm (chi-squared automatic interactiondetector), which can identify structures of higher dimen-sional contingency tables. The empirical significance level(p value), using the chi-square or F-test, between thedependent variable (service utilization in minutes) and thepredicted variables (e.g., items of HoNOS) was used as thesegmentation criteria. The stepwise selection of predictorsforms a tree structure, graphically displaying the results ofthe decision processes of the selected algorithm. Thealgorithm runs through three stages in search of the optimalsplit. We used the sample size of the root node (n = 25), thesample size of the subgroups (n = 10), and a maximum treedepth of five levels (α b .05) as abortion criteria. Theregression tree analysis was conducted using the SPSSversion 15.0 module “tree.”

3. Results

3.1. Descriptive item analyses

Results of the HoNOS ratings in inpatients withsubstance-related disorders are stated in Table 3. Thereare distinct floor effects (answer category 0 = no problem,N30%) for Item 2 “non-accidental self-injury,” Item 1“aggressiveness,” Item 4 “cognitive problems,” Item 5“physical illness or disability problems,” and mostprominently, Item 6 “hallucinations and delusions.” Onthe contrary, there are no distinct ceiling effects for HoNOSitems (answer category 4 = very severe problem, N20%).Furthermore, 8 of the 12 items showed acceptableskewness indices, with an almost normal distribution. Onthe contrary, the following items were not normallydistributed: Item 6 “hallucinations and delusions,” Item 2“nonaccidental self-injury,” Item 4 “cognitive problems,”and Item 1 “aggressiveness.”

Rather low item difficulties (b0.20) were found for 3 ofthe 12 HoNOS items (Item 2 “nonaccidental self-injury,”Item 4 “cognitive problems,” and Item 6 “hallucinations anddelusions”). On the contrary, the highest item difficultycoefficients (N0.50) were found for Item 3 “problem drinkingor drug-taking,” Item 9 “problems with relationships,” andItem 12 “problems with occupation and activities.”Moderatediscriminatory power indices (.30 b r N .50) were found forall HoNOS items, excluding Item 2 “nonaccidental self-injury,” Item 3 “problem drinking or drug-taking,” Item 4“cognitive problems,” Item 5 “physical illness or disabilityproblems,” and Item 6 “hallucinations and delusions.”

Furthermore, as expected, Item 3 “problem drinking ordrug-taking” displayed a higher severity than all other items(all t tests for dependent samples, p b .01), followed by Item9 “problems with relationships” (M = 2.30, SD = 1.14) andItem 12 “problems with occupation and activities” (M = 2.12,SD = 1.26).

Page 5: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

Table 4ICC of HoNOS items prior training (n = 92) and after training (n = 129)

Items HoNOS ICCpre ICCpost

AGGR 0.75 0.82SI 0.82 0.81Addiction 0.76 0.78COG 0.54 0.68PHYSDIS 0.51 0.89Delusions 0.11 0.71DEP 0.49 0.80Other 0.73 0.82REL 0.65 0.64ACT 0.34 0.86Living 0.76 0.79OCC 0.69 0.76

Notes. ICCpre = intraclass coefficients prior to training; ICCpost = intraclasscoefficients after training. For abbreviations of HoNOS scales, see Table 3.

Table 3Distribution properties of the HoNOS items in patients with substance-related disorders

HoNOS items Ceiling (%) Floor (%) M (SD) Skewness Curtosis Item difficulty rit

AGGR 54.2 2.2 0.76 (1.07) 1.35 0.83 0.19 .46SI 90.9 0 0.14 (0.53) 4.26 18.87 0.04 .23Addiction 18 18.5 2.54 (1.38) −0.93 −0.55 0.63 .04COG 48.7 0.7 0.67 (0.86) 1.41 1.76 0.17 .23PHYSDIS 47.4 0.2 0.88 (1.07) 0.95 −0.4 0.22 .14Delusions 94 0 0.07 (0.30) 5.56 36.37 0.02 .19DEP 15.3 3.6 1.70 (1.11) 0.1 −0.77 0.43 .33Other 22.4 9.7 1.96 (1.32) −0.24 −1.16 0.49 .47REL 7.4 12.3 2.30 (1.14) −0.35 −0.74 0.58 .56ACT 27.1 7.7 1.54 (1.30) 0.34 −1.04 0.39 .62Living 34 9.8 1.51 (1.43) 0.41 −1.29 0.38 .55OCC 15.2 7.6 2.15 (1.26) −0.44 −1.0 0.54 .53

Note. rit = moderate discriminatory power (corrected); AGGR = Item 1 “Aggressiveness”; SI = Item 2 “nonaccidental self-injury”; addiction = Item 3 “problemdrinking or drug taking”; COG = Item 4 “cognitive problems”; PHYSDIS = Item 5 “physical illness or disability problems”; delusions = Item 6 “problems associatedwith hallucinations and delusions”; DEP = Item 7 “depressed mood”; other = Item 8 “other problems”; REL = Item 9 “problems with relationships”; ACT = Item 10“problems with activities of daily living”; living = Item 11 “problems with living situation”; OCC = Item 12 “problems with occupation and activities.”

36 S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

3.2. Interrater reliabilities

In line with the above-mentioned criteria, 7 of the 12items exhibited satisfactory coefficients prior to the training(see Table 4). Item 6 “hallucinations and delusions,” inparticular, shows a very low ICC without training. Further,Item 10 “problems with daily activities” and Item 7“depressed mood” show only moderate values prior totraining. After the training, all items displayed satisfactoryvalues (ICC N 0.6), and 6 of the 12 items showed higher thansatisfactory coefficients (ICC N 0.80).

3.3. Factor validity

First, as a basis for the calculation of the confirmatory factoranalysis, an intercorrelation matrix across single items wascalculated. Of 66 possible correlations between the 12 items ofthe HoNOS, only 15 correlations were above the medium ESof r = .30 (Cohen, 1988). The confirmatory factor analysisshowed satisfactory results (RMSEA = 0.09, SRMR = 0.07,CFI = 0.97, TLI = 0.95), thus confirming the original factorstructure. Despite these results, Cronbach's α of three of thefour subscales are quite low (Scale 1 “behavior” α = .05, Scale2 “impairment” α = .20, Scale 3 “symptoms” α = .49, Scale 4“social problems” α = .82). The total score (across all 12 items)was calculated at α = .73, which is considered satisfactory.

Despite the comparable positive characteristics of the CFA,considering the low internal consistency of the subscales, it canbe assumed that the 12 HoNOS items are fairly independentscales. Therefore, a further validity assessment at the singleitem level was conducted. In addition to the validation of singleitems, we decided to report the total score (sum score of the 12items) of the HoNOS for contentwise reasons.

3.4. Prognostic validity

First, the association between single items of the HoNOSand the total service utilization of therapeutic procedures

(KTL) was examined. This revealed a significant highcorrelation with a large ES (r = .53, p b .001, n = 417),particularly for Item 1 “aggressiveness.” Items displaying anearly medium ES were Item 8 “other problems” (r = .32, pb .001, n = 417), Item 10 “problems with daily activities” (r= .33, p b .001, n = 417), Item 11 “problems with livingconditions” (r = .29, p b .001, n = 417), Item 9 “problemswith relationships“ (r = .26, p b .001, n = 417), and Item 12“problems with occupation and activities” (r = .24, p b .001,n = 417). All other items of the HoNOS showed correlationsof r b .20. The correlation coefficient between the total scoreof the HoNOS and the total service utilization of therapeuticprocedures was r = .40 (p b .001, n = 417).

To assess the requirements for the application of aregression tree analysis, the distribution of the total serviceutilization of therapeutic procedures was examined. Satis-factory results emerged regarding the assumption of normaldistribution (M = 2,251 min, SD = 2,542 min, skewness =1.12, kurtosis = 0.42). In addition to normal distribution,high variance of the criteria variables in the sample shouldalso be ensured as a methodological precondition. Therefore,

Page 6: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

37S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

the variation coefficient was calculated at 1.13 as a deviationmeasure (standard deviation/arithmetic mean). According toFischer (2003), variation coefficients greater than 0.5 can beregarded as fairly high (that is, heterogeneity ofcorresponding values can be assumed), whereas variationcoefficients less than 0.3 indicate homogenous distributions.

Regression tree analyses were calculated to examine theprognostic validity of single HoNOS items with otherpredictors. In addition to single items of the HoNOS, thefollowing predictors were also included in the analyses:gender, age at admission, education level, occupationalstatus and marital status, substance-related disorder as maindiagnosis, comorbid personality disorder, and somaticcomorbidity. The decision tree in Fig. 1 displays the results.Of the included predictors, 10 characteristics significantlypredicted the total service utilization of therapeutic proce-dures in inpatient treatment; these accounted for 58% of thetotal variance of the service utilization. Item 1 “aggressive-ness” of the HoNOS was identified as the most importantpredictor. On the second level, the predictors “educationallevel,” “personality disorder (F60, ICD-10) ,” and “occupa-tional status” were relevant predictors. Furthermore, thevariables “marital status,” “gender,” “diagnosis multiple

Fig. 1. Prediction of service utilization with total explanation of variance of 58%.with the lowest resource use.

substance misuse (F19, ICD-10) ,” “age at intake,”“diagnosis cannabinoid dependence (F12, ICD-10),” andItem 3 of the HoNOS “problem drinking and drug-taking”were significant predictors of service utilization (see Fig. 1).Patients with the highest service utilization had higherseverity on Item 1 “aggressiveness,” had a higher educa-tional degree, and were not married (M = 6,159 min, SD =2,239 min, n = 69). On the contrary, patients with the lowestservice utilization exhibited lower severity on Item 1“aggressiveness” and were in occupational rehabilitation(M = 61 min, SD = 343 min, n = 32; see Fig. 1).

A second regression tree analysis was calculated with thetotal score as predictor in place of the single items of theHoNOS. This analysis revealed similar results, and the totalscore was one of the most important predictors of the totalservice utilization of therapeutic procedures. Inpatients withhighest service utilization showed higher scores on theHoNOS total score (M = 1.62–2.08), had less likely acomorbid somatic disorder, and were singles (M = 5,039min, SD = 2,390 min, n = 32). Patients with the lowestservice utilization had lower scores on the HoNOS totalscore (M ≤ 0.82) and were men (M = 274 min, SD = 499min, n = 20).

Dark gray = group with the highest resource use; light gray marked = group

Page 7: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

38 S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

4. Discussion

The HoNOS is a short, fairly easy to complete clinician-rated instrument for the differential assessment of theseverity of mental disorders and is, therefore, suitable forapplication in routine health care. To our knowledge, thisstudy was the first to investigate the specific applicability ofthe HoNOS in a large, multicenter, and to a large degreerepresentative sample of 417 inpatients with substance-related disorders.

The 12 HoNOS items displayed heterogeneous resultsregarding their distribution parameters. Whereas Item 3“alcohol and drug abuse,” Item 7 “depressed mood,” Item8 “other problems,” Item 9 “problems with relationships,” andItem 12 “problems with occupation and activities” (and nearlyItem 10 “problems with daily activities”) showed gooddistribution properties, Item 6 “problems with hallucinationsand delusions” and Item 2 “non-accidental self-injury,” inparticular, did not exhibit satisfactory distributions. Asexpected, these two items exhibit distinct ceiling effectswith unsatisfactory skewness and kurtosis values. This resultis in line with the fact that patients had completeddetoxification treatment and, thus, did not display anymanifest observable or rated symptoms (as could, forexample, be observed during delirium). In addition, patientsdid not exhibit endangerment to self or others at the time ofadmission. This implies the need for further research.Whetherpatients in detoxification treatment differ from patients afterdetoxification on these items should be investigated. Further-more, the distribution properties of the items in a sample ofpatients in detoxification treatment could be examined.

Although Rock and Preston (2001) showed that nospecific HoNOS training is necessary to reach reliable valuesfor single items, this study indicates that training is required.The reliability coefficients of 5 of the 12 items of the HoNOSwere within unsatisfactory limits prior to the training of thetherapists. Following the training, all HoNOS items, exceptItem 9 “problems with relationships,” displayed satisfactoryresults. This result is in line with a number of studies(Brooks, 2000; Orrell, Yard, Handysides, & Schapira, 1999;Trauer et al., 1999) that had shown interrater reliabilitiesbetween ICC = 0.39 and 0.60 for Item 9 “problems withrelationships.” Presumably, this item is difficult to assess atthe beginning of treatment, as during admission and theinitial interview, the focus is on the examination of problemsand the agreement of therapy goals. Hence, whetherreliabilities are greater when the rating is conducted duringthe course of inpatient treatment remains to be investigated.Considering the differences between interrater coefficientsprior to and after training, a future study should investigatewhether HoNOS training is required, taking time andpractice effects into account.

Fairly consistent with empirical findings of other studies(Lauzon et al., 2001; McClelland, Trimble, Fox, Stevenson,& Bell, 2000; Trauer, 1999), the present sample of inpatientswith substance-related disorders did not support the factor

validity of the a priori defined scale structure of the HoNOS(Wing et al., 1998). An analysis of the item intercorrelationsshowed that only 15 of 66 possible correlations were at leastof medium size. Accordingly, no satisfactory results werefound for the internal consistencies of the subscales, whichis consistent with results of previous studies (Lauzon et al.,2001; Orrell et al., 1999; Page, Hooke, & Rutherford, 2001).In light of these results, a further analysis of the prognosticvalidity of the HoNOS at the single item level wasconducted while simultaneously investigating the applica-bility of the HoNOS items for patients with mental disordersin inpatient treatment.

Satisfactory prognostic validity was supported for anumber of HoNOS items. In the correlation analysis and inthe regression tree analysis, Item 1 “aggressiveness,” inparticular, proved to be an important predictor of the totalservice utilization of therapeutic procedures. Although Item3 “alcohol and drug abuse” did not correlate with serviceutilization, it was a relevant predictor in combination withother predictors in the decision tree analysis (see Fig. 1). Ourempirical results show that symptom severity is one of themost important predictors of service utilization. However, toexplicitly examine the relationship between symptomseverity and service utilization, a further study would benecessary where resources were truly meted out in relation totheir need.

Further significant predictors of the total service utiliza-tion were “occupational status,” “personality disorder,”“cannabinoid consumption related disorder,” “marital sta-tus,” “educational level,” “gender,” “diagnosis multiplesubstance misuse,” “age at intake,” and Item 3 of theHoNOS “problem drinking and drug-taking.” Consistentwith earlier investigations, the result of the decision treeanalysis shows that inpatients with higher severity alsodisplay higher service utilization (Ashcraft et al., 1989;Buckingham et al., 1998; Gaines et al., 2003; Horn,Chambers, Sharkey, & Horn, 1989; Mitchell, Dicey, Liptzin,& Sederer, 1987; Taube, Lee, & Forthofer, 1984). In linewith empirical findings of other studies (Ashcraft et al.,1989; Buckingham et al., 1998; Gaines et al., 2003; Hornet al., 1989; Mitchell et al., 1987; Taube et al., 1984), thisstudy found that inpatients in rehabilitation clinics exhibitinglower severity (e.g., lower severity on Item 1 “aggressive-ness,” “occupational status: rehabilitation”) also tend toexhibit lower service utilization. However, the sample sizeper group was relatively small in this study, and therefore,these findings must be replicated in a larger sample. Yet, theextent of the explained total variance of the model is worthmentioning. Empirical studies investigating the relationshipbetween service utilization and sociodemographic andclinical characteristics typically explain approximately20% of the variance in service utilization (Buckinghamet al., 1998; Gaines et al., 2003). The extracted model in thisstudy shows a much larger explained variance of 58%,indicating that the stated predictors substantially predict totaltherapeutic service utilization in inpatient treatment. Our aim

Page 8: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

39S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

was, to analyze the association between symptom severity(among other predictors) and service utilization for patientswho are admitted to a clinic. Further studies are needed toexamine what other factors are influencing service utilization(e.g., insurance coverage) and whether patients withsubstance-related disorders are getting what they need onthe system level.

There are a few numbers of limitations of this study. First,further specific external criteria for the validation of singleitems were not included in this study (e.g., Brief PsychiatricRating Scale [Overall & Gorham, 1976] or AddictionSeverity Index [Mäkela, 2004; McLellan et al., 1992]).However, in another study, we examined the validity of theGerman version of the HoNOS in inpatients with awidespread of mental disorders (Andreas et al., 2010).Evaluation of validity, including factor validity, convergentand discriminant validity, and sensitivity to change, wasconducted on a large, virtually representative, clinicalsample of patients with mental disorders in inpatientpsychotherapy (Study 1, N = 3,169). Additional assessmentof criterion-based validity was completed using anotherclinical sample of patients with mental disorders (Study 2, N= 55). In this study (Study 2), we found that the Item 3 of theHoNOS “problem drinking and drug taking” showed asignificant difference between inpatients with substance-related disorders and inpatients without substance-relateddisorders (Cohen's d = 1.30, Mann–Whitney U test p b.001). In addition, Item 1 “aggressiveness,” Item 2 “self-injury,” and Item 8 “other problems” showed significantdifferences between patients with and without substance-related disorders. Furthermore, the convergent and discrim-inative validity of the HoNOS with the SF-8 (self-rated andclinician-rated), the SCL-90R, and another clinician-ratedinstrument for symptom severity used in Germany could beconfirmed (Andreas et al., 2010).

Second, a further validity criterion is the sensitivity tochange. This study did not examine the extent to which itemsof the HoNOS are sensitive to change in inpatients withsubstance-related disorders; therefore, this should be inves-tigated in future studies.

4.1. Implications for practice

Although, as expected, the a priori postulated scalestructure of the HoNOS could not be confirmed, most of thesingle items seemed to be suitable to validly and reliablyassess the severity of mental problems in inpatients withsubstance-related disorders, but further studies are needed.Hence, the HoNOS can be recommended in inpatients withsubstance-related disorders in routine health care at present.

Acknowledgments

We would like to thank all participants of the study. Thisstudy was funded by the German Research Foundation (GZ:AN 382/3-1).

References

Andreas, S., Dirmaier, J., Harfst, T., Kawski, S., Koch, U., & Schulz, H.(2009). Development and evaluation of a case group concept forinpatients with mental disorders in Germany: Using self-report andexpert-rated instruments. European Psychiatry, 24, 105−111.

Andreas, S., Harfst, T., Dirmaier, J., Kawski, S., Koch, U., & Schulz, H.(2007). A psychometric evaluation of the German version of the “Healthof the Nation Outcome Scales, HoNOS-D”: On the feasibility andreliability of a clinician-rated measure of severity in patients with mentaldisorders. Psychopathology, 40, 116−125.

Andreas, S., Harfst, T., Rabung, S., Mestel, R., Schauenburg, H.,Hausberg, M., Kawski, S., Koch, U., & Schulz, H. (2010). Thevalidity of the German version of the Health of the Nation OutcomeScales (HoNOS-D): a clinician-rating for the differential assessment ofthe severity of mental disorders. International Journal of Methods inPsychiatric Reseach, 19, 50−62.

Arbuckle, J. L. (1999). Amos 4.0 (computer program). Chicago: SPSS.Ashcraft, M. L., Fries, B. E., Nerenz, D. R., Falcon, S. P., Srivastava, S. V.,

Lee, C. Z., et al. (1989). A psychiatric patient classification system: Analternative to diagnosis-related groups. Medical Care, 27, 543−557.

Bech, P., Bille, J., Schutze, T., Sondergaard, S., Wiese, M., & Waarst, S.(2003). Health of the Nation Outcome Scales (HoNOS): Implement-ability, subscale structure and responsiveness in the daily psychiatrichospital routine over the first 18 months. Nordic Journal of Psychiatry,57, 285−290.

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psy-chological Bulletin, 107, 238−246.

Bentler, P. M. (1995). EQS: A structural equations program manual.Encino, CA: Multivariate Software, Inc.

Brooks, R. (2000). The reliability and validity of the Health of the NationOutcome Scales: Validation in relation to patient derived measures.Australian and New Zealand Journal of Psychiatry, 34, 504−511.

Buckingham, B., Burgess, P., Solomon, S., Pirkis, J., & Eagar, K. (1998).Developing a casemix classification for mental health services.Summary. Canberra: Commonwealth Department of Health & FamilyServices.

Bundesversicherungsanstalt für Angestellte (BfA). (2000)). Klassifikationtherapeutischer Leistungen in der medizinischen Rehabilitation (Theclassification of therapeutic procedures in medical rehabilitation), 4th ed.Berlin: BfA.

Cacciola, J. S., Koppenhaver, J. M., Alterman, A. I., & McKay, J. R. (2009).Posttraumatic stress disorder and other psychopathology in substanceabusing patients. Drug & Alcohol Dependence, 101, 27−33.

Cohen, J. (1988). Statistical power and analysis for the behavioral sciences,2nd ed. Hillsdale, NJ: Lawrence.

Drake, R. E., Bartels, S. J., Teague, G. B., Noordsy, D. L., & Clark, R. E.(1993). Treatment of substance abuse in severely mentally ill patients.Journal of Nervous & Mental Disease, 181, 606−611.

Fischer, W. (2003). Die Bedeutung von Pflegediagnosen in Gesundheitsö-konomie und Gesundheitsstatistik ([the importance of diagnoses inhealth economics and health statistics].) Retrieved April 2003, fromhttp://www.fischer-zim.ch/artikel/Pflege-Diagnosen-9901-WEG.htm.

Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and theintraclass correlation coefficient as measures of reliability. Educationaland Psychological Measurement, 33, 613−619.

Gaines, P., Bower, A., Buckingham, W., Eager, K., Burgess, P., & Green, J.(2003). New Zealand mental health classification and outcomes study:Final report. Auckland: Health Research Council of New Zealand.

Horn, S. D., Chambers, A. F., Sharkey, D. P., & Horn, R. A. (1989). Psy-chiatric severity of illness: A case mix study. Medical Care, 27, 69−84.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariancestructure analysis: Conventional criteria versus new alternatives. Struc-tural Equation Modeling, 6, 1−55.

Kisely, S., Campbell, L. A., Crossman, D., Gleich, S., & Campbell, J.(2007). Are the Health of the Nation Outcome Scales a valid andpractical instrument to measure outcomes in North America? A three-site

Page 9: Is the Health of the Nation Outcome Scales appropriate for the assessment of symptom severity in patients with substance-related disorders?

40 S. Andreas et al. / Journal of Substance Abuse Treatment 39 (2010) 32–40

evaluation across Nova Scotia. Community Mental Health Journal, 43,91−107.

Landis, J. R., & Koch, G. G. (1977). The measurement of observeragreement for categorical data. Biometrics, 33, 159−174.

Lauzon, S., Corbiere, M., Bonin, J. P., Bonsack, C., Lesage, A. D., &Ricard, N. (2001). Validation of the French version of the Health of theNation Outcome Scales (HoNOS-F). Canadian Journal of Psychiatry,46, 841−846.

Lora, A., Bai, G., Bianchi, S., Bolongaro, G., Civenti, G., Erlicher, A., et al.(2001). The Italian version of HoNOS (Health of the Nation OutcomeScales), a scale for evaluating the outcome and severity in mental healthservices. Epidemiologia Psichiatria Sociale, 10, 198−212.

Magidson, J. (1997). The chaid approach to segmentation modeling: Chi-squared automatic interaction detection. In R. P. Bagozzi (Ed.), Advancedmethods of marketing research (pp. 118−160). Malden, MA: Blackwell.

Mäkela, K. (2004). Studies of the reliability and validity of the addictionseverity index. Addiction, 99, 398−410.

McClelland, R., Trimble, P., Fox, M. L., Stevenson, M. R. S., & Bell, B.(2000). Validation of an outcome scale for use in adult psychiatricpractice. Quality in Health Care, 9, 98−105.

McLellan, A. T., Kushner, H., Metzger, D., Peters, R., Smith, I., Grissom,G., et al. (1992). The fifth edition of the Addiction Severity Index.Journal of Substance Abuse Treatment, 9, 199−213.

Mitchell, J. B., Dicey, B., Liptzin, B., & Sederer, L. J. (1987). Bringingpsychiatric patients into the Medicare prospective payment system:Alternatives to DRGs. American Journal of Psychiatry, 144, 610−615.

Orrell, M., Yard, P., Handysides, J., & Schapira, R. (1999). Validity andreliability of the Health of the Nation Outcome Scales in psychiatricpatients in the community. British Journal of Psychiatry, 174, 409−412.

Overall, J. E., & Gorham, D. R. (1976). BPRS. Brief Psychiatric RatingScale. In W. Guy (Ed.), ECDEU assessment manual for psychophar-macology (pp. 157−169). Rockville: National Institute of Mental Health.

Page, A. C., Hooke, G. R., & Rutherford, E. M. (2001). Measuring mentalhealth outcomes in a private psychiatric clinic: Health of the NationOutcome Scales and medical outcomes Short Form SF-36. Australianand New Zealand Journal of Psychiatry, 35, 377−381.

Pirkis, J. E., Burgess, P. M., Kirk, P. K., Dodson, S., Coombs, T., &Williamson, M. K. (2005). A review of the psychometric properties ofthe Health of the Nation Outcome Scales (HoNOS) family of measures.Health and Quality of Life Outcomes, 3, 1−12.

Rock, D., & Preston, N. (2001). HoNOS: Is there any point in trainingclinicians? Journal of Psychiatric and Mental Health Nursing, 8,405−409.

Steiger, J. H. (1990). Structural model evaluation and modification: Aninterval estimation approach. Multivariate Behavioral Research, 25,173−180.

Strobl, C., Malley, J., & Tutz, G. (2009). An introduction to recursivepartitioning: Rationale, application, and characteristics of classificationand regression trees, bagging, and random forests. PsychologicalMethods, 14, 323−348.

Taube, C., Lee, E. S., & Forthofer, R. N. (1984). DRGs in psychiatry. Anempirical evaluation. Medical Care, 22, 597−610.

Trauer, T. (1999). The subscale structure of the Health of the NationOutcome Scales (HoNOS). Journal of Mental Health, 8, 499−509.

Trauer, T., Callaly, T., Hantz, P., Little, J., Shields, R., & Smith, J. (1999).Health of the Nation Outcome Scales: Results of the Victorian field trial.British Journal of Psychiatry, 174, 380−388.

Wing, J. K., Beevor, A. S., Curtis, R. H., Park, S. B. G., Hadden, S., &Burns, A. (1998). Health of the Nation Outcome Scales (HoNOS):Research and development. British Journal of Psychiatry, 172,11−18.

World Health Organisation (Ed.). (1992). The ICD-10 classification ofmental and behavioural disorders. Clinical descriptions and diagnosticguidelines. Geneva: World Health Organisation.