Effectiveness and cost-effectiveness of

individually tailored Internet-delivered

cognitive behavior therapy for anxiety disorders

in a primary care population: A randomized

controlled rial

Lise Bergman Nordgren, Erik Hedman, Julie Etienne, Jessica Bodin, Åsa Kadowaki, Stina

Eriksson, Emelie Lindkvist, Gerhard Andersson and Per Carlbring

Linköping University Post Print

N.B.: When citing this work, cite the original article.

Original Publication:

Lise Bergman Nordgren, Erik Hedman, Julie Etienne, Jessica Bodin, Åsa Kadowaki, Stina

Eriksson, Emelie Lindkvist, Gerhard Andersson and Per Carlbring, Effectiveness and cost-

effectiveness of individually tailored Internet-delivered cognitive behavior therapy for anxiety

disorders in a primary care population: A randomized controlled rial, 2014, Behaviour

Research and Therapy, (59), 1-11.

http://dx.doi.org/10.1016/j.brat.2014.05.007

Copyright: Elsevier

http://www.elsevier.com/

Postprint available at: Linköping University Electronic Press

http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-100964

lable at ScienceDirect

Behaviour Research and Therapy 59 (2014) 1e11

Contents lists avai

Behaviour Research and Therapy

journal homepage: www.elsevier .com/locate/brat

Effectiveness and cost-effectiveness of individually tailoredInternet-delivered cognitive behavior therapy for anxiety disordersin a primary care population: A randomized controlled trial

Lise Bergman Nordgren a, *, Erik Hedman b, c, Julie Etienne a, Jessica Bodin a,Åsa Kadowaki d, Stina Eriksson e, Emelie Lindkvist e, Gerhard Andersson a, b,Per Carlbring f

a Department of Behavioural Sciences and Learning, Link€oping University, SE-581 83 Link€oping, Swedenb Department of Clinical Neuroscience, Psychiatry Section, Karolinska Institutet, 171 77 Stockholm, Swedenc Department of Clinical Neuroscience, Osher Center for Integrative Medicine, Karolinska Institutet, 171 77 Stockholm, Swedend County Council of €Osterg€otland, 581 91 Link€oping, Swedene Department of Psychology, Umeå University, 901 87 Umeå, Swedenf Department of Psychology, Stockholm University, 106 91 Stockholm, Sweden

a r t i c l e i n f o

Article history:Received 25 June 2013Received in revised form19 April 2014Accepted 21 May 2014Available online 2 June 2014

Keywords:Primary careAnxietyDepressionComorbidityInternet-delivered cognitive behaviortherapyCognitive behavior therapyCost-effectiveness

* Corresponding author. Tel.: þ46 13 28 20 37.E-mail address: lise.bergman.nordgren@liu.se (L.B

http://dx.doi.org/10.1016/j.brat.2014.05.0070005-7967/© 2014 The Authors. Published by Elsevier

a b s t r a c t

A significant proportion of the general population suffers from anxiety disorders, often with comorbidpsychiatric conditions. Internet-delivered cognitive behavior therapy (ICBT) has been found to be apotent treatment for patients with specific psychiatric conditions. The aim of this trial was to investigatethe effectiveness and cost-effectiveness of ICBT when tailoring the treatment to address comorbiditiesand preferences for primary-care patients with a principal anxiety disorder. One hundred participantswere recruited through their primary-care contact and randomized to either treatment or an activecontrol group. The treatment consisted of 7e10 weekly individually assigned modules guided by onlinetherapists. At post-treatment, 46% of the treatment group had achieved clinically significant improve-ment on the primary outcome measure (CORE-OM) and between-group effect sizes ranged from d ¼ 0.20to 0.86, with a mean effect of d ¼ 0.59. At one-year follow-up, within-group effect sizes varied betweend ¼ 0.53 to 1.00. Cost analysis showed significant reduction of total costs for the ICBT group, the resultswere maintained at one-year follow-up and the incremental cost-effectiveness ratio favored ICBTcompared to control group. Individually tailored ICBT is an effective and cost-effective treatment forprimary-care patients with anxiety disorders with or without comorbidities.Trial Registration: Clinicaltrials.gov: NCT01390168.© 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND

license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

Background

In Europe, anxiety disorders have an estimated 12-monthprevalence of nearly 14 percent (Alonso et al., 2004b; Wittchenet al., 2011). Comorbidity across different anxiety and mood dis-orders is high in the general population (Alonso et al., 2004a) andthe occurrence of an additional Axis-I diagnosis is even higher inindividuals seeking treatment for their anxiety in primary carecenters for treatment of stress, anxiety and related disorders(Brown, Campbell, Lehman, Grisham, & Mancill, 2001). The most

. Nordgren).

Ltd. This is an open access article u

common comorbid disorder for individuals with one anxiety dis-order is a mood disorder or another anxiety disorder. All of theseconditions are associated with reduced quality of life (Mendlowicz& Stein, 2000; Olatunji, Cisler, & Tolin, 2007), an increased risk ofdeveloping somatic disorders (Denollet, Maas, Knottnerus, Keyzer,& Pop, 2009), and high societal costs (Smit, Cuijpers, et al., 2006).

Despite these high numbers of individuals suffering from anx-iety disorders, only a small proportion receive adequate treatment(Weisberg, Dyck, Culpepper, & Keller, 2007; Wittchen & Jacobi,2005). Primary care facilities often lack resources to meet theneed for psychological treatments in general and evidence-basedtreatments in particular. One way of increasing access toevidence-based psychological treatments in primary care could beto use guided Internet-delivered cognitive behavior therapy (ICBT)

nder the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e112

(Andersson, 2009; Carlbring & Andersson, 2006). Guided ICBT hasbeen shown to be effective for a variety of anxiety and mood dis-orders (Andrews, Cuijpers, Craske, McEvoy, & Titov, 2010; Speket al., 2007), and has also been found to be cost-effective whencompared to no treatment or to conventional cognitive behaviortherapy (Hedman, Andersson, Lj�otsson, Andersson, Rück, M€ortberg,et al., 2011; Hedman, Andersson, Lj�otsson, Andersson, Rück, &Lindefors, 2011; Hedman, Lj�otsson, & Lindefors, 2012). There issome evidence for the long-term effects of ICBT for anxiety disor-ders (Carlbring et al., 2011; Carlbring, Nordgren, Furmark, &Andersson, 2009; Hedman, Furmark, et al., 2011; Paxling et al.,2011), and the evidence base for the clinical effectiveness of ICBTis growing, showing promising results (Bell, Colhoun, Carter, &Frampton, 2012; Hedman, Lj�otsson, et al., 2013; Hoifodt, Strom,Kolstrup, Eisemann, & Waterloo, 2011; Mewton, Wong, &Andrews, 2012; Roy-Byrne et al., 2010; Ruwaard, Lange,Schrieken, Dolan, & Emmelkamp, 2012). One potential problemwhen disseminating guided ICBT in primary care is that manyrandomized controlled trials (RCTs) on guided ICBT make use ofstrict single diagnosis protocols and suffer from high exclusionrates. One example is a study by Carlbring et al. (2005), where over80% of the individuals applying for the study were excluded, thelion's share because panic disorder was not their primary problem.The impact of existing co-morbidity on these protocols is largelyunknown. In the field of guided ICBT, there is some evidence thatcomorbidity is reduced following ICBT for social anxiety disorder(Titov, Gibson, Andrews, & McEvoy, 2009). Overall, however, it isreasonable to assume that patients with comorbidities presentwith more deficits in their functioning and that treatments thatfocus on only one aspect of the problem may run the risk of goingagainst patient preferences, which potentially can affect outcome.In CBT, different approaches have been developed to deal withcomorbidity and high exclusion rates. One such approach is theunified transdiagnostic treatment protocol (Barlow, Allen, &Choate, 2004), in which common shared characteristics acrosstreatments for specific conditions are assumed to serve as activeingredients across disorders. But still there is not much evidencethat a unified treatment approach is more effective than conven-tional CBT (Craske, 2012).

There have been some promising results from the ICBT formatfor the delivery of transdiagnostic treatment (Dear et al., 2011;Titov, Andrews, Johnston, Robinson, & Spence, 2010; Titov et al.,2011). Another way of dealing with comorbidities is by address-ing them directly, using individually tailored treatments. Whentailoring the treatment both the patient and the therapist areallowed to influence the protocol.

A similar tailored program used in this study has been shown tobe effective previously in a heterogeneous self-recruited sample,both directly at post-treatment and at a long-term follow-up(Carlbring et al., 2011). In another study, the same protocolapproach was tested for young adults and adults with panicsymptoms (Silfvernagel et al., 2012). Moreover, tailored ICBT wasfound to be more effective for more severely depressed patientscompared to moderately depressed patients in a controlled trialcomparing standard ICBT versus tailored ICBT for depression(Johansson et al., 2012). All of these previous studies were efficacytrials and tailored ICBT has not been tested in any publishedeffectiveness study. The current study, designed to be clinicallyrepresentative (Shadish, Matt, Navarro, & Phillips, 2000), aims toinvestigate whether this approach could be beneficial for a samplerecruited through their primary-care contact and to explore if it is acost-effective alternative. We expected, on the basis of previousefficacy trials, that the treatment would be moderately effective,both at post-treatment and at a one-year follow-up. We also ex-pected that the treatment would be cost-effective.

Method

Ethics statement

The study was approved by the regional Ethical Board and hasbeen registered in clinicaltrials.gov (NCT01390168). Writteninformed consent was collected from all participants by surfacepost.

Study design

The present study was an effectiveness study examining theeffects of ten weeks of ICBT with scheduled e-mail guidancecompared against an active waiting list control group who hadaccess to health care and can be viewed as equivalent to care-as-usual when comparing pre- to post-treatment and pre-to-one-year follow-up results. For the full study design, we refer to thepublished study protocol (Nordgren, Andersson, Kadowaki, &Carlbring, 2012). We made two changes to the protocol in thepresent study by not using ANCOVA for data analysis, but insteadusing a linear mixed model because of the large amount of missingdata in the follow-up measurements (Gueorguieva & Krystal,2004). Moreover, we managed to include only 100 participants,instead of the 128 outlined in the protocol. This change was madebecause of time limits of the project and funding. Inclusion tookplace in 18 different primary care settings, in ten different cities,between January 2010 and April 2011. A new power calculationbased on 100 participants revealed a power of 84%, given an alpha-level of .05 and an effect size of d ¼ 0.6.

A total of 100 patients were recruited through their primary carecontact (e.g., physician) and were randomly assigned by an onlinetrue random-number service independent of the investigators andtherapists to either immediate treatment or control. The controlgroup received treatment after tenweeks, leaving no control groupfor the follow-up measurements.

Inclusion and exclusion criteria

To be included in the study, participants had to fulfill the DSM-IV (American Psychiatric Association, 2000) criteria for any anxietydisorder as a primary diagnosis. Hence, all participants had a clin-ically significant impairment because of at least one anxiety dis-order with or without comorbid problems. Participants had to be atleast 18 years old, have Internet access, not be participating in anyongoing psychological treatment, and if taking any medication foranxiety or depression, this had to be at an unchanged dosage for atleast 12 weeks pre-treatment. Participants with a suicidal risk,defined as scoring >3 on item 9 of the Montgomery ÅsbergDepression Rating Scale (MADRS-S; Svanborg & Åsberg, 1994),were further interviewed using the SAD PERSONS scale (Patterson,Dohn, Bird, & Patterson, 1983) during a telephone-administereddiagnostic interview and were excluded if their risk of suicidewas confirmed (they then were referred back to their original pri-mary care setting). Scoring above 30 on the MADRS-S or below 9 onthe Beck Anxiety Inventory (BAI; Beck, Epstein, Brown, & Steer,1988) also served as reasons for exclusion.

Recruitment and participants

Participants were recruited from a clinical population, typicallyby their general practitioner or nurse, when seeking help at theirprimary care setting. Prescribing clinicians were informed aboutthe intervention and the inclusion criteria in order to decide forwhom this might be adequate, regardless if the patient presentedthemselves with anxiety as their primary complaint. Based on the

Fig. 1. Flowchart of study participants, point of random assignment, and dropouts at each stage of a study of individually-tailored, internet-delivered cognitive behavior therapy foranxiety disorders.

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 3

clinical judgment of the clinician eligible participants were given aninformation folder containing brief information about the trial anda link to the study website. On the website they found further in-formation about the treatment, an informed consent form, a link tothe questionnaires serving as screening, and the application formfor participation.

Initial selection took place through computerized screeningconsisting of the Clinical Outcomes in Routine Eval-uationdOutcome Measure (CORE-OM; Barkham et al., 2001), theself-rated version of the Montgomery Åsberg Depression Rating

Scale (MADRS-S; Svanborg & Åsberg, 1994), the Beck Anxiety In-ventory (BAI; Beck et al., 1988), the Quality of Life Inventory (QOLI;Frisch, Cornell, Villanueva, & Retzlaff, 1992), and ten additionalquestions regarding demographics and current and past treat-ments. These results were used as the pre-treatment assessment.The outcome measures all show good psychometric propertieseven when administered online (Carlbring et al., 2007; Hedmanet al., 2010).

Participants who fulfilled the initial inclusion criteria accordingto the computerized screening were contacted to take part in a

Table 1Demographic characteristics of participants at baseline.

Treatment(n ¼ 50)

Control(n ¼ 50)

Gender Female 33 30Male 17 20

Age Mean (SD) 35 (13) 36 (12)Min-max 20e68 19e60

Marital status Married/living together 25 38In a relationship 3 3Single 20 9Other 2 0

Highesteducation

Secondary school 13 10

level Vocational school (compl) 4 5College/university(not compl.)

13 14

College/university (compl) 20 21Employment

statusEmployed full- or part-time 21 26

Registered sick 4 8Unemployed 6 1Student 16 15Other 3 0

Psychotherapy None 17 (34%) 19 (38%)Earlier 33 (66%) 31 (62%)

Medication None 21 (42%) 22 (44%)Earlier 15 (30%) 16 (32%)Ongoing 14 (28%) 12 (24%)

Primarydiagnosis

Generalizedanxiety disorder

4 (8%) 6 (12%)

Social phobia 16 (32%) 16 (32%)Panic Disorder þ/�agoraphobiaAgoraphobia

16 (32%)4 (8%)

15 (30%)4 (8%)

Anxiety NOS 10 (20%) 9 (18%)Comorbidity Any comorbidity 30 (60%) 28 (56%)

Mood disorder 28 (56%) 15 (30%)Generalized anxietydisorder

2 9

Social Phobia 6 7Panic disorder þ/� agoraphobia 1Agoraphobia 2 2Specific phobia 1Obsessive compulsive disorder 1Hypochondriasis 1

Note: NOS, Not otherwise specified. Participants could have more than one co-

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e114

telephone interview where they were interviewed using theStructured Clinical Interview for DSM-IVdAxis I Disorders (SCID-I;First, Gibbon, Spitzer, & Williams, 1997). Seven MSc clinical psy-chology students in the latter part of their clinical training con-ducted the interviews. All received training, including role-playingand coding of videotapes, and supervision in using the interview.Telephone administration of the SCID-I interview has been found tobe equally valid as a face-to-face administration of the SCID-I(Rohde, Lewinsohn, & Seeley, 1997). A psychiatrist and a licensedclinical psychologist together with the interviewers assessed theSCID protocols, and together they tailored the treatment accordingto each participant's unique set of problems and, to some extent,preferences. Both the psychiatrist and clinical psychologist hadexperience of working in primary care facilities, and the assess-ments and judgments were aimed to mirror how this procedurecould be made in usual care. In summary, participants were firstjudged by their general practitioner to suffer from anxiety and afteronline screening were diagnosed thoroughly by the research teamusing SCID-I.

Of the 152 individuals who initially applied to participate in thestudy, 100 were included after a telephone-administrated diag-nostic interview and randomized by an online true random-number service independent of the investigators and therapists.The reasons for exclusion are specified in the CONSORT flowchart(Fig. 1).

The mean age was 35.4 years, and 63 percent were female (63/100). Thirty six percent (36/100) reported their marital status asmarried or living together, with children in the household. Sixtyfour per cent had a history of previous psychological treatment, and26 per cent (26/100) were using antianxiety or antidepressantmedication, while 31 percent reported a history of medical treat-ment for their condition. Demographic data on the included par-ticipants are presented in detail in Table 1.

All included participants had an anxiety disorder as their prin-cipal diagnosis. A majority (58/100) had one (n ¼ 39) or more co-morbid Axis-I disorder. For 43 participants, a comorbid mooddisorder (major depressive disorder or dysthymia) was present.There were no difference between the treatment and control groupregarding patients presenting comorbidities (n ¼ 30 in the treat-ment condition and n ¼ 28 in the control condition).

morbid condition.

Outcome measures

Primary outcome measureThe CORE-OM (Barkham et al., 2001) served as the primary

outcomemeasure. This is a 34-item scale covering four subscales ofgeneric psychological well-being (Barkham et al., 2001). The sub-scales are subjective well-being, symptoms (anxiety, depression,physical problems, trauma), functioning (general functioning, closerelationships, social relationships), and risk (to self and others).Items are scored from 0 to 4 and relate to the preceding week.CORE-OM clinical scores can range from 0 to 40, with higher scoresindicating greater severity. Internal consistency has been reportedas a ¼ .94 (Barkham et al., 2001), with a cut-off score between aclinical and non-clinical population of 12 for men and 12.6 forwomen on the composite scale (Elfstrom et al., 2012).

Secondary outcome measuresAs secondary outcome measures, the BAI (Beck et al., 1988), the

MADRS-S (Svanborg & Åsberg, 1994), and the QOLI (Frisch et al.,1992) were used.

BAI measures anxiety symptoms and contains 21 short ques-tions. Total scores range between 0 and 63, with a score of 30 orhigher indicating severe anxiety symptoms. Internal consistency is

high (Beck et al., 1988), with Cronbach's alpha ranging from .90 to.94, and test-retest reliability at r ¼ .75.

The MADRS-S (Svanborg & Åsberg, 1994) measures depressivesymptoms over the past three days and consists of nine items ratedon a seven-grade scale. Total scores range between 0 and 54, withscores above 35 (Snaith, Harrop, Newby, & Teale, 1986) consideredsevere. Concurrent validity is shown to be good with a test-retestreliability of r ¼ .83 (Mattila-Evenden, Svanborg, Gustavsson, &Åsberg, 1996).

The QOLI (Frisch et al., 1992) assesses the subjective degree ofsatisfaction in 16 different areas of life (e.g., health and work). Thescale shows high internal consistency (a ¼ .77e.89), and a test-retest reliability from r ¼ .80 to .91 (Frisch et al., 1992; Lindner,Andersson, €Ost, & Carlbring, in press).

Cost assessment

We used the Trimbos and Institute of Medical TechnologyAssessment Cost Questionnaire for Psychiatry (TIC-P; Hakkaart-VanRoijen, Van Straten,&Donker, 2002) to obtain health economic costdata. The TIC-P measures monthly direct medical costs, that is,health care consumption (e.g., primary care visits) as well as

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 5

nondirect medical costs, that is, costs of other health-related ser-vices not directly associated with health care (e.g., time spent inself-help groups). Nonmedical costs, which are costs pertaining towork and domestic productivity loss, were also measured with theTIC-P.

The costs of ICBT and the control condition was representedmainly by the costs of therapists, and we used the tariff of visits tolicensed clinical psychologists when estimating costs for bothconditions. This tariff was multiplied with time spent treatingpatients.

Health related quality of lifeThe EuroQol (EQ-5D) was used to assess quality of life from a

health perspective (EuroQol-Group, 1990). The EQ-5D measuresfive health domains related to quality of life: mobility, self-care,usual activities, pain/discomfort, and anxiety/depression (Rabin &Charro, 2001). The measure has demonstrated good psychomet-rics properties, including acceptable test-retest reliability (intra-class coefficient ¼ .82e.83) and acceptable convergent validity(Ravens-Sieberer et al., 2010).

Intervention

The ICBT treatment were a form of text-based guided self-helptreatment with therapist support via the internet that consistedof seven-to-ten treatment modules, or chapters, per participantcovering a period of tenweeks. Themodules have been shown to beeffective previously when used on specific diagnosed samples (i.e.,panic disorder; Carlbring et al., 2006), social phobia (Anderssonet al., 2006), generalized anxiety disorder (Paxling et al., 2011),and depression (Vernmark et al., 2010) and were adapted to be ableto be presented together in the tailored format by being slightlyrewritten (e.g., words relating to specific conditions wereremoved). Tailoring included both the order of treatment modules,the amount of text presented to the participant and how manymodules to include in the ten week treatment protocol. The sameseven MSc students who conducted the SCID-interviews served asInternet therapists. The therapists had access to supervision by alicensed clinical psychologist both regarding the format and client-specific questions throughout the study period. All communicationbetween participants and therapists was made through theinternet, using a messenger system within the treatment platformsimilar to e-mail, and the main nature of the feedback was toanswer any questions regarding the module and homeworkassignments.

Each module consisted of text and illustrations (9e39 A4 pages)presenting a specific symptom and exercises and were to becompleted by three-to-eight essay questions to be worked throughduring a period of one week. Some of the modules (i.e., relaxationand mindfulness) had audio files attached to them for the partici-pants to listen to. The homework questions were intended toencourage learning and to help the Internet therapist assesswhether the participants had assimilated the material or not. Par-ticipants were asked to answer the questions and provide work-sheets and report on outcomes of different exercises to theirtherapist once a week. Following submission of the report, theywere given individual feedback, most often within 24 h. When thetherapist received a homework assignment showing that theparticipant has assimilated the material, the next module wasmade accessible through an encrypted message exchange system.The therapists were instructed not to spend more than 15 min perparticipant per week in reading and communicating feedback.

Prescribed modules were available for download in PDF format,and participants were advised to print out or to download the self-help material to have the material readily available.

The first module (introduction) and the last (relapse prevention)were fixed, which gave the possibility to tailor the treatment byadding any of the following: cognitive restructuring (2 modules),social anxiety (2 modules), generalized anxiety (3 modules), panicdisorder (2 modules), agoraphobia, behavioral activation (2 mod-ules), applied relaxation, mindfulness, assertiveness, problemsolving, stress management, and sleep. All modules, except forcognitive restructuring, applied relaxation, assertiveness, problemsolving and stress management, were disorder specific in that theytargeted core symptoms of each of the diagnoses.

Attention control groupParticipants in the control conditionwere asked questions about

their well-being on a weekly basis by their therapist. Each partici-pant e-mailed his or her answers to the therapist once a week inorder to track possible deterioration, but was generally not givenany specific feedback on the answers unless the therapist judgedthe answer to warrant further action.

Statistical analysis

Treatment effectsIn order to handle missing data and be able to use the full

sample, we used intention-to-treat (ITT) analysis. This method ac-counts for missing data without assuming that the last measure-ment was stable (as is assumed when using the last observationcarried forward; Salim, Mackinnon, Christensen, & Griffiths, 2008)and is appropriate for designs with few time points. For pre- topost- changes, the sample size was all 100 participants; for thecalculation of pre- to one-year follow-up effects, sample size wasset to 92 participants, excluding the patients withdrawing theirapplication during their time on the waiting list (n ¼ 8). For bothtreatment effects and cost-effectiveness, SPSS 20.0 (SPSS Inc., Chi-cago) was used, and for further calculations of costs and cost-effectiveness, STATA IC/11.0 (Stata Corp) was used.

Treatment effects were calculated using a linear mixed modelwith restricted maximum-likelihood estimation (REML) and anunstructured (UN) covariance structure. Analyses of post-treatmenteffects were made using the treatmentdcontrol group design. Andfor the analyses for effects at one-year follow-up, we used thetreatment and control group as one. Measure points were indi-vidual, using each participant's baseline as time one and thenassessing post and twelve month follow-up measures using thistime as starting point. This resulted in that datawere collected fromfirst inclusions in January 2010 until the spring of 2013 for the lastfollow-up measures. Effect sizes (Cohen's d) were calculated bydividing the differences between means by the pooled standarddeviation. Standard deviations were calculated from the standarderrors by using the formula SD ¼ SE*√N. An effect size of 0.20 wasconsidered to be small, of 0.50 to be moderate, and 0.80 and aboveto be large (Cohen, 1988).

To define responders on the main outcome measure, we usedthe criteria for clinically significant improvement proposed byJacobson and Truax (1991). The reliable change index was calcu-lated using the test-retest reliability coefficient of 0.85 and thenusing the cut-off scores for the nonclinical sample (Elfstrom et al.,2012) to decide whether or not the changes were clinicallysignificant.

Costs and cost-effectiveness. Costs were assessed at baseline, post-treatment, and at 12-month follow-up. Because the control con-dition was crossed over to treatment after post-treatment, be-tween-group comparisons were made only for the post-treatmentdata and not at 12-month follow-up. In this analysis, missing datafrom baseline to post-treatment were imputed using the last

Table 2Estimated means, standard error (SE), on all outcome measures at pre-treatmentand post-treatment, effects, and between-group Cohen’s effect sizes (d) at post-treatment.

Measure Time Treatment(n ¼ 50)

Control(n ¼ 50)

Effects F(1,89e97)a

Between-group d

CORE-OM Pre 18.44(0.63) 17.65(0.63) T: 95.51***Post 10.97(0.75) 17.65(0.63) G: 5.097* 0.86

I: 28.82***BAI Pre 21.18(1.37) 21.32(1.37) T: 68.334***

Post 11.81(1.10) 16.30(1.10) G: 2,305ns 0.58I: 6.23*

MADRS-S Pre 19.62(0.96) 17.84(0.96) T: 59.734***Post 10.84(1.04) 15.94(1.04) G: 1.825ns 0.70

I: 24.77***QOLI Pre 0.446(0.23) 0.865(0.23) T: 12.424***

Post 1.320(0.21) 1.033(0.21) G: 0.064ns 0.20I:5.70*

Abbreviations: CORE-OMOutcomes in Routine Evaluatione OutcomeMeasure; BAI,Beck Anxiety Inventory; MADRS-S, Montgomery Åsberg Depression Rating Scale-Self rated; QOLI, Quality of Life Inventory; T, Time; G, Group; I, Interaction time*-group.ns non significant * ¼ p < .05 ** ¼ p < .01; *** ¼ p < .001.

a denominator degrees of freedom varied between 89.336 and 96.972.

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e116

observation carried forward (LOCF), which has been shown toproduce the same cost-effectiveness outcomes as imputation basedon linearmethodswhen applied on cost data assessedwith the TIC-P (Andersson et al., 2011, Hedman, Andersson, Lj�otsson, Andersson,Rück, M€ortberg, et al., 2011; Hedman, Andersson, Lj�otsson,Andersson, Rück, & Lindefors, 2011), but does not rely onassumption of linearity of cost data. However, considering the longperiod between post-treatment and 1-year follow-up LOCF was notused at the latter assessment point. Instead, 1-year follow-up es-timates are based only on participants who provided data. Thestatistical analysis was conducted in four steps.

Incremental cost effectiveness ratio (ICER), which was the pri-mary health economic outcome, was estimated using the followingformula: (DC1 � DC2)/(DE1 � DE2),where C1eC2 is the difference incosts between ICBT and the control condition, and E1eE2 refers tothe difference of the average effectiveness of the two conditions(Drummond, Sculpher, Torrence, O'Brien, & Stoddart, 2005). Thecosts, from all cost domains, of the participants in the ICBT condi-tion were subtracted from the costs of the participants in thecontrol condition. This difference was then divided by the sub-tracted effects, i.e. improvement of the CORE-OM scores. Thisprocedure was bootstrapped 5000 times, generating an estimatedfigure of the treatment groups' incremental costs in relation to theirincremental health benefit. Bootstrap analysis was used because itis considered to generate reliable cost distribution estimates (Efron& Tibshirani, 1993).

A cost-utility analysis was also conducted, which is identical tothe cost-effectiveness analysis with the exception that the cost ofan additional quality-adjusted life year (QALY; Bravo Vergel &Sculpher, 2008) is calculated instead of an additional unit of theCORE-OM. The analysis was performed applying the population-based index weights proposed by the EuroQol Group (Dolan,1997). A QALY of 1 is equivalent to one year of full health andscore of 0 equals death, which means that five years lived with aquality of life of 0.20 yields a QALY of 1 (Bravo Vergel & Sculpher,2008).

Two sensitivity analyses were performed to test the robustnessof the results. The main objective in these analyses was to inves-tigate how sensitive the ICERs would be to changes of cost esti-mates within a reasonable uncertainty range. In a first step, werepeated the analysis but increased the estimated intervention costof ICBT with $200. In a second step, we performed an analysiswhere $600was added to the cost of ICBT, corresponding to the costof ICBT during the first year of providing the service and thusincluding developmental costs and costs of establishing the treat-ment unit. These costs were expected to decrease rapidly afterestablishing the treatment unit as they are one-time costs. Inaddition to analysis of ICERs, repeated measures ANOVA and t-testswere used to test cost changes for each cost domain.

3. Results

Adherence and attrition

Adherence was measured by using the percentage of completedprescribed modules for the treatment group. Completion wasdefined as when the participant had sent in his or her homeworkassignment to the therapist for the prescribed module. All pre-scribed modules were completed by 32 per cent (16/50) of thetreatment group. One participant (2%) did not complete any of theprescribed modules, while the mean completion rate was 53.5percent of the prescribed modules.

In the treatment group three persons dropped out, and twowere excluded (for details, see Fig. 1), leaving 45 participantsremaining in the treatment group at the end of the ten-week

treatment period. In the control group, eight persons withdrewtheir application before starting the treatment. During the treat-ment of the control group, three participants dropped out, and twowere excluded.

Treatment effects

Pre to post changesEffect sizes varied from d ¼ 0.20 to 0.86 with a mean effect size

of d ¼ 0.59. In terms of treatment response, 46% (23/50) of theparticipants in the treatment group showed a clinically significantimprovement on the CORE-OM, compared to 12% (6/50) in thecontrol condition. Detailed data on the pre to post results arepresented in Table 2.

When taking a comorbid mood disorder into consideration, nointeraction effects of time and group and mood disorder werefound on either the primary measure (F (1,90.026) ¼ 1.666,p ¼ .200) or on secondary measures (BAI; F (1,93.632) ¼ 0.266,p ¼ .608; MADRS-S; F (1,89.482) ¼ 0.280, p ¼ .598; QOLI;F(1,89.318) ¼ 0.832, p ¼ .364).

Taken together, the results indicate that participants in thetreatment group reported an improvement on all outcomes and themean between-group effect was moderate. Comorbid depressiondid not moderate the outcome.

Pre to follow-upResults from the follow up measures (see Table 3) revealed that

results from post-treatment were sustained at a one-year follow-upwith within-group effect-sizes ranging from d ¼ 1.00e0.53, with amean of d ¼ 0.71, indicating that the treatment had sustained ef-fects one year after completion. On the primary measure (CORE-OM), 33.3% (25/75) showed a clinical significant improvement oneyear after completion. If non responding persons would be regar-ded as treatment failures, these figures would have changed to27.2% (25/92).

Further exploration of interaction effects of comorbid mood dis-order revealed no moderating effect on the CORE-OM (F(1,77.974) ¼ 2.353, p ¼ .102) or on the BAI (F (1,75.120) ¼ 1.301,p¼ .278) or theMADRS-S (F (1,71.172)¼ 2.492, p¼ .090). Therewas asignificant interaction effect of time and mood disorder on the QOLI,however (F (1,67.021) ¼ 3.969, p ¼ .023). This indicates that patientswith a comorbid mood disorder had a larger gain in their quality-of-life assessments from pretreatment to one year follow-up.

Table 3Estimated means, standard error (SE), standard deviation (SD) on all outcome measures at pre-treatment, post-treatment and one-year follow up, and within-group Cohen’seffect sizes (d) at one-year follow-up.

Measure Time Estimated means (SE) SD p (time) Significant pairwise comparisons Within-group d

CORE-OM Pre 15.634 (.494) 4.74 p < .000 Pre > Post ¼ 1 yr 0.75Post 11.314 (.579) 5.551Fup 11.487 (.643) 6.17

BAI Pre 16.559 (.920) 8.82 p < .000 Pre > Post ¼ 1 yr 0.53Post 12.312 (.746) 7.161Fup 11.941 (.914) 8.77

MADRS-S Pre 18.098 (.722) 6.93 p < .000 Pre > Post ¼ 1 yr 1.00Post 10.603 (.837) 8.031Fup 10.550 (.851) 8.16

QOLI Pre .703 (.145) 1.39 p < .000 Pre > Post ¼ 1 yr 0.57Post 1.491 (.155) 1.491Fup 1.597 (.182) 1.75

Abbrevations: CORE-OM Outcomes in Routine Evaluation e Outcome Measure; BAI, Beck Anxiety Inventory; MADRS-S, Montgomery Åsberg Depression Rating Scale-Selfrated; QOLI, Quality of Life Inventory; SE, standard error; SD, Standard deviation.

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 7

Psychological treatment and medication

At the one-year follow-up, based on observed data (n¼ 75), 36%(27/75) reported that they had started another psychologicaltreatment for their condition, and 16% (12/75) reported increasingor starting anxiolytic or antidepressant medication. A majority90.7% (68/75) reported no changes in medication, and 12% (9/75)reported lowering their dosage. In these data the persons not usinganxiolytic or antidepressant medication at baseline were reportedas “no changes in medication” if this had not changed. Divided intothe original groups 31% (13/42) of the treatment group started anadditional psychological treatment, 16.3% (7/42) increased or star-tedmedication, 9.5% (4/42) decreased or stopped and 69.1% (29/42)reported unchanged medication. In the control group 39.4% (13/33)started an additional psychological treatment, 3% (1/33) increasedor started medication, 6.1% (2/33) decreased and 51.5% (17/33) re-ported no changes in medication. No differences were found onaltering medication (c2(2) ¼ .952, p ¼ .621) or starting psycho-logical treatment (c2(1) ¼ .319, p ¼ .572) between the two groups.

Costs

Table 4 presents the per capita costs at baseline, post-treatment,and at a 12-month follow-up. Analysis of change in gross total costsfrom baseline to post-treatment showed a significant interactioneffect of time and group, indicating that the ICBTgroupmade largercost reductions than the control condition (F (1,95) ¼ 7.80;p ¼ .006). Analysis of subtype of the cost domain revealed that this

Table 4Per capita costs in US Dollars at pre-treatment, post-treatment and 1-year follow-up. Co

Cost Pre-treatment Post-tre

ICBT CC ICBT

Mean SD Mean SD Mean

Direct medical 333 357 257 305 234Health care visits 322 354 250 303 224Medication 11 27 7 12 9

Direct non-medical costs 73 190 42 69 83Indirect costs 2142 1691 1505 1593 1443Unemployment 1674 1784 1073 1644 1177Sick leave 214 608 193 626 124Work cutback 104 311 160 388 77Domestic 150 231 78 98 65

Gross total costs 2548 1812 1803 1694 1757Intervention costs 507Net total costs 2264

Abbreviations: ICBT, Internet-based cognitive behavior therapy; CC, control condition. N

effect was primarily driven by larger indirect cost reductions in theICBT group compared to the control condition (F (1,95) ¼ 7.54;p¼ .007). There were no significant interaction effects of group andtime in the other cost domains ((F (1,95) ¼ 0.41e1.42); p ¼ .24e53).Within-group tests showed that participants who received ICBTmade significant gross total cost reductions from baseline to post-treatment (t(47) ¼ 3.23; p ¼ .002), and from baseline to the 12-month follow-up (t(40) ¼ 4.02; p < .001), indicating stability ofcost reductions. Participants in the control condition did notsignificantly reduce their gross total at post-treatment compared tobaseline (t(47) ¼ �0.94; p ¼ .35). The intervention costs perparticipant were estimated to be $507 for ICBT and $68 for thecontrol condition.

Cost-effectivenessAt post-treatment, the incremental cost-effectiveness ratio

(ICER) was �616/0.34 ¼ �$1824, favoring ICBT over the controlcondition. This meant that each incremental improvement onCORE-OM for participants in ICBT relative to the control conditiongenerated a societal earning of $1824. This gain was because thetotal net costs were lower in the ICBT condition compared to thecontrol condition and because clinically significant improvementswere more likely to occur in the ICBT condition. The scatter ofsimulated ICERs across the four quadrants of the ICER plane indi-cated the degree of uncertainty around the ICER (see Fig. 2 in theSupplementary Material online). From a cost-effectivenessperspective, the most favorable outcome is a concentration ofscatter in the southeast quadrant, indicating superior treatment

st data at each assessment point.

atment 1-Year follow-up

CC ICBT CC

SD Mean SD Mean SD Mean SD

397 215 399 250 543 229 309395 210 397 246 543 225 30824 6 12 5 7 4 7

158 86 172 73 204 22 511668 1776 1787 1100 1499 995 14501697 1376 1767 827 1533 666 1412559 199 893 70 190 251 704198 120 341 127 321 45 127111 74 107 77 148 33 52

1870 2078 1868 1423 1806 1247 150168 507 575

2146 1930 1822

ote: at 1-year follow-up, participants in the control condition had received ICBT.

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e118

effects and lower costs of the experimental treatment (ICBT)compared to the control condition. A concentration of ICERs in thesouthwest quadrant would indicate lowered treatment effects andlowered costs of ICBT compared to the control condition. If a ma-jority of the simulated ICERs appeared in the northwest quadrant,ICBT would be associated with higher costs and lowered effec-tiveness compared to the control condition, thus making it unac-ceptable from a cost-effectiveness perspective. A majority of thesimulated ICERs are located in the southeast quadrant (95.3%),indicating that ICBT is highly cost-effective compared to the controlcondition.

The same data are used to plot the acceptability curve (see Fig. 3in the Supplementary Material online for a graphic vision of thecurve). The curve indicates that ICBT has a 95% probability of be-ing cost-effective if society was willing to pay $0 for one additionalimproved patient. If society was willing to pay $1000 for one case ofimprovement, the probability of ICBT being cost-effective wouldincrease to 99%.

Cost-utility analysisAt post-treatment, the cost-utility ICER was �474/

0.063 ¼ �$7523. This meant that one additional QALY generated asocietal earning of $7523 when comparing ICBT to the controlcondition.

A majority of the cost-utility ICERs are located in the southeastquadrant of the ICER-plane (77%), suggesting that the most prob-able outcome is that ICBT leads to additional QALYs while pro-ducing societal cost reductions. Of the remaining cost-utility ICERs,13.7% are located in the southwest quadrant, 7.5% in the northeastquadrant, and 2.1% in the northwest quadrant (see Fig. 4 in theSupplementary Material online).

The same data are used for the acceptability curve (see Fig. 5 inthe Supplementary Material online). The curve indicates that ICBThas a 90% probability of being cost-effective if the society wouldpay $0 for one gained QALY. If the society werewilling to pay $3000for one additional QALY, the probability of ICBT being cost-effectivewould increase to 95%.

Sensitivity analyses

The acceptability curves assuming additional costs of ICBT ($200and $600) correspond to (a) a scenario of low productivity and (b) ascenario assuming the cost of ICBT during the first year of deliv-ering the service. The latter analyses included all one-time costs ofdevelopment of ICBT and establishing the treatment in a health-care context. ICBT would remain the most cost-effective treat-ment if $200 were added to the costs of ICBT even if willingness topay for an additional case of improvement were $0. If $600 wereadded to the intervention cost, ICBT would be most likely to becost-effective even if thewillingness to paywas $0 if using clinicallysignificant improvement as ICER outcome (Fig. 3). If adding $600 tothe costs of ICBT and using QALY as outcome, the treatment wouldhave a 37% probability of being cost-effective in comparison to notreatment if willingness to pay for an additional QALY was $0. Thisprobability would increase to 60% if society were willing to pay$3500 for one additional QALY.

Discussion

The aim of this study was to investigate if ten weeks of tailoredICBT could be an effective treatment for a heterogeneous sample ofpatients, referred by their primary care contact, and if treatmenteffects were sustained at a one-year follow-up. We also intended toanswer questions regarding whether treatment could lower directand indirect societal costs in this population. Findings show that

tailored ICBT was effective in our sample of primary care patientswho had comorbidities and approximately one-third of whomwere on medication for their condition. There were small-to-largebetween-group effect sizes at post-treatment on the primary andsecondary outcome measures, and a clinically significantimprovement was obtained by 46 percent of the patients on theprimary outcome measure (CORE-OM). At the one-year follow-up,results were sustained on both primary and secondary measures.About one-third of the sample sought additional treatment. Finally,health economic analyses suggested that the treatment can be cost-effective. Our results thus suggest that tailored ICBT could be afeasible approach in treating patients with anxiety disorders,regardless of comorbidity, in primary care settings, and that it couldbe a cost-effective alternative.

ICBT has been found to be effective for a range of disorders whentested as manualized treatment targeting specific disorders in ef-ficacy trials (Andrews et al., 2010; Cuijpers et al., 2009; Reger &Gahm, 2009; Spek et al., 2007). There is less evidence for effec-tiveness, even if studies suggest that ICBT canwork in more regularclinical settings (Bergstr€om et al., 2010; Hedman, Lj�otsson, et al.,2013; Ruwaard et al., 2012). Although there are a few previous ef-ficacy studies on tailored ICBT (Carlbring et al., 2011), this study isthe first to test this new form of ICBT with more regular clinic pa-tients. From a clinical perspective, where resources for conductingextensive diagnostic test procedures may be limited and hetero-geneity in symptom presentation is common, tailored ICBT may bea feasible approach to use because it considers patient preferencesas well as diagnostic information and allows more patients to beoffered treatment even if they do not present with specific condi-tions. Moreover, by using tailored ICBT, clinicians could make use ofthe different manuals already existing, without having to learn anew one for each condition diagnosed, and this might facilitateimplementation in regular health care. The tailored approach isdifferent than the of for example a unified protocol (Barlow et al.,2004). The differences lie in that the patients’ specific comorbid-ities are being addressed explicitly through what are believed to becore symptoms of each separate diagnosis and that patient pref-erences guide the course of treatment together with clinical char-acteristics. In a unified protocol one single protocol is beingfollowed regardless of comorbidities and patient preferences.

For the time being, limited data are accessible for the cost-effectiveness for ICBT (Hedman et al., 2012); this study providessome evidence of cost savings without compromising clinicaleffectiveness. We found that the incremental improvementgenerated by ICBT was associated with a societal cost reduction,which is a promising finding as it suggests that ICBT could poten-tially be administered at no extra cost from a societal perspectivecompared to no treatment. Incremental cost-effectiveness esti-mates in this study were similar to those of previously reportedstudies investigating cost-effectiveness of ICBT for depression(Gerhards et al., 2010; Warmerdam, Smit, van Straten, Riper, &Cuijpers, 2010), social anxiety disorder (Hedman, Andersson,Lj�otsson, Andersson, Rück, M€ortberg, et al., 2011; Hedman,Andersson, Lj�otsson, Andersson, Rück, & Lindefors, 2011), and se-vere health anxiety (Hedman, Andersson, Lj�otsson, Andersson,Rück, M€ortberg, et al., 2011; Hedman, Andersson, Lj�otsson,Andersson, Rück, & Lindefors, 2011). Given that evidence-basedpsychological treatments often are lacking in primary care (Steinet al., 2004), it is in the interest of both the patients and societyto provide as effective treatment as possible, within the limits ofbudget and competences. As results are emerging that guided ICBTcan be as effective as face-to-face delivered CBT (Carlbring et al.,2005; Hedman, Andersson, Lj�otsson, Andersson, Rück, M€ortberg,et al., 2011; Hedman, Andersson, Lj�otsson, Andersson, Rück, &Lindefors, 2011; Kiropoulos et al., 2008), a crucial factor to

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 9

consider for future implementation is the role of moderators andmediators of outcomedin other words, for which patients is ICBTsuitable and how can good outcomes be obtained (Andersson,Carlbring, Berger, Alml€ov, & Cuijpers, 2009)? The possibility isthat moderators may reveal subgroups with different effects oftreatment, even if these relations are complex it might be of help indecision making upon which treatment to offer (Kraemer, Wilson,Fairburn, & Agras, 2002). Mediators on the other hand, whichanswer the question how treatment works, would possibly help usunderstand not only the way in which specific parts of a treatmentwork, but also add to the knowledge of the nature of clinical psy-chological distress (Kraemer et al., 2002). The present study adds tothe literature by investigating whether comorbid depressionmoderates outcomes. We did not find this to be the case, andindeed there are more studies that suggest that comorbid condi-tions may even improve following ICBT (Titov et al., 2009). How-ever, given the small sample size in this study, larger clinical seriesof patients will be needed to investigate moderators of outcomewith adequate power.

This study has limitations. First, we did not compare guidedtailored ICBT against another active treatment or treatment asusual, even if we did not prevent control group participants fromseeking treatment or continuing with medication (which is themost common treatment option in GP settings in Sweden). There isa possibility that adding an extra three or six months to the waitingperiod would have provided reliable information about the effec-tiveness of ICBTcompared to passage of time or usual care. This wasnot done because of ethical considerations of having patientswaiting for treatment. Also, it would have been interesting tocompare tailored ICBT against either disorder-specific treatment(i.e., for one disorder) or against a unified treatment protocol suchas the one by Titov and colleagues (Titov et al., 2010). However, wemade use of an active control group as a control condition, which isprobably more effective than only waiting because at least some ofthe nonspecific factors in therapy (e.g., being monitored and havinga contact person to send messages to) are controlled for. Second,the follow-up measurements suffered from a significant amount ofmissing data. Only 75 of the original 100 completed the onlinemeasurements at the one-year follow-up, forcing us to estimatemeans even though we were missing data. While this dropout rateis not worse than in other studies on ICBT, it would have beenpreferable to have more completers of the measures. Third, we didnot administer the SCID-interview at post treatment or at follow-up, giving us no possibility to answer questions regarding remis-sion or recovery from the initial diagnoses. We rely on self-reportmeasures as describing the outcome, a method commonly usedbut possibly affecting the reliability (e.g. Eaton, Neufeld, Chen, &Cai, 2000). Further the SCID-interviews were not recorded, so nointer-rater reliability of the diagnostic interviews could be calcu-lated. As a fourth limitation, it needs to be stated that this studywasonly partly an effectiveness study if one considers the recommen-dations by Shadish et al. (1997), because we did not used experi-enced, professional therapists with regular caseloads who workedin the settings our patients were referred from (criterion c inShadish et al., 1997). However, because the role of the therapist inguided ICBT is probably less important than in regular CBT (Alml€ovet al., 2011), it is not unlikely that the way we administered ICBT inthis study will be the way it is delivered in future health care (e.g.,with an external therapist providing the guidance). Fifth, using theprimary care contact as gatekeeper for this treatment, in order tomirror usual procedure might have biased the sample in that wemight have included less (or more) impaired patients in the study.However, in order to address this we invited several primary carefacilities to participate, and GPs were allowed to refer patients toother treatments if they wished for this. We provided all

participating settings with thorough information about the treat-ment and the project. All caregivers were also advised to contactthe teamwith any upcoming questions. A sixth limitation might bethat the tailoring of the treatment was based on clinical judgment,and not on empirical evidence. Since there is limited data on howthese decisions are to be done we decided to rely on the clinicalexperience of the psychiatrist and the licensed clinical psychologist.A seventh limitation was that between-group comparisons couldonly be made at post-treatment and not at long-term follow-up.Finally, an eight limitation is the fact that we did not collect anydata on how many participants actually sought other treatmentsduring the trial giving us no possibility to chart or analyze thereasons or timing for this.

In spite of these limitations, we conclude that tailored, guidedICBT appears to work with patients referred from general practiceand that tailoring can be a feasible approach to handle the issuewith comorbidity across anxiety and mood disorders. We alsoconclude that tailored ICBT can be cost-effective and that theadditional costs of providing treatment clearly are compensated forby the societal gains. Future studies should investigate the effects oftailored ICBT in larger clinical samples, in comparison to anotheractive treatment, and the cost-effectiveness over longer follow-upperiods. It might also be interesting to further explore the fre-quency and exact amount of time for the participants use of theirtherapists.

Acknowledgments

We thank the health-care providers in the primary care settingswho were willing to recommend this treatment to their patients.We also like to thank the following students; Sara Edstr€om, ThereseWågstr€om, Ika Wahlgren, Linnea Jarl, Jesper Arvidsson and EllinorBånkestad, for their contribution. This study is funded by Forte (FAS2008-1145), the Swedish Research Council for Health, Working Lifeand Welfare.

Appendix A. Supplementary data

Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.brat.2014.05.007.

References

Alml€ov, J., Carlbring, P., K€allqvist, K., Paxling, B., Cuijpers, P., & Andersson, G. (2011).Therapist effects in guided internet-delivered CBT for anxiety disorders.Behavioural and Cognitive Psychotherapy, 39(03), 311e322. http://dx.doi.org/10.1017/S135246581000069X.

Alonso, J., Angermeyer, M. C., Bernert, S., Bruffaerts, R., Brugha, T. S., Bryson, H., &,EsemeD/Mhedea Investigators, E. S. o. t. E. o. M. D. P. (2004a). 12-Month co-morbidity patterns and associated factors in Europe: results from the EuropeanStudy of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psy-chiatrica Scandinavica. Supplementum, 109(420), 28e37. http://dx.doi.org/10.1111/j.1600-0047.2004.00328.x.

Alonso, J., Angermeyer, M. C., Bernert, S., Bruffaerts, R., Brugha, T. S., Bryson, H., &,EsemeD/Mhedea Investigators, E. S. o. t. E. o. M. D. P. (2004b). Prevalence ofmental disorders in Europe: results from the European Study of the Epidemi-ology of Mental Disorders (ESEMeD) project. Acta Psychiatrica Scandinavica.Supplementum, 109(420), 21e27. http://dx.doi.org/10.1111/j.1600-0047.2004.00327.x.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mentaldisorders (4th ed., text revision ed.). Washington, DC: American PsychiatricPress.

Andersson, G. (2009). Using the Internet to provide cognitive behaviour therapy.Behaviour Research and Therapy, 47(3), 175e180. http://dx.doi.org/10.1016/j.brat.2009.01.010.

Andersson, E., Lj�otsson, B., Smit, F., Paxling, B., Hedman, E., Lindefors, N., et al.(2011). Cost-effectiveness of internet-based cognitive behavior therapy for ir-ritable bowel syndrome: results from a randomized controlled trial. BMC PublicHealth, 11, 215.

Andersson, G., Carlbring, P., Berger, T., Alml€ov, J., & Cuijpers, P. (2009). What makesinternet therapy work? Cognitive Behaviour Therapy, 38(SUPPL.1), 55e60.

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e1110

Andersson, G., Carlbring, P., Holmstrom, A., Sparthan, E., Furmark, T., Nilsson-Ihrfelt, E., et al. (2006). Internet-based self-help with therapist feedback andin vivo group exposure for social phobia: a randomized controlled trial. Journalof Consulting and Clinical Psychology, 74(4), 677e686. http://dx.doi.org/10.1037/0022-006X.74.4.677.

Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010). Computertherapy for the anxiety and depressive disorders is effective, acceptable andpractical health care: a meta-analysis. PLoS ONE, 5(10), e13196. http://dx.doi.org/10.1371/journal.pone.0013196.

Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., et al.(2001). Service profiling and outcomes benchmarking using the CORE-OM:toward practice-based evidence in the psychological therapies. Journal ofConsulting and Clinical Psychology, 69(2), 184e196.

Barlow, D. H., Allen, L. B., & Choate, M. L. (2004). Toward a unified treatment foremotional disorders. Behavior Therapy, 35(2), 205e230.

Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuringclinical anxiety: psychometric properties. Journal of Consulting and ClinicalPsychology, 56(6), 893e897. http://dx.doi.org/10.1037/0022-006X.56.6.893.

Bell, C. J., Colhoun, H. C., Carter, F. A., & Frampton, C. M. (2012). Effectiveness ofcomputerised cognitive behaviour therapy for anxiety disorders in secondarycare. Australian and New Zealand Journal of Psychiatry, 46(7), 630e640. http://dx.doi.org/10.1177/0004867412437345.

Bergstr€om, J., Andersson, G., Lj�otsson, B., Rück, C., Andr�eewitch, S., Karlsson, A., et al.(2010). Internet-versus group-administered cognitive behaviour therapy forpanic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry, 10,54. http://dx.doi.org/10.1186/1471-244X-10-54.

Bravo Vergel, Y., & Sculpher, M. (2008). Quality-adjusted life years. PracticalNeurology, 8(3), 175e182. http://dx.doi.org/10.1136/pn.2007.140186.

Brown, T. A., Campbell, L. A., Lehman, C. L., Grisham, J. R., & Mancill, R. B. (2001).Current and lifetime comorbidity of the DSM-IV anxiety and mood disorders ina large clinical sample. Journal of Abnormal Psychology, 110(4), 585e599. http://dx.doi.org/10.1037/0021-843x.110.4.585.

Carlbring, P., & Andersson, G. (2006). Internet and psychological treatment. Howwell can they be combined? Computers in Human Behavior, 22, 545e553.

Carlbring, P., Bohman, S., Brunt, S., Buhrman, M., Westling, B. E., Ekselius, L., et al.(2006). Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Amer-ican Journal of Psychiatry, 163(12), 2119e2125. http://dx.doi.org/10.1176/appi.ajp.163.12.2119.

Carlbring, P., Brunt, S., Bohman, S., Austin, D., Richards, J., Ost, L. G., et al. (2007).Internet vs. paper and pencil administration of questionnaires commonly usedin panic/agoraphobia research. Computers in Human Behavior, 23(3), 1421e1434.http://dx.doi.org/10.1016/j.chb.2005.05.002.

Carlbring, P., Maurin, L., T€orngren, C., Linna, E., Eriksson, T., Sparthan, E., et al. (2011).Individually-tailored, Internet-based treatment for anxiety disorders: a ran-domized controlled trial. Behaviour Research and Therapy, 49(1), 18e24. http://dx.doi.org/10.1016/j.brat.2010.10.002.

Carlbring, P., Nilsson-Ihrfelt, E., Waara, J., Kollenstam, C., Buhrman, M., Kaldo, V.,et al. (2005). Treatment of panic disorder: live therapy vs. self-help via theInternet. Behaviour Research and Therapy, 43(10), 1321e1333. http://dx.doi.org/10.1016/j.brat.2004.10.002.

Carlbring, P., Nordgren, L. B., Furmark, T., & Andersson, G. (2009). Long-termoutcome of Internet-delivered cognitive-behavioural therapy for social phobia:a 30-month follow-up. Behaviour Research and Therapy, 47(10), 848e850. http://dx.doi.org/10.1016/j.brat.2009.06.012.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).Hillsdale, NJ, USA: Lawrence Erlbaum Associates.

Craske, M. G. (2012). Transdiagnostic treatment for anxiety and depression.Depression and Anxiety, 29(9), 749e753.

Cuijpers, P., Marks, I. M., van Straten, A., Cavanagh, K., Gega, L., & Andersson, G.(2009). Computer-aided psychotherapy for anxiety disorders: a meta-analyticreview. Cognitive Behaviour Therapy, 38(2), 66e82. http://dx.doi.org/10.1080/16506070802694776.

Dear, B. F., Titov, N., Schwencke, G., Andrews, G., Johnston, L., Craske, M. G., et al.(2011). An open trial of a brief transdiagnostic internet treatment for anxietyand depression. Behaviour Research and Therapy, 49(12), 830e837. http://dx.doi.org/10.1016/j.brat.2011.09.007.

Denollet, J., Maas, K., Knottnerus, A., Keyzer, J. J., & Pop, V. J. (2009). Anxiety pre-dicted premature all-cause and cardiovascular death in a 10-year follow-up ofmiddle-aged women. [Research Support, Non-U.S. Gov't]. Journal of ClinicalEpidemiology, 62(4), 452e456.

Drummond, M. F., Sculpher, M. J., Torrence, G. W., O'Brien, B. J., & Stoddart, G. L.(2005). Methods for the evaluation of health care programmes. Oxford: OxfordUniversity Press.

Dolan, P. (1997). Modeling valuations for EuroQol health states.Medical Care, 35(11),1095e1108.

Eaton, W. W., Neufeld, K., Chen, L.-S., & Cai, G. (2000). A comparison of self-reportand clinical diagnostic interviews for depression. Archives of General Psychiatry,57, 217e222.

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York:Chapman & Hall.

Elfstrom, M. L., Evans, C., Lundgren, J., Johansson, B., Hakeberg, M., & Carlsson, S. G.(2012). Validation of the Swedish version of the Clinical Outcomes in RoutineEvaluation Outcome Measure (CORE-OM). Clinical Psychology & Psychotheraphy.http://dx.doi.org/10.1002/cpp.1788.

EuroQol-Group. (1990). EuroQol e a new facility for the measurement of health-related quality of life. Health Policy, 16(3), 199e208.

First, M. B., Gibbon, M., Spitzer, R. L., & Williams, J. B. W. (1997). Structured clinicalinterview for DSM-IV Axis I Disorders (SCID-I). Washington, D.C.: American Psy-chiatric Press.

Frisch, M. B., Cornell, J., Villanueva, M., & Retzlaff, P. J. (1992). Clinical validation ofthe Quality of Life Inventory. A measure of life satisfaction for use in treatmentplanning and outcome assessment. Psychological Assessment, 4(1), 92e101.

Gerhards, S. A., de Graaf, L. E., Jacobs, L. E., Severens, J. L., Huibers, M. J., Arntz, A.,et al. (2010). Economic evaluation of online computerised cognitive-behavioural therapy without support for depression in primary care: rando-mised trial. Brittish Journal of Psychiatry, 196, 310e318.

Gueorguieva, R., & Krystal, J. H. (2004). Move over ANOVA: progress in analyzingrepeated-measures data and its reflection in papers published in the Archives ofGeneral Psychiatry. Archives of General Psychiatry, 61(3), 310e317. http://dx.doi.org/10.1001/archpsyc.61.3.310.

Hakkaart-Van Roijen, L., Van Straten, A., & Donker, M. (2002). Manual:Trimbos/iMTAQuestionnaire for costs associated with psychiatric illness. Rotterdam: ErasmusUniversity.

Hedman, E., Andersson, E., Lj�otsson, B., Andersson, G., Rück, C., & Lindefors, N.(2011). Cost-effectiveness of Internet-based cognitive behavior therapy vs.cognitive behavioral group therapy for social anxiety disorder: results from arandomized controlled trial. Behaviour Research and Therapy, 49(11),729e736.

Hedman, E., Andersson, G., Lj�otsson, B., Andersson, E., Rück, C., M€ortberg, E., et al.(2011). Internet-based cognitive behavior therapy vs. Cognitive behavioralgroup therapy for social anxiety disorder: a randomized controlled non-inferiority trial. PLoS ONE, 6(3), e18001. http://dx.doi.org/10.1371/journal.pone.0018001.

Hedman, E., Furmark, T., Carlbring, P., Lj�otsson, B., Rück, C., Lindefors, N., et al.(2011). A 5-Year follow-up of internet-based cognitive behavior therapy forsocial anxiety disorder. Journal of Medical Internet Research, 13(2), e39. http://dx.doi.org/10.2196/jmir.1776.

Hedman, E., Lj�otsson, B., & Lindefors, N. (2012). Cognitive behavior therapy via theInternet: a systematic review of applications, clinical efficacy and cost-effec-tiveness. Expert Review of Pharmacoeconomics and Outcomes Research, 12(6),745e764.

Hedman, E., Lj�otsson, B., Rück, C., Bergstrom, J., Andersson, G., Kaldo, V., et al.(2013). Effectiveness of Internet-based cognitive behaviour therapy for panicdisorder in routine psychiatric care. Acta Psychiatrica Scandinavica. http://dx.doi.org/10.1111/acps.12079.

Hedman, E., Lj�otsson, B., Rück, C., Furmark, T., Carlbring, P., Lindefors, N., et al.(2010). Internet administration of self-report measures commonly used inresearch on social anxiety disorder: a psychometric evaluation. Computers inHuman Behavior, 26(4), 736e740. http://dx.doi.org/10.1016/j.chb.2010.01.010.

Hoifodt, R. S., Strom, C., Kolstrup, N., Eisemann, M., & Waterloo, K. (2011). Effec-tiveness of cognitive behavioural therapy in primary health care: a review.Family Practice, 28(5), 489e504. http://dx.doi.org/10.1093/fampra/cmr017.

Jacobson, N. S., & Truax, P. (1991). Clinical significance: a statistical approach todefining meaningful change in psychotherapy research. Journal of Consultingand Clinical Psychology, 59(1), 12e19.

Johansson, R., Sjoberg, E., Sjogren, M., Johnsson, E., Carlbring, P., Andersson, T., et al.(2012). Tailored vs. standardized internet-based cognitive behavior therapy fordepression and comorbid symptoms: a randomized controlled trial. PLoS ONE,7(5), e36905.

Kiropoulos, L. A., Klein, B., Austin, D. W., Gilson, K., Pier, C., Mitchell, J., et al. (2008).Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? Journal of Anxiety Disorders, 22(8), 1273e1284.

Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators andmoderators of treatment effects in randomized clinical trials. Archives of GeneralPsychiatry, 59, 877e883.

Lindner, P., Andersson, G., €Ost, L.-G., & Carlbring, P. (2013). Validation of theInternet-administered Quality of Life Inventory (QOLI) in different psychiatricconditions. Cognitive Behaviour Therapy, 42, 315e327.

Mattila-Evenden, M., Svanborg, P., Gustavsson, P., & Åsberg, M. (1996). De-terminants of self-rating and expert rating concordance in psychiatric out-patients, using the affective subscales of the CPRS. Acta Psychiatrica Scandi-navica, 94(6), 386e396.

Mendlowicz, M. V., & Stein, M. B. (2000). Quality of life in individuals with anxietydisorders. American Journal of Psychiatry, 157(5), 669e682.

Mewton, L., Wong, N., & Andrews, G. (2012). The effectiveness of internet cognitivebehavioural therapy for generalized anxiety disorder in clinical practice.Depression and Anxiety. http://dx.doi.org/10.1002/da.21995.

Nordgren, L. B., Andersson, G., Kadowaki, A., & Carlbring, P. (2012). Tailoredinternet-administered treatment of anxiety disorders for primary care patients:study protocol for a randomised controlled trial. Trials, 13, 16. http://dx.doi.org/10.1186/1745-6215-13-16.

Olatunji, B. O., Cisler, J. M., & Tolin, D. F. (2007). Quality of life in the anxiety dis-orders: a meta-analytic review. Clinical Psychology Review, 27(5), 572e581.http://dx.doi.org/10.1016/j.cpr.2007.01.015.

Patterson, W. M., Dohn, H. H., Bird, J., & Patterson, G. A. (1983). Evaluation of suicidalpatients: the SAD PERSONS scale. Psychosomatics, 24(4). http://dx.doi.org/10.1016/S0033-3182(83)73213-5, 343e345, 348e349.

Paxling, B., Almlov, J., Dahlin, M., Carlbring, P., Breitholtz, E., Eriksson, T., et al.(2011). Guided internet-delivered cognitive behavior therapy for generalized

L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 11

anxiety disorder: a randomized controlled trial. Cognitive Behaviour Therapy,40(3), 159e173. http://dx.doi.org/10.1080/16506073.2011.576699.

Rabin, R., & Charro, F. d (2001). EQ-SD: a measure of health status from the EuroQolGroup. Annals of Medicine, 33(5), 337e343. http://dx.doi.org/10.3109/07853890109002087.

Ravens-Sieberer, U., Wille, N., Badia, X., Bonsel, G., Burstr€om, K., Cavrini, G., et al.(2010). Feasibility, reliability, and validity of the EQ-5D-Y: results from amultinational study. Quality of Life Research, 19(6), 887e897. http://dx.doi.org/10.1007/s11136-010-9649-x.

Reger, M. A., & Gahm, G. A. (2009). A meta-analysis of the effects of internet- andcomputer-based cognitive-behavioral treatments for anxiety. Journal of ClinicalPsychology, 65(1), 53e75. http://dx.doi.org/10.1002/jclp.20536.

Rohde, P., Lewinsohn, P. M., & Seeley, J. R. (1997). Comparability of telephone andface-to-face interviews in assessing axis I and II disorders. American Journal ofPsychiatry, 154(11), 1593e1598.

Roy-Byrne, P., Craske, M. G., Sullivan, G., Rose, R. D., Edlund, M. J., Lang, A. J., et al.(2010). Delivery of evidence-based treatment for multiple anxiety disorders inprimary care: a randomized controlled trial. JAMA, 303(19), 1921e1928. http://dx.doi.org/10.1001/jama.2010.608.

Ruwaard, J., Lange, A., Schrieken, B., Dolan, C. V., & Emmelkamp, P. (2012). Theeffectiveness of online cognitive behavioral treatment in routine clinical prac-tice. PLoS ONE, 7(7), e40089. http://dx.doi.org/10.1371/journal.pone.0040089.

Salim, A., Mackinnon, A., Christensen, H., & Griffiths, K. (2008). Comparison of dataanalysis strategies for intent-to-treat analysis in pre-test-post-test designs withsubstantial dropout rates. Psychiatry Research, 160(3), 335e345.

Shadish, W. R., Matt, G. E., Navarro, A. M., & Phillips, G. (2000). The effects ofpsychological therapies under clinically representative conditions: a meta-analysis. Psychological Bulletin, 126(4), 512e529.

Shadish, W. R., Navarro, A. M., Crits-Christoph, P., Jorm, A. F., Nietzel, M. T.,Robinson, L., et al. (1997). Evidence that therapy works in clinically represen-tative conditions. Journal of Consulting and Clinical Psychology, 65(3), 355e365.

Silfvernagel, K., Carlbring, P., Kabo, J., Edstrom, S., Eriksson, J., Manson, L., et al.(2012). Individually tailored internet-based treatment for young adults andadults with panic attacks: randomized controlled trial. Journal of MedicalInternet Research, 14(3), e65.

Smit, F., Cuijpers, P., Oostenbrink, J., Batelaan, N., de Graaf, R., & Beekman, A. (2006).Costs of nine common mental disorders: implications for curative and pre-ventive psychiatry. Journal of Mental Health Policy and Economics, 9(4), 193e200.

Snaith, R. P., Harrop, F. M., Newby, D. A., & Teale, C. (1986). Grade scores of theMontgomery-Asberg depression and the clinical anxiety scales. British Journal ofPsychiatry, 148(MAY), 599e601.

Spek, V., Cuijpers, P., Nyklicek, I., Riper, H., Keyzer, J., & Pop, V. (2007). Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: ameta-analysis. Psychological Medicine, 37(3), 319e328. http://dx.doi.org/10.1017/S0033291706008944.

Stein, M. B., Sherbourne, C. D., Craske, M. G., Means-Christensen, A., Bystritsky, A.,Katon, W., et al. (2004). Quality of care for primary care patients with anxietydisorders. American Journal of Psychiatry, 161(12), 2230e2237. http://dx.doi.org/10.1176/appi.ajp.161.12.2230.

Svanborg, P., & Åsberg, M. (1994). A new self-rating scale for depression and anxietystates based on the Comprehensive Psychopathological Rating Scale. Acta Psy-chiatrica Scandinavica, 89(1), 21e28.

Titov, N., Andrews, G., Johnston, L., Robinson, E., & Spence, J. (2010). TransdiagnosticInternet treatment for anxiety disorders: a randomized controlled trial.Behaviour Research and Therapy, 48(9), 890e899.

Titov, N., Dear, B. F., Schwencke, G., Andrews, G., Johnston, L., Craske, M. G.,et al. (2011). Transdiagnostic internet treatment for anxiety and depres-sion: a randomised controlled trial. Behaviour Research and Therapy, 49(8),441e452.

Titov, N., Gibson, M., Andrews, G., & McEvoy, P. (2009). Internet treatment for socialphobia reduces comorbidity. Australian and New Zealand Journal of Psychiatry,43(8), 754e759. http://dx.doi.org/10.1080/00048670903001992.

Vernmark, K., Lenndin, J., Bjarehed, J., Carlsson, M., Karlsson, J., Oberg, J., et al. (2010).Internet administered guided self-help versus individualized e-mail therapy: arandomized trial of two versions of CBT for major depression. Behaviour Researchand Therapy, 48(5), 368e376. http://dx.doi.org/10.1016/j.brat.2010.01.005.

Warmerdam, L., Smit, F., van Straten, A., Riper, H., & Cuijpers, P. (2010). Cost-utilityand cost-effectiveness of internet-based treatment for adults with depressivesymptoms: randomized trial. Journal of Medical and Internet Research, 12, e53.

Weisberg, R. B., Dyck, I., Culpepper, L., & Keller, M. B. (2007). Psychiatric treatmentin primary care patients with anxiety disorders: a comparison of care receivedfrom primary care providers and psychiatrists. American Journal of Psychiatry,164(2), 276e282. http://dx.doi.org/10.1176/appi.ajp.164.2.276.

Wittchen, H. U., & Jacobi, F. (2005). Size and burden of mental disorders inEuropeea critical review and appraisal of 27 studies. European Neuro-psychopharmacology, 15(4), 357e376. http://dx.doi.org/10.1016/j.euroneuro.2005.04.012.

Wittchen, H. U., Jacobi, F., Rehm, J., Gustavsson, A., Svensson, M., Jonsson, B., et al.(2011). The size and burden of mental disorders and other disorders of the brainin Europe 2010. European Neuropsychopharmacology, 21(9), 655e679. http://dx.doi.org/10.1016/j.euroneuro.2011.07.018.