Informing best practice in mental health: Using feedback ... · Informing best practice in mental health: Using feedback to improve clinical outcomes. Elizabeth A Newnham, BSc (Hons)

Informing best practice in mental health: Using feedback to improve clinical outcomes.

Elizabeth A Newnham, BSc (Hons)

School of Psychology

The University of Western Australia

Year of submission: 2009

This thesis is presented for the degree of Doctor of Philosophy at

The University of Western Australia

2

Table of Contents

Abstract...........................................................................................................................................4

Publications Arising from this Thesis............................................................................................6

Author Contributions......................................................................................................................7

Acknowledgments..........................................................................................................................8

Section One: General Introduction

General Introduction..................................................................................................................10

Setting the scene..............................................................................................................10

Thesis aims and outline...................................................................................................11

Chapter One: Bridging the gap between best evidence and best practice in mental health.......16

Introduction.....................................................................................................................17

What treatment by whom.................................................................................... ............17

Is most effective...............................................................................................................20

For this individual...........................................................................................................22

With that specific problem...............................................................................................26

Implementing changes in practice...................................................................................27

New directions for the scientist practitioner...................................................................28

Section Two: The clinical value of routine outcome measures in psychiatric care

Foreword.......................................................................................................................................33

Chapter Two: Evaluating the clinical significance of responses by psychiatric inpatients

to the mental health subscales of the SF-36....................................................................38

Foreword.......................................................................................................................................45

Chapter Three: The subscale structure and clinical utility of the Health of the Nation

Outcome Scale.................................................................................................................53

3

Section Three: The effectiveness of progress monitoring and feedback in psychiatric

care

Foreword.......................................................................................................................................63

Chapter Four: Client-focused research: New directions in outcome assessment......................68

Chapter Five: Monitoring treatment response and outcomes using the World Health

Organization’s Wellbeing Index in psychiatric care.......................................................74

Chapter Six: Patient monitoring and feedback in psychiatric care reduces depressive

symptoms.........................................................................................................................80

Section Four: General Discussion

General Discussion...................................................................................................................104

Findings.........................................................................................................................104

Can we identify poor responders in psychiatric care?.....................................104

Is it possible to track patients’ progress during psychiatric care?..................107

What effect does feedback have on clinical outcomes?....................................112

Future research.............................................................................................................114

How does feedback work?................................................................................114

How can information technology best be used to benefit mental healthcare?.117

General conclusions......................................................................................................120

4

Abstract

Physical healthcare uses a suite of tools for measuring response to treatment. However,

reliable systems of regular patient monitoring are rare in mental healthcare. Mental health

services often measure a treatment response from pre- to post- therapy, yet measurement

between those occasions is less common. This omission is problematic since arguably there is a

need for an alarm system in psychotherapy (Andrews & Page, 2005). A substantial minority of

patients do not experience reliable change following treatment, and a small proportion

deteriorates (Hansen, Lambert, & Forman, 2002; Newnham, Harwood, & Page, 2007). Without

monitoring, it is not always possible to know which patients are progressing poorly. Since the

publication of Howard and colleagues’ (1996) proposal that patient progress be monitored

routinely during therapy and the results fed back to clinicians to direct treatment, this

monitoring regime has garnered attention in the United States and Europe (Lambert, 2007; Lutz,

et al., 2006). Findings in outpatient psychotherapy have demonstrated that providing real-time

feedback on patient progress to clinicians and patients significantly improves clinical outcomes

for those patients demonstrating a negative response to treatment (Harmon et al., 2007; Lambert

et al., 2001; Lambert et al., 2002). What is not yet apparent is how these processes would

generalize to inpatient and day patient (i.e. patients attending hospital for a whole day of

treatment) psychiatric care. Inpatients often present with greater severity and are treated in an

intensive setting. Therefore they will experience therapy differently to outpatients, may have

many treatments simultaneously and be managed by a team of professionals (usually under the

leadership of a medical practitioner). The thesis aims to contribute to the field of best practice

by describing the development and evaluation of a program for monitoring patient progress in a

hospital-based psychiatric setting. It was hypothesized that presenting feedback during therapy

to clinicians and patients would significantly improve wellbeing and significantly reduce

symptom distress for those patients at risk of poor outcome. Deviations from this expected

pattern would highlight possible differences between inpatient and outpatient care.

5

To develop an appropriate system for monitoring patient progress, it was important to

first define clinically significant recovery in inpatient psychiatric care, and provide criteria for

clinicians to judge outcome in routine practice (Newnham, Harwood, & Page, 2007). Second, a

quick and easy-to-administer system of progress monitoring and real-time feedback was

developed to enhance treatment decision making (Newnham, Hooke, & Page, 2009). Third, the

system was evaluated to determine clinical effectiveness. Using the World Health

Organization’s Wellbeing Index, a program for monitoring patient progress and providing

feedback to clinicians and patients was established at Western Australia’s largest private

psychiatric service. The sample consisted of 1308 consecutive inpatients and day patients whose

primary diagnoses were predominantly depressive (67.7%) and anxiety (25.9%) disorders.

Feedback to patients and clinicians was effective in reducing depressive symptoms (F (1,649) =

6.29, p<.05) for those patients at risk of poor outcome, but not effective in improving wellbeing

(F (1,569) = 1.14, p>.05). The findings support the use of progress monitoring and feedback in

psychiatric care to improve symptom outcomes, but raise questions about changes in wellbeing

during psychotherapy. The effectiveness study was conducted as a historical cohort trial,

consistent with quality improvement efforts, and replication with a randomized controlled

design is warranted. Feedback of progress information appears to be an important process

within psychotherapy, and further investigation of the means by which clinicians and patients

use that information is necessary.

6

Publications Arising from this Thesis

Chapter 1

Newnham, E. A. & Page, A. C. (In press). Bridging the gap between best evidence and best practice in mental health. Clinical Psychology Review. doi:10.1016/j.cpr.2009.10.004

Chapter 2

Newnham, E. A., Harwood, K. E. & Page, A. C. (2007). Evaluating the clinical significance of responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal of Affective Disorders, 98, 91-97.

Chapter 3

Newnham, E. A., Harwood, K. E. & Page, A. C. (2009). The subscale structure and clinical utility of the Health of the Nation Outcome Scales. Journal of Mental Health, 18, 326-334.

Chapter 4

Newnham, E. A., & Page, A. C. (2007). [Invited essay] Client-focused research: New directions in outcome assessment. Behaviour Change, 24, 1, 1-6.

Chapter 5

Newnham, E. A., Hooke, G. R. & Page, A. C. (In press). Monitoring treatment response and outcomes using the World Health Organization’s Wellbeing Index in psychiatric care. Journal of Affective Disorders. doi:10.1016/j.jad.2009.1006.1005

Chapter 6

Newnham, E. A., Hooke, G. R. & Page, A. C. (under review). Patient monitoring and feedback in psychiatric care reduces depressive symptoms.

7

Author Contributions

Chapter 1: Elizabeth Newnham (90%) and Andrew Page (10%) developed the design of the

manuscript; Elizabeth Newnham wrote the first draft; and both authors contributed to and approved of the final manuscript.

Chapter 2: Elizabeth Newnham (75%) and Andrew Page (20%) designed the study; Elizabeth Newnham conducted the statistical analyses and wrote the first draft; Elizabeth Newnham, Kate Harwood (5%) and Andrew Page all contributed to and approved of the final manuscript.

Chapter 3: Elizabeth Newnham (70%) and Andrew Page (25%) designed the study and conducted the statistical analyses; Elizabeth Newnham wrote the first draft; Elizabeth Newnham, Kate Harwood (5%) and Andrew Page all contributed to and approved of the final manuscript.

Chapter 4: Elizabeth Newnham (80%) and Andrew Page (20%) contributed to the manuscript design; Elizabeth Newnham wrote the first draft; and both authors contributed to and approved of the final manuscript.

Chapter 5: All authors contributed to the study design. Geoff Hooke (10%) managed the data collection. Elizabeth Newnham (75%) and Andrew Page (15%) undertook the statistical analyses and Elizabeth Newnham wrote the first draft of the manuscript. All authors contributed to and have approved of the final manuscript.

Chapter 6: All authors contributed to the study design and data collection. Elizabeth Newnham (70%) and Andrew Page (20%) conducted the statistical analyses and Elizabeth Newnham wrote the first draft of the manuscript. Elizabeth Newnham, Geoff Hooke (10%) and Andrew Page contributed to and have approved of the final manuscript.

8

Acknowledgments_______________________________________________________

I feel very fortunate to have had the opportunity to write a PhD thesis, and more so, to love the research field even more now than when I started. I have many people to thank for this.

First, my deepest gratitude to Professor Andrew Page for his wisdom, guidance, and humour. I would never have embarked upon a PhD without his encouragement, and his support has been constant. Andrew made the process of writing a PhD exciting and interesting, and inspired me to commence a scientific career. I look forward to many more years of collaborative work together.

Thank you to Perth Clinic and the Clinical Improvement Team. I very much appreciate the generous support of Moira Munro and the clinical staff, particularly the CBT therapists. I am indebted to Greg Beard and Paul Hussein, and the administration staff whose assistance helped the project run so smoothly. I am very proud to have worked with such an impressive and effective team.

A special thank you to Geoff Hooke, who helped to implement every research plan, provided inspired discussion, and noticed every new hairstyle.

In particular, I would like to thank the many patients who have contributed to this research.

This research was supported by a number of grants. I would like to acknowledge the Medicare Private Safety and Clinical Improvement Incentive Pool. The research was also supported by a University Postgraduate Award; UWA Completion Scholarship; Geoffrey Kennedy Postgraduate Research Travel Award; a UWA Grant for Research Student Training; and travel grants from the Graduate Research School and School of Psychology at UWA.

I wish to thank Professor Mike Anderson who read a draft of this thesis and whose comments improved it.

I was privileged to share an office with three very bright and entertaining PhD students who have become my close friends. Thank you Pia, Emma and Karina.

To my wonderful girlfriends for their constant encouragement and entertaining distractions. A special thank you to Alex, Heather, Liz, Sarah and Madeleine.

I am deeply grateful to my partner Willis Samson, for his unwavering support and love. Thank you for listening to every idea, for your wise advice, for travelling the world to support me, and for making me laugh. You have been my pillar of strength.

Thank you to my family. To my sister Prudence, for her friendship and humour. To my brother Richard, for the technical support, political debates, and strength when I needed it. To my father, John, who is the most wonderful mentor; and my mother, Susie, my closest confidante; who supported and celebrated every achievement. Thank you. My love of science is the product of being raised in such a close family, for whom scientific debate is regular dinner table discussion.

9

Section One General Introduction

10

General Introduction__________________________________________

Monitoring patient response is integral to modern medical care. Most areas of

healthcare are aided by the use of an instrument, such as the thermometer, that provides clear

and reliable information on the patient’s health status. A thermometer alerts clinicians to

changes in a patients’ health, and thus complements clinical judgment of progress and recovery.

As yet, there is no such instrument for gauging health status in psychiatric care. Yet the

rationale is clear. A mental health thermometer would complement current clinical practice by

assessing patient response to treatment and alerting clinicians to unexpected changes.

Accordingly, clinicians would be empowered to act upon subtle (or substantial) deviations in

the patient’s progress during their treatment.

Development of a mental health thermometer would involve three steps (Stritzke &

Page, 2009). The first is the definition of recovery: a cut-off point that would indicate positive

response or conversely, poor response to treatment. The second involves the generation of

trajectories that illustrate the patient’s expected response to treatment. Deviation from this

pattern of improvement would thus highlight poor response and the need for intervention. The

third step is the implementation of a feedback system that alerts clinicians to poor response,

providing an opportunity for change with minimal interference to the therapeutic process

(Stritzke & Page, 2009).

The current thesis comprises a series of studies that outline the process described above.

To develop and implement a mental health thermometer in clinical practice, recovery was

defined, expected response trajectories were created, and a feedback system implemented; all

within a hospital-based psychiatric service. In doing so, the research program aimed to extend

findings in patient-focused research and contribute to bridging the gap between science and

clinical practice.

Setting the Scene

The widening gap between science and practice is particularly problematic because of

the sheer scale of the level of mental health need in Australia. One in five adults (20%) report

11

experiencing a mental health problem in the previous twelve-month period (Australian Bureau

of Statistics, 1998; Slade et al., 2009). Depressive disorders account for the greatest level of

burden of disability for both males and females in Australia (Mathers, Vos, & Stevenson, 1999),

and are a leading cause of disability burden on a global scale. Great costs, both personal and

financial (Kessler & Frank, 1997), are associated with depressive disorders. Effective evidence-

based treatments for depression are widely available, but not all patients improve during therapy

(Newnham, Harwood, & Page, 2007), and relapse rates are high (Daniels, Kirkby, Hay, Mowry,

& Jones, 1998). A cost-effective instrument that highlights poor response during therapy thus

has the potential to improve clinical practice and consequently, treatment outcomes.

Thesis aims and outline

The primary aim of the present thesis is to develop and assess the effectiveness of a

system for monitoring patient progress and providing feedback within a psychiatric inpatient

and day patient population. Second, the thesis looks to explore new directions in outcome

assessment in the field of mental health, and contribute to the area by critically assessing the

psychometric and clinical validity of routinely used outcome measures, and defining criteria for

recovery in psychological health care.

The thesis is comprised of four parts that address the primary concern of improving the

effectiveness of treatment in psychiatric services. Section One outlines the state of evidence in

mental health outcome assessment. Chapter 1 consists of a review of the literature in

psychotherapy outcomes research and provides a context for patient-focused methodologies,

with an aim to bridge the gap between science and practice. Section Two addresses the use of

outcome assessment measures in psychiatric care, and specifically their clinical value in routine

practice. The specific aims are to determine:

1. That recovery can be defined using clinical significance methodologies with outcome

measures commonly used in Australia, such as the SF-36, for the benefit of practising

clinicians.

2. A factor structure that is both psychometrically sound and clinically meaningful for the

Health of the Nation Outcome Scale.

12

The section comprises two published manuscripts that investigate the psychometric

properties and clinical function of two measures, the Medical Outcomes Short Form (SF-36)

and the Health of the Nation Outcome Scales (HoNOS). Both have been recommended for use

as outcome assessment measures in routine clinical practice in Australia by the Consumer

Outcomes Project Advisory Group advising the Commonwealth Government (Stedman,

Yellowlees, Mellsop, Clarke, & Drake, 1997). However, despite this recommendation,

assessment of their psychometric properties and clinical validity in psychological settings in

Australia has been minimal. The papers comprising Section One (Newnham et al., 2007;

Newnham, Harwood, & Page, 2009) attempt to expand upon the Advisory Group’s guidance by

providing clinically meaningful information on recovery criteria and appropriate scoring

methods for the measures, so that they may be more readily used in routine practice.

The third section of the thesis comprises three papers, of which two are published and one

has been submitted for publication. These papers outline the development and evaluation of a

system for monitoring patient wellbeing and providing feedback to clinicians and patients

within an inpatient and day patient setting. Specifically, the following hypotheses are

investigated:

1. The World Health Organization Wellbeing Index (WHO-5) is an appropriate and

clinically useful monitoring instrument for use with a psychiatric sample, in that is it

reliable, sensitive to change in time-limited treatment, and early scores predict final

outcome.

2. Monitoring patient progress using the WHO-5, and providing individualized feedback

to clinicians and patients during group therapy will improve (i) wellbeing and (ii)

symptom relief as measured on a series of standardized self report and clinician rated

outcome assessment instruments.

Chapter 4 (Newnham & Page, 2007) reviews the literature in the area of patient-focused

research and highlights the challenges remaining. Chapter 5 (Newnham, Hooke, & Page, 2009)

extends previous work conducted with the WHO Wellbeing Index with an investigation of the

psychometric properties of the WHO-5 within a psychiatric setting, finding it to be a reliable

and valid measure, appropriate for use as a monitoring and outcome assessment tool in mental

13

health services. The final study (Newnham, Hooke, & Page, submitted) reports a non-

randomized effectiveness trial that demonstrated that providing feedback on patient progress in

psychiatric care was effective in reducing depressive symptoms, but did not improve wellbeing

for those at risk of poor outcome. This trial addresses two issues that have not been previously

examined in the mental health literature: the implementation a freely available monitoring

system within a psychiatric hospital setting, and the use of feedback within a group therapy

format. The impact of both features is outlined in Chapter 1.

The Discussion (Section Four) contextualizes the findings of the current thesis within

the field of patient-focused research and clinical outcomes assessment in mental health.

Changes in wellbeing during psychotherapy are examined, and new directions for the

assessment of feedback processes are outlined. The thesis therefore contributes to a growing

field of evidence-based practice by providing an innovative and sustainable means by which

clinicians can assess and improve their practice.

14

References

Australian Bureau of Statistics. (1998). Mental Health and Wellbeing: Profile of Adults,

Australia, 1997 Canberra: ABS Cat No. 4326.0.

Daniels, B. A., Kirkby, K. C., Hay, D. A., Mowry, B. J., & Jones, I. H. (1998). Predictability of

rehospitalisation over 5 years for schizophrenia, bipolar disorder and depression.

Australian and New Zealand Journal of Psychiatry, 32, 281-286.

Kessler, R. C., & Frank, R. G. (1997). The impact of psychiatric disorders on work loss days.

Psychological Medicine, 27, 861-873.

Mathers, C., Vos, T., & Stevenson, C. (1999). The burden of disease and injury in Australia.

Canberra: Australian Institute of Health and Welfare.

Newnham, E. A., Harwood, K. E., & Page, A. C. (2007). Evaluating the clinical significance of

responses by psychiatric inpatients to the mental health subscales of the SF-36. Journal

of Affective Disorders, 98, 91-97.

Newnham, E. A., Harwood, K. E., & Page, A. C. (2009). The subscale structure and clinical

utility of the Health of the Nation Outcome Scale. Journal of Mental Health, 18, 326-

334.

Newnham, E. A., Hooke, G. R., & Page, A. C. (2009). Monitoring treatment response and

outcomes using the World Health Organization's Wellbeing Index in psychiatric care.

Journal of Affective Disorders, doi:10.1016/j.jad.2009.1006.1005.

Newnham, E. A., Hooke, G. R., & Page, A. C. (submitted). Patient monitoring and feedback in

psychiatric care reduces depressive symptoms.

Newnham, E. A., & Page, A. C. (2007). Client-focused research: New directions in outcome

assessment. Behaviour Change, 24(1), 1-6.

Slade, T., Johnston, A., Teesson, M., Whiteford, H., Burgess, P., Pirkis, J., et al. (2009). The

Mental Health of Australians 2. Report on the 2007 National Survey of Mental Health

and Wellbeing. Canberra: Department of Health and Ageing.

15

Stedman, T., Yellowlees, P., Mellsop, G., Clarke, R., & Drake, S. (1997). Measuring Consumer

Outcomes in Mental Health. Canberra, ACT: Department of Health and Family

Services.

Stritzke, W. G. K., & Page, A. C. (2009). Electronic patient monitoring in mental health

services. In A. Dwivedi (Ed.), Handbook of Research on Information Technology

Management and Clinical Data Administration in Healthcare (pp. 87-103). Hershey,

New York: IGI Publishing.

16

Chapter One_________________________________________________

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

Section Two The clinical value of routine outcome measures in psychiatric care.

33

Foreword

Clinical psychology is now building upon the processes of data collection and

assessment, to improve the utility of outcome evaluation programs and their interpretation.

Thus, outcomes information must be presented in a way that enables its routine use by

practising clinicians (Clancy & Eisenberg, 1998). With automated or at least easily calculated

interpretation methods available, clinicians will be able to integrate outcomes information into

clinical work without the usual challenges to time, resources and the therapeutic relationship

(Hatfield & Ogles, 2004). When such methods are available to clinicians, outcome assessment

takes on a much greater significance not only for the assessment of treatments or services, but

also for individual patients.

The measurement of recovery is not new. Reports dating from the 1920s outline

attempts by clinicians to classify the outcomes of their patients as ‘cured’, ‘improved’, ‘much

improved’ and ‘uncured’. The Berlin Psychoanalytic Institute took care to collect clinician-rated

data on the proportion of patients in each category (Bergin & Lambert, 1978), thus providing

some indication of therapeutic effectiveness. Despite the methodological flaws (Ogles, Lunnen,

& Bonesteel, 2001), the early data signals an attempt to address a dilemma that has troubled the

field since its origins – the definition and measurement of psychological recovery. More

recently, a rise in the use of statistical methodologies to capture “significant” change, has

included the employment of an alpha criterion to determine reliable improvement, or effect size

calculations to address the magnitude of change. However, the question remained – has the

client experienced a meaningful improvement as a result of the treatment?

Thus the status quo in outcomes assessment is not sufficient. Providing an averaged

group score (or even an individual’s assessment or outcome score) is not adequate information

for meaningful interpretation of progress or useful for treatment planning. It is not sufficient

because average scores can mask wide individual differences. The average may also not be the

best measure of central tendency for a particular individual or sub-group. Fortunately, given the

progress made in the large-scale collection of outcomes data, we now have an exciting

34

opportunity to provide clinicians (and also patients) with more meaningful information about a

particular patient’s treatment progress.

The development of clinical significance methodologies (Jacobson & Truax, 1991)

enables the categorization of outcome according to the individual patient’s pathway of recovery.

Utilizing scores at two points in time (for inpatients this is often admission and discharge) we

can determine whether a patient has made change large enough to be statistically reliable, and

whether that change has involved a clinically meaningful reduction in symptoms. Alternatively,

it enables knowledge of whether a patient’s functioning has deteriorated over the time period.

Recent advances have been made in calculating the criteria for recovery on widely and routinely

used measurement tools such as the Medical Outcomes Questionnaire Short Form (SF-36;

Ware, Snow, Kosinski, & Gandek, 1993) and the Health of the Nation Outcome Scale (HoNOS;

Wing et al., 1998) which will provide clinicians with a much richer level of interpretation of

patients’ outcomes (see Newnham, Harwood, & Page, 2007; Parabiaghi, Barbato, D'Avanzo,

Erlicher, & Lora, 2005).

The provision of individualized outcome data empowers patients and positions them as

co-managers of their own healthcare (Calman, 1998). It enables a dialogue between clinician

and patient about treatment progress and expectations, such as whether the patient has made a

meaningful recovery and should consider termination of therapy, or an objective indication that

expected progress has not been achieved. Whilst increasing accountability for service providers,

the open discussion of outcomes data may also increase responsibility on the patient’s part so

that they may reflect upon their own treatment progress and put necessary changes into action.

Only with the appropriate and psychometrically valid assessment of recovery, will people be

able to make informed choices about their mental healthcare.

The mental health subscales of the SF-36 are widely and routinely used at a national

and international level, and their use as a measure of outcome is mandatory in the psychiatry

private sector in Australia. It was anticipated that the following study (Newnham et al., 2007)

would provide clinicians with information specific to their individual patient’s recovery. Further

to this, assessment of a patient group’s clinically significant recovery provides valuable

information on treatment effectiveness, which is potentially useful for service providers, health-

35

care managers, and administrative and legislative policy makers. Criteria outlined for the mental

health subscales of the SF-36 provide clinicians with a simple formula for determining a

patient’s level of improvement. By entering a patient’s pre- and post-treatment scores and

determining whether they have passed the level of statistically significant and meaningful

change, clinicians are positioned to assess the progress of their caseload and to provide valid

information on their own effectiveness.

An important factor in encouraging the use of outcome assessment by clinicians and

service providers is the ease of application. So in practical terms, how would the clinician assess

and interpret their patient’s change over the course of treatment? The first step is to subtract the

patient’s admission score from their discharge score, and divide the answer by the subscale’s

Standard Error of Difference (as outlined in Newnham et al., 2007). If the answer is greater than

1.96, then the patient has achieved statistically reliable change during the therapy period.

Additionally, if the patient’s discharge score is greater than or equal to the subscale’s cut-off

point (also illustrated in Newnham et al., 2007); then the patient has made clinically significant

change. With this information, clinicians have the opportunity to make decisions about the

treatment plan, their patient’s progress, or their own effectiveness with ease.

Chapter 2 also illustrates the rate of recovery for inpatients receiving psychiatric

treatment in Australia. While the majority of patients made a reliable improvement, a notable

proportion experienced a negative outcome as measured on the SF-36 mental health subscales.

It is important to identify patients who respond poorly to treatment so that action can be taken to

address this issue. Findings in outpatient therapy suggest that a significant proportion of patients

do not achieve reliable improvement in treatment (Hansen, Lambert, & Forman, 2002), however

a corresponding assessment of recovery rates has not yet been conducted with inpatient

samples. Chapter 2 illustrated rates of no change in therapy that ranged from 43.6% to 56.6%;

and that approximately 2% of inpatients deteriorated during treatment (Newnham et al., 2007).

This finding indicates that a significant proportion of patients are not experiencing the

improvement expected from treatment, and that a need exists for an intervention that addresses

poor response prior to the termination of therapy (in this case, discharge from hospital). Thus

36

Chapter 2 marks the first step in identifying poor responders in psychiatric care, so that their

progress may be addressed during treatment.

37

References

Bergin, A. E., & Lambert, M. J. (1978). The evaluation of therapeutic outcomes. In S. L.

Garfield & A. E. Bergin (Eds.), The handbook of psychotherapy and behavior change

(2nd ed.). New York: John Wiley.

Calman, K. C. (1998). Potential for health. Oxford: Oxford University Press.

Clancy, C. M., & Eisenberg, J. M. (1998). Outcomes research: Measuring the end results of

health care. Science, 282(5387), 245-246.

Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response

effect and its implications for treatment delivery services. Clinical Psychology: Science

and Practice, 9, 329-343.

Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in

clinical practice. Professional Psychology: Research and Practice, 35(5), 485-491.

Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining

meaningful change in psychotherapy research. Journal of Consulting and Clinical

Psychology, 59, 12-19.




Ogles, B. M., Lunnen, K. M., & Bonesteel, K. (2001). Clinical significance: History, application

and current practice. Clinical Psychology Review, 21, 421-446.

Parabiaghi, A., Barbato, A., D'Avanzo, B., Erlicher, A., & Lora, A. (2005). Assessing reliable

and clinically significant change on HoNOS: A method for displaying longitudinal data.

Australian and New Zealand Journal of Psychiatry, 39(719-724).

Ware, J. E., Snow, K. K., Kosinski, M., & Gandek, B. (1993). SF-36 Health Survey: Manual

and Interpretation Guide. Boston: The Health Institute, New England Medical Centre.

Wing, J. K., Beevor, A. S., Curtis, R. H., Park, S. B. G., Hadden, S., & Burns, A. (1998). Health

of the Nation Outcome Scales (HoNOS): Research and development. British Journal of

Psychiatry, 172, 11-18.

38

Chapter Two_________________________________________________

39

40

41

42

43

44

45

Foreword

The expansion of health outcomes research and implementation of evaluation programs

in service settings is changing the face of psychological care. The use of routine outcome

assessment in practice benefits clinical, ethical and scientific domains (Slade, 2002; Stedman,

Yellowlees, Mellsop, Clarke, & Drake, 1997). Clinically, evaluating patient progress has the

potential to inform practice and treatment planning; whilst the assessment of outcomes ensures

the highest quality of care – an ethical responsibility for all clinicians. The evaluation of

treatment has traditionally been conducted with efficacy and effectiveness research. Efficacy

research provides an indication of therapeutic benefit for the average patient under ideal

conditions in randomized controlled trials. If the treatment is shown to be efficacious, its

effectiveness within a clinical setting with less controlled variables may then be investigated.

Whilst efficacy research maximises internal validity at the cost of external validity,

effectiveness research strengthens a treatment’s potential to generalise to other clinical settings.

Although efficacy and effectiveness studies complete one side of the scientist-

practitioner equation, the ongoing evaluation of ordinary practice is an important

complementary process. Now that the age of outcome assessment is maturing, we have the

opportunity to use that information not only for the assessment of practice, but also to improve

the services we provide.

However, despite advances, there are a range of difficulties that hinder the assessment

of outcomes in mental health. First, defining a positive outcome may be problematic. It’s not

always the case that the clinician and patient agree upon a desired outcome for therapy (Slade,

2002), despite a common goal of optimal functioning for the individual. For example, a

clinician may encourage a patient to reduce their alcohol intake, which may not align with the

patient’s treatment goals. Second, different types of outcome can be desynchronous and

different indices may change at different rates during intervention. The phase model (Howard,

Lueger, Maling, & Martinovich, 1993) proposes that the remediation of hope occurs first,

followed by a reduction in symptoms, and then a gradual rehabilitation and improvement in

quality of life. Changes in patients’ wellbeing, symptomatic distress and life functioning at

46

different sessions across the duration of therapy have been shown to be consistent with the

three-phase model (Howard et al., 1993). To complicate things further, for many patients, some

things get better while others may get worse (Trauer, 1998). For example, medication may

improve mood, but at the risk of unpleasant side effects. Capturing this complex progression of

wellbeing is not possible in a single measure, and requires significant consideration when

deciding upon the timing of assessments. Multiple assessments or organizing administration at

clinically meaningful points during therapy are essential for the appropriate evaluation of

outcomes.

Third, there are a number of reasons why a patient may not improve, not all of which

equate to a negative outcome. Occasionally, as may be the case in the treatment of

schizophrenia, the aim of therapy may be to maintain a level of functioning, or to slow a decline

in functioning, and thus the expected outcome will not be recovery. Additionally, the purpose of

hospitalization may be to monitor a patient’s wellbeing whilst changing medication, to keep

them safe from suicide attempts, to remove them from a dangerous environment or to reduce

risk to others. In each of these cases, an improvement in symptom scores may not reflect all

facets of an outcome.

These issues highlight the significance of collecting outcome data from a variety of

sources. A clinician’s perspective of the patient’s progress will differ from the patient’s

perspective and vice versa. It would be insufficient to measure only one viewpoint and assume

an objective assessment of outcome. Therefore, to complement the collection of patient self-

report data, the use of clinician-rated measures has been widely encouraged (Stedman et al.,

1997).

Enthusiasm about evidence-based practice and accountability in health care has driven a

vast improvement in outcome assessment methods and measures. In Australia, the first national

mental health strategy involved a systematic review of outcomes data collection policies and

practices (Andrews, Peters, & Teesson, 1994), the result of which was a set of recommendations

for specific measures to be used in routine care. This review was followed by an independent

field-assessment of the recommended measures and procedures (Stedman et al., 1997). Despite

shifting the focus towards developing optimal methods of outcome evaluation; the consultancy

47

provided specific suggestions for the use of measures when assessing treatment outcomes in

mental health. Of six measures proposed, two were endorsed for routine use in clinical practice:

the mental health subscales of the SF-36 and the Health of the Nation Outcome Scale (Stedman

et al., 1997). Where service providers had an outcomes collection system already established,

continuance was encouraged. However, for those not already collecting outcomes data, the

implementation of a program using the SF-36 subscales and the Health of the Nation Outcome

Scale (HoNOS) in combination was advised. Accordingly, both patient and clinician

perspectives would be captured and the data available could contribute to outcome evaluation,

as well as treatment planning and quality improvement efforts. Yet despite enthusiasm for using

the HoNOS to complement patient health status data, the field report asserted that at the time,

the HoNOS was a new measure, and further assessment of its psychometric properties and

clinical utility were required (Stedman et al., 1997).

For an outcome measure to provide meaningful and valid data, Slade (2002) has

proposed a number of criteria that must be met: (1) the measure assesses a desired outcome, a

process issue, problem severity or enables comparison between staff and patient views; (2) it is

specifically designed for a mental health population; (3) it is freely available; (4) brief to

administer; and (5) the measure has peer-reviewed published evidence of sound psychometric

properties.

The HoNOS assesses severity and breadth of presenting problems, as well as

functioning and environmental factors that may influence wellbeing; and when used in

combination with self-report measures, allows for comparison between staff and patient views.

This provides an important level of information not only about the patient’s difficulties, but also

their insight into those difficulties. Created by the Royal College of Psychiatrists for the purpose

of assessing national health status in the United Kingdom (College Research Unit, 1996), the

HoNOS meets criteria for specific mental health assessment. It is freely available, quick to

administer (less than 15 minutes) and to increase reliability of the measure, training is required

for its use. Since its development, the psychometric properties of the HoNOS have been an issue

of great debate. In accordance with the authors’ intentions (Wing et al., 1998), the items are

heterogenous and cover a wide range of symptoms and behaviours, resulting in low internal

48

consistency (Preston, 2000). However, despite this, evidence suggests that the HoNOS is a valid

and useful measure in clinical practice (Hansen & Kingdon, 2006; Kisely, Campbell, Crossman,

Gleich, & Campbell, 2007; Page, Hooke, & Rutherford, 2001; Wing, Lelliott, & Beevor, 2000).

The HoNOS’ value has certainly been reflected in its use. The use of the HoNOS in

outcomes studies and clinical assessment reports from the United Kingdom (Wing et al., 2000),

Europe (Lora, Bezzi, & Erlicher, 2007; Parabiaghi, Barbato, D'Avanzo, Erlicher, & Lora, 2005),

the United States (Kisely et al., 2007), Australia (Pirkis, Burgess, Kirk, Dodson, & Coombs,

2005) and New Zealand (Theuma, Read, Moskowitz, & Stewart, 2007) are a testament to the

need for a clinician’s judgment of treatment outcome when evaluating services and treatments.

The collection of HoNOS data is mandatory within both private and public psychiatric services

in Australia, and yet very little has been published to support its use and enable meaningful

interpretation by clinicians, service providers and government policy makers.

Further criteria originally identified by a committee of experts assembled by the

National Institute of Mental Health in the United States of America (NIMH; Ciarlo, Brown,

Edwards, Kiresuk, & Newman, 1986) refer to the measure’s use on a wider scale. They

proposed that an effective outcome measure should have the capacity to be understood by non-

professional audiences, that feedback should be easily provided and interpreted, that it should be

useful in clinical services and be compatible with clinical theories and practices (Newman &

Ciarlo, 1994). HoNOS data have been used in practice to provide information to the patient

about their treatment progress or outcome, changes that have been made or progress still

expected, and what areas have been influenced by the intervention (Page et al., 2001). On a

larger scale, HoNOS data are collected across all private psychiatric hospitals in Australia, and

these data are used to inform service providers and policy makers (Morris-Yates & The

SPGPPS Data Collection and Analysis Working Group, 2000). The Private Mental Health

Alliance, (PMHA, formerly SPGPPS) provides quarterly reports on service effectiveness which

allows for comparison between services with casemix accounted for. HoNOS data are integral

to this reporting, as they provide a clinician’s perspective on patient outcomes. Accordingly,

HoNOS results have the capacity to be provided as feedback for patients, clinicians, third-party

payors, and policy makers at all levels of government.

49

However, at present, the feedback of results can be problematic. The current models for

subscale structure and subsequent scoring methods of the HoNOS have received much criticism

(Preston, 2000; Trauer, 1999) and do not appear to provide patient information consistent with

clinical theory and practice. The twelve items were designed to provide information on distinct

and disparate contributing factors. They range from rating aggressive, agitated behaviours to

activities of daily living and the availability of appropriate accommodation. It appears that the

College Research Unit (1996) developed the conventional four subscales upon the basis of item

similarity (Trauer, 1999). In contrast, Trauer (1999, 2000) has investigated the HoNOS’ utility

in largely psychotic samples and proposed a novel five-subscale structure based on these

findings. Despite its demonstrated validity within some patient groups, it is unlikely that a

subscale structure developed within a psychotic sample will provide an adequate fit for a largely

mood disordered patient group. Depression is the most common mental health problem reported

in Australia, and now accounts for the greatest burden of disease for all mental disorders,

followed by substance abuse and anxiety disorders respectively (Mathers, Vos, & Stevenson,

1999). Depressive disorders affect a substantial proportion of the population and thus

appropriate outcome evaluation for patients with depression, substance use and anxiety is

important information for patients, families, health-care providers, and the public. It is vital that

the assessment of outcome for these patients cater to their clinical characteristics in order to

provide a valid interpretation of the data. It is for this reason that the current study aimed to

investigate the factor structure and interpretation of the HoNOS within a predominantly mood

and anxiety disordered population. Previously, despite administration and data collection

requirements for mental health services in Australia, insufficient empirical evidence has been

available to assess the HoNOS’ validity in psychiatric samples. The following study (Newnham,

Harwood, & Page, 2009) illustrates the most appropriate means for scoring and interpreting the

HoNOS within a predominantly mood-disordered psychiatric sample.

50

References

Andrews, G., Peters, L., & Teesson, M. (1994). The measurement of consumer outcome in

mental health. Canberra: Australian Government Publishing Service.

Ciarlo, J. A., Brown, T. R., Edwards, D. W., Kiresuk, T. J., & Newman, F. L. (1986). Assessing

mental health treatment outcome measurement techniques (DHHS Pub. No. (ADM)86-

1301). Washington, DC: Superintendent. of Documents., U.S. Government Printing

Office.

College Research Unit. (1996). Health of the Nation Outcome Scales: Report on Research.

London: Royal College of Psychiatrists.

Hansen, L., & Kingdon, D. (2006). Rating Suicidality in Schizophrenia: Items on Global Scales

(HoNOS and CPRS) Correlate with a Validated Suicidality Rating Scale (InterSePT).

Archives of Suicide Research, 10(3), 249-252.

Howard, K. I., Lueger, R. J., Maling, M. S., & Martinovich, Z. (1993). A phase model of

psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical


Kisely, S., Campbell, L. A., Crossman, D., Gleich, S., & Campbell, J. (2007). Are the Health of

the Nation Outcomes Scales a valid and practical instrument to measure outcomes in

North America? A three-site evaluation acorss Nova Scotia. Community Mental Health

Journal, 43(2), 91-107.

Lora, A., Bezzi, R., & Erlicher, A. (2007). Estimating the prevalence of severe mental illness in

mental health services in Lombardy (Italy). Community Mental Health Journal, 43(4),

341-357.



Morris-Yates, A., & The SPGPPS Data Collection and Analysis Working Group. (2000). A

National Model for the Collection and Analysis of a Minimum Data Set with Outcome

Measures for Private Hospital-based Psychiatric Services. Canberra: Commonwealth

of Australia.

51

Newman, F. L., & Ciarlo, J. A. (1994). Criteria for selecting psychological instruments for

treatment outcome assessment. In M. E. Maruish (Ed.), The use of psychological testing

for treatment planning and outcome assessment (pp. 98-110). Hillsdale, NJ: Lawrence

Erlbaum Associates Inc.



334.

Page, A. C., Hooke, G. R., & Rutherford, E. M. (2001). Measuring mental health outcomes in a

private psychiatric clinic: Health of the Nation Outcome Scales and Medical Outcomes

Short Form SF-36. Australian and New Zealand Journal of Psychiatry, 35, 377-381.




Pirkis, J., Burgess, P., Kirk, P., Dodson, S., & Coombs, T. (2005). Review of standardised

measures used in the National Outcomes and Casemix Collection. Version 1.2.

Canberra: National Mental Health Strategy.

Preston, N. (2000). The Health of the Nation Outcome Scales: Validating factorial structure and

invariance across two health services. Australian and New Zealand Journal of


Slade, M. (2002). What outcomes to measure in routine mental health services, and how to

assess them: a systematic review. Australian and New Zealand Journal of Psychiatry,

36, 743-753.



Services.

Theuma, M., Read, J., Moskowitz, A., & Stewart, A. (2007). Evaluation of a New Zealand early

intervention service for psychosis. New Zealand Journal of Psychology, 36(3), 136-145.

Trauer, T. (1998). Issues in the assessment of outcome in mental health. Australian and New

Zealand Journal of Psychiatry, 32, 337-343.

52

Trauer, T. (1999). The subscale structure of the Health of the Nation Outcome Scales (HoNOS).

Journal of Mental Health, 8, 499-509.

Trauer, T. (2000). Update from Down Under. British Journal of Psychiatry, 176, 392-395.


of the Nation Outcome Scales (HoNOS): Research and development. British Journal of


Wing, J. K., Lelliott, P., & Beevor, A. S. (2000). Progress on HoNOS. British Journal of


53

Chapter Three________________________________________________

54

55

56

57

58

59

60

61

62

Section Three The effectiveness of progress monitoring and feedback in psychiatric care.

63

Foreword

The National Mental Health Strategy outlines a cycle of quality improvement in mental

health service, by which data are collected and analysed, and the resulting findings inform and

improve practice. Considerable progress has been made in each of these individual areas;

however, a full completion of the cycle is rarely realized in practice. The recent establishment of

large-scale outcome assessment programs in health care (e.g. CORE Partnership, 2007;

Lambert, 2007) have made significant progress in the collection and assessment of data. A

multitude of evaluation measures that are both psychometrically sound and clinically useful

have been reviewed in the scientific literature and adopted in routine practice. Further to this,

data collection at an individual (Lambert, 2007), service-wide, state and national level

(Department of Health and Ageing, 2008) has provided findings that have the potential to

inform practice and policy. Increasingly useful and meaningful methods of interpretation are

being developed for use by professionals and the public alike (e.g. Newnham, Harwood, &

Page, 2007, 2009; Parabiaghi, Barbato, D'Avanzo, Erlicher, & Lora, 2005). We are now in a

position to answer the most important question in psychotherapy: Is this treatment working for

this particular patient?

In a seminal work, Howard and colleagues (1996) outlined the three central questions in

clinical psychology research. First, does this treatment work? Second, does this treatment work

in real-practice settings? Third and most importantly, a question that has been largely neglected

within psychological science is whether this treatment is working for a particular patient. The

first question has been answered by randomised controlled trials evaluating the efficacy of

psychological treatments. Effectiveness research answers the second question. And the third

question, of particular interest to clinicians and patients, is the subject of patient-focused

research.

Clinical significance has broadened our understanding of psychotherapy outcomes, and

demonstrated that not all patients achieve reliable and meaningful change, despite use of

evidence-based treatments (Lambert, Hansen, & Bauer, 2008). In fact, some patients deteriorate

64

over the course of treatment (Lambert & Ogles, 2004). Clearly, receiving this information at the

end of therapy is not useful for that particular patient, and thus feedback on treatment progress

in real time is required. At an individual level, patient progress information enhances the

opportunity for change during the course of therapy.

The ongoing monitoring of individual patients’ progress and the feedback of that

information to clinicians and patients significantly improves outcome in outpatient therapy

(Harmon, Hawkins, Lambert, Slade, & Whipple, 2005; Harmon et al., 2007; Hawkins, Lambert,

Vermeersch, Slade, & Tuttle, 2004; Lambert, Hansen, & Finch, 2001; Lambert, Whipple et al.,

2001; Lambert et al., 2002). This process of providing feedback to clinicians about their

patient’s level of improvement, in comparison to their expected level of improvement, is a new

development in clinical and research strategy. Based upon the assertion that the foundation of

change is knowledge (Calman, 1998), and in a therapeutic partnership which requires

continuous revision and reassessment, patient-focused research provides education as to a

patient’s level of progress, contrasted against their expected improvement. With a goal to

complement clinical judgment, feedback provides an avenue for assessment of treatment and the

patient’s relationship to change.

However, despite the promising findings demonstrated for outpatient, individual

therapy (Lambert, 2007), a program for patient-focused research has not yet been developed for

use within an inpatient setting. Further to this, there is potential for its use in a group-therapy

format, and this has not been assessed either. Accordingly, the aim of the current project was to

develop and evaluate a patient-focused research program within a group-format inpatient and

day patient psychiatric setting. The first paper, (Newnham & Page, 2007), describes patient-

focused research and its role within psychotherapy research and practice. The manuscript was

published with a view to inform CBT therapists of recently developed methodologies as well as

the potential benefits of patient monitoring. Notably, patient monitoring is a suitable vehicle for

conducting research within one’s practice, and in turn, using those findings to inform process.

The second paper in this section (Newnham, Hooke, & Page, 2009), assesses the

psychometric properties of the World Health Organization’s Wellbeing Index (WHO-5) within

a psychiatric inpatient setting, and illustrates its clinical validity as a monitoring measure. The

65

final paper (Newnham, Hooke, & Page, submitted), evaluates the effectiveness of the wellbeing

monitoring program conducted within an inpatient psychiatric setting, and demonstrates the

potential for patient-focused research to improve treatment outcomes in hospital-based settings.

The effectiveness trial is unique not only in its format and applicable patient-group, but

also in its method of evaluation. Other trials of patient-focused research have evaluated

effectiveness with comparison against the same measure (Hawkins et al., 2004; Lambert,

Whipple et al., 2001). The current feedback trial assessed effectiveness not only on the same

measure, the WHO-5, but also against psychometrically valid measures of function and

psychological distress. Having illustrated the sound properties and clinical validity of the self-

rated mental health subscales of the SF-36, and the clinician-rated HoNOS, these, together with

the Depression and Anxiety Stress Scales were ample measure of the program’s effect.

66

References

Calman, K. C. (1998). Potential for health. Oxford: Oxford University Press.

CORE Partnership. (2007). Is initial overall CORE-OM score an indicator of likely outcome? In

CORE Partnership Occasional Paper, No 1. CORE IMS: Rugby.

Department of Health and Ageing. (2008). 2007-08 Annual Report. Canberra: Commonwealth

of Australia.

Harmon, C., Hawkins, E. J., Lambert, M. J., Slade, K., & Whipple, J. L. (2005). Improving

outcomes for poorly responding clients: The use of clinicial support tools and feedback

to clients. Journal of Clinical Psychology, 61(2), 175-185.

Harmon, C., Lambert, M. J., Smart, D. W., Hawkins, E. J., Nielsen, S. L., Slade, K., et al.

(2007). Enhancing outcome for potential treatment failures: Therapist/client feedback

and clinical support tools. Psychotherapy Research, 17(4), 379-392.

Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. (2004). The effects

of providing patient progress information to therapists and patients. Psychotherapy

Research, 14, 308-327.

Howard, K. I., Moras, K., Brill, P., Martinovich, Z., & Lutz, W. (1996). Evaluation of

psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist,

51, 1059-1064.

Lambert, M. J. (2007). Presidential address: What we have learned from a decade of research

aimed at improving psychotherapy outcome in routine care. Psychotherapy Research,

17(1), 1-14.

Lambert, M. J., Hansen, N. B., & Bauer, S. (2008). Assessing the clinical significance of

outcome results. In A. M. Nezu & C. M. Nezu (Eds.), Evidence-based Outcome

Research: A Practical Guide to Conducting Randomized Controlled Trials for

Psychosocial Interventions. New York: Oxford University Press.

Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: Using patient

outcome data to enhance treatment effects. Journal of Consulting and Clinical


67

Lambert, M. J., & Ogles, B. M. (2004). The efficacy and effectiveness of psychotherapy. In M.

J. Lambert (Ed.), Bergin and Garfield's Handbook of Psychotherapy and Behaviour

Change (5 ed., pp. 139-193). New York: Wiley.

Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkins, E.

J. (2001). The effects of providing therapists with feedback on patient progress during

psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-68.

Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L.,

et al. (2002). Enhancing psychotherapy outcomes via providing feedback on patient

progress: A replication. Clinical Psychology and Psychotherapy, 9, 91-103.






334.











68

Chapter Four_______________________________________________

69

70

71

72

73

74

Chapter Five_________________________________________________

75

76

77

78

79

80

Chapter Six__________________________________________________

Patient monitoring and feedback in psychiatric care reduces depressive symptoms.

Abstract

Background: To date, the monitoring of patient progress using standardized assessments

has been neglected in hospital-based psychiatric care. Findings in outpatient psychotherapy

have demonstrated clinically significant benefits for providing feedback to the sizeable minority

of patients who were otherwise unlikely to experience positive outcome (Lambert, 2007).

However, a similar system for presenting feedback on patient progress has not yet been assessed

within psychiatric inpatient settings. The current study aimed to develop and evaluate the

effectiveness of a feedback system suitable for use in psychiatric services. Methods: In a

nonrandomized trial, 1308 consecutive inpatients and day patients, whose diagnoses were

primarily depressive and anxiety disorders according to ICD-10 criteria, completed the World

Health Organization’s Wellbeing Index (WHO-5) routinely during a ten-day cognitive

behavioral therapy group. The first cohort (n=461) received treatment as usual. The second

cohort (n=439) completed monitoring measures without feedback, and for patients in the third

cohort (n=408), feedback on progress was provided to both clinicians and patients midway

through the treatment period. Results: Feedback was effective in reducing depressive symptoms

(F(1,649)=6.29, p<.05) for those patients at risk of poor outcome, but not effective in improving

wellbeing (F(1,569)=1.14, p>.05). Conclusions: Similar to outpatient settings, feedback

appears to be beneficial for improving symptom outcomes but further time may be required for

wellbeing to be affected. The current findings may be generalized to patient samples that exhibit

largely depressive disorders, and rigorous follow-up is warranted.

81

Psychiatric care, unlike physical healthcare, does not yet have the instruments available

to reliably monitor patient health and alert clinicians to a negative response to treatment.

Physical healthcare relies upon a suite of tools, such as the thermometer, to guide treatment.

Monitoring complements and informs the treatment process by providing the clinician with

quick, easy to interpret feedback on a patient’s response. Unfortunately, psychiatric care is in

the early phase of developing standard monitoring instruments that may alert clinicians to a

poor treatment response and signal a need to evaluate and possibly alter therapy.

This absence is concerning because relatively high rates of negative outcome occur in

mental health care. Despite the widespread use of evidence-based treatments, a large proportion

of patients still fail to demonstrate reliable or clinically meaningful improvement. Estimations

of deterioration rates have been as high as 23%, and up to 40% of patients show no change as a

result of therapy, as illustrated in both clinical settings (Lambert et al., 2002; Newnham,

Harwood, & Page, 2007; Parabiaghi, Barbato, D'Avanzo, Erlicher, & Lora, 2005) and

randomized controlled trials (Hansen, Lambert, & Forman, 2002), where a highly controlled

treatment is assessed under optimal conditions and with homogenous patient groups. It appears

that there is no typical pattern of change for all patients (Lutz, Stulz, & Köck, 2009), and that

despite the success of efficacy and effectiveness studies in identifying valuable treatments; a

large minority of patients are not benefiting from psychological therapy. One response to these

findings has been to seek better and more refined treatments. Another, not mutually exclusive

response has been to highlight the need to complement the delivery of psychological treatment

with additional progress information to improve rates of clinical recovery. By monitoring

progress during the therapy period, clinicians have the opportunity to improve outcomes in real

time for the benefit of each particular individual (Lambert, Harmon, Slade, Whipple, &

Hawkins, 2005; Lutz, 2003; Page & Stritzke, 2006).

Monitoring programs suitable for outpatient psychological therapy have been developed

in the United States and Europe, with notable success. In the US, Lambert and colleagues have

developed a program of feedback to alert therapists and patients to deviations from expected

response (Lambert, 2007). Administration of the 45-item Outcomes Questionnaire (OQ-45) at

each therapy session allows for the ongoing monitoring of progress, mapped against the

82

patient’s expected response trajectory. The expected trajectory is a function based upon the

patient’s severity at intake. Five studies (Harmon et al., 2007; Hawkins, Lambert, Vermeersch,

Slade, & Tuttle, 2004; Lambert et al., 2001; Lambert et al., 2002; Whipple et al., 2003), of

which four were randomized controlled trials, have demonstrated that providing feedback on

patients’ treatment response improved rates of clinically significant outcomes, but perhaps more

importantly, reduced negative outcomes from 20% to 8%. Those studies that have demonstrated

a significant improvement in outcomes have also revealed cost-efficient benefits for the use of

ongoing monitoring. Whilst the minority of cases that were identified as ‘not on track’ were

provided two to three sessions extra, the majority were ‘on track’ and received one session less

(Lambert, 2007). Thus progress monitoring and feedback appears to improve treatment

outcomes in an efficient and cost-effective manner.

However, one issue not addressed by the studies to date relates to the quantity of

psychotherapy provided. Although participants were randomly assigned to conditions, the

number of sessions of therapy delivered varied across conditions. As mentioned, the cost-

effectiveness of the different numbers of sessions received could be highlighted, but it does

mean that the research has not revealed that the same benefits would be observed if patients in

the feedback and non-feedback groups received the same quantity of therapy. Consequently,

research is needed in which the amount of treatment delivered is constant across feedback and

non-feedback groups to address this issue.

At present, monitoring patient response has been illustrated as an important addition to

clinical practice and research within the scientific literature; however it has been greatly

underutilized in practice. There are a number of arenas in which it would be beneficial to

monitor treatment response more closely. The argument is particularly compelling for fields in

which early response to treatment is useful information, such as drug trials; or the monitoring of

patient health is particularly important, such as suicide risk management or treatment for

trauma. Time-intensive psychotherapy and inpatient psychiatric care also require ongoing

monitoring but as yet, an appropriate and efficient monitoring system for short-term assessment

has not been identified. This is a surprising reality, given that Lambert rightly heralds the

ongoing monitoring of patient progress a clinician’s ethical responsibility (Lambert, 2007).

83

Despite the temptation to extend the methods to all areas of mental healthcare, of the

models that have been developed for the ongoing monitoring of treatment response (Lambert,

2007; Lutz et al., 2005) none operate over a brief time-frame. For instance, treatment offered in

inpatient settings is often intensive and brief, being measured in days, as opposed to outpatient

psychotherapy that takes place over weeks. Although a number of health status measures are

available for monitoring treatment response, all measure progress within a timeframe of 1 week

to 3 months, and most fail to demonstrate appropriate psychometric features for individual

monitoring (Lambert & Hawkins, 2004; McHorney & Tarlov, 1995). Accordingly, a system of

monitoring appropriate for settings in which changes may occur in days, not weeks, is required.

In addition, current programs have been developed within outpatient samples, and

therefore the outcomes may not necessarily generalize to an inpatient group setting (Newnham

& Page, 2007). Inpatient care differs in that it can be more intensive, the population may

experience a greater level of disturbance, psychotherapy is delivered in the context of a variety

of other interventions (e.g., pharmacotherapy, ongoing nursing care) and patients are often

discharged with the expectation that further improvement will occur with ongoing community

care. Furthermore, while the average length of stay in psychiatric facilities differs between

nations, Australia, like the United States, tends to have relatively short admissions (Page,

Hooke, & Rampono, 2005). It is therefore important to develop a monitoring program that has

the capacity to provide feedback within a shorter time-frame.

The current study comprised the development and evaluation of a monitoring system

suitable for use in an acute psychiatric setting. The World Health Organization’s Wellbeing

Index (WHO-5; Bech, Gudex, & Johansen, 1996) is a five-item self-report measure of positive

wellbeing that has performed reliably and sensitively in psychiatric samples (Newnham, Hooke,

& Page, 2009). The monitoring program involved the routine administration of the WHO-5 as a

measure of patient progress, evaluated within a Cognitive Behavioral Therapy group at a private

psychiatric hospital. The aim of the study was to assess the effectiveness of monitoring patient

progress, using that information as feedback for clinicians and patients. It was anticipated that

patients receiving feedback on their progress during group therapy would exhibit significantly

improved outcomes on measures of wellbeing and symptom distress at completion of the group.

84

Method

Research setting and participants

Participants were recruited from a 98 bed, private psychiatric hospital in Australia.

Eligible participants were English speaking inpatients or day patients who were participating in

the hospital’s two week cognitive behavioral therapy program. The sample comprised of 1308

patients who participated in the trial; 408 who received feedback on their WHO-5 scores, 439

completed the WHO-5 routinely but did not receive feedback on scores until completion of the

group, and a control group of 461 who were not administered the WHO-5 while participating in

the CBT treatment program.

Participants were diagnosed by their treating psychiatrist according to ICD-10-AM

criteria (National Centre for Classification in Health Publications, 2002) and primary diagnoses

consisted mostly of mood (67.7%), anxiety (25.9%), and substance use (3.0%) disorders. Ages

ranged from 16 years to 76 years with a mean of 39.8, and 63.1% were female. There were no

significant differences between groups in diagnoses (F (2,1302) = 1.77, p > .05), sex (F

(2,1305) = .476, p > .05) or age (F (2,1305) = .295, p > .05). The investigation was carried out

in accordance with the Declaration of Helsinki. The University of Western Australia Human

Research Ethics Committee approved the study protocol prior to commencement, and patients

provided informed consent as part of the routine admission procedure at the hospital.

Monitoring and Outcome Measures

The World Health Organization’s Wellbeing Index (WHO-5; Bech et al., 1996) is a 5-

item scale of positive wellbeing. Originally designed to screen for depression in diabetic

samples (Bech et al., 1996), the WHO-5 has also demonstrated reliability and validity in

screening for depression in primary care (Löwe et al., 2004) and elderly samples (Bonsignore,

Barkow, Jessen, & Heun, 2001). Further to this, the psychometric properties and clinical utility

of the WHO-5 has warranted its use as a measure for monitoring patient progress and treatment

outcomes in psychiatric care (Newnham et al., 2009). The Index is rated on a five point scale,

and for the purpose of the current study, wellbeing was measured over the past day rather than

previous two weeks (Newnham et al., 2009). Scores thus range from 0 to 25 with higher scores

indicating more positive wellbeing.

85

The Mental Health subscales of the Medical Outcomes Questionnaire Short Form (SF-

36; Ware, Snow, Kosinski, & Gandek, 1993), are a reliable self-report measure of patient

mental health status. They comprise of Vitality, Social Function, Role Emotion and Mental

Health scales, which have previously demonstrated validity as measures of patient outcomes in

psychiatric care (Newnham et al., 2007). The scales have sound internal consistency, exceeding

0.8 for each of the scales (Scott, Tobias, Sarfati, & Haslett, 1991), together with strong content

and construct validity (Newnham et al., 2007; Scott et al., 1991; Ware et al., 1993).

The Depression Anxiety Stress Scale (DASS-21; Lovibond & Lovibond, 1995a) is a 21-

item self-report measure of psychopathology. A short form of the of the 42 item scale, the

DASS-21 has strong internal consistency (Crawford & Henry, 2003) sound construct validity

(Lovibond & Lovibond, 1995b), and a cleaner factor structure (Antony, Bieling, Cox, Enns, &

Swinson, 1998). It is rated on a 5-point scale and high scores indicate more severe

psychopathology.

The Health of the Nation Outcome Scale (HoNOS; Wing et al., 1998) is a 12-item

measure of patient mental health that covers a heterogeneous range of presenting issues,

including aggression and agitated behavior, hallucinations, depressed mood, difficulties with

activities of daily living, social relationships and housing issues. The HoNOS is completed by

therapy staff, psychiatric nurses or the treating psychiatrist, all of whom have completed a

training program for its administration. The Scale measures health as reported for the previous

two weeks when rated at admission and over the preceding 72 hours when rated at discharge.

All items on the scale are rated from 0 (no problem) to 4 (severe problem); thus low scores

indicate healthier functioning.

The SF-36, DASS-21 and HoNOS were administered as routine clinical practice at

admission and discharge for each patient.

Trial Design

The trial comprised a historical cohort design so that between January 2005 and March

2006, patients were not administered the WHO-5; from April 2006 to July 2007 the WHO-5

was routinely administered during treatment, but feedback was not presented to the patient or

clinician until the final day of therapy; and between August 2007 and January 2009, participants

86

completed the WHO-5 routinely during therapy and were provided with feedback on their

progress at Day 5 and Day 10. The cohorts that were administered the WHO-5 were matched

according to the severity of their WHO-5 score at Day 1 of treatment.

Group Treatment

The Cognitive Behavioral Treatment Program is a closed group of 6-8 members that

runs from 9am to 5pm over 10 working days. Each group is run by two therapists, and covers

depression and anxiety management, cognitive disputation, behavioral activation and

experiments, identification and modification of negative core beliefs, self esteem,

communication skills, stress management, and dealing with setbacks. Each group member sets

personal therapy goals and participates in homework tasks. The treatment program has

demonstrated effectiveness comparable to randomized controlled trials (Page & Hooke, 2003).

Monitoring Intervention

Prior to the commencement of the group therapy, on days 1 (Monday), 3 (Wednesday),

5 (Friday), 7 (Tuesday), and 9 (Thursday) of the program, patients completed the WHO-5.

Occasionally, if a day was missed, the questionnaire would be completed on the following day.

For those in the No Feedback cohort, scores were graphed and provided to patients on Day 10

(Friday), where they were given the opportunity to discuss their scores with the therapist.

Using the data from the No Feedback group’s wellbeing scores, expected treatment

response cures were generated. The sample was divided into five groups according to severity

of wellbeing scores at Day 1, so that each group consisted of 20% of the sample. For each

group, means and standard deviations were calculated for each measurement point, which

illustrated a dose-response curve across the ten days of therapy. A log linear curve was

generated, one standard deviation around the mean for each day, for each group (see Table 1).

This curve became the trajectory of expected response against which each patient’s actual

scores were mapped (see Figure 1).

Consistent with the procedure of administration for No Feedback, those in the Feedback

condition completed the WHO-5 on days 1 (Monday), 3 (Wednesday), 5 (Friday), 7 (Tuesday),

and 9 (Thursday) of the program. According to the patient’s score at Day 1, an expected

treatment response curve was generated, and their actual scores mapped against it. This graph

87

and an accompanying explanation was provided as feedback to the clinician and patient at Day

5 (Friday of the first week of therapy) and Day 10 (Friday of the final week of therapy).

Feedback graphs were distributed during group, and clinicians spent time discussing what the

graphs meant, exploring the meaning of fluctuations in scores, and discussing the opportunity

that feedback provides to examine one’s progress and re-assess the treatment goals. Clinicians

were not given specific directions on the use of feedback but clinical management meetings

were held during the trial to assess adherence to protocol and discuss the clinical management

of cases.

Day 5 was considered the most appropriate point for feedback for clinical and evidence-

based reasons. Regression analyses (see Newnham et al., 2009) deemed Day 5 to be the point at

which Day 9 scores could be best predicted and it provided sufficient time to modify the

treatment plan. Further to this, it was suggested by the clinical management team that providing

feedback on a Friday morning presented the optimal opportunity to review the week’s work, set

homework exercises for the weekend ahead, and evaluate and devise treatment goals for the

second week of therapy.

Results

Equivalence of Groups

To compare the effectiveness of providing feedback, groups were matched according to

severity at Day 1 on the WHO-5. This was calculated by comparing each cohort according to

their expected trajectory of response groupings. Accordingly, for the analyses, each condition

had a sample size of 379, with no significant differences in severity between groups at Day 1.

To reduce the skew of the data, a square root of WHO-5 scores at each day was taken.

This score was used for all further WHO-5 analyses. Quality control charts revealed three

therapy groups with outlying scores (i.e. three standard deviations beyond the mean) which

were removed from the sample for further analyses.

Definition of Alarm

Consistent with Lambert’s research (Harmon et al., 2007; Lambert et al., 2001; Lambert

et al., 2002; Whipple et al., 2003), the data were analysed according to whether a person was

deemed ‘on track’ at the point of feedback. Following the feedback processes used, those who

88

were deemed to be ‘not on track’ at Day 5 (alarm cases) were patients whose Day 1 scores were

below 12 (thus within the unwell range at admission to the CBT Program), and whose scores

fell below the expected trajectory of improvement at Day 5.

Intervention Outcomes

The distribution of raw scores was inspected and means and standard deviations were

calculated for each cohort across all outcome measures (see Table 2). The data suggests that on

average, patients in all cohorts move from the ‘unwell’ range to the ‘well’ range on the WHO-5

(Newnham et al., 2009) and SF-36 (Newnham et al., 2007) as a result of therapy.

Patient self-report outcomes

To assess the effectiveness of feedback in reducing patients’ symptoms and improving

wellbeing, a series of repeated-measures ANOVAs were conducted with the outcome measure

as the dependent variable, and condition and alarm status as between group factors. A

significant improvement over time was evident for both conditions so that on average, patients

improved in wellbeing as a result of therapy (F (1,569) = 237.1, p<.05). However, there was no

significant difference in wellbeing scores at Day 9 between feedback conditions for those

patients on track or not on track (F (1,569) = 1.14, p>.05). Thus feedback to staff and patients

about scores on the WHO-5 did not significantly improve patients’ wellbeing by Day 9.

In contrast, feedback was of benefit on some measures of symptom distress. Again,

significant improvement over time was evident for all conditions on the Depression (F (1,649) =

438.5, p<.05), Anxiety (F (1,649) = 305.2, p<.05), and Stress (F (1,649)=421.2, p<.05) scales

of the DASS-21. A significant interaction was evident, arising because those patients who were

not on track and received feedback exhibited relatively greater improvements in Depression

scores on the DASS-21 (F (1,649) = 6.29, p<.05). However no significant difference resulting

from feedback was evident for any patients on the Anxiety (F (1,649) = .496, p>.05) or Stress

(F (1,649) = .628, p>.05) subscales of the DASS-21.

Those patients not on track who received feedback exhibited a significant improvement

on the Vitality subscale (F (1,639) = 5.53, p<.05), and Role Emotion subscale (F (1,635) = 4.11,

p<.05) of the SF-36. Yet no significant interaction was evident for Mental Health (F (1,639) =

2.28, p>.05) or Social Function subscales (F (1,643) = 1.94, p>.05). The results suggest that

89

measures of depressive symptoms, such as vitality and emotion, demonstrate a significant

improvement for those patients who received feedback when at risk of poor outcome.

The clinical validity of the results was investigated using the Jacobson and Truax

(1991) criteria for clinical significance (Newnham et al., 2009). Consistent with the finding that

feedback does not affect outcomes for those patients on track during therapy, positive outcomes

on the Vitality subscale of the SF-36 remained constant across feedback conditions (40.8%

achieved reliable improvement without feedback and 40.2% with feedback). However, the

provision of feedback reduced deterioration rates from 5.3% in the No Feedback cohort to 3.3%

in the Feedback cohort.

In contrast, no significant change in rates of clinically significant improvement or

deterioration was evident for the DASS-21 Depression subscale. Positive outcomes remained

constant across cohorts (51.3% achieved reliable improvement without feedback and 51.1%

with feedback). Similarly, 48.7% of participants exhibited a negative outcome (no change or

deterioration) without receiving feedback, compared to 48.9% with feedback. Deterioration

rates showed no significant change across cohorts.

The calculation of clinical significance was not appropriate for the Role Emotion

subscale due to the limited variability in potential scores (Newnham et al., 2007).

Clinician-rated outcomes

When examining the clinician-rated scores, again a significant effect of improvement

over time was evident so that patients were regarded as much improved following therapy (F

(1,615) = 639.7, p<.05). Yet, no effect of feedback was evident on HoNOS total scores

regardless of predicted outcome (F (1,615) = 3.20, p>.05). Interestingly, the data depict an

overall elevation of HoNOS scores for those patients not on track regardless of whether

feedback is provided, but no significant effect is evident (F (1,615) = 3.15, p=.08).

To investigate the hypothesis that feedback improves the convergence of clinician-rated

scores with patient self-report, correlations were conducted between HoNOS scores and the

other outcome measures (see Table 3). There is some indication of convergence between

clinician-ratings and self-report as the correlations reliably increase over time, however there is

no significant change in correlations across cohorts.

90

Discussion

The current study aimed to assess the effectiveness of monitoring patient progress and

providing feedback about wellbeing in an inpatient and day patient psychiatric setting. It was

proposed that those patients who received feedback on their response to treatment would exhibit

significant improvements in symptom relief and wellbeing, when compared to those who did

not receive feedback. The study found, unexpectedly, that providing feedback on wellbeing did

not significantly improve patients’ wellbeing scores at conclusion of the group. This result

represents a divergence from previous findings in patient-focused research that have illustrated

large treatment effects for feedback when measured against the same questionnaire at outcome

(Hawkins et al., 2004; Lambert et al., 2001; Lambert et al., 2002; Whipple et al., 2003) and

therefore warrants further consideration.

There are a number of plausible reasons for this discrepancy. It may be the case that

wellbeing is a construct less susceptible to the effects of feedback, or that it requires a longer

period of time to realise significant change. The stages of change outlined in the Phase Model

(Howard, Lueger, Maling, & Martinovich, 1993) suggest that an improvement in general

positive wellbeing and function occur in a later stage of therapy, following periods of

remoralization and remediation of symptoms. Thus, the time-intensive program of measurement

required for this study, as opposed to the weekly points of measurement depicted in previous

studies, may not capture changes in wellbeing required to show a significant improvement

resulting from feedback.

Another reason may be that the WHO-5 scores illustrate a ceiling effect which may

indicate that a large proportion of the sample move into the ‘well’ range early in therapy and

thus their difficulties are not captured by the measure. To test this hypothesis, the effects of

feedback were assessed on two measures of symptom distress.

In an extension of previous studies in the field of patient-focused research, the

effectiveness of feedback in reducing patients’ presenting symptoms was assessed on two

convergent measures: the DASS-21 and the mental health subscales of the SF-36. Traditionally,

the effects of feedback have been investigated using a single measure for monitoring and

outcome assessment (Harmon et al., 2007; Lambert et al., 2001; Lambert et al., 2002; Whipple

91

et al., 2003). Analyses revealed that those patients who were not on track for improvement in

therapy, but received feedback on their progress demonstrated a significant improvement in

depression, vitality and role emotion scores. For a predominantly depressive disordered group,

these treatment gains are of great consequence. The gains suggest that providing depressed

patients with feedback that they are not improving as expected presents an opportunity for a

crucial change in direction. Importantly, these findings demonstrated clinical impact in that

deterioration rates in therapy reduced from 5.3% to 3.3% when feedback was provided. This

finding complements and extends Lambert and colleagues’ work in patient-focused research

that has illustrated the significant benefits of providing feedback to clinicians and patients

during therapy (Harmon et al., 2007). Thus an improvement in depressive symptoms is evident

for those patients demonstrating poor progress when the feedback and non-feedback groups

received the same quantity of therapy.

In contrast, feedback did not significantly improve outcomes as measured on the broad

mental health and social function subscales of the SF-36, or the anxiety and stress subscales of

the DASS-21. Thus, the benefits of feedback appear limited to the symptoms which were the

principal reasons for treatment. This pattern of results is consistent with previous findings that a

reduction in depressive symptoms may not necessarily generalize to improvements in general

function (Bolton et al., 2007; Hirschfeld et al., 2000; Lin et al., 2000). Furthermore, the

improved outcomes associated with feedback were not evident when measured on the clinician-

rated HoNOS. Interestingly, a trend in the data suggested that clinicians rated ‘alarm’ patients

worse off, regardless of whether they received feedback, which suggests that they are attuned to

their patients’ symptoms, despite the low correlations between clinician ratings and patient self-

report outcomes.

The current study aimed to extend the field of patient-focused research by assessing the

use of feedback in an inpatient and day patient setting, using a group therapy format.

Accordingly, a number of discrepancies exist between the current research design and previous

work that may account for the alternative findings. First, for the current study, feedback was

provided within a group format. Previous attempts to provide feedback within a group format

have not used individualized feedback, and have failed to demonstrate any clinical benefit

92

(Davies, Burlingame, Johnson, Gleave, & Barlow, 2008). Group treatment differs from

individual treatment in that the individual is able to process their own response to treatment in

the context of others’ progress. Group therapy offers a dynamic environment in which a

patient’s lack of change may be highlighted or normalized, and achievements encouraged,

depending on the progress of the group in which they are treated. Thus the process of feedback

within group settings may be a qualitatively difference experience for patients and clinicians,

and this requires further examination.

Second, inpatient and day patient therapy differs to outpatient therapy in that it is time-

limited, the therapy provided is intensive, and the severity of the sample is likely to be greater,

which may limit the utility of feedback in this setting. Lambert and colleagues’ work suggests

that treatment for those patients deemed on track during therapy concludes sooner than those

not on track when feedback is provided (Lambert, 2007). Thus when the dose of therapy is

fixed, it appears that feedback presents the opportunity to attend more closely to those patients

not responding.

Third, the current study employed the WHO-5 as a measure of patient progress, which

is a freely-available, quick, reliable and valid measure of individual progress and outcome in

psychiatric settings (Newnham et al., 2009); but has a ceiling effect. The low cut-off for

movement into the healthy range means that a large proportion of patients’ distress is not

captured by the measure, and similarly, the range of movement is restricted. It is recommended

that future patient-focused research includes the measurement of symptoms which may be more

sensitive to acute changes. In addition to this point, outcome on the WHO-5 was measured at

the beginning of Day 9, which does not accurately reflect the patient’s health status at

conclusion of the group (end of Day 10). Thus, the symptom measures portray outcome more

precisely.

The present study was conducted with a historical cohort design, and thus patients were

not randomized to conditions. The reliable finding that progress monitoring and feedback

improves outcomes has been demonstrated in a series of randomized controlled trials (Harmon

et al., 2007; Hawkins et al., 2004; Lambert et al., 2001; Whipple et al., 2003), and therefore the

current study moved to extend the patient-focused methodology to a novel setting. Whilst the

93

study design reflects the real-world nature of the setting, conclusions cannot be drawn on the

reliability of improvements in symptom reduction among inpatients and day patients without

rigorous follow-up.

Patient-focused research has the potential to bridge the scientist-practitioner gap and

improve patient outcomes (Howard, Moras, Brill, Martinovich, & Lutz, 1996; Lutz, 2003). The

current findings provide limited support for this proposition. Patient monitoring and feedback

presents an opportunity to improve safety issues, increase the reliability of outcome

measurement, and foster a more collaborative relationship between clinician and patient. In this

regard, the patient is informed about their progress in relation to their expected progress, and

accordingly, empowered to make decisions about their treatment management; an objective that

should be highlighted more frequently in psychotherapy research.

Acknowledgments

The authors would like to thank Mrs Moira Munro, Perth Clinic, for ongoing support and

assistance. This study was supported by a grant from the Medibank Private Safety and Clinical

Improvement Incentive Pool, and a PhD Completion Scholarship awarded by the University of

Western Australia.

94

References

Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric

properties of the 42-item and 21-item versions of the Depression Anxiety and

Stress Scales (DASS) in clinical groups and a community sample. Psychological

Assessment, 10, 176-181.

Bech, P., Gudex, C., & Johansen, K. S. (1996). The WHO (Ten) Well-being Index: Validation

in Diabetes. Psychotherapy and Psychosomatics, 65, 183-190.

Bolton, P., Bass, J., Betancourt, T., Speelman, L., Onyango, G., Clougherty, K. F., et al. (2007).

Interventions for depression symptoms among adolescent survivors of war and

displacement in Northern Uganda. Journal of the American Medical Association,

298, 519-527.

Bonsignore, M., Barkow, K., Jessen, F., & Heun, R. (2001). Validity of the five-item WHO

Wellbeing Index (WHO-5) in an elderly population. European Archives of

Psychiatry and Clinical Neuroscience, 251(Supplement 2), II27-II31.

Crawford, J., & Henry, J. D. (2003). The Depression Anxiety Stress Scales (DASS): Normative

data and latent structure in a large non-clinical sample. British Journal of Clinical


Davies, D. R., Burlingame, G. M., Johnson, J. E., Gleave, R. L., & Barlow, S. H. (2008). The

effects of a feedback intervention on group process and outcome. Group Dynamics:

Theory, Research, and Practice, 12, 141-154.


effect and its implications for treatment delivery services. Clinical Psychology:

Science and Practice, 9, 329-343.


(2007). Enhancing outcome for potential treatment failures: Therapist/client

feedback and clinical support tools. Psychotherapy Research, 17(4), 379-392.

95



Research, 14, 308-327.

Hirschfeld, R. M. A., Montgomery, S. A., Keller, M. B., Kasper, S., Schatzberg, A. F., Moller,

H. J., et al. (2000). Social functioning in depression: A review. Journal of Clinical


Howard, K. I., Lueger, R. J., Maling, M. S., & Martinovich, Z. (1993). A phase model of

psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical



psychotherapy: Efficacy, effectiveness, and patient progress. American

Psychologist, 51, 1059-1064.





aimed at improving psychotherapy outcome in routine care. Psychotherapy

Research, 17(1), 1-14.

Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing

feedback to psychotherapists on their patients' progress: Clinical results and

practice suggestions. Journal of Clinical Psychology, In Session 61(2), 165-174.

Lambert, M. J., & Hawkins, E. J. (2004). Measuring outcome in professional practice:

Considerations in selecting and using brief outcome instruments. Professional

Psychology: Research and Practice, 35(5), 492-499.


J. (2001). The effects of providing therapists with feedback on patient progress

during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-

68.

96




Lin, E. H. B., VonKorff, M., Russo, J., Katon, W., Simon, G. E., Unützer, J., et al. (2000). Can

depression treatment in primary care reduce disability? A stepped care approach.

Archives of Family Medicine, 9, 1052-1058.

Lovibond, S. H., & Lovibond, P. F. (1995a). Manual for the Depression Anxiety Stress Scales

(2nd ed.). Sydney: Psychology Foundation.

Lovibond, S. H., & Lovibond, P. F. (1995b). The structure of negative emotional states:

Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck

Depression and Anxiety Inventories. Behavior Research and Therapy, 33, 335-343.

Löwe, B., Spitzer, R. L., Grafe, K., Kroenke, K., Quenter, A., Zipfel, S., et al. (2004).

Comparative validity of three screening questionnaires for DSM-IV depressive

disorders and physician's diagnoses. Journal of Affective Disorders, 78, 131-140.

Lutz, W. (2003). Efficacy, effectiveness, and expected treatment response in psychotherapy.

Journal of Clinical Psychology, 59(7), 745-750.

Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., et al. (2005).

Predicting change for individual psychotherapy clients on the basis of their nearest

neighbors. Journal of Consulting and Clinical Psychology, 73(5), 904-913.

Lutz, W., Stulz, N., & Köck, K. (2009). Patterns of early change and their relationship to

outcome and follow-up among patients with major depressive disorders. Journal of

Affective Disorders, doi:10.106/j.jad.2009.2001.2019.

McHorney, C. A., & Tarlov, A. R. (1995). Individual-patient monitoring in clinical practice:

Are available health surveys adequate? Quality of Life Research, 4, 293-307.

National Centre for Classification in Health Publications. (2002). The International Statistical

Classification of Diseases and Related Health Problems, Tenth Revision,

Australian Modification (ICD-10-AM). Geneva: National Centre for Classification

in Health Publications.

97


responses by psychiatric inpatients to the mental health subscales of the SF-36.

Journal of Affective Disorders, 98, 91-97.


outcomes using the World Health Organization's Wellbeing Index in psychiatric

care. Journal of Affective Disorders, doi:10.1016/j.jad.2009.1006.1005.



Page, A. C., & Hooke, G. R. (2003). Outcomes for depressed and anxious inpatients discharged

before or after group cognitive behavior therapy: A naturalistic comparison.

Journal of Nervous and Mental Disease, 191(10), 653-659.

Page, A. C., Hooke, G. R., & Rampono, J. (2005). A methodology for timing reviews of

inpatient hospital stay. Australian and New Zealand Journal of Psychiatry, 39, 198-

201.

Page, A. C., & Stritzke, W. G. K. (2006). Clinical Psychology for Trainees: Foundations of

Science-Informed Practice. Cambridge, UK: Cambridge University Press.


and clinically significant change on HoNOS: A method for displaying longitudinal

data. Australian and New Zealand Journal of Psychiatry, 39(719-724).

Scott, K. M., Tobias, M. I., Sarfati, D., & Haslett, S. (1991). SF-36 health survey reliability,

validity and norms for New Zealand. Australian and New Zealand Journal of

Public Health, 23, 401-406.

Ware, J. E., Snow, K. K., Kosinski, M., & Gandek, B. (1993). SF-36 Health Survey: Manual

and Interpretation Guide. Boston: The Health Institute, New England Medical

Centre.

Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E.

J. (2003). Improving the effects of psychotherapy: The use of early identification of

treatment failure and problem solving strategies in routine practice. Journal of

Counseling Psychology, 58, 59-68.

98


of the Nation Outcome Scales (HoNOS): Research and development. British

Journal of Psychiatry, 172, 11-18.

99

Figure 1.

Example feedback graph of WHO-5 scores for a patient responding to feedback at day 5.

100

Table 1

Log linear upper and lower scores for the expected treatment response trajectories for feedback.

WHO-5 Score at Admission Day 1 Day 3 Day 5 Day 7 Day 9

0-3 Upper 3.43 7.66 10.13 11.89 13.25

Lower 1.64 3.97 5.33 6.30 7.05

4-6 Upper 6.65 10.54 12.81 14.43 15.68

Lower 4.95 6.97 8.15 8.99 9.64

7-9 Upper 8.79 12.22 14.23 15.66 16.76

Lower 7.37 8.62 9.35 9.87 10.27

10-13 Upper 12.20 14.45 15.77 16.71 17.43

Lower 10.27 11.11 11.61 11.96 12.23

14-25 Upper 17.54 18.85 19.62 20.17 20.59

Lower 14.73 14.65 14.60 14.57 14.55

101

Table 2

Means and standard deviations (in parentheses) for each cohort across all outcome measures.

Measure Control No Feedback Feedback

WHO-5 Day 1 8.10 (5.15) 8.12 (5.31)

Day 9 13.21 (5.93) 12.98 (6.07)

DASS-21

Depression (Pre) 26.04 (12.92) 25.28 (13.17) 24.13 (12.69)

Depression (Post) 13.08 (11.29) 13.86 (11.07) 12.40 (10.79)

Anxiety (Pre) 20.00 (12.00) 19.24 (11.36) 19.57 (11.94)

Anxiety (Post) 11.41 (9.77) 11.41 (9.37) 10.92 (9.37)

Stress (Pre) 26.93 (11.38) 26.29 (11.23) 26.03 (11.19)

Stress (Post) 15.95 (10.47) 16.37 (10.67) 15.21 (10.07)

SF-36 Mental Health (Pre) 42.00 (20.68) 43.46 (20.65) 43.69 (20.98)

Mental Health (Post) 62.21 (19.92) 64.27 (19.90) 65.70 (19.49)

Vitality (Pre) 32.00 (21.81) 33.39 (21.68) 33.94 (22.69)

Vitality (Post) 48.88 (22.33) 51.01 (22.24) 52.77 (22.10)

Role Emotion (Pre) 28.25 (36.77) 26.55 (35.89) 26.54 (35.41)

Role Emotion (Post) 57.05 (41.23) 58.40(41.47) 60.40 (40.16)

Social Function (Pre) 38.71 (27.00) 40.33 (26.12) 40.47 (26.81)

Social Function (Post) 61.73 (27.39) 61.82 (27.69) 65.58 (25.86)

HoNOS Total (Pre) 11.17 (5.50) 10.32 (4.53) 12.15 (3.89)

Total (Post) 5.44 (4.06) 5.13 (3.39) 6.65 (4.05)

102

Table 3

Correlations between HoNOS post-treatment scores and patient self-report outcome measures across cohorts.

WHO-5 Day 1

WHO-5 Day 9

Depression Anxiety Stress Role Emotion

Vitality Mental Health

Social Function

HoNOS

Score

Control - - .385** .350** .326** -.276** -.297** -.390** -.334**

No Feedback -.218** -.397** .467** .366** .444** -.366** -.373** -.441** -.482**

Feedback -.275** -.476** .518** .485** .477** -.434** -.417** -.549** -.498**

** p<.01

103

Section Four General Discussion

104

General Discussion____________________________________________

Providing optimal healthcare is a dynamic process. It is vital that we act upon the best

scientific information available to improve patient care, with an ongoing openness to new

initiatives and findings so that the bridge between science and practice is continually

maintained. In doing so, we must implement systems of quality improvement for psychological

services, and incorporate new advances in knowledge, techniques and technologies as they

become accessible. At a treatment level, this evidence-based evolution of clinical practice is

epitomized in patient-focused research – the process of monitoring patient progress and

providing feedback on the patient’s treatment response. The ongoing assessment of patients’

health status allows clinicians to act upon deviations from expected response, and accordingly

alter the course of treatment. It enables the ongoing incorporation of knowledge with a view to

improve patient outcomes.

The current thesis outlined a program of patient-focused research, and in doing so,

aimed to address three important questions in psychotherapy research:

1. Can we identify poor responders in psychiatric care?

2. Is it possible to track patients’ progress during psychiatric care?

3. What effect does feedback have on clinical outcomes?

To address these questions, a series of studies investigated the validity of existing measures for

the evaluation of clinical outcomes in Australia, before designing and implementing a

monitoring system appropriate for use in psychiatric care, and evaluating its effectiveness. The

specific findings from these studies are outlined below.

Findings_______________________________________________________________

Can we identify poor responders in psychiatric care?

To improve treatment outcomes, it is important to first define recovery and conversely,

identify those patients who do not respond well to treatment. Various estimates of deterioration

rates have been published (Boisvert & Faust, 2003; Hansen, Lambert, & Forman, 2002;

105

Lambert & Ogles, 2004) however, the corresponding values have not been identified for

inpatient hospital-based psychiatric care. The detection of patient groups that are not

progressing well in therapy would enable clinicians to alter future treatment.

Outcomes are routinely measured in Australia via administration of the mental health

subscales of the SF-36 and the Health of the Nation Outcome Scale (HoNOS) (Morris-Yates &

Page, in press; Stedman, Yellowlees, Mellsop, Clarke, & Drake, 1997). Chapter 2 (Newnham,

Harwood, & Page, 2007) employed the Jacobson and Truax (1991) method for calculating

clinical significance, to determine criteria for recovery on the SF-36 for psychiatric inpatients

and day patients. A reliable change index (RCI) was calculated, as well as cut-off scores for

each subscale to convey the point at which a patient has moved from the ‘unwell’ to the ‘well’

range of scores. This information allows clinicians to determine the outcome for each patient, or

graph the pattern of outcomes for all of their patients. Application of the criteria to a sample of

inpatients in Western Australia revealed that between 35.7% and 41.8% recover, 5.5% to 12.8%

improve, 43.6% to 56.6% show no change, and up to 2.1% deteriorated. Thus almost 60% of

patients showed no improvement on one measure of mental health (Vitality) as a result of

inpatient treatment.

Deterioration rates identified in the psychiatric sample are significantly lower than those

shown for outpatient samples for a number of possible reasons. First, the inpatient sample

exhibits more severe psychopathology, and so it is more difficult to illustrate a decline in health

when many people are close to the measurement instrument’s ceiling. Second, inpatient

psychiatric treatment is intensive, and in Australian private psychiatric centres, average length

of stay is 20 days (Morris-Yates & Page, in press), while the hospital where the present research

was conducted is shorter at 12 days (Page & Hooke, 2009). It is therefore difficult to compare

these statistics with those for outpatient therapy which comprises weekly sessions measured in

hours rather than days.

The findings of Chapter 2 (Newnham et al., 2007) are directly applicable to clinical

practice in a variety of ways. Clinicians and mental health services could use the data to

evaluate outcomes and benchmark against other services. This process encourages optimal

clinical practice by enhancing transparency and accountability, highlighting individual

106

differences, and enabling a more meaningful approach to evaluating treatment effectiveness.

There are however, a number of different methods for calculating clinical significance

(Hageman & Arrindell, 1999; Hsu, 1989; Speer, 1992) and these would produce alternative

results. The Jacobson and Truax method was used in conjunction with recommendations for its

routine use in clinical practice (Lambert, Hansen, & Bauer, 2008), and convergent findings with

measures of quality of life and psychopathology outlined in Chapter 2 provide validity for this

approach. However, when assessing outcomes in mental healthcare, it important to consider a

variety of perspectives.

Fortunately, in Australian mental health services, self-reported outcomes are

complemented by clinician-rated scores. The HoNOS provides clinically-meaningful

information on a variety of health and psychosocial domains, and is widely used as a measure of

treatment outcome (Wing, Lelliott, & Beevor, 2000). Due to the diversity of the items, a total

aggregate score or individual items are often used to interpret the HoNOS; however, a new

parsimonious and clinically-useful subscale structure has emerged (Newnham, Harwood, &

Page, 2009). The third chapter of the thesis outlined the psychometric and clinical validity of the

HoNOS modified four-factor scoring model, which is applicable to psychiatric settings that treat

patients with predominantly depressive symptoms. The modified four factor model

demonstrated greater sensitivity to change, higher internal consistency, parsimony, and a better

fit to both the sample in which it was developed, and a second test sample, than previously

proposed models. In addition, the new model should facilitate the swift scoring of HoNOS

questionnaires, and the straightforward interpretation of the scores, enhancing its clinical utility.

The findings of Chapter 3 are important because the HoNOS is widely used both nationally and

internationally (e.g. Eager, Trauer, & Mellsop, 2005; Parabiaghi, Barbato, D'Avanzo, Erlicher,

& Lora, 2005; Pirkis, Burgess, Kirk, Dodson, & Coombs, 2005; Wing et al., 2000), and the

scores provide a different perspective on treatment outcomes, encompassing the clinician’s view

of a patient’s health status. As outlined in Chapter 6, the HoNOS exhibits a small to moderate

correlation with self-report measures of health status, thus providing more information than self-

report alone.

107

The HoNOS accounts for clinical significance in its scoring procedure (i.e., a score of 2

or more on any item indicates clinically significant distress or dysfunction), and Parabiaghi and

colleagues (2005) have investigated the use of clinical significance with longitudinal HoNOS

data; thus the issue of clinical significance was not further investigated for the current thesis.

In summary, the first section of the thesis outlines the clinical value of two routinely

used measures of outcome which provide not only a valid means for assessing treatment

effectiveness, but also enable benchmarking across services. Chapters 2 and 3 described

recovery criteria and scoring methods respectively; that would facilitate the use of patients’

scores to best inform treatment outcomes. In doing so, use of the SF-36 mental health subscales

and the HoNOS was validated in psychiatric outcome assessment. Further to this, a group of

poor responders could be identified using the clinical significance calculations outlined in

Chapter 2. This group of patients clearly requires intervention during therapy to address poor

treatment progress. The findings of Section One thus enable the valid and rigorous assessment

of whether a feedback intervention in psychiatric care improves clinical outcomes.

Is it possible to track patients’ progress during psychiatric care?

The second chapter of the thesis (Newnham et al., 2007) depicts the small but

noteworthy proportion of poor responders in psychiatric care, and highlights the level of need

for new innovations that complement effective treatments and improve outcomes for those

particular patients. This finding is further supported by the calculations of clinically significant

outcomes for wellbeing scores described in Chapter 5 (Newnham, Hooke, & Page, 2009).

Building upon the strong foundations for patient-focused research outlined by Michael Lambert,

Wolfgang Lutz and their colleagues (e.g. Howard, Moras, Brill, Martinovich, & Lutz, 1996;

Lambert, 2007; Lutz et al., 2006), a system for monitoring patient progress in a psychiatric

hospital setting was developed. Whilst incorporating the processes that have demonstrated

clinical effectiveness in a number of randomized controlled trials (Harmon et al., 2007;

Hawkins, Lambert, Vermeersch, Slade, & Tuttle, 2004; Lambert et al., 2001; Lambert et al.,

2002; Whipple et al., 2003), the system was designed to address the nuances of a novel clinical

setting. The hospital setting required a program that demonstrated reliability, validity and

108

sensitivity to change over a short time-frame (due to the intensive nature of inpatient and day

patient care), it had to be freely-available, quick to administer, and demonstrate robust

psychometric properties. Chapter 5 (Newnham, Hooke et al., 2009) outlines the process of

development and Chapter 6 (Newnham, Hooke, & Page, submitted), the implementation of the

monitoring program.

The present research program employed the World Health Organization’s Wellbeing

five-item Index (WHO-5; Bech, Gudex, & Johansen, 1996) to monitor patients’ wellbeing

across a ten-day cognitive behavioural therapy group. The WHO-5 is a freely-available measure

of positive wellbeing that demonstrated reliability, validity and sensitivity to change in a

psychiatric sample (Newnham, Hooke et al., 2009). The ongoing monitoring of patient progress

was implemented in the hospital in a historical cohort design, consistent with quality

improvement efforts. Given the literature on implementation science, it was anticipated that

clinicians would raise concerns about the system’s impact on the therapeutic process, as well as

concerns that it would affect the amount of control clinicians’ may have over the treatment plan,

and burden them with extra tasks (Grizenko, Poinsier, Mercier, Perreault, & Latimer, 2000).

To address these points, we worked towards giving clinicians a voice in the design and

implementation of the program. This transpired in seven main actions.

1. Prior to the implementation of the feedback trial, we ran a series of presentations at

the hospital, to describe the rationale and methods of the program, for therapists,

nursing staff, doctors and the executive board, to ensure that all contributors to the

treatment team were aware of the program, had an opportunity to ask questions and

make suggestions for improvement. The presentations were followed up with

handouts that comprised timelines, a rationale statement, a review of the literature

(Chapter 4: Newnham & Page, 2007), and example patient graphs. This facilitated

communication between therapists, nurses and doctors about the feedback graphs

and their impact potential on the treatment plan.

2. The feedback trial was initiated in a small sample comprising only the CBT group at

the hospital. Two CBT groups run concurrently, with two therapists each.

Accordingly, we were working with only four therapists (with little turnover during

109

the trial period) and thus clinical concerns could be attended to quickly and often

among the therapists themselves. Starting small enabled us to establish an effective

and efficient system.

3. The timing of feedback was decided upon with the assistance of empirical data and

clinical opinion. The findings highlighted in Chapter 5 indicated that Day 5

Wellbeing scores (i.e., at the mid-point of therapy) significantly predicted outcome,

and the possibility of feedback on Day 5 was discussed with clinicians in a series of

meetings. Therapy staff agreed that Day 5 (Friday of the first week of therapy)

would be an optimal point to provide feedback as it presented an opportunity to

discuss the week’s progress, set homework tasks for the weekend, and validate or

modify the treatment goals for the coming week. Accordingly, clinicians were

encouraged to take greater control over the use of the feedback graphs, and this

promoted their engagement in the research trial.

4. The monitoring program was run first as a pilot trial which provided the opportunity

for clinicians to raise any concerns about the methodology and have them addressed

prior to the commencement of the research trial.

5. Once the trial was underway, regular clinical management meetings were held for

the therapy staff to discuss any concerns or difficulties they had experienced with

providing feedback to patients. Early in the trial period, clinicians raised concerns

about telling patients that they were ‘not on track’, fearing that it would dishearten

patients and cause them to feel as though they had “failed”. The clinical management

meetings were thus employed to normalize these concerns, reaffirm the rationale and

previous findings in the area (e.g. Harmon et al., 2007; Lambert, 2007) and problem

solve difficult cases and patient queries. A patient’s resistance to feedback may

highlight their concerns about the accuracy or appropriateness of the information,

and discussions about its interpretation are likely to enrich the therapeutic process

(Claiborn & Goodyear, 2005). Thus, the process of feedback is important, even if the

content is rejected; and this sentiment was emphasized with therapists. A team

approach was used to communicate ideas in the clinical management meetings, so as

110

to avoid directive guidance of the clinical staff. The meetings also provided an

opportunity to ensure adherence to protocol throughout the trial period.

6. The use of information technology such as touch-screen computers to administer the

WHO-5 enabled reliable and immediate data entry and feedback of results. There is

potential for information technology to play a large role in the implementation of

quality improvement programs in healthcare, and this will be expanded in the Future

Directions section of the General Discussion. However, even without the use of

technology to enable monitoring, reliance on administrative assistance is beneficial

in lessening the burden on clinicians’ time.

Accordingly, the measures taken to involve clinicians in the design of the monitoring

program and engage them in the implementation, significantly enhanced the management of the

research trial. In a reflection of the clinicians’ involvement, and the monitoring program’s

impact on clinical outcomes, it has since been extended to the other therapy groups offered by

the hospital. Tailoring the feedback program to the particular setting in which it is to be

implemented is vital. Without the inclusion of staff perspectives in quality improvement

programs and health services research, implementation within health settings can be difficult, if

not impossible (Grol & Grimshaw, 2003).

Chapters 5 and 6 describe the first program for individual feedback conducted with

group therapy. Graphs were created for each patient in the feedback cohort, depicting their

progress scores across the duration of therapy, from which they could assess their rate of

improvement and make changes to their treatment plan. Feedback was provided as individual

graphs, accompanied by a written description. Therapists complemented this process in session

by describing potential shapes of change and what they might mean, and provided an

opportunity for patients to share their views on their own progress, or ask questions, should they

wish to. Further opportunity to discuss the graphs privately with the therapists was available to

all patients. Therapists reported spending between five and thirty minutes discussing feedback

in session, dependent upon the level of interest within the group.

The wellbeing monitoring program demonstrated strong psychometric properties within

a psychiatric setting, as described in Chapter 5. Despite not having been used in psychiatric care

111

previously, the WHO-5 exhibited reliability, convergent validity with measures of

psychopathology and mental health, and specificity to depressive symptoms at discharge.

Clinical significance calculations revealed a cut-off score appropriate for the definition of

recovery in a psychiatric sample. The findings outlined in Chapter 5 enable the straightforward

execution of the wellbeing monitoring system in new settings. They highlight benchmarks for

clinicians to judge progress and outcome for individual patients when using the widely available

WHO-5 in mental healthcare.

Expected treatment response curves were created for the purpose of tracking patient

progress and alerting clinicians to “alarm” responses during therapy. These trajectories were

calculated according to the log-linear curves of the standard deviation around the group mean,

for each of five severity groups. The simplicity of this approach has advantages and limitations.

It provided clear and meaningful guides to treatment progression for clinicians and patients in

the current study, and it may be easily replicated in other data samples, however, sophistication

in prediction of patient profiles and accuracy of modelling were sacrificed. The modelling of

patient progress data would be greatly enhanced by multilevel modelling (Tasca & Gallop,

2009), or the nearest neighbour approach outlined by Lutz and colleagues (2005), which would

generate a more accurate depiction of the patient’s expected trajectory during the course of

therapy. With the addition of more data on progress in psychiatric care, these methods should be

explored in the future.

One limitation with the present research is that the WHO-5 is a measure of positive

wellbeing and thus it does not capture a full range of psychopathology or distress that patients

may present with. It is not intended to be a comprehensive and all-encompassing measure of

health status, rather, a perspective to complement clinical judgment. Initially, we had hoped to

achieve the same benefits that had been achieved with symptom measures, with a succinct and

quick-to-administer positive wellbeing assessment. However, it appears that incorporation of

both symptom and wellbeing information would provide a more comprehensive picture of

patient progress. Nonetheless, it was hoped that the monitoring process would help patients to

verbalise issues that they may not otherwise raise in therapy (perhaps in particular, group

112

therapy). Thus, the feedback is not an objective measure of the patient’s health, but a

supplement to the therapeutic process.

Chapter 5 therefore suggests that it is possible to track patient progress during group

psychotherapy, and that this information can be assembled as feedback, which leads to the

question of whether this process is effective in improving outcomes for those patients who do

not respond to treatment.

What effect does feedback have on clinical outcomes?

The clinically significant improvements resulting from progress feedback have been

well-documented in randomised controlled trials conducted in outpatient psychotherapy

(Harmon et al., 2007; Hawkins et al., 2004; Lambert et al., 2001; Lambert et al., 2002; Whipple

et al., 2003). It is therefore time to extend these findings to novel settings, such as psychiatric

hospital services, and examine their effectiveness. The sixth chapter outlines the effectiveness of

feedback in a sample exhibiting three characteristics not previously examined: Australian

patients seen in a hospital setting serving inpatients and day patients attending group therapy.

Extension of the feedback methodology to an inpatient setting resulted in a significant

improvement in clinical outcomes. Those patients who were not on track at Day 5 demonstrated

significantly greater improvements in depression, vitality and role emotion when given

feedback on their progress. Further to this, the rate of deterioration in vitality was reduced from

5.3% to 3.3%. The finding that the treatment of depression can be complemented by monitoring

and feedback to assist poor responders is noteworthy for mental healthcare. In Australia,

depression is the most common mental health difficulty reported (Mathers, Vos, & Stevenson,

1999) and it is costly to treat, with an estimated 48% of patients rehospitalized within five years

(Daniels, Kirkby, Hay, Mowry, & Jones, 1998). By improving outcomes specifically for those

patients who do not appear to respond well to psychotherapy; the costs (both financial and

personal) of depression in Australia could be greatly reduced.

Conversely, feedback did not improve outcomes when assessed on broad measures of

mental health and wellbeing, nor measures of anxiety and stress. There are three potential

reasons for why no change was evident in wellbeing scores. First, the final point of

113

measurement on the WHO-5 was at the beginning of Day 9, a rating of the previous 24 hours.

Accordingly, patients’ status over Day 8 of the ten-day group is unlikely to be representative of

their final outcome, and thus the other measures (SF-36 and the DASS-21, administered on Day

10 of group) present a more typical illustration of outcome. Second, the WHO-5 demonstrates a

ceiling effect in the present sample, so that many patients may have moved into the “well” range

early in therapy, and their distress not identified on a measure of positive wellbeing. Third, it

may be that wellbeing is less receptive to the effects of feedback. Information concerning a

patient’s progress may not be sufficient to effect change in their general wellbeing. Rather, the

feedback appears to elicit changes in mood and vitality, and more time may be required to

improve general functioning and quality of life as measured on the WHO-5 and mental health

subscale of the SF-36. Similarly, no significant benefits of feedback were evident on measures

of anxiety and stress. This may reflect the smaller proportion of patients presenting with

primary diagnoses of anxiety or stress, but could also indicate that these symptoms are more

difficult to improve with feedback. Further investigation into the effects of feedback for patient

samples exhibiting mostly anxiety disorders may provide more detail on its effectiveness within

that population.

Inpatient group psychotherapy is an effective means of treating depression (Koesters,

Burlingame, Nachtigall, & Strauss, 2006). The American Psychological Association’s Clinical

Practice Guidelines for Group Psychotherapy (Bernard et al., 2008) recommend the ongoing

monitoring of patient progress, and use of feedback to improve outcomes. However, despite

recommending a battery of tests (Strauss, Burlingame, & Bormann, 2008) that may be used with

groups to assess process, engagement and outcome, no evidence for use of monitoring in group

psychotherapy was cited. Feedback has been trialled in group outpatient psychotherapy on one

occasion previously, with little impact on clinical outcomes (Davies, Burlingame, Johnson,

Gleave, & Barlow, 2008). Indeed, the provision of group process and cohesion feedback to

group members was ineffective in improving outcomes, and harmful to those in high conflict

groups. However, Davies and colleagues (2008) did not provide individualized feedback, and it

may be the disconnect between feedback and treatment goals that failed to elicit an

114

improvement in outcomes. Thus, Chapter 6 highlights the first trial of feedback in a group

format to make a positive impact on treatment outcomes.

Group therapy is subject to different dynamics to those that play out in individual

therapy. The cohesion of the group and individuals’ engagement in the group process appear to

interact with feedback processes (Davies et al., 2008), however it is unclear what role these

therapeutic factors play. It may be the case that highly cohesive groups encourage the use of

feedback, or discourage it; or that a person’s progress in relation to the rest of the group may

alter their interpretation of individualised scores. Accordingly, the qualitative processes

involved in providing progress feedback with group psychotherapy are an intriguing area

requiring investigation.

Australian mental healthcare is moving towards an era of improvements based on

outcomes information (Australian Mental Health Outcomes and Classification Network, 2005).

Psychiatric care accounts for a significant proportion of mental healthcare, and private hospitals

account for one quarter of all psychiatric overnight admissions (Morris-Yates & Page, in press).

Of these, the majority of admissions are presentations of depressive disorders (Morris-Yates &

Page, in press). The findings of the thesis suggest that monitoring patient progress is not only

possible in Australian private psychiatry; the process improves clinical outcomes for those

exhibiting depressive symptoms. The wider application of this work is an important addition to

evidence-based practice. However, it is unclear as yet whether monitoring progress is possible

with all patient groups. Those patients experiencing difficulties with insight into their mood or

behaviours (including those with psychotic symptoms or personality disorders), may not be able

to provide valid information on their treatment progress. Further to this, it is not clear how

feedback is interpreted by patients, and this raises questions about whether it would be as

beneficial for other patient samples. In fact, very little is known about the process of feedback,

or how it is received and used to alter behaviour.

Future research_________________________________________________________

How does feedback work?

115

Feedback communicates information about a behaviour that has occurred, and

influences the probability and nature of its recurrence (Claiborn & Goodyear, 2005). Thus

feedback is intended to shape behaviour, through the recipient’s acceptance, partial acceptance

or rejection of the information. While feedback is a routine process inherent in all domains of

psychotherapy, the provision of formalized feedback to patients, with a view to improve the

treatment offered, is a relatively new technique. The feedback program outlined in Chapters 5

and 6 (Newnham, Hooke et al., 2009; Newnham et al., submitted) used descriptive and

interpretive feedback outlining the individual patient’s pattern of progress, and how it related to

similar patients’ progress. Very little research has been conducted within psychotherapy to

inform the necessary components required for feedback to be useful; and the way feedback is

interpreted has rarely been investigated within clinical samples. However, the few studies that

have examined the effects of various feedback formats provide some clues as to the

characteristics required for effective behavioural change (Claiborn & Goodyear, 2005).

For feedback to be optimally received, it should be clear and provide information that is

useful and of personal relevance to the receiver. It should come from an influential source, who

is judged to be not only an authority, but also trustworthy (Sapyta, Reimer, & Bickman, 2005;

Strong & Matross, 1973). Thus the therapeutic relationship is of great importance to the process

of change. Without a trusting and secure relationship, feedback may not be well received, and

without a shared curiousness, modification of the treatment plan may not be considered

(Claiborn & Goodyear, 2005). Despite this, the course of feedback does not require immediate

and total acceptance of the information, in fact, critical interpretation of the message is the key

to active participation and thus insight into one’s mood and behaviours. Another important

factor suggested to be critical in the effectiveness of feedback is the accompaniment of progress

information with a suggested course of action. Davies and colleagues (2008) proposed that their

group feedback program may not have worked because feedback was not individualized, and

did not identify possible options for enacting improvement. The study outlined in Chapter 6

(Newnham et al., submitted) responded to both issues. By providing individualized feedback on

wellbeing status (rather than group cohesion) and encouraging the use of this information in

116

treatment planning and evaluation of progress, outcomes were significantly improved for those

patients who were unlikely to respond otherwise.

The contextual feedback intervention theory (Sapyta et al., 2005), proposed a model of

feedback that encompasses a self-directed learning theory. It is argued that a number of features

within the therapeutic encounter interact with feedback in order to exact successful behaviour

change. First, the clinician (and patient) must be motivated to reach a goal, and feedback should

highlight their position in relation to attaining it. The goal must be both attractive and realistic.

Considerable planning is required to determine the most appropriate and useful feedback

content, direction and format, so that it meets the needs of the recipient and can be rapidly

processed. Further to this, Sapyta and colleagues (2005) highlight the importance of cognitive

dissonance in the process of behaviour change, but are quick to point out that dissonance does

not necessarily create change in the intended direction. There are a number of ways to reduce

cognitive dissonance, and reduced motivation and engagement in the therapeutic process are

potential outcomes. Continued commitment to the goal, personal responsibility for the outcome,

and a sense of control over the process are all important factors in optimizing the effects of

feedback within clinical settings (Sapyta et al., 2005).

An area yet to receive research attention is how feedback is interpreted, and what

factors influence its interpretation. As highlighted in Chapter 5 (Newnham, Hooke et al., 2009),

mood biases attention to and interpretation of negative information (Mathews, Ridgeway, &

Williamson, 1996). Research conducted with university students demonstrated that those with

dysphoric mood were more likely than those with non-dysphoric mood to attend to negative

messages, and were more suggestible, aligning their answers in a negative direction

(MacFarland & Morris, 1998). It is likely that negative messages fit with a confirmatory bias for

those with low mood (Claiborn & Goodyear, 2005), but this does not explain why feedback

suggestive of “alarm” status helps to turn progress around and improve outcomes. Feedback is

most successful when the information highlights a discrepancy between the current behaviour

and expected or desired outcome (Sapyta et al., 2005). Further clinical research that explores

how feedback interacts with a receiver’s mood, motivation to change, expectations for therapy

and self-esteem is undoubtedly warranted.

117

Further research is also necessary for investigating the qualitative experience of group

psychotherapy members who receive feedback. Group psychotherapy is inherently a social

experience, and implicit feedback occurs continuously at all levels of dynamic interaction.

However, the formal monitoring of progress and individualized feedback to group members

adds a complexity not yet examined. It is unclear whether members compare their progress with

others (in that particular group), and whether this impacts upon their personal interpretation. In

addition, motivation and expectations for therapy may impact upon one’s interpretation of

feedback. Feedback programs have now demonstrated effectiveness in outpatient and inpatient

settings, and with individual and group therapies. Thus, future research should investigate the

qualitative experience of feedback and how it is used by clinicians and patients, in order to

further our understanding of feedback processes and how they can be best used to improve

patient care.

How can information technology best be used to benefit mental healthcare?

The recent surge in information technology calls for the integration of electronic

apparatus in health service delivery and research. To match the evolution of information

technology, access to communication networks has become widespread and more than two

thirds of Australian households have internet access (Australian Bureau of Statistics, 2008). The

implications for health promotion and service delivery stem from American research that

suggests that 80% of internet users report searching for health information, and that this

information has influenced their decisions about healthcare access and interactions with their

doctor (Atienza et al., 2007). While many businesses have adopted recent technologies to

improve efficiency (such as online shopping, telephone and internet banking, and online check-

in and touch-screen kiosks at airports), health services are slowly adapting information and

communication technologies (ICT) to improve access and costs for patients and providers

(Griffiths, Farrer, & Christensen, 2007; Marks, 2004). Considerable opportunities exist for the

use of ICT in assessment and diagnosis, monitoring of patient health, complementing

interventions and the audit of clinical outcomes in mental health.

118

The current thesis adapted touch-screen technology to monitor patients’ wellbeing

during therapy, and provide real-time feedback. The benefits of using touch-screens in the

therapy rooms included: (1) increased interest and engagement with the monitoring process for

patients, because they can see what happens to the questionnaires they complete; (2) greater

reliability and efficiency for the process of data collection, as data are automatically submitted

and collated within an electronic database; (3) reducing the burden on clinical staff to administer

measures; (4) increased security with data encryption, a ‘kiosk’ running mode, and a Wyse “thin

client” terminal that can be locked in a cabinet; (5) greater efficiency in producing feedback

graphs; (6) the amount of missing data is reduced because the touch-screen requires completed

responses to previous items in order to move to the next screen; and (7) the automated growth of

a health database for future clinical audit and research evaluation. The use of touch-screens and

an electronic data network within the clinic also led to a rapid expansion of services integrated

into the technology, such as monitoring group attendance, creating outcome graphs (using the

SF-36 mental health subscales) and the integration of financial management data. Accordingly,

touch-screen technology was widely and enthusiastically adopted within a psychiatric setting,

and this raises the question of how new information technologies can best complement clinical

practice in future.

To date, ICT has been adopted in diagnosis and assessment of patient health, in both

physical and psychological healthcare. An abundance of self-evaluation measures of

psychopathology are now available on the internet, either through healthcare provider or

independent websites (Atienza et al., 2007); and electronic assessment has extended to the use

of touch-screens in waiting rooms (Hunter, Travers, & McCulloch, 2003), and interactive-voice-

response technology available through telephones (Stritzke & Page, 2009). High diagnostic

correspondence has been demonstrated between assessments conducted with interactive-voice-

response systems and clinician-administered structured clinical interviews (Kobak et al., 1997),

and the procedure has potential to improve efficiency and costs per patient (Stritzke & Page,

2009).

Technological advancements have also been incorporated into psychological

interventions. Telepsychiatry is particularly valuable in Australia where regional and remote

119

patients are disadvantaged due to great geographical isolation and difficulty with professional

recruitment, but is not yet being employed to its full potential (McLaren, 2005). Computer-

aided therapies such as the VirtualClinic supported by the Clinical Research Unit for Anxiety

and Depression (Titov, Andrews, & Schwencke, 2008), and the internet-based MoodGYM

program for adolescents (O'Kearney, Kang, Christensen, & Griffiths, 2009) are becoming

increasingly available and have demonstrated effectiveness at lower cost to the patient (Marks,

2004). In addition, doctors are increasingly using the applications available on smart-phones

such as the BlackBerry and Apple’s iPhone to provide immediate and accurate information for

patients, and to communicate with wider treatment teams regarding patient health (i.e. by

sending patient data such as patient charts that can be acted upon immediately) (Bhanoo, 2009).

Internet-based monitoring of patients’ progress is the next step in eHealth, and has the potential

to highlight a patient’s decline in health more quickly than current care procedures. However,

despite the benefits in cost efficiency and ease of use, there is considerable potential for

technological advancements to do harm as well as good.

The continuing advancement of technological improvements to health service provision

will require collaborations between IT specialists, industry partners, government and researchers

(Atienza et al., 2007). It is important that electronic systems are used to engage patients in their

healthcare decisions and promote collaborative management of the treatment plan, rather than

alienate them. The risk of alienation stems from the potential for ICT to increase the differential

of power between doctor and patient, increase the distance in communication between them,

and reduce the patient’s confidence in their own critical assessment. Technological

advancements are only a useful addition to healthcare if the clinician and patient are

comfortable using it. It is vital that technology is used to complement evidence-based practice,

not replace it. There is also a need for further randomized controlled trials conducted by

investigators not commercially invested in the technology. Despite concerns that an increasing

reliance on technology in health will widen the “digital divide,” recent employments of touch-

screen programs in Australia suggest otherwise.

Touch-screen technology provides an easy and interesting way to both deliver and

solicit health information. While the installation and programming of touch-screens incurs a

120

cost, their use can generate an overall saving in time for both health and administrative staff.

Touch-screen use improves reliability in data collection, enables the quick dissemination of

educational material, engages people in their health management, and allows clinical staff to

address more complex concerns. One example in a class of possible applications is the use of

touch-screens in remote communities. Providing health education through touch-screens set-up

in remote health clinics is a step towards reducing health disparities between Indigenous and

non-Indigenous Australians. By supplying health information through a touch-screen kiosk

adapted with local Indigenous voices and easy-to-access buttons, Hunter and colleagues (2003)

were able to engage a remote Indigenous community in their health education. This technology

has great potential for mental health promotion and monitoring, and provides the opportunity to

address unmet need in currently disadvantaged groups – the most exciting application of health

technologies yet.

Thus the current thesis outlines the novel application of touch-screen technology to

progress monitoring, which together with advancements in assessment and intervention

highlights the ongoing integration of health and technology for the improvement of health

services.

General conclusions_____________________________________________________

There is a very clear need for the extended application of monitoring and feedback

methodologies in mental healthcare. The current thesis outlined the level of need for

interventions to complement evidence-based practice, and illustrated the implementation,

validation, and evaluation of a wellbeing feedback program that significantly improved

outcomes for poor responders in psychiatric care. Feedback was effective in improving patient

outcomes for depressive symptoms, but further work is required to determine the effects on

wellbeing, and how feedback is processed and experienced by patients and clinicians. The

current findings justify the application and rigorous evaluation of monitoring systems in other

clinical settings.

Feedback to patients and clinicians appears to improve the therapeutic process for those

unlikely to improve as expected. This process is not only useful at a treatment level, but reflects

121

an important course of action, influential at many levels of mental healthcare. Quality

improvement developments highlight a cycle of feedback for patients and clinicians, but also to

the organization, to policy advisors and administrators, the scientific community, and the public.

An ongoing communication between all interest groups should complement healthcare

improvement and inspire future research questions. In doing so, the patient may be empowered

in their experience of mental healthcare, and this should continue to be the goal for

psychotherapy research.

122

References

Atienza, A. A., Bradford, W. H., Baker, T. B., Abrams, D. B., Rimer, B. K., Croyle, R. T., et al.

(2007). Critical issues in eHealth research. American Journal of Preventive Medicine,

32, S71-S74.

Australian Bureau of Statistics. (2008). Household Use of Information Technology, Australia,

2007-08: Cat. No. 8146.0.

Australian Mental Health Outcomes and Classification Network. (2005). Reporting Framework

for the National Outcomes and Casemix Collection, Version 1.0. Brisbane, Queensland:

Australian Government Department of Health and Ageing.

Bech, P., Gudex, C., & Johansen, K. S. (1996). The WHO (Ten) Well-being Index: Validation

in Diabetes. Psychotherapy and Psychosomatics, 65, 183-190.

Bernard, H., Burlingame, G. M., Flores, P., Greene, L., Joyce, A., Kobos, J. C., et al. (2008).

Clinical practice guidelines for group psychotherapy. International Journal of Group

Psychotherapy, 58, 455-542.

Bhanoo, S. N. (2009). New tool in the MD's bag: A smartphone. The Washington Post, p. 6,

Boisvert, C. M., & Faust, D. (2003). Leading researchers' consensus on psychotherapy research

findings: Implications for the teaching and conduct of psychotherapy. Professional

Psychology: Research and Practice, 34, 508-513.

Claiborn, C. D., & Goodyear, R. K. (2005). Feedback in psychotherapy. Journal of Clinical


Daniels, B. A., Kirkby, K. C., Hay, D. A., Mowry, B. J., & Jones, I. H. (1998). Predictability of

rehospitalisation over 5 years for schizophrenia, bipolar disorder and depression.

Australian and New Zealand Journal of Psychiatry, 32, 281-286.

Davies, D. R., Burlingame, G. M., Johnson, J. E., Gleave, R. L., & Barlow, S. H. (2008). The

effects of a feedback intervention on group process and outcome. Group Dynamics:

Theory, Research, and Practice, 12, 141-154.

123

Eager, K., Trauer, T., & Mellsop, G. (2005). The performance of routine outcome measures in

adult mental health care. Australian and New Zealand Journal of Psychiatry, 39(728-

731).

Griffiths, K., Farrer, L., & Christensen, H. (2007). Clickety-click: E-mental health train on

track. Australasian Psychiatry, 15, 100-108.

Grizenko, N., Poinsier, B., Mercier, C., Perreault, M., & Latimer, E. (2000). Developing a

patient care monitoring system. [Letter to the Editor]. The Canadian Journal of


Grol, R., & Grimshaw, J. (2003). From best evidence to best practice: Effective implementation

of change in patients' care. Lancet, 362, 1225-1230.

Hageman, W. J., & Arrindell, W. A. (1999). Enhancing clinically significant change: Increment

of precision between individual and group level of analysis. Behaviour Research and

Therapy, 37, 1169-1193.


effect and its implications for treatment delivery services. Clinical Psychology: Science

and Practice, 9, 329-343.


(2007). Enhancing outcome for potential treatment failures: Therapist/client feedback

and clinical support tools. Psychotherapy Research, 17(4), 379-392.



Research, 14, 308-327.


psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist,

51, 1059-1064.

Hsu, L. (1989). Reliable changes in psychotherapy: Taking into account regression toward the

mean. Behavioral Assessment, 11, 459-467.

Hunter, E., Travers, H., & McCulloch, B. (2003). Bridging the information gap: IT and health in

Indigenous populations. Australasian Psychiatry, 11(Supplement), S51-S56.

124




Kobak, K. A., Taylor, L. V. H., Dottle, S. L., Greist, J. H., Jefferson, J. W., Burroughs, D., et al.

(1997). A computer-administered telephone interview to identify mental disorders.

Journal of the American Medical Association, 278(905-910).

Koesters, M., Burlingame, G. M., Nachtigall, C., & Strauss, B. (2006). A meta-analytic review

of the effectiveness of inpatient group psychotherapy. Group Dynamics: Theory,

Research, and Practice, 10, 146-163.


aimed at improving psychotherapy outcome in routine care. Psychotherapy Research,

17(1), 1-14.

Lambert, M. J., Hansen, N. B., & Bauer, S. (2008). Assessing the clinical significance of

outcome results. In A. M. Nezu & C. M. Nezu (Eds.), Evidence-based Outcome

Research: A Practical Guide to Conducting Randomized Controlled Trials for

Psychosocial Interventions. New York: Oxford University Press.

Lambert, M. J., & Ogles, B. M. (2004). The efficacy and effectiveness of psychotherapy. In M.

J. Lambert (Ed.), Bergin and Garfield's Handbook of Psychotherapy and Behaviour

Change (5 ed., pp. 139-193). New York: Wiley.


J. (2001). The effects of providing therapists with feedback on patient progress during

psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49-68.




Lutz, W., Lambert, M. J., Harmon, S. C., Stultz, N., Tschitsaz, A., & Schurch, E. (2006). The

probability of treatment success, failure and duration - What can be learned from

empirical data to support decision making in clinical practice? Clinical Psychology and

Psychotherapy, 13, 223-232.

125

Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., et al. (2005).

Predicting change for individual psychotherapy clients on the basis of their nearest

neighbors. Journal of Consulting and Clinical Psychology, 73(5), 904-913.

MacFarland, W. L., & Morris, S. J. (1998). Are dysphoric individuals more suggestible or less

suggestible that nondysphoric individuals? Journal of Counseling Psychology, 45, 225-

229.

Marks, I. (2004). Psychiatry in the future: Information technology can pull mental health care

into the 21st century. Psychiatric Bulletin, 28, 319-320.



Mathews, A., Ridgeway, V., & Williamson, D. A. (1996). Evidence for attention to threatening

stimuli in depression. Behavior Research and Therapy, 34, 695-705.

McLaren, P. (2005). Information and communications technology in mental health care. Journal

of the Royal Society of Medicine, 98, 232-234.

Morris-Yates, A., & Page, A. C. (in press). Routine measurement of outcomes by Australian

private hospital–based psychiatric services. In T. Trauer (Ed.), Outcome measurement

in mental health. Cambridge: Cambridge University Press.






334.






126



O'Kearney, R., Kang, K., Christensen, H., & Griffiths, K. (2009). A controlled trial of a school-

based internet program for reducing depressive symptoms in adolescent girls.

Depression and Anxiety, 26, 65-72.

Page, A. C., & Hooke, G. R. (2009). Effects of increased inpatient group psychotherapy on

outcomes, readmission rates, and satisfaction levels. Psychiatric Services, 60, 426-428.




Pirkis, J., Burgess, P., Kirk, P., Dodson, S., & Coombs, T. (2005). Review of standardised

measures used in the National Outcomes and Casemix Collection. Version 1.2.

Canberra: National Mental Health Strategy.

Sapyta, J., Reimer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research and

practice. Journal of Clinical Psychology, 61, 145-153.

Speer, D. C. (1992). Clinically significant change: Jacobson & Truax (1991) revisited. Journal

of Consulting and Clinical Psychology, 60, 402-408.



Services.

Strauss, B., Burlingame, G. M., & Bormann, B. (2008). Using the CORE-R battery in group

psychotherapy. Journal of Clinical Psychology: In Session, 64, 1225-1237.

Stritzke, W. G. K., & Page, A. C. (2009). Electronic patient monitoring in mental health

services. In A. Dwivedi (Ed.), Handbook of Research on Information Technology

Management and Clinical Data Administration in Healthcare (pp. 87-103). Hershey,

New York: IGI Publishing.

Strong, S. R., & Matross, R. P. (1973). Change processes in counseling and psychotherapy.

Journal of Counseling Psychology, 20, 25-37.

127

Tasca, G. A., & Gallop, R. (2009). Multilevel modeling of longitudinal data for psychotherapy

researchers: I. The basics. Psychotherapy Research, online, i1-9.

Titov, N., Andrews, G., & Schwencke, G. (2008). Shyness 2: Treating social phobia online:

Replication and extension. Australian and New Zealand Journal of Psychiatry, 42, 595-

605.

Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E.

J. (2003). Improving the effects of psychotherapy: The use of early identification of

treatment failure and problem solving strategies in routine practice. Journal of

Counseling Psychology, 58, 59-68.

Wing, J. K., Lelliott, P., & Beevor, A. S. (2000). Progress on HoNOS. British Journal of
