Introduction to Outcomes Informed Care. The content for this course is offered by A Collaborative Outcomes Resource Network (ACORN)

Introduction to

Outcomes Informed Care

The content for this course is offered by A Collaborative Outcomes Resource Network (ACORN).

https://www.psychoutcomes.org ACORN is a non-profit organization promoting Outcomes

Informed Care in behavioral health care and related fields.

TWiki site provides information, fosters collaboration, and offers support for organizations launching and nurturing outcomes-informed care initiatives.

The organization is supported by health plans, managed care companies, and employers seeking to improve treatment outcomes in mental health care.

Course Content

1. Required and Recommended Readings2. What is “Outcome Informed Care?”3. Background Research4. Measurement Concepts5. Clinician’s Use of Feedback6. Financial Implications

1. Required Readings

a. Outcomes Informed Care: An overviewb. Outcomes Measurement 2.0: Emerging

technologies for managing treatment outcomes in behavioral healthcare

c. ACORN Criteria for Effectiveness (ACE) Treatment Outcomes Evaluation Method

d. Outcomes Measurement: Concepts and definitions

1. Recommended Reading

This is an up-to-date review of what really makes a difference in psychotherapy outcomesDuncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2nd ed.). Washington, DC: American Psychological Association

2. What is “Outcomes Informed Care?”

Outcomes Informed Care is the philosophy and method of providing treatment and care that is informed by outcomes reported by patients.

2. What is “Outcomes Informed Care?”

Outcomes Informed Care is done by: Collecting the patient’s clinical symptoms

and perception of therapeutic alliance throughout the course of treatment using questionnaires.

Questionnaires are administered frequently so as to actively inform the patient’s course of treatment.

Clinicians are provided continuous feedback on these outcomes to actively improve treatment.

3. Background Research

a. The “Dodo Bird Effect”b. Clinician effectsc. The relationship between the patient and

cliniciand. Clinician’s use of feedbacke. Outcomes benchmarking

The “Dodo Bird Effect” Taken from the article by Rosenzweig S.

(1936) titled, “Some implicit common factors in diverse methods of psychotherapy: At last the Dodo said, ‘Everybody has won and all must have prizes.’”(Am J Orthopsychiatry 6:412-5)

Refers to the conjecture that all psychotherapies intended to be effective are roughly equivalent in their treatment outcomes due to factors that are common to all psychotherapies

3. Background Researcha. The “Dodo Bird Effect”

Common factors Three decades of meta-analytic studies have

served to reinforce Rosenzweig’s (1936) observation (e.g., Wampold et al., 1997)

Lack of evidence for specific treatment effects bolster the argument that almost all of the effects of psychotherapy are due to factors common to all psychotherapies

No evidence that effects of treatments have increased over the past three decades in psychotherapy and pharmacotherapy research

3. Background Researcha. The “Dodo Bird Effect”

Meta-analyses of psychotherapy outcomes reveal that the treatment effect due to the clinician are larger than the techniques used in the treatment Recent analyses show that who prescribes

the medication also affects treatment outcomes in pharmacotherapy outcomes

Other recent analyses suggest that the psychotherapist even effects the effect of medication in treatments that combine both psychotherapy and pharmacotherapy!

3. Background Researchb. Clinician Effects

Relationship building is an Evidence Based Practice!

“Practitioners are encouraged to routinely monitor patients’ responses to the therapy relationship and ongoing treatment. Such monitoring leads to increased opportunities to repair alliance ruptures, improve the relationship, modify technical strategies, and avoid premature termination.”

—John Norcross & Michael Lambert (2006) from “The Therapy Relationship” in Norcross, Beutler, & Levant (Eds.), Evidence-Based Practices in Mental Health, p. 218

3. Background Researchc. Relationship Between The Patient and Clinician

Practice implications for relationship building Listen to client Privilege the client’s experience Request feedback on the therapy relationship Avoid critical or pejorative comments Ask what has been most helpful in this

therapy—John Norcross (2009) from “The Therapeutic Relationship” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2nd ed.), pp. 116-117

3. Background Researchc. Relationship Between The Patient and Clinician

Research indicates that routine use of questionnaires provides clinicians ability to identify “at risk” cases and prevent premature termination Statistical algorithms compare actual patient

improvement against “expected” improvement based on large normative samples and provide this information to clinicians

Strong evidence from controlled studies and real world applications that patients benefit

3. Background Researchd. Clinician’s Use of Feedback

“Yes, it is time for clinicians to routinely monitor treatment outcome”“The use of outcomes management systems is ushering in a significant change in how psychotherapy is conducted. This review underscores the value of monitoring treatment response, applying statistical algorithms for identifying problematic cases, providing timely feedback to therapists (and clients), and providing therapists with problem-solving strategies. It is becoming clear that such procedures are well substantiated, not just matters for debate or equivocation. When implemented, these procedures enhance client outcome and improve quality of care.”

—Michael Lambert (2009) from “Yes It Is Time for Clinicians to Routinely Monitor Treatment Outcomes” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2nd ed.), p. 259.

3. Background Researchd. Clinician’s Use of Feedback

Benchmarking refers to the practice of comparing one set of outcomes to a criterion (the benchmark) Commonly, comparison of effect sizes to

“benchmarks” obtained from meta-analyses of clinical trials

An individual clinician’s average effect size could also be compared against effect sizes of other clinicians in a large normative sample

3. Background Researche. Outcomes Benchmarking

4. Measurement Concepts

a. Measurement 2.0b. Questionnaire development

i. Global distress factorii. Therapeutic allianceiii. Reliability & validityiv. Item response theory

c. Effect sized. Analysis

i. Case mix adjustmentii. Inadequacies of analyses used in clinical trialsiii. Hierarchical linear modeling

Measurement 1.0 Reliance on copyrighted

and published questionnaires

Copyright holder may charge fees for the use of questionnaires

Copyright holder may place conditions or restrictions on the use of questionnaires

Measurement 2.0 Item banks and

questionnaires that use them belong to the community of users

No fees for questionnaires constructed from items in the shared item bank

Each organization is responsible for determining the appropriate content (items) and implementation of questionnaires

General differences:

4. Measurement Conceptsa. Measurement 2.0

Measurement 1.0 A pool of items are tested in various

samples Item analysis used to select items for

final questionnaire Questionnaire is usually validated

through factor analysis and studies correlating it with other questionnaires measuring the same construct

Questionnaire published in final and unchangeable form

Manual for questionnaire published in paper form

Many years may pass before a new version is published

Measurement 2.0 A pool of items are tested in various

samples Item analysis used to select items;

multiple versions of the questionnaires are created by combining a set of items that reflect the needs of the users

Questionnaire validated by factor analysis and correlational studies

Questionnaires are constantly updated as data accumulate and needs of users change

Online manual constantly updated as data accumulate and needs of users change

New version is immediately available if the other versions do not capture the unique needs of the users

Differences in questionnaire development:

4. Measurement Conceptsa. Measurement 2.0

4. Measurement Conceptsb. Questionnaire Development

i. What is “global distress?” Most measures commonly used in mental

health research correlate strongly with a common factor, commonly called the “global distress factor”

Global distress includes items measuring: Symptoms of depression and anxiety Attention and concentration problems Family and interpersonal relations Work place productivity and functionality


ii. Therapeutic alliance Measuring the quality of the therapeutic

relationship between the patient and the clinician during the session

Three Components: Task: Behaviors and processes within the therapy

session that constitute the actual work of therapy Bond: The positive interpersonal attachment

between therapist and client of mutual trust, confidence, and acceptance

Goal: Objectives of therapy that both client and therapist endorse


ii. Therapeutic alliance Sample items:

I felt like we talked about the things that were important to me

I felt like the therapist liked and understood me I felt the session was helpful I felt confident that the therapist and I worked

well together Did you feel that the clinician understood what it

was like to be you?


iii. Reliability and validity Reliability

It asks, “how consistent is this questionnaire?” Cronbach’s coefficient alpha (α) estimates how

consistent the items within this questionnaire is α =.9 or higher preferred for measures

Validity It asks, “does the questionnaire measure what it is

supposed to measure?” Measures of global distress will correlate highly with

other similar measures (construct validity) Items are representative of what is supposed to be

measured (content validity)


iv. Item Response Theory (IRT) Evaluates how clients at varying

levels of distress reply to each items Identifies items that are endorsed

by clients with high levels of distress E.g., thoughts of suicide; worthlessness

Questionnaires can be uniquely tailored to the users or setting

Questionnaires for outpatient treatments are appropriately calibrated for patients with moderate to severe symptoms

Sample ACORN Questionnaire:


Global distress

Suicidal IdeationSubstance abuse

ProductivityGlobal distress

Alliance

Questionnaires faxed to data center for cost effective scoring, analysis & data warehousing

What is effect size? Effect size is simply the magnitude of the

treatment effect Of many effect size units, Cohen’s d is very

informative for treatments An effect size of d= 1 means that the client improved

one standard deviation on the outcome measure On a global distress scale, an effect size of d= 0.8 or

higher is considered large Meta-analyses of large samples of psychotherapy

studies suggest the effect size for psychotherapy using a global distress scale is approximately d= 0.8

4. Measurement Conceptsc. Effect Size

4. Measurement Conceptsd. Analysis

i. Case mix adjustment Statistical adjustments are used to control for

case mix differences (severity adjusted effect size)

General linear model (GLM) is commonly used GLM use both categorical (e.g., diagnosis) and

continuous (e.g., severity) variables in predictive model Predictive model employs clinically relevant

variables collected at the start of treatment to predict improvement during treatment

Initial severity, as measured by the intake score, is the strongest predictor

Other predictors: diagnosis, treatment history, health


ii. Inadequacies of analyses used in clinical trials

Clinical trials typically: Randomly assign patients to treatment conditions Conduct analysis of variance (ANOVA) to compare

among different treatment conditions to see whether or not treatment results are “statistically significant”

Analysis assumes that the clinicians are all the same ANOVA as commonly employed in clinical trials

cannot be used in assessing real-world treatment effects because there are clear differences among clinicians in their average treatment outcome


iii. Hierarchical linear modeling (HLM) Clinicians are treated as a factor that

influences outcomes Patients are “nested” within clinician Estimates percentage of variance due to clinician

Use of HLM becomes a necessity because we cannot ignore the effects due to the individual clinicians

Ignoring effects due to clinicians can lead to erroneously concluding that the type of treatment makes a difference in outcomes

5. Clinician’s Use of Feedback

Monitoring trajectory of change:

Graphing change permits clinician to quickly identify cases that are “off track”

Expected change based on normative sample

Signal score: patient “off track” and at risk for poor outcome

Patient’s actual scores: probability of good outcome remains high if patient in the “clinical” range remains engaged in treatment.

Clinical Boundary: Scores above this line are in a clinical range


Monitoring effect size: Data warehouse and web based tool

permits health plans and managed care companies to monitor effect size

Many health plans and managed care companies permit clinicians to access their data and monitor their own outcomes

Outcomes informed clinicians can use data to market their services


Monitoring effect size:

Severity Adjusted Effect Size: d = 0.5~0.8 (effective) d > 0.8 (highly effective)


Therapeutic alliance scale Items are heavily skewed in positive direction

I.e., clients report “perfect” or “near perfect” therapeutic alliance with their therapist

Scale scores are not normally distributed Cannot calculate reliability & validity using

parametric statistics that assume normality of distribution

Items are only as “valid” as clinician’s ability to illicit honest and frank responses!

0

0.2

0.4

0.6

0.8

1

1.2

Alliance items completedat start of treatment

(n=1924)

No items alliance at startof treament (n=1192)

Eff

ect

Siz

e Effective Range

Highly effective range


Therapeutic alliance scale results Alliance analyses performed by Jeb Brown,

PhD using data from the ACORN data repository

0.00

0.20

0.40

0.60

0.80

1.00

1.20

Alliance Change forWorse

No Change Alliance Change forBetter

Eff

ect

Siz

e Effective Range

Highly effective range


Therapeutic alliance scale results Measuring alliance clearly makes a

difference

6. Financial Implications

Clinicians vary in their “value” Value Index estimates effect size per $1000

Effective clinicians produce greater “return on investment” of employers Increased productivity gains for treated employees Reduced medical costs

Effective clinicians enhance the “value” of medications

Clinicians with good outcome data could have competitive advantage in attracting business

6. Financial Implications

Certificate of Effectiveness Similar to concept of “Certified Organic” Process

Data analyzed by independent party Applies agreed upon criteria, including minimum

effect size and sample size Purpose

Increase customer confidence of “value” Enable clinicians to demonstrate effectiveness to

referral sources such as employers, health plans, and HMOs

Empower clinicians to compete for business and negotiate contract based on demonstrated “value”

Psychotherapy Works. It Works!

“Monitoring combined with feedback is a simple method, divorced of theoretical baggage, for providing accountability. The results are apparent to all who have an interest in the outcome: therapists, consumers, administrators, and payers. Accepting the premise that therapeutic factors constitute the engine of change, then monitoring and feedback offers the means to deliver them. Many are anxious about the future and deservedly so. At the same time, the profession has the opportunity to establish itself in its own right. Psychotherapy works. It works. Therapists now have the ability to show it and the means to banish the despair in the workforce. The challenge is to put it into practice.”

—Barry Duncan, Scott Miller, Bruce Wampold, & Mark Hubble (2009) from “Introduction” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2nd ed.), p. 40

Documents

Introduction to Outcomes Informed Care. The content for this course is offered by A Collaborative Outcomes Resource Network (ACORN)