Transcript
Page 1: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

From Concept to Implementation: The

Challenges Facing Evidence-Based Social Work

Faculty Research & Insights: A Series Featuring CUSSW Faculty Research

Edward Mullen & Aron ShlonskySeptember 22, 2004

Page 2: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Evidence-Based Social Work Is A Concept Awaiting Implementation

• Much discussed

• Required in EPAS

• Not clearly Defined

• Not yet implemented in pure form

“--- the conscientious, explicit and judicious use of current best evidence

in making decisions regarding the welfare of

service-users and carers” (Sheldon, 2003, p. 1).

“Placing the client’s benefits first, evidence-based practitioners adopt a process of lifelong learning that

involves continually posing specific questions of direct

practical importance to clients, searching objectively and

efficiently for the current best evidence relative to each question,

and taking appropriate action guided by evidence” (Gibbs et al.,

2003, p. 6).

“Evidence-based medicine has been defined as “--- the

conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual

patients” (Sackett, et al., 1996, p. 71); and the "integration of best research evidence with clinical expertise and patient values" (Sackett, Straus, Richardson,

Rosenberg, & Haynes, 2000, p. 1).

Page 3: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Origin of Evidence-Based Social Work

– Adaptation of evidence-based approaches found in medicine and healthcare

– Evolution from earlier attempts to join policy-practice-research

– Effectiveness research– Eclecticism– Empiricism & the scientist-practitioner model– Social intervention research– Outcomes measurement– Systematic review methodologies – meta-analysis

– Eileen Gambrill early translator (1999)– Leonard Gibb’s first workbook-text (2003)– Aaron Rosen & Enola Proctor on guidelines (2003)

Page 4: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

EBP

The EBP Model

BestEvidence

Client Values and Expectations

Practitioner’s IndividualExpertise

Sackett et al., 1997

Page 5: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Newest EBP Model

Clinical state and

circumstances

Clinical Expertise

Client Preferences and actions

ResearchEvidence

Haynes, Devereaux, and Guyatt, 2002

Page 6: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Steps of EBP Become Motivated to Apply EBP Step 1—Convert information need (prevention,

assessment, treatment, risk) into an answerable question.

Step 2—Track down current best evidence. Step 3—Critically appraise the evidence. Step 4—Integrate critical appraisal with practice

experience, client’s strengths, values, and circumstances.

Step 5—Evaluate effectiveness and efficiency in exercising steps 1-4 and seek ways to improve them next time.

Step 6—Teach others to follow the same process

Gibbs, 2003

Page 7: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

The Cycle of EBP

Clinical state and

circumstances

Client Preferences and

actions

Current Best

Evidence

Clinical Expertise

Actuarial risk

assessment

Contextual assessmentContextual

assessment

Other valid assessment measures?

Effective services

Appropriate for this client?

Barriers (e.g., cultural conflict

Client’s preferred course or at least

willing to try? Adapated from Haynes, Devereaux, and

Guyatt, 2002; as presented in Shlonsky and Wagner, in press.

Page 8: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Hallmarks of EBP

• Individualized assessment (strengths, needs, values, preferences of client)

• Well-formulated question (database query)

• Well-executed search of research literature

• Deciding its applicability to clients

• Considering the evidence together with the values and preferences of the client.

Sackett et al., 1997; Gibbs, 2003

Page 9: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Origins of Evidence-Based Healthcare & Public Policy

• Used to inform health care policy at hospital, regional, & national levels (Davies & Boruch, 2001; Romanow, 2002; Gray, 2001; Davies, Nutley, & Smith, 2000)

• Became component of New Labour’s modernizing government strategy - extended to other sectors including social care (Davies, 2004; Walker, 2001)

• Facilitated by:– Cochrane & Campbell Collaborations– UK ESRC’s evidence-based practice & policy centres– UK Social Care Institute for Excellence– Sweden Institute for Evidence-Based Social Work– Publications – e.g., Evidence-based Policy: A Journal of

Research, Debate and Practice

Page 10: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Policy Applications

• An approach to policy that:– helps people make well informed decisions

about policies, programs & projects

• EBPolicy contrasts with opinion-based policy– which relies heavily on either the selective

use of evidence (single studies irrespective of quality)

– untested views of individuals or groups

Philip Davies (2004), Director of Policy Evaluation in the Cabinet Office of the UK Prime Minister’s Strategy Unit

Page 11: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

What is evidence-based healthcare?

• Discipline centered on evidence-based decision-making about:

-Groups of patients-Populations

• Evidence from wide range of disciplines

• Use of scientific research findings and logic

– Healthcare problems

– Health improvement

– Decision-making in healthcare

Gray 2001

Page 12: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Evidence-Based Policy Definition

Decision-making process in which policy-makers,

purchasers, managers, or practitioners together with

key stakeholders make decisions about problem or need assessment, goals & objectives, interventions &

outcomes

Page 13: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

EBPolicy Process (Gray, 2001)

Finding &Appraising Evidence

CapacityBuilding

Getting Evidenceinto Practice

Page 14: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Sources of Evidence in EBP

AttitudinalStudies

EthicalAnalysis

EconomicAnalyses

StatisticalModeling

DescriptiveStudies

ImplementationEvaluation

ImpactEvaluation

Page 15: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

EBPolicy Steps• Finding and appraising evidence pertaining to the

decision at hand: – Assessment or intervention options– Intended outcomes– Research method

• Developing organizational capacity• Getting the evidence into practice

– Preparing a policy– Making cultural changes– Designing systems for implementation– Implementation– Monitoring– Audit

Page 16: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Examples of Evidence-based Policy Initiatives

• EVIDENCE AND POLICY: A JOURNAL OF RESEARCH, DEBATE AND PRACTICE

• ESRC Evidence Network - UK Centre for Evidence Based Policy & Practice

• U.K. Cabinet Office• University of St. Andrews• Romanow Commission – Canada• U.S. Implementing Evidence-based Practices

Project

Page 17: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Arguments For Evidence-Based Social Work (EBSW)

• Enhances decision quality• Fosters learning of assessment skills• Incorporates client values and expectations• Fosters evidence search and appraisal skills• Makes best use of best evidence• Framework for self-directed, life-long learning• Identifies gaps in knowledge• Common interdisciplinary language

Sackett et al, 2000

Page 18: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Arguments For Evidence-Based Social Work (EBSW)

• Eschews authoritarian practices and policies

• Promotes SW ethics through:

– Informed consent

– Using effective services

– Wisely allocating scarce resources

– Involving clients in practice process

– Individualizing & personalizing decisions

Gambrill, 2003

Page 19: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Arguments For Evidence-Based Social Work (EBSW)

– Helping clients develop critical appraisal skills– Involving clients in design and critique of

practice and policy related research– Involving clients as informed participants who

share in decision making– Recognizing client’s unique knowledge in terms

of application concerns• Promoting transparency and honesty• Encouraging a systemic approach for integrating

practical, ethical and evidentiary issues• Maximizing the flow of knowledge and information

about knowledge gapsGambrill, 2003

Page 20: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Challenges Facing EBSW• Evidence-of-effectiveness• Authority• Conflicting hierarchy• Definition of evidence• Shortage of evidence

– Variation by field of practice– Inflation of evidence– Small & transient effects– Assessment validity

• Nomothetic versus ideographic knowledge• Efficacy versus effectiveness• Diffusion, adoption and implementation• Policy & system level issues• Training & sustaining

Page 21: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Evidence-of-effectiveness Challenge

• A key assumption of EBP is that it results in better outcomes than other forms of practice– Yet, this assumption is not empirically testable:

• “This proof is no more achievable for the new paradigm (EBM) than it is for the old, for no long-term randomized trials of traditional and evidence-based medicine are likely to be carried out.” (Evidence-Based Medicine Working Group, 1992, 2424)

Page 22: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Authority Challenge

• As originally conceived the founders of evidence-based medicine envisioned that medical practitioners would seek out evidence & appraise the quality of that evidence, alone or in teams of practitioners

• “The new paradigm puts a much lower value on authority. The underlying belief is that physicians can gain the skills to make independent assessment of evidence and thus evaluate the credibility of opinions being offered by experts” (Evidence-Based Medicine Working Group, 1992)

Page 23: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Authority Challenge

• Lack of evidence that EBM can be transmitted into practice– Medical students have not shown an interest

in becoming “evidence-based practitioners” – only “evidence users”

• As evidence users rather than independent appraisers, practitioners are simply replacing one authority with another – researchers are the new authorities!

Page 24: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Authority Challenge

• To date it has not been shown that average practitioners can gain the skills needed to conduct independent assessments of the evidence

• Practitioners can become “evidence-users” not “evidence-based practitioners” say Upshur & Tracy

• “Evidence-users” accept evidence that has been vetted by others – thus replacing one authority for another! But what is the basis for the authority of the evidence appraisers?

Page 25: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Conflicting Hierarchy Challenge

• Hierarchy of Evidence Ranked by Quality– N of 1 randomized trials– Systematic reviews/Meta-analysis of

randomized trials– Randomized Controlled Trials– Systematic review of observational studies

addressing patient-important outcomes– Physiological/Laboratory experiments– Unsystematic clinical observation

Page 26: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Conflicting Hierarchy Challenge

• Practice recommendations (e.g., practice guidelines) are to be based on the soundness of the evidence

• Many evidence hierarchies have been published & these hierarchies are not commensurable (e.g., randomized, controlled trials and meta-analysis versus well designed epidemiological studies)

• Fundamentally these hierarchies do not rest on evidence but rather on the consensus of experts – the beliefs of the few!

Page 27: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Definition of Evidence Challenge

Because contemporary health care and human services are multidisciplinary there are many differing perspectives that require consideration

Accordingly, the concept of a univocal understanding of evidence is not sustainable

A restrictive view of evidence is not sustainable

Page 28: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Shortage of Evidence Challenge

• The question that faces proponents of EBP is whether there are enough high-quality studies so that evidence-based decisions can be made.

• Surprisingly for a field that places a high premium on research, few studies have examined this.

• The conclusion at this point, based on just a few studies, is that there are still many decisions that are made that are not based on good evidence, but the picture is not nearly as bleak as opponents to EBP would have us believe.

• Professionals must remember, though, that when they make decisions for which little or no evidence exists, that they should exercise caution and perhaps be even more vigilant in monitoring outcomes.

Page 29: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Shortage of Evidence

• Varies by field of practice

• Confounded by inflation of evidence by proponents and users

• Clouded by dominance of small and transient effects

• Applies to assessment evidence as well as intervention evidence

Page 30: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Shortage of Evidence Varies by Field of Practice

• Adequacy varies considerably by field of practice

• Most robust is mental health

• In other fields of practice much more of a patchwork

• Unevenness of the research base across fields of practice looms as a major challenge

Page 31: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Inflation of Evidence Challenge

• Bias in reported research due to:– Tendency for investigators or assistants to

give “a leg up ” to interventions they favour - “expectation for improvement”

– Dependence on client self-report to measure intervention outcomes which may reflect social desirability, expectancy, & cognitive dissonance effects

Page 32: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Small & Transient Effect Size Challenge

• The rule is small effect sizes which may not be clinically or social important

• Gains frequently do not persist, especially for complex and chronic problems such as addressed by social workers

Page 33: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Assessment Challenge

• Reliable, valid, & relevant assessment tools are needed to support evidence-based social work practice

• Rapid assessment measures have contributed greatly (Corcoran & Fisher, 2000)

• Social work practitioners generally do not use any standardized assessment procedures to guide their practice (Mullen & Bacon, 2004)

Page 34: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Shortage of Evidence Challenge

• Evidence-based social workers must remember that when they make decisions for which little or no evidence exists, that they should exercise caution and perhaps be even more vigilant in monitoring outcomes

• Knowledge gaps point the way to needed research• Methodological corrections should be instituted to deal

with the inflation of evidence issue• Where interventions are found to have small with

transient effects of limited importance this should be acknowledged and addressed

• The underutilization of assessment instruments should be corrected by enhance training in the use of appropriate instruments & dissemination of information about available instruments.

Page 35: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Challenge of Applying Results to Individuals

• Results of RCTs are analyzed by comparing the mean score of the experimental group against that of the placebo or control group (or some comparable summary statistic). This masks the fact that there is always individual variability around the means, & overlap in the distributions of scores for the two groups. The result of this is that a proportion of people in the experimental group actually do worse than some in the control group and, conversely, some in the comparison group improve more than some people in the active treatment group.

• Practitioners cannot blindly apply a “proven” procedure and assume that a particular individual receiving that procedure will benefit

Page 36: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Challenge of Applying Results to Individuals

• We are at least able to quantify the probability with which an individual person will respond to a given procedure (Number Needed to Treat)

• Alternative to using evidence-based interventions – with their known rate of failure – is to use unproven procedures, based only on the hope that they may work

• Practitioners can and should view each case or situation as an “N = 1” study and collect data

Page 37: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Efficacy Versus Effectiveness Challenge

• The efficacy versus effectiveness challenge requires a careful consideration of the trade off between internal and external validity

• Claims to evidence-based practice typically must be grounded in random, controlled trials which usually translate into efficacy studies

• Such studies do not address how effective such interventions would be in real world contexts

• At the present time there is a paucity of evidence of relevance to social work based on effectiveness studies conducted in real world contexts

• it may be more efficient to begin with studies based on conditions as realistic as possible so as to avoid the misinformation that too frequently comes from misinterpretations of efficacy studies

Page 38: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Adoption & Implementation Challenge

• Two major challenges:

(1) adoption & effective implementation of evidence-based practice in social agencies

(2) educating for evidence-based practice

Sisyphus

Page 39: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Training, Time, Resources Challenges

EBP requires:• Training in search

techniques• Training in critical

appraisal• Computer

resources• Electronic

resources

Page 40: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Killer B’s

• Low BASE rate

• Incompatible client or community BELIEFS

• BAD BARGAIN in terms of scarce resources

• BARRIERS too high

Page 41: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Adoption & Implementation Challenge

• Motivate core group• Acceptance of EBP• Facilitate adoption• Implementation of effective services• Evaluate outcomes

2.5% 13.5% 34% 34% 16%

Innovators Early Adopters

EarlyMajority

LateMajority

Laggards

Page 42: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Gira, Kessler and Poertner (2004)• Survey of reviews of

adoption of research evidence medical practice

• Educational outreach visits and audit and feedback showed weak to moderate effects

• Certain types of continuing education and the use of computers as decision aids showed moderate effects

• There are no ‘magic bullets’ (Oxman et al., 1995)

•“The literature from health care suggests that

disseminating information alone is insufficient. Many interventions have been

designed to improve practitioners’ adherence to

EBP guidelines and are differentially effective. To date, no intervention has demonstrated powerful

effects” (p. 77-78).

Page 43: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Adoption & Implementation Challenge

• Can social agencies adopt EBP approach?• Can social workers become EB practitioners? • What are the barriers? Facilitators?• What resources are needed?• What training is required?• What systems need to be established?• How will EBP change client experience?• What outcomes will be achieved?

Page 44: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Challenges in Teaching Evidence-based Social Work

• Fear of Loss of Human Context

• Disconnect between classroom and field

• Overwhelmed by new skills and knowledge needed

(Bilsker & Goldner, 2004)

Page 45: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

A Challenge to Critics

• Knowledge and information revolution

• Valuable resource• Cannot be ignored• Must be harnessed

Page 46: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

A Measured Optimism

• Real world constraints-limitation of approach-social and political realities

• Great promise, but need for caution-harnessing knowledge and information-management of scarce resources-unknowns abound

• Hold EBP to its own evidentiary standards-the jury is still out

Page 47: From Concept to Implementation: The Challenges Facing Evidence-Based Social Work

Knowledge needs to be managed much more effectively than it has been in the past

The challenge is to rethink our view of what knowledge is & how best

to facilitate its rapid generation, sharing, and application in a manner that closes the policy-research-practice gap


Recommended