VA HSR&D Center for the Study of Healthcare Provider Behavior Maren T. Scheuner, MD, MPH, FACMG Chief, Medical Genetics VA Greater Los Angeles Healthcare

VA HSR&D Center for the Study of Healthcare Provider Behavior

Maren T. Scheuner, MD, MPH, FACMGMaren T. Scheuner, MD, MPH, FACMGChief, Medical GeneticsChief, Medical Genetics

VA Greater Los Angeles Healthcare SystemVA Greater Los Angeles Healthcare SystemDirector, Clinical Genetic Services, VISN 22Director, Clinical Genetic Services, VISN 22

Dept of Medicine, David Geffen School of Medicine at UCLADept of Medicine, David Geffen School of Medicine at UCLA

[email protected]@va.gov

Evaluation of Genetic Education Evaluation of Genetic Education

Programs for Healthcare ProvidersPrograms for Healthcare Providers


DisclosuresDisclosures

No conflicts of interest relating to the content of this presentation.

Funding source for content:

• CDC Office of Public Health Genomics

• VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior


33

OverviewOverview

Educating healthcare providersEducating healthcare providers

Evaluating success of an education programEvaluating success of an education program

– Formative evaluationFormative evaluation

– Summative evaluationSummative evaluation

– Use of theoryUse of theory

Case studyCase study


What is the goal of healthcare What is the goal of healthcare provider education?provider education?


Behavior change

Improved health outcomes

What is the goal of healthcare What is the goal of healthcare provider education?provider education?


Is education (didactic teaching) Is education (didactic teaching) sufficient to change healthcare sufficient to change healthcare

provider behavior?provider behavior?


Generally, not.

Oxman et al., 1995; Mazmanian and Davis, 2002; Mansouri and Lockyer, 2007

Is education (didactic teaching) Is education (didactic teaching) sufficient to change healthcare sufficient to change healthcare

provider behavior?provider behavior?


Informational interventions

Clinical interventions

Behavioral interventions

Components of an effective Components of an effective

education programeducation program

Continuing medical education objectives as defined by Mazmanian and Davis, 2002.


Evaluation of Genetics Evaluation of Genetics Education ProgramsEducation Programs


How do we measure success?

Measures of implementation success

Implementation strategies

Clinical Innovation

Process Outcomes

HealthOutcomes

Intervention

Other factors affecting progress and success

Adapted from: Luska CV, Hall C. Challenges in measuring implementation success. 3rd Annual NIH Conference on the Science of Implementation and Dissemination. Methods and Measurement. March 15-16, 2010, Bethesda, MD


Implementation DefinedImplementation DefinedEfforts designed to get evidence-based practices and related products into use

Implementation typically follows dissemination and includes:– Identifying barriers, facilitators and strategies

to reduce, overcome, leverage them– Adapting the targeted practice to the context– Developing a tailored implementation strategy


Implementation ResearchImplementation Research

‘the scientific study of methods to promote the uptake of research findings for the purpose of improving quality of care” McDonald et al., 2004. Toward a Theoretical Basis for Quality Improvement

Interventions in K.G. Shojania et al., Closing the Quality Gap.

“…scientific investigations that support movement of evidence-based, effective health care approaches (e.g., as embodied in guidelines) from the clinical knowledge base into routine use.” Rubenstein & Pugh, 2006.


Types of EvaluationTypes of Evaluation

Formative evaluation– Rigorous assessment process designed to

identify potential and actual influences on the progress and effectiveness of implementation efforts

Summative (impact) evaluation– Systematic process of collecting and analyzing

data on impacts, outputs, products, outcomes and costs in an implementation study

Stettler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 2006;21(Suppl 2):S1-8.


Summative Evaluation in Summative Evaluation in Implementation ResearchImplementation Research

Outcomes Assessment– A priori measures defined at outset of project to

assess intervention impact or effectiveness– Defined at patient-, provider-, clinic-, facility-, and/or

system-level– Involves use of administrative data, chart review,

and/or primary data collection

Cost Assessment– Assess costs associated with implementation strategy

to inform decision makers on value and feasibility of implementing the intervention


Four Stages of Formative EvaluationFour Stages of Formative Evaluation

Developmental

Implementation-focused

Progress-focused

Interpretive


Formative evaluation according to implementationFormative evaluation according to implementation

1. Developmental

Identify determinants of current practice

Identify barriers and facilitators

Assess feasibility of proposed intervention

Integrate findings into intervention design, and refinement prior to implementation

2. Implementation-focused

Assess discrepancies between implementation plan and execution, exploring issues of fidelity, intensity, exposure

Understand and document nature and implications of local adaptation

3. Progress-focused

Monitor impacts and indicators of progress toward project goals

Use data to inform need for modifying original strategy

Provide positive reinforcement to high performers; negative reinforcement to low performers

4. Interpretive

Assess intervention usefulness/value from stakeholders perspectives

Elicit stakeholder recommendations for further intervention refinements

Assess satisfaction with intervention and implementation process

Identify additional barriers / facilitators

Pre-Implementation Implementation Post-Implementation


Developmental




Integrate findings into intervention design and refinement prior to implementation

Implementation-focused



Progress-focused

Monitor impacts and indicators of progress toward project goals

Use data to inform need for modifying original strategy

Provide positive reinforcement to high performers; negative reinforcement to low performers

Interpretive

Assess intervention usefulness/value from stakeholders perspectives




Pre-Implementation Implementation Post-Implementation

Formative evaluation according to implementationFormative evaluation according to implementation


Need for Formative Evaluation in Need for Formative Evaluation in Implementation ResearchImplementation Research

Captures information on factors that hinder or facilitate successful implementation

Addresses interpretive weaknesses– Avoid “implementation assessment failure”– Avoid explanation and outcome attribution

failure– Enhance understanding of study outcomes


Assessment Methods / Tools for Assessment Methods / Tools for Formative EvaluationFormative Evaluation

Quantitative– Structured surveys / tools– Instruments assessing organizational culture, readiness to change,

provider receptivity to evidence-based practices– Intervention fidelity measures– Audit / feedback of clinical performance data

Qualitative– Semi-structured interviews with clinical stakeholders (pre-/post-)– Focus groups– Direct observation of clinical structure and processes in site visits– Document review

Mixed methods (i.e., quantitative + qualitative)


Usefulness of TheoryUsefulness of Theory

In terms of…

– Planning the implementation strategy

– Conducting evaluations

– Identifying unanticipated elements critical to successful implementation, but may be unexplained by selected theory

– Gaining additional insights about the theory

– Helping to understand findings, including relationships between domains or constructs


Types of TheoriesTypes of Theories

Multiple theories often needed

Explanatory theories (aka descriptive, impact)

– Hypotheses and assumptions about how implementation activities will facilitate a desired change as well as the facilitators and barriers for success

Process theories (aka prescriptive, planned action)

– How implementation should be planned, organized and scheduled

Mixed theories– Elements of both


Choosing TheoryChoosing TheoryConsider nature of the theory– Process vs. explanatory– Context (e.g., policy, organization)– Discipline (e.g., social science, psychology)

Consider level at which it will be applied– Individuals– Teams– Organization– System

Consider previous findings, experience

Consider greatest potential for adding to the knowledge-base


2323


RE-AIM Evaluation FrameworkRE-AIM Evaluation Framework

Purposes

– Broaden and standardize criteria used to evaluate clinical QI programs

– Evaluate issues relevant to program adoptions, implementation and sustainability

– Help close the gap between research studies and practice

Glasgow et al., Ann Behav Med 2004;27(1):3-12.


RE-AIM to help plan, evaluate and RE-AIM to help plan, evaluate and report studiesreport studies

R Increase Reach

E Increase Effectiveness

A Increase Adoption

I Increase Implementation

M Increase Maintenance(Sustainability)

www.re-aim.org


SummarySummaryGenetics education is necessary but not sufficient for change in healthcare provider behavior

Effective education programs are multi-faceted, including: informational, clinical and behavioral interventions.

Measuring success of education programs:

– Implementation success

– Process outcomes

– Health outcomes


SummarySummaryFormative evaluation can be key to effective intervention design and success, interpretation and replication of results

Summative evaluation is used to assess relevant clinical outcomes and costs associated with implementation of evidence-based practices

Formative and summative evaluation are complementary in developing, implementing, evaluating and refining implementation interventions

Important to use theory in planning and evaluation of implementation research/QI, and evaluate theory


Case StudyCase Study


CDC OPHG Translation ProgramsCDC OPHG Translation Programs

Goal: Promote evidence-based clinical and public health practice in genomics

Focus: Genetic testing applications with evidence of clinical utility (e.g., CDC EGAPP reviews, USPSTF recommendations)

Supported activities: education, policy, surveillance


““Family History Education to Improve Risk Family History Education to Improve Risk

Assessment for Hereditary Cancer”Assessment for Hereditary Cancer”

Translating Clinical Guidelines for Translating Clinical Guidelines for

Family History Risk Assessment into PracticeFamily History Risk Assessment into Practice


GoalGoal

To develop a multi-component

education program for primary care

clinicians that improves recognition

and referral of patients at risk for

hereditary cancer syndromes.


PopulationFamilial Risk Stratification

Genetic Consultation for Hereditary Cancer

Morbidity & Mortality

Strategies to Increase Genetics Referrals

Strategies to Increase

Familial Risk Assessment

Strategies to Increase Risk-

Appropriate Interventions

Interventions

Outcomes

Logic Model


Formative Evaluation - Developmental workFormative Evaluation - Developmental work




Integrate findings into intervention design, and refinement prior to implementation

Pre-implementation PhasePre-implementation Phase


3434

Priority Setting PanelPriority Setting Panel13 Experts13 Experts


3535

Availability of:Availability of:

– Genetics education for health professionalsGenetics education for health professionals

– Clinical decision support tools in the electronic Clinical decision support tools in the electronic health recordhealth record

– Prevention/management options available for Prevention/management options available for genetic conditionsgenetic conditions

– Clinical guidelinesClinical guidelines

Key Factors for Adoption of Key Factors for Adoption of Genomic Medicine at the VA Genomic Medicine at the VA


3636

Highest Priorities for Health Services Highest Priorities for Health Services Research at VA in the Next 5 YearsResearch at VA in the Next 5 Years

Genetics education Genetics education

Development of clinical guidelines Development of clinical guidelines

Development of tools in CPRS for:Development of tools in CPRS for:

– Familial risk assessment Familial risk assessment

– Ordering and interpreting genetic testsOrdering and interpreting genetic tests


3737

Family History Content Review in the Family History Content Review in the Electronic Health RecordElectronic Health Record


Our EHR lacks standards for family Our EHR lacks standards for family history documentation history documentation

1,416 shared templates for progress notes (Aug 11, 2007 - Aug 12, 2008)– Family history mentioned in 8%

– Disease checklist most common format, 46%

– Family history open text box, 38%

– List of first-degree relatives with text box, 14%

None captured information about specific diseases in specific relatives.


3939

Key Informant Interviews with Key Informant Interviews with Primary Care ProvidersPrimary Care Providers


4040

To Improve Process of To Improve Process of Family History DocumentationFamily History Documentation

PCPs want:

– Template in the electronic health record

– Better organization of the family history in the electronic health record

– Patient-provided data (through kiosk or personal health record)


High Ratings for Clinical Reminders

Stratify familial risk

Recognize inherited conditions

Prompt referrals for consultation or testing

Reasons for high ratings: – Lack of knowledge, familiarity and confidence in

genetic risk assessment, diagnosis and testing


First draft: Family History Red First draft: Family History Red Flags for Hereditary CancerFlags for Hereditary Cancer


Focus Group FeedbackFocus Group Feedback

Not useful

As primary care providers, we need to document complete family history

Once history is documented, we can recognize the red flags

Tool should have a few stem questions that can be completed quickly for most patients



Baseline Knowledge & Attitudes of Enrolled Clinicians

% Correct answers

Somewhat relevant, %

Relevant, %

Very relevant, %

Basic molecular genetics, concepts, terminology

77 50 25 25

Familial/genetic risk assessment 47 4 46 50

Recognizing hereditary cancer syndromes 45 0 50 50

Genetic testing 33 63 17 20

Management of hereditary cancers, including referral

58 6 13 81

Ethical issues for patients and clinicians 67 0 37 63


Implementation StrategyImplementation StrategyClinical interventions– Cancer family history reminder in EHR

– Self-administered, patient questionnaire

– Practice-feedback reports each quarter

Informational interventions:– 7-part CME lecture series on cancer genetics

– Information sheets for providers

– Information brochures for patients

– Web site

Behavior interventions– Review of family history generated by reminder with

feedback by opinion leaders


Implementation-focused:Implementation-focused:



Implementation Phase Implementation Phase


Progress-focused:Progress-focused: Monitor impacts and indicators of progress

toward project goals Use data to inform need for modifying original

strategy Provide positive reinforcement to high

performers; negative reinforcement to low performers

Implementation Phase Implementation Phase


Pre/Post design:Pre-implementation Oct - Dec 2009Post-implementation Apr 2010 - Jun 2011

Abstraction of random 10% of progress notes each month. Assessed change in documentation of:Cancer family historyReferral for genetic consultation

Monthly monitoring of health factors generated by cancer family history reminder, and quarterly practice-feedback reports for enrolled clinicians

Evaluation Plan:Evaluation Plan:


Completion of Cancer Family History ReminderCompletion of Cancer Family History ReminderApril 2010 - March 2011April 2010 - March 2011

3,548 patients seen with reminder due

– Avg, 413 per provider; range, 54 - 771

1,087 reminders completed when due

– Avg, 30% per provider; range, 23% - 98%

105 (10%) referred for genetic consult

– 54% of patients with a strong familial risk

– 14% of patients with a moderate familial risk

– 1% of those with a weak familial risk


Cancer Family History Documented in Progress NotesCancer Family History Documented in Progress Notes

0

10

20

30

40

50

60

70

(n=76) Q1 (n=101) Q2 (n=109) Q3 (n=112)

By text By Template Previously By Template

%

Post-implementationPre-implementation

28%

50%54%

60%


Quality of Cancer Family History Documentation Quality of Cancer Family History Documentation in Progress Notes with Cancer Family Historyin Progress Notes with Cancer Family History

Pre-implementation

(n=21)

Post-implementationa

(n=117)

1st degree relatives, % 76 81

2nd degree relatives, % 48 62

Lineage of relatives, % 14 62

Age of cancer onset, % 19 43

Jewish ancestry, % 0 45


Trends in Genetics Referral Generated by Template Trends in Genetics Referral Generated by Template

0

5

10

15

20

25

30

Apr May Jun Jul Aug Sep Oct Nov Dec

%

134 131 130 90 78 84

On average, a genetic consult was requested for 9% of patients and 3.5% declined a genetic consult.

Missed opportunities decreased over time.

82

Consult requested Patient declined consult Familial risk but consult “not indicated”

N = 75 82



Interviews with Primary Care ProvidersInterviews with Primary Care Providers

“My documentation of cancer family history has improved… I had a template I was using and it was limited to the colon, breast, uterine and ovarian cancer, so now it’s expanded because we have all those other options.”

“Now my documentation is very detailed, whereas before I would just mainly ask about mom and dad.”

It helps a lot. I had a couple of patients who never mentioned before that their grandmother died of ovarian cancer.”


“I probably wasn’t doing that in-depth of a family history before, especially not focused on cancer.”

“The template is much broader and more detailed than what I probably would have gotten before. I don’t know if I would have gone down to all those relatives…, and it certainly triggered a number of consultations in some people who probably deserved it a long time ago. So I think this has greatly improved my history-taking.”

Interviews with Primary Care ProvidersInterviews with Primary Care Providers


Interpretive:Interpretive: Assess intervention usefulness/value from

stakeholders perspectives




Post Implementation Phase Post Implementation Phase


Evaluation plan:Evaluation plan:

Semi-structured interviews ongoing.

Repeat knowledge and attitudes survey.

Look at theory used


ConclusionsConclusions

Our implementation strategy is working.Some interventions effective, others not so much.

Goals met:More comprehensive family history documentation

necessary for familial risk assessment. Improved recognition and referral of high-risk patients. High-risk patients utilizing genetic services.

Future studies: Summative evaluationDisseminate implementation strategyAssess utilization of preventive services, health

outcomes and costs according to use of reminder


SummarySummary

When developing education program, consider determinants of provider behavior

Select theoretical model to inform development, implementation and evaluation

Conduct formative evaluation in addition to summative evaluation


AcknowledgementsAcknowledgementsVA Greater Los AngelesVA Greater Los Angeles Advisory BoardAdvisory BoardElizabeth M. Yano, PhD, MSPH Deborah Duquette, MS, CGCLisa V. Rubenstein, MD, MSPH Michael Kelley, MDBrian Mittman, PhD Barbara Lerner, MS, CGCStuart Gilman, MD, MPH Douglas Olson, PhD, RNCaroline Goldzweig, MD, MPH Holly Peay, MS, CGCAlison Hamilton, PhD Ann Chou, PhDColletta Austin, RN, MSN Barbara Simon, MA CDC, OPHGMartin Lee, PhD Rodolfo Valdez, PhD, MScShannon Rhodes, PhD Shelley Reyes, PhD, MSNina Smith, MPH Mack Anders, MPAErin Schalles, MS, CGC Cecelia Bellcross, PhD,Heather ZmyewskiHeather Zmyewski MS Daurice Grossniklaus, PhDAdrie Young

Documents

VA HSR&D Center for the Study of Healthcare Provider Behavior Maren T. Scheuner, MD, MPH, FACMG Chief, Medical Genetics VA Greater Los Angeles Healthcare