Systematic Review Module 1: Refining Key Questions C. Michael White, PharmD, FCP, FCCP Professor and Director University of Connecticut / Hartford Hospital

Systematic Review Module 1: Systematic Review Module 1: Refining Key QuestionsRefining Key Questions

C. Michael White, PharmD, FCP, FCCPC. Michael White, PharmD, FCP, FCCPProfessor and DirectorProfessor and Director

University of Connecticut / Hartford HospitalUniversity of Connecticut / Hartford HospitalEvidence-based Practice CenterEvidence-based Practice Center

Speaker has no actual or potential conflicts of interest in relation to this activity

2

Learning ObjectivesLearning Objectives At the conclusion of this activity, the

successful learner will be able to– Describe what the Effective Healthcare

Program is– Describe how topics are selected for key

question refinement– Identify the importance of key questions– Describe the role of key informants in topic

refinement– Describe a longitudinal and transparent topic

refinement process

3

Effective Healthcare Program Principles

The work of the EHC program is intended to be– Relevant and timely

– Objective and scientifically rigorous

– Transparent with public participation

Whitlock EP, Lopez SA, Chang S, et al. Identifying, selecting, and refining topics. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

4

Topic Refinement in the CER Process

Develop topic:

Identify and triage topics

based on appropriateness,

importance, desirability of new research/duplication,

feasibility, and potential impact.

Refine topic:

Identify patient, intervention, comparator,

outcomes, timing and setting for

each topic.

Review topic:

Prepare topic, search for and select studies, abstract data, analyze and

synthesize data, present findings.

Engage stakeholders in clarifying areas

for future research:

Identify future research needs to inform real-world

healthcare decisions

5

Topic RefinementTopic Refinement

Topics for refinement will be – Appropriate, high priority, feasible, not

duplicative, and of high potential value

A key feature of topic refinement is the formulation of key questions

Key questions are objective demonstrations of EHC program principles– Should reflect uncertainty stakeholders have

Decisionmakers, clinicians, patients, others

Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

6

Key Questions (I)

Key questions guide the entire systematic review process– Literature search

– Inclusion and exclusion criteria

– Types of data extracted

– Data synthesized and reported

Key questions must be– Clear and precise

– Relevant to stakeholdersWhitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

7

Key Questions (II) Key questions should not be posed because

– The answer is assumed to be known– There seems to be adequate literature to evaluate it

An understanding of the topic area is critical– Understand PICOTS (explained in next slide),

understand the topic – Background reading and key informants are critical to

understanding PICOTS Elements of PICOTS outlined in general form

during topic development– Comprehensive development of PICOTS is required

during topic refinement before key questions are posed

Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

8

Key Questions DerivedKey Questions DerivedFrom Assessing PICOTSFrom Assessing PICOTS

Population—Who is being evaluated? Intervention—What intervention is being

evaluated? Comparator—What is the intervention

being compared with? Outcomes—What are the benefits and

harms being evaluated? Timing—What is the follow-up time? Setting—What are the settings of interest?


9

Devise Analytic Framework Devise Analytic Framework (See Module 2)(See Module 2)

Intermediate Outcomes Maximal concentrations Minimal concentrations at steady state Area under the curve Average steady state concentrations Dose needed to control seizure Switchback rate from generic to innovator antiepileptic medication[Pharmacokinetic data sought in

epileptic patients only]

Final Health Outcomes Mortality Hospitalization Office/emergency department visits Composite of ambulance services, hospitalization, or emergency department visits for epilepsy Health-related quality of life Breakthrough seizures Frequency of seizures Secondary seizure injury (fracture, laceration, head injury, aspiration pneumonia) Status epilepticus Loss of driver’s license Loss of employment

Total adverse events Neurological adverse

effects Hypotension Rash Suicidal ideation Dizziness

(KQ 1)

(KQ 3)

(KQ 2)

Patients with epilepsy using generic versus innovator antiepileptic medication

White CM. AHRQ Topic Refinement Report, Submitted 7.15.09.

sd03835

This diagram leaves out KQ 4, which asks how intermediate outcomes may be related to healht outcomes. When direct evidence is lacking (for KQ 2), both links for KQ 1 and KQ4 must be established to support any conclusions about health outcomes.

sd03835

KQ 4 goes here.

10

Key Informants (I) Key informants (stakeholders) serve a

variety of important functions:– Help formulate key questions that address

real-world dilemmas Key questions should be devised to address

important health care dilemmas, not focus on interests pertinent to researchers

– Provide context to help discern content area and applicability

– Ensure transparency in the process


11

Key Informants (II)

Beliefs about advantages or disadvantages of various alternative treatments are an important target for exploration with key informants– Some beliefs will be based on long-term RCTs

– Others will be based on dogma, extrapolations from surrogate end points, pathophysiology of disease, or pharmacologic profiles of drugs Key questions addressing these areas can

profoundly affect practice


12

Key Informants (III)

Clinicians– Clarify (sub)populations to focus on– Identify areas in which studies differ that

can affect applicability– Identify what contemporary practice looks

like Patients

– Provide insight not usually appreciated by other stakeholders May focus more on quality of life and harms


13

Public Comment (I)

After key informants provide insight, a final draft of the topic refinement document is devised

This document will undergo public comment – Anyone with an opinion and access to a

computer can comment Public comment is a critical final check

of underlying assumptions about relevance and ensures transparency for all stakeholders


14

Public Comment (II)

All public comments that can be answered with the existing Methods Guide are answered generically

All other public comments are answered individually by the EPC– Changes to the topic refinement document are

made or further explanations of why certain aspects of the topic refinement project are not being changed are laid out

Revised topic refinement document and dispensation of public comments are reviewed and approved


.

sd03835

Should read "disposition" not "dispensation".

15

Topic Refinement Process: UCONN/HH EPC Example (I)

Step 1: Core Leadership team reviews topic development document with title, preliminary PICOTS, brief review of what systematic reviews and studies exist, and topic refinement template– Topic refinement on treatments for epilepsy

Step 2: Core Leadership team selects a team from within the EPC that has knowledge, interest, and/or contacts within the topic area– Lead investigator, senior scientist, research fellow identified

Step 3: Selected team performs general review of the topic area, focusing on tertiary texts describing the disease process or interventions of interest and then any practice guidelines– General review on epilepsy conducted, presented by senior

scientist as a lecture with a neurologist invited

16

Topic Refinement Process: UCONN/HH EPC Example (II)

Step 4: Team reviews the topic development documents and continues more focused background reading

Step 5: Preliminary search strategy proposed– Approved by medical librarian and conducted

Step 6: Citations reviewed, body of literature identified, data on PICOTS extracted; NO RESULTS ARE REVIEWED

Step 7: Team determines PICOTS and analytic framework

17

Analytic FrameworkAnalytic Framework

White CM. AHRQ Topic Refinement Report. Submitted 7/15/2009.

Intermediate Outcomes Maximal concentrations Minimal concentrations at steady state Area under the curve Average steady state concentrations Dose needed to control seizure Switchback rate from generic to innovator antiepileptic medication[Pharmacokinetic data sought in

epileptic patients only]

Final Health Outcomes Mortality Hospitalization Office/emergency department visits Composite of ambulance services, hospitalization, or emergency department visits for epilepsy Health-related quality of life Breakthrough seizures Frequency of seizures Secondary seizure injury (fracture, laceration, head injury, aspiration pneumonia) Status epilepticus Loss of driver’s license Loss of employment

Total adverse events Neurological adverse

effects Hypotension Rash Suicidal ideation Dizziness

(KQ 1)

(KQ 3)

(KQ 2)

Patients with epilepsy using generic versus innovator antiepileptic medication

sd03835

This slide is a duplicate of slide #8. Unlcear why it is repeated here. KQ4 should be included.

18

Topic Refinement Process: UCONN/HH EPC Example (III)

Step 8: Preliminary key questions posed Step 9: List of questions for key

informants posed Step 10: Key informants shown the

background, PICOTS, analytic framework, and preliminary key questions

Step 11: After 2 weeks, the topic refinement document discussed with key informants, questions posed

19

Topic Refinement Process: UCONN/HH EPC Example (IV)

Step 12: Based on feedback, revised topic refinement document generated

Step 13: Topic refinement document sent to AHRQ

Step 14: After editorial review, posted for public comment

20

Topic Refinement Process: UCONN/HH EPC Example (V)

Step 15: Public comment sent to the EPC, revisions made, comments responded to

Step 16: EPC sent final version to AHRQ for final approval and official posting

21

The topic refinement process is devised to ensure quality, clinical relevance, and transparency

Understanding the topic area is important to describing the PICOTS and devising the analytic framework

When studies are evaluated, the focus is on the methods and demographics, not the results

Key informants provide context and ensure relevance and transparency

Key informants cannot simply include researchers in the field

Public comment is a final verification of relevance and ensures transparency

Key Messages

sd03835

This point is not addressed in previous slides, but should be elaborated.

Documents

Systematic Review Module 1: Refining Key Questions C. Michael White, PharmD, FCP, FCCP Professor and Director University of Connecticut / Hartford Hospital