Critical Appraisal of Health Critical Appraisal of Health Information and Its Place in Information and Its Place in

Evidence-Based Public HealthEvidence-Based Public Health

Milos Jenicek, MDMilos Jenicek, MDMcMaster UniversityMcMaster University

University of Waterloo Department of Health Studies & Gerontology Seminar Series

October 17, 2007

What’s in focus?What’s in focus?1. Defining the words

2. Today’s world of Evidence-Based Public Health

3. Critical appraisal in public health and community medicine. Of what: evidence, its uses or both?

4. Is the critically appraised ‘best evidence’ everything and enough?

5. Towards the critical appraisal of the best evidence uses in argumentations about health programs planning, implementation and evaluation

1. Past thoughts about definitions:1. Past thoughts about definitions:

• The beginning of wisdom is the definition of terms.

Socrates (470? – 399 B.C.)N.B. Philosophers cannot find where.

• How many a dispute could have been deflated into a single paragraph if the disputants had dared to define their terms?

Aristotle (384-322 B.C.)

Definition of EBM Definition of EBM (one of many)(one of many)

The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.

Sackett DL, Rosenberg WM, Gray JAM, Haynes RB,

Richardson WS, 1996

Defining the world of Defining the world of Evidence-Based MedicineEvidence-Based Medicine

Out of many, our definition of EBM:

The process of systematically finding, appraising and using contemporaneous clinical and community research findings

as the evidential basis for decisions in clinical and community medicine.

What is evidence?What is evidence?

• Any data or information, whether solid or weak, obtained through experience, observational research or experiment (trials). This data or information must be relevant and convincing to some degree either to the understanding of the problem (case) or to the clinical and other decisions made about the case.

‘Evidence’ is not automatically correct, complete, satisfactory and useful. It must first

be evaluated, graded and used on its own merit.

Living in the world of ‘evidence’.Living in the world of ‘evidence’.What is it in general?What is it in general?

• Intuitions• Personal experience• Personal observations• Narratives• Testimony• Appeals to authorities• Case examples• Research studies• Analogies

What is the What is the bestbest evidence? evidence?

That’s a question!

The cause-effect relationship is not the only problem of interest in

health sciences, however essential and primordial it might be.

2. Evidence-Based Public Health2. Evidence-Based Public Health(EBPH)(EBPH)

The process of systematically finding, appraising, and using contemporaneous clinical and community research findings as the basis in making decisions about the care of communities and populations in the areas of health protection, disease prevention, and health maintenance and improvement (health promotion).

Evidence-Based Policy and Evidence-Based Policy and PracticePractice

• Rational uses of evidence in defining, formulating, implementing, and evaluating health policies and health programs

- in research

- in practice

The EBPH goal:The EBPH goal:

The best possible management

and control of health and disease and their determinants

at the community level.

EBPH stepsEBPH steps• Formulation of an answerable (research) and/or

evaluable (health programs) question arising from a public health problem

• Searching for evidence• Critical appraisal of evidence• Selection of the best evidence for public health

decisions• Linking evidence with public health experience,

knowledge, and practice and with community values and preferences

• Implementation of useful evidences in public health research and practice (health policies and programs)

• Evaluation of implementations and EBPH practitioners

• Teaching EBPH others

3. What is critical appraisal3. What is critical appraisal

Application of rules of evidence to a study to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards etc. The rules of evidence vary with circumstances.

The IEA Dictionary of Epidemiology (2001)

Steps of the EBM/EBN/EBPH processSteps of the EBM/EBN/EBPH process

• Identifying need(s) for evidence (problem formulation in question or other form)

• Evidence finding and production (practical experience, original research, research synthesis – systematic reviews)

• Critical appraisal of the new evidence• Evidence (research findings) dissemination• Evidence uptake and adoption by its users

(cont’d)

Critical appraisal step 3 Critical appraisal step 3 in the EBM process means:in the EBM process means:

Critically appraising evidence for its• validity (closeness to truth)• impact (size of the effect) and• applicability (usefulness in our

clinical and public health practice) Strauss et al (2005) - modifed

Steps of the EBM/EBN/EBPH Steps of the EBM/EBN/EBPH processprocess (cont’d)(cont’d)

• Uses of evidence by EBM/EBN/EBPH practitioners through logical and critical ways of reasoning, understanding and decision-making (evidence-based critical thinking and its appraisal)

(cont’d)

Steps of the EBM/EBN/EBPH Steps of the EBM/EBN/EBPH processprocess (cont’d)(cont’d)

• Evaluation of the focus (problem, question, objectives), soundness, relevance (priorities), structure, process, and impact of the entire above-mentioned EBM/EBN/EBPH pathway in terms of efficacy, effectiveness and efficiency as well as benefits for the patient(s) and the community(ies)

(end)

Research question PICO(T) styleResearch question PICO(T) style

A workable and meaningful research question that is also usable in an argument must contain four or five elements (in any order depending on the problem under study):

• Problem in patient/population setting (target population)• Intervention (beneficial or noxious exposure)• Comparison (controls; if relevant)• Outcome(s) or results of intervention in a defined time frame• Time frame itself

PICO(T) question:PICO(T) question:An exampleAn example

Does drug therapy (intervention)decrease (expected effect, if relevant)long-term morbidity (outcome)in older persons (population setting)with mild to moderate hypertension

(condition of interest)as compared to younger individuals

(comparison)over a period of …. ? (time frame)

Three aspects of critical appraisal Three aspects of critical appraisal of a study or health programof a study or health program

• Validity of results (credibility relying on question, design, analysis, interpretation)

• What are the results (relevance for practice or magnitude for understanding and decision-making)

• Applicability of results to clinical and community health care (individuals or target populations and communities)

Aspects of activity -Aspects of activity - Snapshot evaluation (single event)Snapshot evaluation (single event)

• Soundness: “Does it make sense?”

Plausibility and its reasons.

• Structure: “How is it organized?” Direction and logical structure of the elements of a given activity.

(cont’d)

Aspects of activity -Aspects of activity - Snapshot evaluation (single event) Snapshot evaluation (single event) - cont’d- cont’d

• Process: “How does it work? Does it work as desired and expected?”

Functionality within a given structure.

• Impact: “What kind of effect does itproduce (if any)?”Efficacy, effectiveness, efficiency of interventions and care.

(cont’d)

Aspects of activityAspects of activity – –constancy or evolution in constancy or evolution in

time/persons/placetime/persons/place

• Constancy: “Does the above informationchange or not over time, in different places, patients, care providers, and settings?”Fluctuations of soundness, structure, process and/or impact.

Aspects of activityAspects of activity – –consistency in our understandingsconsistency in our understandings

• Consistency:“Does it make the same

sense?” Do the results follow the line of

previous experience? Do they have a meaning similar to what we already know in terms of biological, contextual and practice plausibility based on previous studies and their findings?

4. Is critically appraised 4. Is critically appraised evidence enough for good evidence enough for good

evidence-based public health evidence-based public health practice?practice?

Critical appraisal of uses of Critical appraisal of uses of the best evidencethe best evidence

• It means critical appraisal of the decision making process in which the already critically appraised

best evidence is used as elements leading to right conclusions, claims, and decisions

Critical ThinkingCritical Thinking General definitionGeneral definition

• Reasonable reflective thinking that is focused on deciding what to do or believe (RH Ennis)

• The ability to solve problems by making sense of information (evidence) using creative, intuitive, logical and analytical mental processes.

Also worthy of mentioningAlso worthy of mentioning

• Critical thinking is the intellectually disciplined process of actively and skilfully synthesizing, evaluating and/or applying information

gathered from and generated by observation, experience, reflection,

reasoning, or communication in research and/or practice as a guide to belief and action.

Modified from Michael Scriven and Richard Paul

Critical Thinking in MedicineCritical Thinking in Medicine

Reasonable reflective thinking about health, disease, and care that is focused on what to believe ( to understand their mechanisms and nature) and/or what to do (at any level and domain of application in prevention and care). Modified RH Ennis’ general definition

of critical thinking

It relies on the best evidence and its uses within the framework (model) of modern argumentation.

Do we need training and Do we need training and experience in critical thinking?experience in critical thinking?

Consider the following:Consider the following:

‘… The proper mix of proven complementary, traditional and modern remedies, which emphasizes the active participation of the patient can help to create a powerful healing force for our world. … The question to my mind, should no longer be whether health-care services should be integrated, but how and how soon it can be done? …’

Whose statement is this one?

Reference for the above Reference for the above quotationquotation

His Royal Highness The Prince of Wales. Address by His Royal Highness The Prince of Wales to the Fifty-ninth World Health Assembly.

Geneva: World Health Organization, Fifty-ninth World Health Assembly, Agenda item 4, A59/DIV/7, 26 May 2006.

The essentials we should knowThe essentials we should know• How critical thinking fits into the evidence-based

medicine and other health sciences• How important are the definitions of what we are

doing• Critical appraisal of evidence vs. critical thinking

of uses of evidence• Informal logic and argumentation in critical

thinking in medicine• Ancient and modern models of argumentation,

their structure and components: some basic rules• Integration of modern argumentation into the

evidence-based reasoning, understanding and decision making in medicine

Informal LogicInformal Logic

• A branch of logic that uses methods and techniques to identify, analyze, interpret and evaluate reasoning and argument in the context of natural language used in everyday life

Contrary to formal, symbolic, or mathematical logic, informal logic deals directly with reasoning and argument in natural, everyday language.

Logic in MedicineLogic in Medicineand Public Healthand Public Health

• System of thought and reasoning that governs understanding and decisions in clinical and community medicine (public health)

• Logic in health sciences defines valid reasoning, which helps us understand health problems and leads us to the justification of our health decisions about how to act on such phenomena.

Evidence in health sciencesEvidence in health sciences• Any data or information, whether solid or

weak, obtained through experience, observational research or experimental work (clinical trials or public health interventions), convincing to some degree and relevant either to the understanding of the health problem under scrutiny or to clinical or public health decisions made about the case

‘Evidence’ is not automatically correct, complete, satisfactory and useful. It must first be evaluated, graded and used based on its own merit.

Evidence as proofEvidence as proof

Proofs are interpretations drawn from evidence to provide readers or listeners with good reasons for

- changing an attitude or

- following a course of conduct

or action.

Argument in health sciencesArgument in health sciences• A proponent’s structured and organized monologue

(reflection)or

a dialogue between stakeholders in health and disease (physicians, patients, other health care and services

providers)with a predetermined purpose and aim:

• To improve the understanding of health problems

and• To make correct decisions regarding the care of individuals as well as of groups of patients and others in community medicine and public health

5.Towards the critical 5.Towards the critical appraisal of the best appraisal of the best

evidence uses in evidence uses in argumentations about argumentations about

health programs planning, health programs planning, implementation and implementation and

evaluationevaluation

Basic vehicle of critical thinking Basic vehicle of critical thinking and domains of its usesand domains of its uses

• ArgumentationProcess of proposing, defining,

explaining and valuing considerations to support and justify some claim

• Domains of its usesTalking to peers, patients or groups in the

community, or social, economical or political stakeholders in individual and community health

Critical Appraisal Critical Appraisal vs. Critical Thinkingvs. Critical Thinking

• Critical Appraisal of Evidence:

Application of rules of evidenceto a study in order to assess the validity of data,

completeness of reporting, methods and procedures, conclusions, compliance with

ethical standards, etc. The rules of evidence vary with circumstances.

Critical appraisal means‘assessment of evidence’

Critical Appraisal of Evidence Critical Appraisal of Evidence vs. Critical Thinking Appraisalvs. Critical Thinking Appraisal

• Critical Thinking Appraisal:

Application of rules of logic and modern argument to assess the validity of claims in medical and public health communication in light of the completeness and validity of the argument building blocks and coherence between them on the path from original propositions to final claims

This means ‘appraisal of the entire thinking process’ and not simply appraisal of evidence only.

Classical layout of arguments:Classical layout of arguments:categorical syllogismcategorical syllogismTwo premises and a conclusion:

• Major premise (A): statement about something in general• Minor premise (B): statement about something specific• Conclusion: Statement about the

‘specific’ in the light of the ‘general’.

Modern argument by Stephen ToulminModern argument by Stephen Toulmin

Who is Stephen Toulmin?

English/US philosopher, born 1922, Henry R. Luce Professor, University of South California College of Letters, Arts & Sciences

What to read?

Toulmin S, Rieke R, Janik A. An Introduction to Reasoning. New York and London: Macmillan Publishing Co Inc; 1986.

Toulmin SE. The Uses of Argument. Cambridge, England: Cambridge University Press, 1958.

Modern argument Modern argument adapted for medicineadapted for medicine

• Problem in context (thesis, proposition)

• Grounds• Backing• Warrant• Qualifier modulators (adducts)• Qualifier• Claim• Rebuttals

Problem in contextProblem in context

• Domain of interest• Thesis (initial proposal, hypothesis,

research question)• Problem to solve• Target population• Setting (clinical, community, social,

cultural, denominational, etc.)

Grounds - definitionsGrounds - definitions• Specific facts from a study supporting

the claim as the basis for our reasoning and argument (“internal” evidence)

• Belief stemming from a study that when accepted, can lead to the endorsement (or rejection) of another belief or of a policy (claim

of the study)

GROUNDS - examplesGROUNDS - examples

• In medical and nursing research: findings in the study (data and information gathered by observation)

• In medical and nursing practice: Data and information in patient charts and other records

• In civic communication about health matters: Hard (critically appraised) medical evidence, narratives, opinions, laic testimonies, legal/administrative/ economical/political statements built on various foundations

Grounds – what they areGrounds – what they are

• Direct observation, its analysis and interpretation

• Personal experience• Learnt notions• Best evidence• Accepted rules and orders

(substantiated or not)

BackingBacking

• Body of experience and evidence that supports the warrant

• “External evidence” such as a belief stemming from other studies and experience (medical literature, learnt knowledge, past experience – personal or otherwise) that when accepted, can lead to the endorsement (or rejection) of another belief or of a policy (final claim)

WarrantWarrant• General rule inferring a claim from grounds• A principle certifying the reasonableness of

moving from external and/or internal evidence to the claim

• A set of criteria or general principles justifying the move from evidence to claim

• A bridge between two beliefs showing that the adherence to one stands as justification for the other

SupportSupport

• Components of grounds and backing that strengthen or weaken our certainty or conviction about the ensuing (final) claim

• “Qualifier modulators” in the above-

mentioned sense

QualifierQualifier

• Word or phrase indicating the strength of the connection between grounds and claim (conclusions)

• Degree of confidence with which the claim should or might be accepted

• Quantification of our certainty that our claim is right

Examples of qualifiers Examples of qualifiers in cause-effect argumentationin cause-effect argumentation

• This factor .. may be .. the cause of• .. is a potential cause of ..• .. cannot be excluded as a cause of ..• .. is definitely a cause of ..• .. probably causes ..• .. we are more sure than not that ..• .. the etiological fraction of this factor is ..

ClaimClaim• Conclusion as produced by the argument• Proposition that is reached through our

reasoning path• An idea that an arguer wants others to

accept• View(s) as formulated only for the study

itself or as generalizations beyond the study• Ensuing recommendations, clinical orders,

guidelines, health policy and program formulations Conclusion as produced by the argument.

RebuttalsRebuttals

• Exclusionary conditions for the claim• Reservations as circumstances in which the

claim as a result of our reasoning (inference) would not be valid

• All elements against the claim (conclusion)• Special circumstances disallowing or

limiting the applicability of the grounds, backing, and warrant to the conclusion (claim)

AdductsAdducts• Support and rebuttals, i.e. all

elements strengthening, weakening, or nullifying the claim (conclusion): causality criteria, validity, and others as elements weighted “for” (“yea”) and “against” (“nay”) the quantified certainty (qualifier) about the claim (conclusions)

• Inclusion and exclusion criteria

The ultimate challengeThe ultimate challenge

Once EBM is mastered and applied,how can we evaluate its impact

and superiority compared to other alternative methods of practicing and researching clinical and community medicine?

We still have no answers!

Value-based clinical and Value-based clinical and community medicinecommunity medicine

Practicing clinical and community medicine by applying individual (patient) or community preferred values and outcomes along with

other evidence sources

How can we integrate such elements of decision-making?

How can we integrate evidence, How can we integrate evidence, specific community circumstances, specific community circumstances,

preferences and values?preferences and values?

Through various decision analysis and decision making models (decision trees and others)

branching the sequence of• evidence

• circumstances• community considerations• outcomes and their values

in line or not with what the best evidence

originally proposes

In this context:In this context:

Evidence-Grounded

and

Argument-Based Medicine, Veterinary Medicine,

Dentistry, Nursing, Public Health, Community Medicine?