Upload
russell-pearson
View
222
Download
0
Tags:
Embed Size (px)
Citation preview
Appraisal, Extraction and Pooling of Qualitative Data and Text
- Evidence from qualitative studies, narrative and text.
JBI/CSRTP/2013-14/0003
Introduction• Recap of Introductory
module– Developing a question (PICo)– Inclusion Criteria– Search Strategy– Selecting Studies for Retrieval
• This module considers how to appraise, extract and synthesize evidence from qualitative studies and text.
Program Overview
Program Overview
Session 1: Introduction to Qualitative Evidence Synthesis
Qualitative Research is…• “…a situated activity that locates the observer in the world. It
considers a set of interpretive, material practices that make the world visible…[it] involves an interpretive, naturalistic approach to the world. This means qualitative researchers study things in their natural settings, attempting to make sense of, or to interpret, phenomena in terms of the meanings people bring to them” (Denzin & Lincoln, 2000)
Qualitative research and health care• Health care is a people centered process concerned with
health and healing– Health care derives much of it’s knowledge from a range of
biomedical sciences
• However this does not supply all the knowledge that is necessary to provide holistic care for the patients and clients
• Qualitative researchers use humanistic frameworks to examine ordinary activities of everyday life
Qualitative Research Findings as Evidence for Practice
• Qualitative evidence is of increasing importance in health services policy, planning and delivery.
• It can play a significant role in: – understanding how individuals / communities perceive health,
manage their own health and make decisions related to health service usage;
– increasing our understandings of the culture of communities and of health units;
– Informing planners and providers;– evaluating components and activities of health services that
cannot be measured in quantitative outcomes.
Application of Qualitative Research to Practice
• As with quantitative research, results from a single study only should not be used to guide practice
• The findings of qualitative research should be synthesized in order to develop recommendations for practice
Research Paradigms
• Serve as a model/world view or philosophy that structures knowledge and understanding
• Define what is to be observed and scrutinised• Prioritise questions• Inform the interpretation of results
Dominant Paradigm 1: Positivist/Empirical-Analytical
• Assumptions– There is an external, real world of objects apart from
people, which is comprehensible to us– Researchers can compare their claims against this
objective reality
• Seeks to predict and control
Dominant Paradigm 2: Interpretive• Assumptions
– Reality as we know it is constructed/produced inter-subjectively (between people).
– Meaning and understanding are developed socially and experientially.
– Researchers’ values are inherent in research. – Findings of knowledge claims are created– All interpretations are located in a particular context,
setting and moment.• Seeks to understand
Dominant Paradigm 3: Critical• Assumptions
– Examines issues of power and envisions new opportunities– Main task social critique, whereby the restrictive and alienating
conditions of the status quo are brought to light– Common assumptions with Interpretivism; however, moves past
describing meaning to emancipation and change. • Seeks to critique and emancipate
Qualitative Methodologies
Interpretive Methodoligies
Phenomenology• Assumes that people come to know a phenomena
through their experience of it• Focuses on INDIVIDUAL meaning• Emphasizes a focus on people's subjective
experiences and interpretations of the world. • Understand how the world appears to others.
Ethnography• Study of Culture/specific cultural group.
– What is the way of life of this group of people?– Everyday life is worth of study.
• Focuses on SOCIAL meaning.• Researcher immerses self in cultural group.
– Field Work - primarily participant observation and interview data
• Focuses on the context of communities• Used increasingly in healthcare to explore the relationship
between health and culture.
Grounded theory• Develop theory grounded in real world;• Searches and generates theoretical explanations
from observations of the world (induction);• The constant collection and iterative analysis of data
to enable theory to emerge; and• Data are not constrained by predetermined
theoretical framework, but define boundaries of inquiry.
Critical Methodologies
Action Research• Basis in critical social science;• Researchers interact with the
participants to achieve change;• Often community-based;
– used with implementation studies in healthcare; and
• Treats the individual as an autonomous being, capable of exercising agency
Discourse analysis• Discourse – to talk, converse; hold forth in speech or
writing on a subject;• Discourse as patterns of ways of representing
phenomena in language;• Application of critical thought to social situations and
the unveiling of (hidden) politics within socially dominant or marginalised discourses.
Qualitative data analysis• Is based on principles of thematic analysis for most
methodologies– A process of identifying and presenting meaning in the
form of themes, metaphors or concepts;• there is wide variability in the terms used and how they are
defined;
– Not suited to all methodologies (i.e. discourse analysis)
Qualitative data analysis
• Three strategies:– Data analysis as a separate step following data collection– Data analysis occurs simultaneously with data collection– Data collection and analysis are ‘staged’
Congruity between Paradigm, Methodology and Methods
Group Work 1• In pairs, work through the Group Work 1 task in the
Workbook.• Reporting back
Session 2: The Systematic Review of Qualitative Evidence
Trends and Debates within the International Literature
Approach Purpose Evidence of Interest
Search Strategy
Critical Appraisal
Data Extraction
Method of Synthesis
Outcome Software Available
Narrative Synthesis
To summarise two or more papers in narrative form
Generic Unspecified/Selective
Not specified Not Specified
Unspecified Review article-type report
NO
Meta-ethno-graphy
The generate new knowledge/theory use processes of interpretation
Findings of qualitative research studies
Not comprehen-sive or exhaustive; seeks saturation – theoretical sampling
Opposed; all studies included as each may provide insight into the phenomena of interest
Extraction of key concepts
Refutation-al synthesis;Reciprocal translation;Line of argument synthesis.
Higher order interpretat-ion of study findings
NO(Although QARI can be used)
Meta-synthesis: approaches
Approach Purpose Evidence of Interest
Search Strategy
Critical Appraisal
Data Extraction Method of Synthesis
Outcome Software Available
Realist Synthesis
To develop and test program theories
Generic; but more suited to qualitative research studies
Not comprehen-sive or exhaustive; seeks saturation
Not specified Extracts positive and negative mechanisms/strategies that influence change
Tacit testing of implicit theories; building theory.
A transfer-able theory on “what works, for whom, in which circum-stances”
NO
Thematic analysis
To aggregate findings of 2 or more studies
Findings of qualitative research studies
Not comprehen-sive or exhaustive; seeks saturation – theoretical sampling
Not specified Extracts of major/recurrent themes in literature
Aggregation of themes/metaphors/categories
A summary of findings of primary studies under thematic headings
NO(Although QARI can be used
Approach Purpose Evidence of Interest
Search Strategy
Critical Appraisal
Data Extraction
Method of Synthesis
Outcome Software Available
Content analysis
To analyse then summarise content of papers.;occurrences of each theme counted and tabulated
Generic Usually comprehen-sive with predeterm-ined search strategy
Not specified
Content extracted then coded using extraction tool designed to aid reproducibility
coded data categorised under thematic headings; also counted and tabulated
A summary of findings, and their rate of occurrence, of primary studies under thematic headings
YES
Meta-synthesis/Meta-aggregation
To aggregate the findings of included studies
Findings of qualitative research studies
Comprehen-sive; detailed search strategy at protocol stage required
Required, using standard-ised critical appraisal instrument
Extraction of findings PLUS data that gives rise to finding using data extraction instrument
Aggregation of findings into categories; and of categories into synthesised findings
Synthesised findings that inform practice or policy in the form of a standardised chart
YES
Meta-synthesis: worked examples
Meta aggregationMeta ethnography
Meta aggregation
• A structured and process driven approach to systematic review drawing on the classical understandings and methods associated with systematic review of quantitative literature as practiced by the Cochrane Collaboration
Meta aggregation
• Based on an a-priori protocol– Established, answerable question– Explicit criteria for inclusion– Documented review methods for searching, appraisal,
extraction and synthesis of data
Meta aggregation• Explicitly aligned with:
– Philosophy of pragmatism• Delivers readily useable findings• Informs decision making at the clinical or policy level
– Transcendental phenomenology• Looks for common or “universal” essences of meaning• Attempts to “bracket” pre-understandings of the reviewer
Transcendental Phenomenology
• Based on the philosophic traditions of Husserlian phenomenology:– the intuitive examination of essences that have immediate
validity;– Seeks to avoid undue influence of the reviewer on the text;– Seeks to generate practice level theory that has explanatory
power for policy or practice– Seeks to preserve the intended meaning of text
The needs and experiences of people with a diagnosis of skin cancer: a
systematic review Janet Barker, Arun Kumar, Wendy Stanton and Fiona Bath-Hextall, JBI Library of
Systematic Reviews, 2011; 9(4):104-121
Inclusion Criteria• Participants
– Adults with a diagnosis of skin cancer.
• Phenomena of Interest – The needs and experiences of people with skin cancer.
• Context– Hospital or community-located dwellers.
• Types of studies – All qualitative studies that described or analyzed the needs and experiences
with people who had been diagnosed with skin cancer.
Methods• Each paper was assessed independently by two
reviewers for methodological quality. The internal validity (quality) of research papers was assessed using JBI-QARI.
• There were no disagreements between the two reviewers and therefore a third reviewer was not required.
Results
Results ( cont/…)
• A total of 12 findings were extracted from the 2 included qualitative papers
• These findings were aggregated into 4 categories on the basis of similarity of meaning.
• The categories were synthesized to generate 2 synthesized findings.
Results/cont..
• The 2 studies included in the review used a qualitative approach but neither specified a particular methodological approach.
• Both studies were conducted in the United Kingdom (UK), using semi structured interviews.
Recommendations arising• There is a real need to increase knowledge of skin cancer so
that people do not delay in seeking medical help as early diagnosis can dramatically improve both prognosis and the patient experience since early lesions are treated more simply compared with larger or neglected lesions.
• Health professionals caring for these patients need to understand the psychosocial concerns of this patient group in order to design services appropriately and to provide patients with the support they need and information that they can easily understand.
Meta-Ethnography• Explicitly aligned with philosophy of interpretivism• Searches for new meaning• Focuses on multiple realities
Hermeneutic Interpretivism• Is the basis for meta-ethnography
– Interpretation of text for its inner meaning– Focuses on the reviewers interpretive skills– Seeks to re-interpret the published literature– Seeks to generate new, mid level theoretical
explanations– Seeks engagement between reviewer and text
Meta ethnography
• From within the social sciences to develop theories from existing ethnographic data,
• Iterative development of emic interpretations, • Incorporates 7 phases, these can be aligned to the
systematic review process, but there is no requirement to do so.
Meta ethnography• Identify the research interest• Decide which studies to include• Read the studies• Determine how the studies are related• Translate the studies in to each other• Synthesize the translations• Express the synthesis
Meta ethnography: 3 stages of synthesis
• First order interpretations– Themes, metaphors or concepts identified
• Second order interpretations– The researchers interpretation of how the identified
concepts relate to each other
• Third order interpretations– Seeks to encompass themes within each other
Interpretation Second Order Interpretation:
• Reciprocal– Like interpretations are brought together “this one is like that one..”
• Refutational– Competing discourses; must also have a relationship that can be
explored
Third Order Interpretation:
• Line of argument– Asks what do the parts infer about the whole
Patient adherence to Tuberculosis treatment
Salla A. Munro, Simon A. Lewin, Helen J. Smith, Mark E. Engel1, Atle Fretheim, Jimmy Volmink. ‘Patient Adherence to Tuberculosis Treatment: A Systematic Review of Qualitative Research’.
PLoS Medicine July 2007.
Inclusion criteria
• Patients, carers or health professionals delivering DOTs
• Perceptions of adherence was the phenomena of interest
• Context was patients undergoing DOTs therapy for TB• Qualitative studies on the perception of adherence
Methods
• Studies were assessed using a checklist• Data extracted using a standardized form• Synthesis was reciprocal and line of argument
Results
• Following screening, 44 papers were included• First order interpretations 8• Second order interpretations 6• Third order interpretations 4• Expression of the synthesis: visual model
Recommendations• Increase visibility of TB programs in the community, which may
increase knowledge and improve attitudes towards TB• Provide more information about the disease and treatment to patients
and communities• Increase support from family, peers, and social networks• Minimize costs and unpleasantness related to clinic visits. Increase
flexibility/ patient autonomy
STEP 2: CATEGORIES
STEP 3: SYNTHESISED FINDINGS
FIRST ORDER ANALYSIS
SECOND ORDER INTERPRETATION
THIRD ORDER INTERPRETATION
QARI METAGGREGATION
META ETHNOGRAPHY
Session 3: Critical Appraisal of Qualitative Evidence
Why Critically Appraise?
• Combining results of poor quality research may lead to misleading understandings of issues explored
1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
The Critical Appraisal Process• Every JBI review must set out to use an explicit
appraisal process. Essentially:– A good understanding of philosophical perspectives,
methodologies, methods is required in appraisers; and– An agreed instrument/checklist should be used.
Validity, Reliability and Generalizability
• Judging the quality of qualitative research remains deeply contested
Analogous criteria for paradigmatic assumptions
Quantitative Qualitative
Reliability Dependability
Internal Validity Credibility
External Validity Transferability
Reliability/dependability
• Appropriateness of methodology, methods and implementation of the research methods, regardless of paradigm
• The focus of dependability is on achieving consistent quality rather than repeatability.
Internal validity/credibility• Credibility addresses whether a finding has been
represented correctly– Assessment of credibility is multi-dimensional, including
goodness of fit and representativeness, – Credibility is auditable - the process may be based upon
researcher confirmation, member checks, peer checks, second researcher analysis, or observation.
External validity/transferability• Findings are not generalizable in the quantitative sense of
the word – generalization is “narrowly conceived in terms of sampling and
statistical significance.”– “qualitative research is directed toward naturalistic or idiographic
generalizations, or the kind of generalizations made about particulars”
– Schofield (1990) describe qualitative metasynthesis as “cross-case generalizations created from the generalizations made from, and about, individual cases.” Sandelowski et
al(1997)
Critical Appraisal• General acceptance of need for quality• Ongoing debate around the role of appraisal• Particular focus of debate on role of scales and sum
scores in appraisal• In practice, appraisal instruments for qualitative
research tend to focus on establishing the degree to which the evidence applies to practice (transferability) rather than internal validity (credibility)
The premise
• Methods applied during the systematic review of qualitative evidence should firstly be congruent with the universally accepted process of systematic review.
• The characteristics of a systematic review might be debated in terms of the detail, but there is general acceptance of a series of steps, stages or processes
The Joanna Briggs Approach• Establishing credibility in qualitative research requires:
– A standardized approach that is clearly defined, articulated, and applied, and
• Takes in to consideration the relationship between:– Methodology and methods– Methods and data– Data and conclusions
The Qualitative Assessment and Review Instrument (QARI)
• QARI provides a systematic process mirroring that of systematic reviews of quantitative research whilst sensitive to the nature of qualitative data.
• a significant proportion of qualitative work is not designed to address questions or issues of an immediate practical nature
• this is consistent with the traditions of many qualitative methodologies
• JBI focuses on evidence to improve global health, reviews with a clinical or policy focus are encouraged.
Critical Appraisal in JBI-QARI
Critical Appraisal of Interpretive Research
• In interpretive and critical inquiry validity relates to the rigor of the process of inquiry– limiting bias to establish validity in the appraisal of
quantitative studies is antithetical to qualitative approaches to inquiry
• QARI incorporates a checklist to appraise rigor generically
• Read the 2 papers provided and then undertake critical appraisal of the paper using the JBI Critical Appraisal Checklist for Interpretive and Critical Research.
• Report Back• Enter the details into JBI QARI
Group Work 2
Session 4: Data Extraction
Study data and Data Extraction
Data most frequently extracted
1004 references
832 referencesScanned Ti/Ab
172 duplicates
117 studiesretrieved
715 do not meetIncl. criteria
82 do not meetIncl. criteria
35 studies forCritical Appraisal
26 studies incl.in review
Considerations in Data Extraction • Source - citation and contact details• Eligibility - confirm eligibility for review• Methodology - approach to the inquiry• Methods - how data were collected• Phenomena of Interest - the described experience or activity• Setting - specific context of study eg, hospital• Geographical location - region or country• Culture - cultural characteristics of setting or participants• Participants - demographic data such as gender, age etc• Data analysis - analytic approach taken, eg thematic analysis etc
Extracting Findings• The units of extraction in this process are specific findings (and illustrations from the
text that demonstrate the origins of the findings);• In Meta-aggregation a finding is defined as: A conclusion reached by the
researcher(s) and often presented as themes, metaphors, findings, concepts, conclusions etc;
• Some papers may present themes and sub themes (or findings of the qualitative analysis at different levels); for such papers the reviewer will need to decide which of these levels represent authentic findings; however, only findings of one level should be extracted from a specific paper.
• No matter what level is decided, the actual verbatim words of the researcher must be extracted.
Extracting Illustrations• Meta-aggregation requires reviewers to also extract an
illustrative excerpt that the researcher presents in support of that particular finding;
• It is only necessary to extract one such supporting illustration;• The actual verbatim words the researcher uses as the
illustration must be extracted.
Assigning a Level of Plausibility1 to Qualitative Evidence
1 Currently, the term “credibility” is used in QARI; the next version will refer to plausibility
Levels of Plausibility- Qualitative• Unequivocal - relates to evidence beyond reasonable doubt• Plausible - those that are, albeit interpretations, plausible in light of
data and theoretical framework. • Not Supported - when 1 nor 2 apply and when most notably findings
are not supported by the data– Should not be included in synthesis to inform practice
Session 5: Data Analysis and Meta-synthesis
General Analysis - What Can be Reported and How
– What phenomena of interest has been evaluated– The meaningfulness/appropriateness/feasibility of the
experience/activity– Contradictory findings and conflicts– Limitations of study methods– Issues related to study quality– The use of inappropriate definitions– Specific populations excluded from studies– Future research needs
Meta-Synthesis
• Analysis and synthesis of qualitative studies • Based on processed data• Aim of meta-synthesis
– is to assemble findings; – categorize these findings into groups on the basis of similarity in
meaning; – aggregate these to generate a set of statements that adequately
represent that aggregation.
Synthesis of the Results of Qualitative Studies
• Differing methodologies, such as phenomenology, ethnography or grounded theory, can be mixed in a single synthesis of qualitative studies because the synthesis is of findings and not the data.
• This is a critical assumption of meta-aggregation.
As qualitative studies differ from RCT, meta-synthesis utilizes a different approach from that used during meta-analysis.
Both approaches provide an understanding based on populations, settings and circumstances
Unlike meta-analysis, meta-synthesis deals in multiple realities and so provides but one interpretation of the phenomenon.
Meta Synthesis vs Meta Analysis
Executing Meta Synthesis using QARI
Assembling the findings
• Once findings are extracted from all included papers;– The reviewer transitions from a focus on papers to a
focus on assembling all of the findings– This is executed inside JBI-QARI
Creating Categories
• Categorization involves repeated, detailed examination of the assembled findings• The reviewer identifies groups of findings on the basis of similarity in meaning to
create categories– A category “Name” is the element of analysis that appears in the synthesis and
is therefore a full description that conveys the whole, inclusive meaning of the category; that comprehensively represents the meanings embodied in the findings it encompasses
– A Category “Summary” is a brief explanatory statement created by the reviewer to assist interpretation and auditabililty.
Allocating Findings to Categories
Synthesized Findings• In meta-aggregation a synthesized finding is an overarching
description of a group of categorized findings that allow for the generation of recommendations for practice.
Creating Synthesized findings• Meta-aggregation of categories involves repeated, detailed examination of the
assembled categories• The reviewer identifies groups of categories on the basis of similarity in meaning
– A Synthesized Finding “Name” is the element of analysis that conveys the whole, inclusive meaning of the categories; and that comprehensively represents the meanings embodied in the findings it encompasses
– A synthesized finding “Name” must also draw a conclusion or claim and communicate the probability of that conclusion or claim
– A Synthesized Finding “Summary” is a brief explanatory statement created by the reviewer to assist interpretation and auditabililty.
Creating Synthesized Findings– Can be stated propositionally as “if-then” statements
• e.g. “If a patient is awaiting a final diagnosis, their relatives will sometimes feel as if they are not involved”
– Preference for the indicatory form (i.e. suggesting or demonstrating the expedience or advisability of action - emphasizing the probability of the claim)
• “Relatives of patients awaiting a final diagnosis of brain death may feel as if they are not involved if strategies to include them are not pursued”.
Allocating categories to synthesized findings
Session 6: QARI Trial
Day 2
Program Overview
Overview
• This module considers how to appraise, extract and synthesize evidence from qualitative studies and text and opinion papers.
Session 7: The systematic review of text and opinion
Text, Expert Opinion and Discourse– Narrative and text (e.g. review papers, discussion papers and policy
documents) present information that arises out of experience or observation. Much of the resources drawn on to develop policy are narrative and textual in nature rather than the outcomes of formal research
– The opinions of leaders, professional organizations and learned bodies draw on the collective wisdom of experienced experts and are often referred to for guidance in relation to policy making and professional practice
• Expertise is highly regarded and is linked to the ability of an expert to “have to hand” information in a given area and is associated with the possession of large amounts of knowledge and fluency in applying this knowledge
• Discourse refers the verbal interchange of ideas that is grounded in language and in the context within which it occurs. Discourse in the professional and public domains is a source of knowledge that can be used to inform policy and clinical decision making.
Text, Expert Opinion and Discourse as Evidence for Policy and Practice
• Narrative, opinion, expertise and discourse often represent the best available evidence in areas where research is limited, or where the knowledge that is needed is generally generated through policy-making or other processes rather than through formal research
• This kind of knowledge cannot be ignored as legitimate sources of evidence for policy and practice
• While this kind of evidence is not a product of “good” science it is empirically derived and mediated through the cognitive processes of health professionals or policy makers who have typically been trained to be analytical
• The superior quality of evidence derived from research is not denied rather, in its absence, it is not appropriate to discount opinion as non-evidence
The Narrative, Opinion and Text Assessment and Review Instrument (NOTARI)
Critical Appraisal of Evidence arising out of Opinion and Text
Critical Appraisal of Text and Opinion• This focus on limiting bias to establish validity in the
appraisal of quantitative studies is not possible when dealing with text and opinion
• The assessment of validity within the JBI-NOTARI process focuses on:
examining the opinion; identifying the credibility of the source of the opinion; establishing the motives that underlie the opinion; and locating alternative opinions that give credence to it or, conversely,
question it.
Critical Appraisal of Evidence arising out of Text and Opinion
Report Back
Extracting ConclusionsExtracting Conclusions
• The units of extraction in this process are specific
conclusion stated by the author/speaker and the
text that demonstrate the argument or basis of the
conclusion
Meta-SynthesisMeta-Synthesis
• The aim of meta-synthesis is to assemble conclusions; categorise these conclusions into groups on the basis of similarity in meaning; and to aggregate these to generate a set of statements that adequately represent that aggregation. These statements are referred to as synthesised findings - and they can be used as a basis for evidence based practice.
Session 8: NOTARI Trial
Session 9: Protocol Development
Session 10: Protocol Presentations
Session 11: Assessment