29
Workshop: Framework Synthesis, Meta-Ethnography and Realist Synthesis

Workshop: Framework Synthesis, Meta-Ethnography and Realist Synthesis

  • Upload
    howell

  • View
    34

  • Download
    0

Embed Size (px)

DESCRIPTION

Workshop: Framework Synthesis, Meta-Ethnography and Realist Synthesis . Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in Zambia. - PowerPoint PPT Presentation

Citation preview

Page 1: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Workshop: Framework Synthesis, Meta-Ethnography and Realist Synthesis

Page 2: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in Zambia

• BACKGROUND: Antiretroviral therapy (ART) has significantly improved morbidity and mortality of individuals infected with HIV. However, lack of adherence to highly active antiretroviral therapy (HAART) remains a key challenge to successful management of patients with HIV/AIDS. Adherence rates lower than 95% are associated with development of viral resistance to antiretroviral medications.

• ‘Efforts to sustain adherence in Africa and elsewhere remain important goals to optimize outcomes for individuals and global HIV treatment.’ (Mills, Nachega, Buchan, Orbinski, Attaran, Singh et al., 2006).

Page 3: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in Zambia

• Different Emphases– Barriers and Facilitators to ART (Best Fit

Framework Synthesis)– Theory Explaining Adherence to ART (Meta-

Ethnography)– Under what Contexts and For Whom is Adherence

to ART Effective in Zambia (Realist Synthesis)

Page 4: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Reading and Data Extraction

Page 5: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Data extraction What is it?

An attempt to reduce a mass of material (your included papers) to a much smaller body of text and numbers, amenable to analysis and the interpretation of findings

Page 6: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Data extraction form Location Setting Sample (n) Age Gender Ethnicity Socio-economic status Intervention (if any) Quality assessment criteria Results? Further citations

See Handouts 1-3 Format?????

Page 7: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

What results do you extract? What is your question?

Keep the question in mind as you read:Are the data relevant to this question?Is the question answered by the data?

Page 8: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Data ExtractionFramework Synthesis Meta-Ethnography Realist SynthesisExtracts data against framework. Coding framework with definitions is provided to increase consistency. Data not explained by framework is“parked” for subsequent inductive stage. Distinction typically made between original quotes and author’s analysis.

Key themes and concepts extracted and peer reviewed for inclusiveness. Distinction made between first-order (original quotes) and second-order (author’s analysis) findings. Findings were coded in duplicate. Discrepancy between codes resolved by third person.

Created tool to cover broad range of informationand to consolidate data from across studies. [Pawson argues such tools are too limiting]. Tool flexible to account for diversity. Reviewers returned to papers if new questions arose. Template applied to body of each report (i.e., introduction, methods, results, and discussion sections) by two members of research team. Differences in coding resolved by consensus with other team members.

Page 9: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Quality Assessment

Andrew Booth, Reader in Evidence Based Information Practice, Co-Convenor – Cochrane Collaboration Qualitative

Methods Group

Page 10: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Before You Begin…

• Consider how you will use judgements of quality (cp. 50% of published Cochrane Quantitative Reviews performed quality appraisal but did not make it clear how judgements were used!)– To exclude or to moderate?

• Will chosen instrument militate against certain types of research?

• Quality of reporting or quality of study?

Page 11: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Variability in Practice - 1

21 papers did not describe appraisal of candidate studies

6 explicitly mentioned not conducting formal appraisal of studies

5 papers did a critical appraisal, but did not use a formal checklist

7 described modifying existing instruments1 used an existing instrument without modification

Dixon-Woods, Booth & Sutton (2007)

Page 12: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Variability in Current Practice - 1

23 papers did not describe critical appraisal5 papers explicitly pleaded against quality assessment of

papers or provided valid reason for not conducting quality appraisal.

Criteria used varied between detailed descriptions of relevant items in existing or modified checklists to a set of broad criteria evaluating, for example, rich description of data, credibility or relevance of the original study.

Hannes and Macaitis (2012)

Page 13: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Variability in Current Practice - 2One team used overall judgement (Smith et al., 2005). Five opted for self-developed assessment instrumentThree used previously developed checklists to create own. Two mentioned critical appraisal, but did not specify tool. Most used existing instruments/frameworks. 24 different

assessment tools identified: Critical Appraisal Skills Programme (CASP) (n = 18) Mays and Pope criteria (n = 6)Popay criteria (n = 6) Joanna Briggs Institute (n = 4).

Hannes and Macaitis (2012)

Page 14: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Appraising research quality1. Epistemological criteria: Judgement of ‘trustworthiness’

requires criteria tailored to different research ‘paradigms’. 2. Theoretical Criteria: Explicit theoretical framework shaping

the design of the study and informing claims for generalisability

3. Prima facie ‘Technical’ criteria: Used to assess ‘quality’ common to all research traditions e.g.:Sufficient explanation of background;Method appropriate to question;Succinct statement of objectives/research questions; Full description of methods include approach to analysis; Clear presentation of findings including justification for interpretation of

data etc.

Noyes J (2005)

Page 15: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Two dimensional approach to appraising qualitative research

Technical markers –CASP

Epistemological and theoretical markers – Popay et al

Technical Quality High Description – thicker •Privileges Subjective experience and meanings •Use of theory to build explanations

Technical Quality Low Description - thinner •Imposed pre-determined framework on respondents narratives.•Limited/no/inappropriate use of theory, little explanatory insight

(Noyes, 2005)

Page 16: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Available Tools - 1• CASP – 10 questions to help you make sense of qualitative

research http://www.casp-uk.net/wp-content/uploads/2011/11/CASP_Qualitative_Appraisal_Checklist_14oct10.pdf

• Joanna Briggs Institute - Critical Appraisal Checklist for Interpretive & Critical Research http://www.jbiconnect.org/agedcare/downloads/QARI_crit_apprais.pdf

• National Centre for Social Research. Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence. London: National Centre for Social Research/UK Cabinet Office, 2003 http://www.civilservice.gov.uk/wp-content/uploads/2011/09/a_quality_framework_tcm6-38740.pdf

Page 17: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Available Tools - 2• Dixon-Woods M, Shaw RL, Agarwal S & Smith JA (2004) The

problem of appraising qualitative research. Quality & Safety in Health Care, 13, 223-5.

• Hannes K, Lockwood C, Pearson A (2010). A comparative analysis of three online appraisal instruments' ability to assess validity in qualitative research. Qualitative Health Research. 20(12):1736-43.

• Popay J, Rogers A & Williams G (1998) Rationale & standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8, 341-51.

• Seale C & Silverman D (1997) Ensuring rigour in qualitative research. European Journal of Public Health, 7, 379-84.

Page 18: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

CASP – Technical/Procedural Tool

Page 19: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

JBI – Theoretical Tool

Page 20: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Key Issue

• How are you going to use the quality assessment?– From quantitative assessment we know authors

frequently say they do it – but they don’t incorporate it into results

– Is it technical proceduralism gone mad?– Or can we use the assessments to improve our

synthesis and subsequent interpretation?

Page 21: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Quality AssessmentFramework Synthesis Meta-Ethnography Realist Synthesis

Pragmatic so tends to include all studies . Focuses explicitly on quality of reporting. Qualitative sensitivity analysis used to test robustness of synthesis.

Quality Assessment as Hurdle: Studies using qualitative design and analysis method were included. All studies that focused on ART access and adherence (n = 130) continued to full-text review. Fourteen studies passed quality appraisal (Wallace et al. 2004; Smith et al. 2005; Walsh & Downe 2005, 2006; Rolfe 2006) and were retained.

Rather than appraising and excluding publications prior to review, reviewer mines each publication for evidence to contribute to fuller development of explanatory model. Instead of evaluating entire reports against a priori standards for appraisal, each piece of evidence is appraised for utility and relevance. Value of evidence depends on whether it contributes to better understanding of questions addressed in review.

Page 22: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Data Synthesis

Page 23: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

What is Data Synthesis?

• Process of moving from focus on single studies (cp. Data Extraction and Quality Assessment) to focus on cross-study analysis

• Requires identification of patterns across data, including contradictory findings and data that does not fit

• Iterative and requires ongoing refinement• Acts as prelude to Analysis which seeks to explain

patterns, contradictions and differences

Page 24: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Thematic synthesis; Critical Interpretive Synthesis; Meta-ethnography

1. Only include “good” qualitative studies (?)2. Constant comparison; iterative; interpretations generated from the data

by reviewers3. Create a theory

– Inductive (theory-generating)

Examples: Thomas J, Harden A. Methods for the thematic synthesis of

qualitative research in systematic reviews. BMC Medical Research Methodology 2008; 8.

Campbell R et al. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science & Medicine 2003; 65:671-684.

Methods of qualitative evidence synthesis

Page 25: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Framework synthesis:1. Only include “good” qualitative studies (?)2. Map data from included studies onto an existing framework to test the

framework/theory (a role for theory)3. Build a conceptual model or framework

– Deductive (theory-testing)

Examples: Oliver S et al: A multidimensional conceptual framework for analysing

public involvement in health services research. Health Expectations 2008, 11:72-84.

Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006.

Methods of qualitative evidence synthesis

Page 26: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

“Best-fit” framework synthesis1. Identify relevant pre-existing conceptual models or frameworks2. Identify and extract all relevant qualitative studies satisfying review’s

inclusion criteria3. Code data from included studies against framework4. Use secondary thematic analysis/synthesis to generate completely new

themes to supplement the framework’s themes5. Create new framework and conceptual model or theory

Deductive and Inductive Framework and Thematic synthesis

Carroll C, Booth A, Cooper K. A worked example of “best-fit” framework synthesis: A systematic review of views concerning the taking of potential chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29

Methods of qualitative evidence synthesis

Page 27: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Realist Synthesis (Also The Matrix Method); • Examine Provisional Explanatory Model • Extract Data against Provisional Explanatory Model • Assess for Rigour and Relevance• Develop Propositional Statements• Consider the Strength of Evidence for Each Propositional Statement• Refine and Revise Provisional Explanatory Model

Merten, S., Kenter, E., McKenzie, O., Musheke, M., Ntalasha, H., & Martin Hilber, A. ‐(2010). Patient reported barriers and drivers of adherence to antiretrovirals in sub‐ ‐Saharan Africa: a meta ethnography. ‐ Tropical Medicine & International Health, 15(s1), 16-33.

[Candy B, King M, Jones L, Oliver S. Using qualitative synthesis to explore heterogeneity of complex interventions. BMC Med Res Methodol. 2011 Aug 26;11:124. doi: 10.1186/1471-2288-11-124].

Integration of qualitative and quantitative evidence

Page 28: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis

Data SynthesisFramework Synthesis Meta-Ethnography Realist Synthesis

Original best fit framework is expanded to include new themes. Relationship between themes is examined and the data is used to reconstitute a new model. Particular attention is directed at discrepant cases.

Second-order constructs pertinent to adherence identified and cross-compared and presented in results section. Key themes consolidated into a line of argument (third-order analysis), presented in the synthesis ⁄ discussion section.

Reviewers synthesize extracted evidence and apply it to explanatory model. Then iteratively refine model to bestexplain existing data. Where evidence does not fit, reviewer looks for other factors with more explanatory value. Synthesis ends with summary of how explanatory model has been revised.

Page 29: Workshop:  Framework Synthesis, Meta-Ethnography and Realist Synthesis