Issues and Challenges around Integrating Qualitative and Quantitative Data

Dr Andrew Booth (with Acknowledgements to Professor Angela

Harden & Professor James Thomas)

Overview

• What is Already Known around Integrating Qualitative and Quantitative Data

• Overview of Methodological Issues/Challenges• Recent Developments (with focus on literature

of last two years)• Outstanding Issues/Challenges

Drivers

• Complexity of Context• Complex Interventions• Complexity of Questions: Feasibility

Appropriateness Meaningfulness Effectiveness/Economy (FAME)

• Complexity of the Data

Complex Interventions

• Defined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome.” Craig P et al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337

FAME! (i.e. Feasibility etc…)

Complex Questions!

When You’re Lost You Need a Map!

Complexity of the Data

E.g. Process Evaluations• Elaboration / expansion / explanation • Initiation

– Where different data ‘speak’ to one another, we are able to analyse and explain findings / analyse variation

• Complementarity – Use all evidence at our disposal

• Contextualisation – Some questions are really about re-contextualising findings for

specific use • View problem from multiple directions / perspectives

Approaches and methods in 2004…• Narrative summary• Thematic analysis

• Grounded theory• Meta-ethnography• Meta-study• Miles and Huberman's data

analysis techniques• Content analysis• Case survey• Qualitative comparative

analysis • Bayesian meta-analysis

“Qua

litisi

ng”

“Qua

ntitis

ing

”

2004

Approaches and methods 2015

Meta-narrative

review

Critical Interpretive

synthesis

Realist synthesis

Mixed methodssynthesis

Narrative synthesis

Integration of qualitative

and quantitative

research

Bayesian meta-

analysis

Origins of approaches and methodsMethod Developed by/Exemplars Context and purpose

Narrative synthesis Popay et al. (2005) Cochrane remit – to examine issues of process, implementation and experience

Mixed methods synthesis

Thomas, Harden et al. (2004)Harden and Thomas (2005)

Informing policy to promote children’s health

Bayesian synthesis Roberts et al. (2002) Factors that affect the uptake of childhood immunisation

Critical interpretive synthesis

Dixon-Woods et al. (2006) Access to healthcare for vulnerable groups

Meta-narrative Greenhalgh et al. (2005) To review research on diffusion of innovation to inform healthcare policy

Realist synthesis Pawson (2006) To develop and test theories of change underpinning complex policy interventions

Overview of Methodological Issues/Challenges

• Key Issue remains “Integration”• At what point is integration taking place?:

– Review Question– Methods– Findings– Synthesis– Discuss– Recommendations

Integration of Methods

Qual/Quan and Mixed

Challenges (Boeije et al, 2014)• Extracting the separate qualitative and quantitative components

can be difficult if tightly integrated within different stages of the study, e.g. data collection, data-analysis, and interpretation.

• Where components are integrated, components are more concise and narrow in scope. Specifically aim to compare and integrate well-defined qualitative and quantitative components. Turning these studies into MMS as they are meant to be, makes separate components unfit for inclusion in separate research syntheses.

• As result of integration, MMS outputs are hard to extract and formulate as newly gained insights. Reflects debate about nature/quality of yield in MMS.

• Bottomline: Well integrated primary research makes it more difficult to integrate it within a qualitative evidence synthesis

Different degrees of combining disciplines (Oliver, 2015):

• Distinct disciplines with different strengths for different purposes (multidisciplinary); making links between different disciplines and creating additional knowledge from where they meet (interdisciplinary) and dissolving boundaries between them and around them (transdisciplinary).– Multidisciplinary approach: parallel procedures for process evaluations and controlled trials

(peer-delivered health promotion), then discussed their findings together, latter help to explain former. Process evaluations revealing how teachers often undermine peer delivery by retaining control explained why controlled trials showed that the interventions were not working, but could not offer solutions.

– Interdisciplinary approach: (Barriers and facilitators) - interdisciplinary research, where new learning emerges at the interface. Comparing findings of qualitative studies with others evaluating effects of interventions – some by sound designs and others by flawed designs – revealed appropriate interventions ready for policy consideration or for rigorous evaluation.

– Transdisciplinary approach (children and healthy eating) - merging two methodologies to provide single coherent product, single report in widely read medical journal. Findings of overarching synthesis present statistical meta-analysis in terms determined by the synthesis of children’s ‘views’. In final synthesis and conclusion, contributions of experimental designs and qualitative studies are indistinguishable. Methodological development transcended academic disciplines.

What Tools do we have?

Methods• Bayesian Synthesis• Critical Interpretive

Synthesis• “EPPI-Method”• Meta-Narrative• Narrative Synthesis• Qualitative Comparative

Analysis• Realist Synthesis

Techniques• Logic Models• Frameworks• Matrices

Review questions•What is known about the barriers to, and facilitators of, healthy eating amongst children?

•Do interventions promote healthy eating amongst children?

•What are children’s perspectives on healthy eating?

•What are the implications of the above for intervention development?

REVIEW PROCESS

Searching, screening and mapping

Synthesis 1: Trials (n=33)1. Quality assessment

2. Data extraction3. Statistical meta-analysis

Synthesis 2: Qualitative studies (n=8)1. Quality assessment

2. Data extraction3. Thematic synthesis

Synthesis 3: Trials and qualitative studies

Focus narrowed to ‘fruit &veg’

Children’s views Trials

Recommendation for interventions

Good quality

Other

Do not promote fruit and vegetables in the same way

0 0

Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one

5 5

Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health

5 6

Synthesis 3: Across studies

Increase (standardised portions per day) in vegetable intake across trials

0

0.1

0.2

0.3

0.4

0.5

0.6

War

dle

Liqu

ori

Henry

Ander

son

Reyno

lds

Auld

Auld

(b)

Baran

owsk

i

Perry

Study

Po

rtio

ns

Little or no emphasis on health messages

Synthesis 3: Across studies

In focus: Bayesian synthesis

• Aim is to test theory• Findings from qualitative and

quantitative research are ‘fused’• Only three worked examples to

date• Variation in weight given to the

qualitative evidence• Synthesis product is a set of

weighted factors associated with/predicting the phenomenon under review

Recent examples

Factors affecting the uptake of childhood immunisation

(Roberts et al., 2002)

Factors that influence adherence to HIV

medication regimes (Vollis et al., 2009)

(Crandell et al., 2011)

Factors that influence adherence to HIV medication (Crandell et al., 2011)

In focus: Critical Interpretive Synthesis

• Aim is to generate theory from large and diverse body of literature

• Literature itself is an object of scrutiny (critical)

• Comprehensive search to identify sampling frame

• Purposive and theoretical sampling

• Analysis leads to generation of synthetic constructs and a synthesising argument

Examples

Access to health care for vulnerable groups (Dixon-

Woods et al. 2006)

Use of morphine to treat cancer related pain (Flemming, 2009)

Nurses response to suicide and suicidal patients (Talseth and

Gilje, 2011)

Access to health care for vulnerable groups (Dixon-Woods et al. 2006)

• Based on 119 papers

• Interpretive qualitative analysis of diverse types of studies

• Conceptual and methodological problems with measures of health service utilisation

• Synthesising argument organised around a set of central concepts (e.g. navigation, adjudications) with the synthetic construct of ‘candidacy’ at the core

Access to health care for vulnerable groups (Dixon-Woods et al. 2006)

Core construct – candidacy

“Candidacy describes the ways in which people's eligibility for

medical attention and intervention is jointly negotiated between individuals and health

services…….candidacy is a dynamic and contingent process,

constantly being defined and redefined through interactions

between individuals and professionals, including how "cases" are constructed.”

Central concepts

• Identification of candidacy• Navigation

• Permeability of services• Appearances at health

services• Adjudications

• Offers and resistance• Operating conditions and

local production of candidacy

In focus: Meta-narrative review

• Aim is to make sense of and understand diverse bodies of literature and their findings

• Literature itself is an object of scrutiny (critical)

• Searching is iterative, ‘snowballing’ a key technique

• Analysis leads to production of a set of meta-narratives (‘storylines of research’)

Recent examples

Spread and sustainability of innovations in health service delivery and

organisation (Greenhalgh et al., 2005)

Understanding the use of electronic patient records in health care

organisations (Greenhalgh et al., 2009)

See also the:Realist and meta-narrative evidence

synthesis evolving standards project (RAMASES) (Greenhalgh et al., 2011;

Wong et al., 2013)

A meta-narrative approach

• The influence of Kuhn’s ‘paradigms’ (1962) and the makings of the first meta-narrative

• The essential technique is interpretive synthesis exploring distinct research traditions, each with its own meta-narrative

• Methods of ‘unpacking’ the meta-narrative: exploratory methods; expert consultations; snowballing; database searching

Stages of a meta-narrative review (from Greenhalgh et al., 2009)

Meta-narratives identified in the electronic patient records review (from Greenhalgh et al., 2009)

Comparing approaches and methodsMethod Idealist –

realist continuum

Deconstruct body of literature?

Mixed methods lens Other characteristics

Mixed methods synthesis

Realist No Complementary strengths stance andDialectical stance Little iteration in

methods

Synthetic product aims to directly address policy

Bayesian synthesis

Realist No Alternative or single paradigm stance

Critical interpretive synthesis

Idealist Yes Alternative or single paradigm stance Iterative approach

key

Synthetic product requires interpretation

Meta-narrative Idealist Yes Dialectical stance

Mixed methods systematic reviews

Working definition

Combining the findings of ‘qualitative’ and ‘quantitative’ studies within a single systematic

review, in order to address the same, overlapping or complementary review

questions(Harden and Thomas, 2010)

Three ways in which reviews are mixed….

1. The types of studies included and hence the type of findings to be synthesised (i.e. ‘qualitative/ textual and quantitative/numerical)

2. The types of synthesis method used (e.g. statistical meta-analysis and qualitative synthesis)

3. The mode of analysis: theory testing AND theory building

Mixed Methods Reviews

• Mixed methods reviews have a distinctive heritage – Address complex (and compound) questions – Use different types of evidence in a ‘dialectical’ fashion to grapple with complexity – Mitigate impact of the lack of intervention / evaluation replication – Blend the macro and micro perspective

• Some thinking & methodological development has already taken place – in the primary research mixed methods literature

Three main types of integration

– Sequential explanatory design– Sequential exploratory design – Convergent design

(Pluye and Hong, 2014 – See Next Slide)

Mixed Methods Syntheses (Pluye & Hong, 2014)

Mixed methods synthesis type 1: Sequential explanatory design

Synthesis design: a) QUAN synthesis is followed by, and informs, QUAL synthesis; and b) QUAL synthesis helps to explain some results of QUAN synthesis

• Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012. (http://www.bmj.com/cgi/content/full/328/7446/1010)

Mixed methods synthesis type 2: Sequential exploratory design

Synthesis design: • a) QUAL synthesis is followed by, and informs, the QUAN

synthesis; and • b) QUAN synthesis generalizes or tests findings of the QUAL

synthesis • Aim: – Identification of new hypotheses and knowledge gaps

(e.g. development of a typology) • Sutcliffe K, Stokes G, O’Mara-Eves A, Caird J, Hinds K, Bangpan M, Kavanagh

J, Dickson K, Stansfield C, Hargreaves K, Thomas J (2014) Paediatric medication error: A systematic review of the extent and nature of the problem in the UK and international interventions to address it. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. ISBN: 978-1-907345-73-9

Mixed methods synthesis type 3: Convergent approaches

• Two types: – QUAL data – QUAN data

• Data from primary studies transformed before synthesis begins

• E.g. Qualitative Comparative Analysis

Logic Model for Integrating Evidence

Baxter et al, 2014)

Recent development - Matrix

• Candy et al. (2013) Using qualitative evidence on patients views to help understand variation in effectiveness of complex interventions

• Used qualitative comparative analysis to identify pathways to effectiveness

• Worked example - improving adherence to drug therapy

Recent developments

Future challenges

• Need more worked examples• Focus on methods and tools for the actual integration

– Enhancing transparency– Establishing rigour

• Further conceptual work to illuminate points of difference, strengths and weaknesses, fit for purpose (e.g. aggregation and configuration – Sandelowski et al., 2011; Gough et al., 2012)

• Learning from, and contributing to, the mixed methods literature for primary research.

• Critical appraisal and reporting standards

The Key is Difference = Integration

• “for mixed research synthesis to advance, researchers must solve the problems generated by the methodological diversity within and between qualitative and quantitative studies. Difference has recurrently been identified as the most important factor complicating both the qualitative and quantitative research synthesis enterprises”. (Sandelowski et al, 2007)

In Summary

• Mixed methods research syntheses have dynamic and evolving ‘heritage’

• Address complex (and compound) questions, interventions, wicked problems

• Need to move from “handmaiden” role to a genuine “enhancement” role (Popay; 1998; Petticrew, 2015)

• Methods evolving: fast moving, interesting and rewarding field!

References• Baxter, S. K., Blank, L., Woods, H. B., Payne, N., Rimmer, M., & Goyder, E. (2014). Using

logic model methods in systematic review synthesis: describing complex pathways in referral management interventions. BMC medical research methodology, 14(1), 62.

• Boeije, H., Slagt, M., & van Wesel, F. (2013). The Contribution of Mixed Methods Research to the Field of Childhood Trauma A Narrative Review Focused on Data Integration. Journal of Mixed Methods Research, 7(4), 347-369.

• Candy B, King M, Jones L, Oliver S. Using qualitative synthesis to explore heterogeneity of complex interventions. BMC Med Res Methodol. 2011 Aug 26;11:124. doi: 10.1186/1471-2288-11-124.

• Hannes K. Building a case for mixed-methods reviews. In: Richards & Hallberg. Complex Interventions in Health. Routledge, 2015.

• Heyvaert, M., Maes, B., & Onghena, P. (2013). Mixed methods research synthesis: definition, framework, and potential. Quality & Quantity, 47(2), 659-676.

• Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., ... & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. bmj, 350, h1258.

References• Pluye, P., & Hong, Q. N. (2014). Combining the power of stories and the power of

numbers: mixed methods research and mixed studies reviews. Public Health,35(1), 29.

• Pluye P, Gagnon MP, Griffiths F, et al. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud 2009;46:529–46. doi:10.1016/j.ijnurstu.2009.01.009

• Sandelowski et al. (2007) Comparability work and the management of difference in research synthesis studies. Social Science and Medicine, 64: 236-247 29.

• Shaw RL, Larkin M, Flowers P. Expanding the evidence within evidence-based healthcare: thinking about the context, acceptability and feasibility of interventions. Evid Based Med 2014; 19:201–3. doi:10.1136/eb-2014-101791

• Souto RQ, Khanassov V, Hong QN, et al. Systematic mixed studies reviews: updating results on the reliability and efficiency of the mixed methods appraisal tool. Int J Nurs Stud 2015;52:500–1.