Generalising From Qualitative Research

Professor Lee Parker

The Controversy • Quantitative school’s asserted superiority

o The ability to generalise their findings

• Prejudicial view that the qualitative school o Cannot generalise its findings

o Therefore produces lesser form of knowledge

• Classic qualitative researcher responses o We need to be willing to generalise

o No recipes for success but general cautionary tales

o Reluctance to generalise

o Qualitative community’s uncertainty & reticence

• A way forward o Drawing on multidisciplinary methodology literature

o E.g. Education, health sciences, sociology, information systems, management, marketing, accounting

Ideographic or nomothetic focus? • Ideographic studies of uniqueness • Nomothetic studies for general application • Qualitative researchers tend to ideographic • Quantitative researchers tend to nomothetic • Qualitative cases

o Interest in themselves – particularistic behaviour, change practice within that context

o micro-organisational focus on context, uniqueness, difference

• Qualitative researchers’ generalisation avoidance: o Little acknowledgement of generalisation o Not a priority, avoid giving any hint of it o Focus on rich description & nuances that are presumed not replicable

• Avoidance risks o Findings’ limited applicability & usefulness o Unclear alternative offering

Quantitative Generalisation Problems

• Sample adequacy • Low response rates • Assumptions not matching socio-economic context • Inappropriate statistical methods • Predictions exceeding limits of assumptions & probabilities • Non-generalisability to other time periods, locations, cultural

settings • Three different outcomes:

o Statistical significance, result generalisability, result importance. • Statistical analysis = ‘amulet’ projecting pseudo-scientific

reliability & believability • May not actually be generalisable to the field of practice

The Qualitative Generalisation Challenge

• Qual researchers make generalisations, even unconsciously – without explaining or justifying

• Intuitive generalisation & particularisation in changing balance for laypersons

• So qual researchers reflect everyday life o Drawing on past experiences to build expectations about the future

• Dangers of limited case based expectations o Mismatch to other case profiles & contexts o Overgeneralisation o Caution with intuitive generalisation o Avoid mimicking quantitative generalisation

Different Ways of Rethinking Generalisation

• Intrinsic ideographic cases begin some forms of initial generalisation o E.g. observations about other cases, other locations, other times

• Generalisation is central to social life and discourse • Qualitative researchers are not exempt from this

phenomenon • Inappropriate to mimic quantitative concepts and forms of

generalisation • Generalisation is socially constructed and must account for

environmental complexities across time & space • Other means of drawing inferences and extrapolating

beyond research site actors and context • And who makes the generalisations?

o Only the researchers? o What about the readers?

Apologetic Qualitative Accounting Researchers

• Sidestepping any direct generalising references • Those that do address this, largely offer an apologia • Implicitly assuming the only generalisation definition is

statistical • Declaring statistical generalisation limitation

o Unconditionally or qualified by reference to some broader implications • Examples

o ‘need to replicate the findings in other firms/industries’ o Expand sample ‘to generalize to other empirical settings’ o ‘to offer more detailed examinations in other organisations’ o ‘require comparative case studies across multiple organisations’ o this generalisation limitation is ‘obvious but important’ o ‘subject to the conventional limitations attendant with field work’

• Other qualifications o Application only within the case studied o Need to extend across processes, organisational levels, national

cultures etc. • Impression of methodological uncertainty & reticence

Reinterpreting Qualitative Generalisation

• A VARIETY OF FORMS OF QUALITATIVE GENERALISATION SUCH AS:

• Cross case generalisation o Commonalities across cases

• Naturalistic generalisation o Concepts transferring to and recurring in different settings

• Moderatum generalisation o Moderate scope generalisations open to change – styled as everyday actors’

generalisations

• Engaging actors’ implicit theories and researchers’ theories

• Relate to realities & worldviews of wider audiences

Analytical Generalisation • In contrast to statistical generalisation from sample to

larger population • AG generalises to social forms, relations & processes • Through capturing both actors’ & readers’ experiences

of these • Generalise about and to other social processes in

similar or different settings • Judgment and logical argument re one study speaking

to social processes in other circumstances • Leads to construction of theoretical frameworks:

o Beyond the bounds of the original study o Of more general import

Theoretical Generalisation • Theorise about forms/meanings of social phenomena

o Rather than focussing on frequencies of occurrence & probabilities • Develop from local data observations to ideas at more

general level • More abstract theoretical ideas & understandings about

phenomena studied • Producing concepts, theory & rich insights • Thick description

o Portraying contexts, attitudes, motivations, meanings underpinning observed actions & behaviours

• Theory emerges from analysis and interpretation • Highly contextualised accounts of processes engaging

organisational actors and their lifeworlds

Analogical Generalisation • Generalising from one or more cases to analogous

cases • One or more characteristics in one case may be

adaptable to/actionable in other analogous cases • Must be plausible & acceptable to research site

actors and study audiences • Plausibility enhanced by more similarities than

differences across cases

Communicative Generalisation • Effective communication with target audience to

derive study meanings for them in their circumstances • Adequate contextualisation so reader can assess study

evidence similarity with their own setting • Researcher must communicate findings, context,

similarities & differences to audience • Audience catered for includes:

o General, professional, business, government, non-profit and research communities

• Stimulate community conversations about the study & its potential wider application

Naturalistic Generalisation • Observed tendency of people to make generalisations

from personal or vicarious experiences • Developed incrementally through multiple events &

encounters • Generalisation a function of their experiences and how

they tacitly accumulate knowledge and expectations from those o Far removed from formal predictions of the quantitative school

• Similar to qualitative research that: o accumulates case-based findings o develops experiential & propositional knowledge o Incorporates thick description & arguments into lifeworld & memory

• Empowers the readers and democratises generalisation o Provide sufficient context for reader to judge applicability to their world o Facilitates transfer of study findings to practice settings

Naturalistic Generalisation and Practice Transferability

• Associated risks: o Transferability is somewhat beyond researcher control

o It lies in hands of readers

o Researcher judgement needed re type and depth of contextual information to provide

o Communication style & content of researcher’s publication is crucial to reader uptake

• Naturalistic generalisation focusses on practice transferability o i.e. it must be actionable in practice settings

o Requires more practice language than purely research language

o Practitioners disinterested in population averages

o They prefer findings re contextual settings and specific circumstances that embody variation, extremes, and speak to their own situation

Quality of Generalisation

• Quality of research conducted Influences generalisability

• Three particular influences o Sample selection

o Credibility

o Thick description

Sample Selection • Sampling rationale differs from quantitative sampling • Rather than addressing the question “How many?” • We address:

o Which cases? o Why selected? o What do they represent?

• Sampling aims to focus on significant dimensions & characteristics of a phenomenon

• Closer to sociological sampling for actors’ socio-cultural characteristics, interactions & meanings.

• Small sample for deep contextualised articulation of phenomenon • Focus on selecting samples likely to yield varied characteristics of

the research subject, in high concentration • Variation embraced for explaining types of variation rather than

focussing on simplistic averages

Purposive & Theoretical Sampling • Aim to identify sources & actors best providing richest

account of phenomenon under investigation

• Purposive sampling:

o Select sample most likely providing in-depth information addressing key research questions

• Theoretical sampling

o Select sample of cases, events, actors corresponding to or contributing to the chosen informing theory

o Select sample most likely to directly contribute to inductively developing theory from collected field data

o Sample selection criteria may broaden as theory develops

Credibility • Credibility as term & concept redefined away from quantitative

concept of ‘validity’ • Qualitative focus is on ‘credibility’ or ‘trustworthiness’ of data and

findings • The test of these……………………….. • Do the data analysis & findings:

o Instil confidence that this is an authentic & genuine account? o Deliver a plausible & convincing conclusion about the phenomenon?

• ‘Authenticity’ differentiates the qualitative from quantitative researcher o They have been personally engaged in the field with the actors!

• Various emergent terms also used relating to credibility: o E.g. Integrity, dependability, confirmability, explicitness, sensitivity

• Credibility emerges over time, admitting multiple voices o Authenticity & plausibility implicitly negotiated between actors, researchers &

readers

Thick Description • Thin description = detailed description • Thick description requires the researcher to:

o Develop empathetic understanding of actors & their context

• Researcher must probe, analyse & convey: o Actions, motives, symbols, meanings & situational contexts

• Access actors’ lived experiences: o Portray how and why they do as observed

• Thick description not limited to the world of the particular o Prompts thinking about and proposing qualitative generalisations

o Articulates findings potentially in other contexts

o Allows readers to assess transferability of findings to other settings

Unique – Typical - Exceptional • Structural corroboration

o Convergence of sources to reach same conclusion o Rather than repeated observations

• Single case as a mirror o Reflecting the world around it – potentially generalisable issues & behaviours o Can be a typical or exceptional case

• Typicality as a site selection guide can: o Yield insights transferrable to many similar cases o May require multiple cases to overcome idiosyncrasies within individual cases o Lead to selecting cases in similar contexts

• Exceptional cases can lead to generalisable insights: o Place same/similar actors in extraordinary settings o Examine extraordinary actors in common settings o Examine major change situations where actors may be leading change o Examining the unusual/unique exposes researcher to widest range of variation o Findings holding in extreme/diverse cases, can generalise convincingly o Outliers may challenge and change conventional wisdom

Why Have We Been Reticent to Generalise?

• Three cultural conditioners o Misplaced assumption that most qualitative studies

are ideographic & particularistic

o Continuing (un)conscious deference to quantitative community’s claims to superiority of their methods, findings and generalisability

o Lack of familiarity with unique qualitative concepts and forms of generalisation

Policy & Practice Engagement

• Qualitative generalisation reflects ‘natural’ intuitive laypersons’ approach to generalisation

• Reflects case by case development of expectations

• Addresses change at the particular, situational, holistic, contextual levels

• These are the ways of policy and practice

o Rather than mathematised research approaches & terminology

‘Generalisation’ • Do we persist with this term?...................Yes! • Retained almost unanimously across disciplines by

the qualitative research community • Best suited and most widely employed term for

communicating with practitioners & policymakers • Abandoning the term leaves the quantitative

community with freehold control over it • So the qualitative community must redefine it in

terms appropriate to communicating with and impacting on institutions, regulators, business, government, third sector & wider community