Upload
stephen-reynolds
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Analysing and interpreting cognitive
interview data: a qualitative approach
Presentation structure
• background• aims and objective of QT• design of cognitive interviews
• review of methods of analysis• NatCen approach• issues for discussion
Background
• aims and objectives of QT• does test question meet measurement objectives• if not, what problems arise• implications of problems for survey
• design of cognitive interviews• probe sheet• interviewing techniques• sampling strategy
Review of methods of analysis
• little written on analysis of cognitive interview data• two main methods cited:
• standardised coding scheme • qualitative analysis
• standardised coding scheme approaches are documented (usually)
• qualitative analysis descriptions are often scant on detail
Standardised coding schemes
• can be used to as stand alone question appraisal tool (e.g. QAS)
• can incorporate some elements of behaviour coding• interviewer has a problem reading the question or recording
the answer
• focus on cognitive Q & A model• comprehension/communication• recall/computation• bias/sensitivity (judgement issues)• response category • plus ‘logical issues’
Issues with standardised schemes
PROS• lend themselves to presenting data quantitatively (x Rs had this problem)• perceived (by some) to be more robust• process is replicable• useful in cross-national/ cross cultural settings, where standardisation important
CONS• need a lot of detailed codes under each main heading (particularly comprehension)• time consuming• loose context: why did R interpret Q in that way?• lend themselves to presenting data quantitatively (x Rs had this problem)
Qualitative approach
“Just naming and classify what is out there is usually not enough. We need to understand the patterns, the recurrences, the whys. As Kaplan (1964) remarks, the bedrock of inquiry is the researcher’s quest for ‘repeatable regularities’.”
Miles & Huberman (1994)
Approaches to qualitative analysis
Ethnographic accounts• detailed ‘thick’ descriptions of cultures or organisations
Life histories • analysed as individual cases or mined for common themes
Content analysis • identifies content and context of documents, often involves counting
(not strictly qual)
Grounded theory• generates analytical categories and the links between them through
an iterative process of collecting and analysing data
Approaches to qualitative analysis
Narrative analysis• examines how a story is told and the intention of the teller
Conversation analysis• examines the structure of (usually) naturally occurring
conversations
Discourse analysis• focuses on how knowledge is produced through the use of
language
Interpretative analysis • attempts to present and re-present the world of those
studied, by identifying and describing substantive themes, and searching for patterns between them
Key stages of the analytical process
Data management• identifying themes• sorting and reducing
data
Generation of findings• describing• classifying• finding linkages and
patterns• identifying explanations
Characteristics of ‘good’ analysis system
• Remains grounded in the data• Transparent data reduction process• Facilitates and displays ordering• Permits within and between case analysis
Seeking wider applications
Developing explanations
Detecting patterns of association
Establishing typologies
Identifying elements & dimensions
Summarising / synthesising data
Sorting data
Tagging data
Identifying initial concepts / themes
Primary data
Datamanagement
Descriptiveanalysis
Explanatoryanalysis
The analytical hierarchy in qualitative research
Data collection
?
Thematic analysis - purposes and principles
Structured display of data by theme (Q) across all cases
Creating categories and classifying data within them*
Demonstrating range and diversity
Using examples to illustrate and amplify
Must be comprehensive
Labelling and categorising must be valid
Carrying out thematic analysis - NatCen approach
Familiarisation with dataIdentification of factors
• highlight, summarise, provisionally label
Categorisation• is this a different manifestation of that• is this a subset of that• is this of the same order as that
Iterative process of refinementStart close to the data - become more abstract and
interpretativeMust be comprehensiveAim is analytical coherence
Looking for explanations
Informed by:• hunches and hypotheses• reflections during fieldwork and analysis• other research or theories
Process involves:• detailed within case analysis• comparison between cases• repeated interrogation of the data
milking datamoving back and forth between casessearching for rival explanations
Expect multiplicityMust be comprehensive
Summary of NatCen approach
Detailed notes made on interviews
Notes reviewed
Chart set up
Notes charted (chart revised)
Charts reviewed
Data interpreted
Findings emerge
Recommendations made
Report written
Example chart
Case details Q1Emp status
Q2Job title
Q3No. hrs wrklast wk
Q4No.employees
DC001, M, 52yrs,Self-employed
AnswerCompResponseOther issues
DC002, F, 24, pttemporaryworker & ftstudentDC003, F, 36, ftemployee
DC004, M 68,retired, self-employed (pt)
Issues to consider
• better documentation of qualitative analysis approach
• integration of code frames within thematic approach
• development of best practice for analysis of cognitive data
• cognitive interview data as one component of testing strategy
• collaborate findings using other data sources (e.g. split ballot experiments)