View
213
Download
0
Category
Tags:
Preview:
Citation preview
Innovative Design & Manufacturing Research Centre
University of Bath
“World-leading research in engineering design and manufacture.”
1
IdMRC Social Research MethodsAutumn Lecture-Workshop Series
BATH September 2009 2
Design Research Methods:
Observation and Coding
Janet McDonnell
BATH September 2009 3
Note
This collection of slides supports a lecture on the topic of observation and coding.
The slides in this set are not intended to serve independently of the lecture as a stand-alone introduction to, or overview of, the subject.
The lecture includes the use of video and other visual material excised from this sub-set of slides which is solely intended to support note taking at the lecture.
Janet McDonnell
BATH September 2009 4
Outline
[software tools] CASE STUDY: from the categorical to the conjectural Knowledge claims – from nature to narrative Traceability and reliability What is the data from observation Approaches to research using observation data CASE STUDY: single dataset, multiple approaches to analysis Pointers for quality
BATH September 2009 5
Software Tools - Let’s get them out of the way
Behavioural Research : recording behaviour, eye tracking, facial expression and other types of behaviour recognition, synchronising with transcripts and support for coding, etc.
www.mangold-international.com (Interact) www.tracksys.co.uk (Observer XT)
Qualitative Data Analysis – content analysis of texts – from transcripts, interview data, etc. linking in images and video clips, ‘qualitative data base management’, collaboration
www.qsrinternational.com (Nvivo) www.atlasti.com (ATLAS-ti)
Quantitative data analysis Stats packages, graphing and diagramming tools
Transcription aids
BATH September 2009 6
Anatomy of a small scale enquiry
An investigation into the exchange of ideas and information between an architect and building users in the early stages of a building (re)design project before the design brief or any drawings have been produced.
We look at the type of information users exchange.
We are interested in what influences the information exchanges - context of the meetings and- the conversational strategies of the architect.
Classifications were determined in advance of coding the data.
BATH September 2009 7
Data and method
Participant observation: interaction as it occurs in natural setting
Data consists of transcripts acquired from audio-taped meetings
Focus on manifest content Segment transcripts into units of analysis Apply predetermined codes* Inter coder issues Data analysis and data interpretation Construct an account of the topic
* Based on P. Medway (2000) Writing and Designing in Architectural Education in
A.Pare (ed) Transitions: writing in academic and workplace settings, Hampton Press, pp 89-129.
BATH September 2009 8
Conversation summaries
CONVERSATION 1 2 3 4
Length mins:secs 10:20 25:04 5:27 20:37
No. of turns 149 285 43 247
No. of information contributions
16 43 12 30
BATH September 2009 9
Information contributions
CONVERSATION 1 2 3 4
Building User 9 23 12 9
Architect 6 11 9
Deputy Head Teacher 1 8 12
BATH September 2009 10
Codes for information contributions
CODE DEFINITION
1 Functional or structural naming
Elements and properties of the building or space (floor, exposed brickwork, etc.) – its objective
character
2 Perceptual awareness Attributes of the building or space that can be perceived (visual, acoustics, etc.)
3 Phenomenological experience
Feelings and associations made when experiencing the space (e.g. solidity)
4 Symbolic meaning Evocation of ideas unrelated to the structural form (mystery, memory, etc.)
BATH September 2009 11
“shelving reachable from the floor”“easy to clean surfaces”
“light and quiet study areas”
“a place where it feels safe to think ”
“a link between our past and our future”
BATH September 2009 12
Information contributions
CONVERSATION 1 2 3 4
Building User 9 23 12 9
Architect 6 11 9
Deputy Head Teacher 1 8 12
BATH September 2009 13
It was noticeable that some users were competent at discussing contributions in a more informed, technical manner than others. There were differences between the users’ understanding of the structural properties of buildings and their range of abilities to articulate spatial preferences for the design of the workspace. Compare U3: … my wish is I suppose to take out this joint wall and incorporate this store and this area into a larger area …
withU1: … if I had my wish then I would basically um take this wall down …
Although conversation 3 was shorter than the others, U3’s ability to converse in a designerly language made it easier … to share a common understanding of the properties of the space.
BATH September 2009 14
Types of information contribution
CONVERSATION 1Ar BU
2Ar BU
3Ar BU
4Ar BU
Functional 6 8 9 22
9
9 14
Perceptual 7
Phenomenological 2 2 4
3
2 4
Symbolic 3
BATH September 2009 15
The contribution of information with a symbolic meaning was infrequent. Symbolic references only cropped up in conversation 4 … all contributed by the deputy head teacher rather than the primary user of the space. We note also that the symbolic meaning discussed concerned what it meant to be part of the school and only once was the symbolic meaning of the space being designed mentioned strictly. The three instances are
D: … this is regarded as a privilege … to give them a kind of base so that they can start acting can I say more acceptable, normal circumstances in their reaction and inter-reactions with other peopleand D: … that’s why we still insist on them being in school uniform so they’re still part of the school to stamp identity U1: er yeah the whole idea is that they’re to be re-integrated into schoolandD: … we use this facility as an escape to do course work in the privacy of here as opposed to the pressures in the school
BATH September 2009 16
Claims, evidence and … truth
what is provably true logico-deductive reasoning, exhaustive
search
what is probably true statistical reasoning
what is plausibly truethe evidence of arguments that are
sincere and will convince a reasonable audience
BATH September 2009 17
Knowledge claims from research
Provable and ProbablePositivist : natural sciences : experimental reproducibility;
falsifiable hypotheses
PlausibleInterpretivist : cultural and social sciences : plausibility –
traceability; recoverability; narratives providing understanding
Emancipatory‘Critical’ theorist : social sciences : emancipatory force
BATH September 2009 18
Transparency in enquiry
Declared or shared framework, methodology, methods: allow process by which results are obtained to be recovered by others
The process of enquiry may lead to contributions to knowledge in each aspect (framework, methodology, research area)
Sharing of the meaning-making addresses ‘subjectivity’ objections – a route to agreement
framework of ideas
methodology
area of research concern
BATH September 2009 19
Inter coder reliability
Context: decisions about what is required to adequately serve the claims and evidence
Kappa coefficient is a measure of agreement between coders on the classification of qualitative data
Measures agreement in using a defined coding scheme in a prescribed manner.
Kappa takes into account the potential for chance agreement.
Does not take account of degrees of disagreement. Is a good test of whether categories are sufficiently distinct
from each other and sufficiently clearly defined for more than one coder to code reliably using them. So can be used to test identification of units of analysis and application of codes.
[J Carletta (1996) Computational Linguistics 22(2) 249-254 Assessing agreement on classification tasks: the kappa statistic. ]
BATH September 2009 20
45 Adrian OK ++ so moving into a little bit more detail46 [paper shuffling, 7 seconds pause]47 Adrian erm starting with the entrance, we arrive at the car park here and we 48 move forward to the entrance and waiting area here we've upped the 49 importance of this area one of the items on the minutes last time was 50 to increase the size of the waiting area 51 Angela yes52 Adrian and we virtually doubled in size errm we've actually extended the 53 canopy over it as well to make more of an entrance to make more of a 54 greeting zone so there's lots of space here with seats for people to wait 55 for taxis or whatever at the end of their end of their 56 Angela /hmm-mm\57 0.03.00 Adrian /service\ places for people to to actually stand and mill in if they wish
58 to stand out here before they move into the waiting area then come 59 into the porte-cochere so it's an even grander entrance 60 Angela /OK yes\61 Adrian /than we had\ before 62 Adrian erm one of your next items on the list was to widen the access way so 63 that two vehicles perhaps two limousines or a hearse and limousine 64 could park side by side in this area so this has been widened so its 65 now possible as I say for two vehicles ++ to park here ++ before 66 moving through further on ++ one of the next points made was the size 67 of the sanctuary we've increased the diameter of this by correct me if 68 I'm wrong Toby three hundred millimeters internally69 Toby it’s three point four
Extract from a transcript
BATH September 2009 21
Transcription conventions + pause of one second ( ++ two seconds, and so on)
…/….\… crude indicators of simultaneous speech…/……\…
NO emphatic stress
GOOGLE names of people, organisations, and so on
…… material omitted
- incomplete or cut off utterance
( ) unclear utterance
(over there) unclear utterance, transcriber’s best guess within brackets
[points at drawings] transcriber’s comments
[laughs] paralinguistics
[For a more comprehensive scheme refer to Gail Jefferson’s notation described in J Atkinson and J Heritage (eds) Structures of Social Action (1984) CUP, pp ix-xvi]
BATH September 2009 22
Ways of approaching research using observation data
Starting with a ‘template’ e.g. a model or a theory, classifications determined in advance
that are imposed on the data
Starting with the data e.g. looking for emergent patterns, evolving classification to
cover the phenomena observed (coding scheme not inevitable consequence)
Flexible research agenda vs. rigid research agenda e.g. starting with a scheme but being open to the
unclassifiable (and to new classes); developing hypotheses to test based on indications of possible patterns, causations
BATH September 2009 23
Common Dataset Multi-angle video recordings, plans, drawings, sketches, flip-charts,
transcripts from meetings from two authentic design projects
Architectural Design Project Data Two meetings 8 months apart between architect and
clients/building users Brief to design a crematorium with chapel and related provision
including offices, vestry, waiting rooms, parking, landscaping Engineering / Product Design Project Data
Two meetings 3 days apart among engineers and other specialists from the same company – multi-disciplinary groups
Brief to develop novel product ideas for sort of digital pen to exploit a new technology
24 different studies on themes relevant to research into design thinking
Anatomy of a large collaboration: DTRS7
BATH September 2009 24
How the data was tackled
Styles of approaching the data Starting with a ‘template’ : e.g. a model or a theory Starting with the data Flexible agenda vs. rigid agenda
Themes Understanding Process Values in Designing Aspects of Design Cognition Design Process Models Language, Discourse, and Gesture Designing Contexts Objects, References, and Representations
BATH September 2009 25
Types of analysis : imposition of a model or theory
Focus of attention: uses of analogy in designingFinding: a previously unreported use of analogy,
namely function-finding in creative design[Ball and Christensen, 2009]
Focus of attention: object references during designingFinding: object references in meetings between co-
workers are directed effectively to draw attention efficiently to features
essential to the comparison [Stacey et al., 2009]
Focus of attention: language of appraisalsFinding: positive and negative appraisals have
different effects on the focus of attention and knowledge generation during design
[Dong et al., 2009]
BATH September 2009 26
Types of analysis : fine grained analysis of the data
Focus of attention: gestures and individual turns at talkingFinding: reveals fundamental distinctions between
view of designing as goal-oriented and as goal-directed [Glock, 2009]
Focus of attention: designing through talk-in-interactionFinding: design concept has a special status – less
negotiable – the preserve of the designated ‘design’ expert
[Luck, 2009]
Focus of attention: social order – the ‘rules’ of interactionFinding: adherence to interactional norms interferes
with the ‘rules’ of brainstorming[Matthews, 2009]
BATH September 2009 27
Types of analysis : testing the limitations of prior findings or models
Focus of attention: consideration of context during designingFinding: support for claims that experienced designers explore broad context before close context is considered in depth
[Atman et al., 2009]
Focus of attention: a design team’s development of shared notions about the task and the team’s operation (indicated by verbal communication); based on a prior model of the development of ‘sharedness’ in teamsFindings: some expected findings – predicted by the model; some unexpected findings (not predicted by prior model) – only some of these accounted for by particularities of the data
[Badke-Schaub et al., 2009]
BATH September 2009 28
Quality of work
For in-vivo studies err on the side of over collection of data And/or iterate over observation/data collection and analysis … and don’t make excuses Coding – invest in adequate granularity (beware of findings
which are artifacts of the coding or analysis) An underused question: what else could account for the
findings? Consider the arguments, claims and evidence explicitly at
epistemological level Choose research methods that will provide the supporting
evidence you need (e.g. inter coder agreement if that is critical)
Choose language with care – consider the ‘baggage’ e.g. ‘hypothesis’, ‘subjects’
BATH September 2009 29
Further information
The ‘small scale enquiry’ was conducted with Dr Rachael Luck of Reading University and is published in Design Studies 27 (2006) pp 141-166.
DTRS7 work is published as a book About:Designing J.McDonnell and P.Lloyd (eds), 2009 and parts of it in two journal special issues Design Studies 30 (2) and CoDesign 5 (1) both in March 2009.
Janet McDonnell Central Saint MartinsSouthampton RowLondon WC1B 4APj.mcdonnell@csm.arts.ac.uk
www.csm.arts.ac.uk/janet-mcdonnell.htm
Recommended