Notes are not enough! Why relying on your notes will lead you down the garden path

NOTES ARE NOT ENOUGH!

@ashdonaldson Design Research 2017, Sydney

Resurgence of research

I decided that we shouldn’t even look at the notes. We’d just go

with our gut.

Senior Researcher playing back findings

We can all capture stories…

…but are they true?

*sniff*

Communication feels intuitive…

…but it’s really complex

The great enemy of communication, we

find, is the illusion of it.

William H. Whyte

Communication can be critical

Melbourne tower, this is Alpha Echo Charlie inbound from the South at 5,000. Request descent to join downwind of zero niner at 1,000.

An interview is a directed conversation.J. & L. Lofland

How we think about thinking…

Mental models of the mind

Hydraulic

Mental models of the mind

Hydraulic Mechanistic

Mental models of the mind

Hydraulic Mechanistic Information Processor

Information Processor Model

- Fixed focus - Captures everything

Information Processor Model

- Fixed focus - Captures everything

- Logically interpreted - Systematically classified

Information Processor Model

- Fixed focus - Captures everything

- Logically interpreted - Systematically classified

- Stored faithfully - High definition

Information Processor Model

- Fixed focus - Captures everything

- Logically interpreted - Systematically classified

- Stored faithfully - High definition

How communication works…

It all starts with an intent

Sender

that intent exists within an environment

Sender

Interviewer’s environment

the intent needs to be expressed as an idea

Sender

Interviewer’s environment

Encodes

through a channel

Sender

Interviewer’s environment

EncodesChannel

carrying transmitted information

Sender

Interviewer’s environment

Encodes MessageChannel

which has to be perceived

Sender

Interviewer’s environment

Encodes MessageChannel Channel

[ attention is limited ]• We don’t have a fixed attention

• Serial switching: Can only attend to one thing at a time

• Easily distracted: Directed Attention Fatigue (DAF)

• Expensive and rapidly exhausted

• Affected by sleep, stress, nutrition, medications, age & exposure to external stimuli

[ perception is tiny ]• We don’t take everything in as input

• We perceive only tiny fragments

• Our minds construct a seamless movie

• We more readily perceive things we expect

• Pilots divide the sky into quadrants to scan for aircraft

• Look under a pier: you’ll notice more fish the longer you look

Inattentional Blindness: The Monkey Business Illusion - https://youtu.be/f94o3B3csYI

…and interpreted by the receiver

Sender

Interviewer’s environment

ReceiverEncodes Message Decodes

Participant’s environment

Channel Channel

[ cognition ]• Our brains do not function like CPUs

• Our cognition is based on pattern matching

• We draw heavily on past experiences and present cues

• We satisfice - stopping when we have enough information to make a conclusion

• We are subject to many predictable biases

Nothing is so difficult as not deceiving

oneself.

L. Wittgenstein

https://www.ted.com/talks/michael_shermer_on_believing_strange_things

[ memory ]• We do not store information like a hard drive

• Long-term memory is constructive

• The more you remember it, the further from the truth it is

• Short-term memory is fleeting

• Fills quickly (4 chunks - not 5 to 7)

• Gets replaced by the next thing we attend to

• Has limited duration (10 to 20 seconds)

Interviewer’s environment Participant’s environment

both parties have a history

Sender ReceiverEncodes Message DecodesChannel Channel

then a signal is sent back

Sender

Interviewer’s environment

Receiver

Receiver

Sender

Encodes Message

Message

Decodes

Participant’s environment

EncodesDecodes

Channel Channel

Channel Channel

Feedback Feedback

Noise Noise Noise Noise

…of course, there’s always some type of noise

Sender

Interviewer’s environment

Receiver

Receiver

Sender

Encodes Message

Message

Decodes

Participant’s environment

EncodesDecodes

Channel Channel

Channel Channel

Feedback Feedback

Noise Noise Noise Noise

Transactional model of communication

Sender

Interviewer’s environment

Receiver

Receiver

Sender

Encodes Message

Message

Decodes

Participant’s environment

EncodesDecodes

Channel Channel

Channel Channel

Feedback Feedback

Notes take effort

CPM-GOMS of a note

Add a third perspectiveInterviewer’s environment Participant’s environment

Notetaker’s environment

So what should you do?

1. Record your interview

When reviewing the footage I saw their expressions in the moments when I was looking

away. It was intense, and important to re-live some of these interviews seeing new

things.

Natalie Rowland Qualitative Specialist (redrollers)

Recording interviews: Tips• Craft a good recording release

• Aim for video. If not, use audio

• Skype or Facetime: We use eCamm’s Call Recorder

• Aim for a quiet, well-lit space in context

• Test your recording equipment

Recording interviews: Equipment

2. Get transcripts

We had 3 notetakers for one of our interviews (most

notetakers I’ve ever had) and I just combed the transcript. We missed about 15 critical

points that were said.

Alyce Lythall Senior Researcher (Tobias & Tobias)

Transcripts: Tips• Use participant coding e.g. T1P03

• REV.com US$1/min

• Clients appreciate research documentation

• Clean up your audio

3. Build in enough analysis time

Analysis: Tips• Create a cadence that allows digestion

• Allow at least 3x the research time

• Build the time into projects up front

• If you can’t sell analysis time, hide it

• Use Quicktime Pro or similar that will allow you to playback recordings at 3x

• Comb transcripts for insights and verbatims

Notes are not enough!• Recordings and transcripts give objective feedback:

• Re-live stumbles and correct next time

• Every interview recording is an opportunity to become a better researcher

• Recordings and transcripts = Good data in

Questions?@ashdonaldson

Tobias &

Tobias