View
19
Download
0
Category
Preview:
Citation preview
User interaction engineering
Measuring cognitive load for safety and efficiency
Ronnie Taib11 June 2015 - Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET
Acknowledgements
• Fang Chen
• Natalie Ruiz
• Eric Choi
• Julien Epps
• Yu Shi
• Bo Yin
• Peng Wang
• James Constable
• Asif Khawaja
• Kun Yu
• Ling Luo
• Jeremy Tederry
• Pega Zarjam
• Siyuan Chen
• Yang Wang
• Ben Itzstein
• Jessica Jung
• Anne Hess
2© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET
What is NICTA?
• Australia’s National Centre of Excellence in ICT• 700 Staff, 5 labs, $100m/y revenue
• NICTA objectives• Research Excellence in ICT• Wealth Creation for Australia
• Transforming Industry• Delivering over $3bn/y impact on GDP• Projects of national scale and impact
• New Industries• Fourteen spin-outs, on every three months
• Skills and Capacity• 22 University partners, 280 PhD Students
Kernel
device software
seL4 microkernel
Hardware
seL4
Hardware
seL4 isolates critical components from software failures
critical componentisolated and protected
untrusted, complex user interface
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 3
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 5
“One of the top 5 places on the planet to go
for detailed understanding of ITS”
Outline
• The problem
• Research
• Real-life applications
• Opportunities
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 6
The problemCognitive load theory
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 7
The problem
• The lack of “at least some cognitive availability and understanding of the situation” may have led pilots to ignore continuing alarms during the fatal accident on the Rio to Paris flight AF447
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 8
Baddeley’s modal model of working memory
• Definition• Level of perceived effort
associated with learning, thinking and reasoning (including perception, memory, language, etc)
• Available ‘space’ in working memory in comparison to the ‘space’ needed by a user to complete the task successfully
9
Cognitive Load Theory [Sweller et al. 98]:
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET
Spare capacity
Germane loadFor schema building, effort of learning
Extraneous loadRepresentation of the task, unnecessary for task completion
Intrinsic loadConcepts of the task
Working memory
Measuring cognitive load
• Subjective measures• Questionnaire: introspection, e.g. NASA TLX• Traditionally most consistent, but disrupts the task
• Physiological measures• Heart rate, EEG, galvanic skin response, eye activity• Intrusive/obtrusive, costly, difficult to analyse
• Performance measures• Intrusive, post hoc, requires a task metrics
• Behavioural measures• Eye gaze, mouse/pen movement, speech, posture• Response-based, non-obtrusive, real-time
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 10
Adapted from [Hockey, 2003]
Cognitive load and human interactionLong-term memory
Shared space (7±2)
Perception
Response
Visual processing(Visuospatial sketchpad)
Linguistic processing(Phonological loop)
Central executive
Muscular action
Excitation + vocal tract configuration
Gesture…
Multi-sensory
perception . . .
. . .
Short-term memory
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 11
e.g. latency, pitch, jittering
Response
Stimuli
The big picture: interaction design
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 12
System output
Perception
Memory
Response
System input
Content adaptation
Challenges
• Signal acquisition• Collection: obtrusive
• Processing: video, speech
• Calibration• Complex signals
• No direct mapping to cognitive load
• User, time and location dependent
• Real-time output• Signal sensitivity and time scale
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 13
Approach
• Baseline
• Signal fluctuations
• Classification using machine learning
Measure cognitive load unobtrusively and in real-time
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 14
Multimodal cognitive load measurement
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 15
Cognitive load
Subjective ratings
Task Performance
Physiological Behavioural
Mouse
Body movement
Eye activity
Pen input
Gesture
Speech
Linguistic
Fusion
Data-driven Knowledge-based
EOG
EMG
BVP
Temp
MEG
GSR
EEG
Generic process
Raw signalFiltering, cleaning
Feature extraction
Classification, regression, clustering
Cognitive load
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 16
Models
ResearchCognitive load measurement
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 17
Communicative Indicators of High Load
• Multimodality [e.g. Oviatt et. al., 2007]
• Increased use of integrated multimodal communication as load increases
• Speech signal [e.g. Yin 2008]
• Fundamental frequency, pitch, prosodic features
• Linguistic patterns [e.g. Khawaja 2008]
• Word types, sentence structures, pauses, pronoun use…
• Manual gesture [e.g. Ruiz, 2006]
• Semantic structures (redundant vs. complementary)
• Pen input [e.g. Ruiz, 2007; Yu, 2010]
• Geometric + temporal features of trajectories
• Eye-gaze [e.g. Chen, 2010]
• Pupil dilation
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 18
Study 1: traffic monitoring
• Tasks• Incoming information (text)
• Tagging Incident, Accidents, Events
• Notifying and Deploying crews
• Modalities• Speech
• Freehand gesture
• Hand Shapes
• 4 levels of difficulty
Function Buttons (for point and dwell gestures)
System Feedback Area
Task Display Area Main Map Area
Function Buttons (for point and dwell gestures)
System Feedback Area
Task Display Area Main Map Area
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 19
Difficulty
Study 1: working memory strategy
• As difficulty increases• Redundant Hybrid Complementary
• High load: multimodal input• Semantic chunks channelled to different
modalities• Least replication possible• Shift to areas marked exclusively for modal use• Similar to data acquisition “modality effect”
• Working memory strategy• Maximise modal working memory• Use less central executive resources, which can
be utilised for higher-order processes such as planning, understanding and hypothesis testing
Redundant
Spatial VerbalExecutiveA
B
A
B
Hybrid
Spatial VerbalExecutiveA
B
A
Complementary
Spatial VerbalExecutiveA
B
0
10
20
30
40
50
60
70
80
90
Level1 Level2 Level4
Q1MinMeanMaxQ3
Proportion of Purely Redundant Turns by Level
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 20
Study 1: automated gesture segmentation
• Manual annotation of gestures• Timestamping, annotating
• Pointing, dwelling
• Hand shapes
• Trajectory extraction from video
• Segmentation• Accuracy under 50ms
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 21
13000
Sp
ee
d
Time (ms)
Study 1: automated gesture segmentation
Gesture
identified
Start time End time
Ditch 1
Ditch 2 Ditch 3
Ditch 1
Ditch 2 Ditch 3
Start time before
Start time nowAuto Start time Auto End time
Manual start
time
Manual end time
1. Segment gestures using speed threshold and noise level
2. Compare result with manual segmentation
3. Spot and filter out ditches
4. Trace back to the previous local minima
Start time at the beginning
Start time after ditches were filtered out
Start time now© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 22
Study 2: reading data collection
• Primary task• Read 3 stories (1 page each), then oral Q&A session• Lexile ratings to quantify complexity (www.lexile.com)• From primary school level to post-graduate
• Secondary task• Listening/count beeps/numbers
• 4 levels of difficulty (incl. baseline)
• Modalities and data• Speech: reading + Q&A• Performance• Galvanic skin response (GSR)
Martin Luther King, Jr.'s "I Have a Dream" Speech
March on Washington, DC, August 28, 1963
I am happy to join with you today in what will go down in history as
the greatest demonstration for freedom in the history of our nation.
Five score years ago, a great American, in whose symbolic shadow
we stand today, signed the Emancipation Proclamation. This
momentous decree came as a great beacon light of hope to millions
of Negro slaves who had been seared in the flames of withering
injustice. It came as a joyous daybreak to end the long night of
their captivity.
But one hundred years later, the Negro still is not free. One
hundred years later, the life of the Negro is still sadly crippled by
the manacles of segregation and the chains of discrimination. One
hundred years later, the Negro lives on a lonely island of poverty in
the midst of a vast ocean of material prosperity. One hundred years
later, the Negro is still languishing in the corners of American
society and finds himself an exile in his own land. So we have come
here today to dramatize a shameful condition.
In a sense we have come to our nation's capital to cash a check.
When the architects of our republic wrote the magnificent words of
the Constitution and the Declaration of Independence, they were
signing a promissory note to which every American was to fall heir.
This note was a promise that all men, yes, black men as well as
white men, would be guaranteed the unalienable rights of life,
liberty, and the pursuit of happiness.
It is obvious today that America has defaulted on this promissory
note insofar as her citizens of color are concerned. Instead of
honoring this sacred obligation, America has given the Negro people
a bad check, a check which has come back marked "insufficient
funds." But we refuse to believe that the bank of justice is
bankrupt. We refuse to believe that there are insufficient funds in
the great vaults of opportunity of this nation. So we have come to
cash this check — a check that will give us upon demand the
riches of freedom and the security of justice. We have also come to
this hallowed spot to remind America of the fierce urgency of now.
This is no time to engage in the luxury of cooling off or to take the
tranquilizing drug of gradualism. Now is the time to make real
promises of democracy. Now is the time to rise from the dark and
desolate valley of segregation to the sunlit path of racial justice.
Now is the time to lift our nation from the quick sands of racial
injustice to the solid rock of brotherhood. Now is the time to make
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 23
Study 2: results
• Linguistic features
• Pauses: more and longer, increased latency with difficulty
• Complexity: decreased lexical density, longer and broken
sentences
• Speech-based classification
• 71% accuracy (speaker independent, up to 84% speaker
dependent)
Martin Luther King, Jr.'s "I Have a Dream" Speech
March on Washington, DC, August 28, 1963
I am happy to join with you today in what will go down in history as
the greatest demonstration for freedom in the history of our nation.
Five score years ago, a great American, in whose symbolic shadow
we stand today, signed the Emancipation Proclamation. This
momentous decree came as a great beacon light of hope to millions
of Negro slaves who had been seared in the flames of withering
injustice. It came as a joyous daybreak to end the long night of
their captivity.
But one hundred years later, the Negro still is not free. One
hundred years later, the life of the Negro is still sadly crippled by
the manacles of segregation and the chains of discrimination. One
hundred years later, the Negro lives on a lonely island of poverty in
the midst of a vast ocean of material prosperity. One hundred years
later, the Negro is still languishing in the corners of American
society and finds himself an exile in his own land. So we have come
here today to dramatize a shameful condition.
In a sense we have come to our nation's capital to cash a check.
When the architects of our republic wrote the magnificent words of
the Constitution and the Declaration of Independence, they were
signing a promissory note to which every American was to fall heir.
This note was a promise that all men, yes, black men as well as
white men, would be guaranteed the unalienable rights of life,
liberty, and the pursuit of happiness.
It is obvious today that America has defaulted on this promissory
note insofar as her citizens of color are concerned. Instead of
honoring this sacred obligation, America has given the Negro people
a bad check, a check which has come back marked "insufficient
funds." But we refuse to believe that the bank of justice is
bankrupt. We refuse to believe that there are insufficient funds in
the great vaults of opportunity of this nation. So we have come to
cash this check — a check that will give us upon demand the
riches of freedom and the security of justice. We have also come to
this hallowed spot to remind America of the fierce urgency of now.
This is no time to engage in the luxury of cooling off or to take the
tranquilizing drug of gradualism. Now is the time to make real
promises of democracy. Now is the time to rise from the dark and
desolate valley of segregation to the sunlit path of racial justice.
Now is the time to lift our nation from the quick sands of racial
injustice to the solid rock of brotherhood. Now is the time to make
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 24
Study 3: traffic incident management
• Task• Creating detours • Creating green light corridors• Requires calculations: scratchpad available
• Modalities
• Digital pen input
(tablet monitor)
• Speech • GSR
• Data• Performance• Subjective ratings
3 levels of difficulty1. Easy: 6 streets2. Medium: 10 streets3. Hard: 16 streets
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 25
Study 3: degeneration of interactive shapes
• Geometric analysis of trajectory• 12 features incl. angle at start stroke,
angle and end stroke, duration, length, sharpness etc. [Rubine 1991]
Selection Examples Shape ExamplesStandard form and Sample inputs
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 26
Study 3: degeneration of interactive shapes
• Malahanobis distance (MDIST)• Weighted Euclidean distance
• Stdev away from standard form captured during training
As difficulty increases, the curve moves away from 0, indicating a greater degree of degeneration (statistically significant)
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 27
Study 4: basketball skill acquisition (with AIS)
• Player Formation Recall• 10 second video played• Recalling increasing numbers
• Modalities• Pen input
(Tablet monitor)• Speech • GSR• Eye tracking
• Data• Performance and subjective ratings
3 levels of difficulty- Low (Easy): 3 players- Medium (Med): 6 players- High (Hard): 10 players
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 28
Study 4: pen input
Attacker
Defender
Ball carrier
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 29
Study 4: pen gesture temporal features
• Duration• Decreased with tasks difficulty in
82% of gestures
• Significant decrease from Low to High (t-test, p<0.05) in 50% gestures tested
• Velocity• Increased with difficulty in 78% of
gestures
• Significant increase from Low to High (t-test, p<0.05) in 44% gestures tested
0
100
200
300
400
500
600
cross circle ball carrier
Subject 9 Gesture Duration
easy
medium
hard
0
0.05
0.1
0.15
0.2
0.25
0.3
cross circle ball carrier
Subject 12 Velocity
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 30
Study 5: Stroop test
Simply read aloud each word
RED YELLOW BLUE GREEN BLACK
PINK ORANGE BROWN GRAY PURPLE
GREEN RED BLACK BLUE YELLOW
PURPLE GRAY PINK ORANGE BROWN
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 31
Study 5: Stroop test
Cognitive Load Instructions Example
Low Read colour
words
Blue, green
Yellow, red
Medium Name word
colour
Blue, green,
red
High Same as
medium load
except with time
pressure
Blue, green,
red
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 32
Study 5: Stroop test
• Speech only classification: 79% accuracy (3 levels)
• iPhone implementation
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 33
Real-life applicationsImproving efficiency and safety
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 34
Real-life case studies
• Emergency communications centre in North America (ambulance dispatch)• Speech classification: 82% accuracy
• High load detection rate: 96%
• Contact centre operator in Australia (5000+ seats)• Reduced attrition rate by 50% by automated operator selection
• www.braingauge.com
• Air traffic controllers (training centre)• Speech classification: 88% accuracy
• Estimates produces every 2 minutes
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 35
Our safe future
• Adaptive cruise control
• Adaptive headlamps
• Advanced automatic collision notification
• Automatic parking
• Automotive night vision with pedestrian detection
• Distance control assist
• Traffic sign recognition
• Lane departure warning system
• Drowsiness warning
• Blind spot monitoring
• Driver monitoring system
• Dead man's switch
• Platooning
• Robotic car or self-driving car
Source: EU Commission’s Intelligent Car Initiative (2006)
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 36
But inattention is an omnipresent danger
Sleeping can be dangerous
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 37
Inattention at the wheel
• 10% of car crashes in USA• An Examination of Driver Distraction as Recorded in NHTSA Databases [NHTSA
2009]
• Over 25% of crashes involved a form of driver inattention8.3% of drivers were distracted at the time of their crash• The Role of Driver Distraction in Traffic Crashes [Stutts 2001]
• Technology-based distraction is only 15% of distraction related incidents• The impact of driver inattention on near-crash/crash risk: An analysis using
the 100-car naturalistic driving study data [Klauer 2006]
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 38
Driver mental state monitoring
• Identify mental states at risk• Don’t wait until it is too late
• Not just phones and distractors
• Use existing sensors: dashboard camera, seat/wheel sensors, body sensors
• Unique capability• Collaboration NICTA, Emotiv, Fraunhofer
• Cognitive load expertise Clever simulator tasks
• Machine Learning expertise Models out of noisy data
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 39
Theoretical challenges
• Cognitive load• How to quantify judgement
• Causality between task, performance, physiology and behaviour
• Simulator vs. test track vs. FOT vs. reality
• Machine learning• What features? Physiological, behavioural
• Time series with varying sampling rate
• Features with different delay and duration after stimulus
• Online learning, user independent seed models?
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 40
Practical challenges
• Task design• Realistic tasks• Controlled cognitive load, emotions, UX• Elicit right amount of speech, movement
• Signal acquisition• Sensors• Loggers: NICTA’s OML• Machines and synchronisation issues• Reducing manual annotation
• Implementation• A lot of work to create the virtual scene
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 41
Preliminary user study
• 19 voluntary participants
• 45 min each
• Sensors• Physiological: GSR, EEG, Pupil dilation, temperature
• Behavioural: wheel, pedals, speech, eye gaze, wrist and head motion
• Performance: task markers, repeat loops with varying conditions, use of devices (in/out)
• Audio-visual: speech, face video, glasses camera, pupils
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 43
Driving simulator
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 44
Instrumentation
Eye gazePupil dilationSkin conductanceEEG
PostureUpper body videoSpeechWheel and pedals
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 45
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 47
Adjusting the radio
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 48
Sending a text message
Preliminary study: glossy outcome
www.forthebetter.com.au
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 49
Frustration study
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 50
Frustration study: challenges
• Frustration in simulator • Simulated frustration?
• Sensors• Positioning?
• Features?
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 51
Frustration study: challenges
0
0.2
0.4
0.6
0.8
1
1.2
-20000 0 20000 40000 60000 80000 100000 120000 140000
actions
P01S04T0
P01S04T1
P01S04T2
P01S04T3
P01S04T4
P01S04T5
P01S04T6
P01S04T7
P01S04T8
P01S04T9
Action on the pedals Moving on the seat
Action on the wheel
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 52
Frustration study: results
• Experiment• 11 subjects
• 128 = 16 sensors * 8 features
• 1241*128 samples (10s each)
• Analysis• Multinomial regression with forward feature selection
• Support Vector Machine
• Infinite Gaussian Mixture Model
• Neural nets
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 53
Frustration study: results
• Accuracies
• Features selection• Two of the classes perform much better
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 54
Method MRMR + FFS
SVMSVM+GM
MGMM
Neural Network
Accuracy 41% 39% 33% 1% 15% 35%
class
1
class
2
class
3
class
4
class
5
class
6
class
7
class
8
class
9
class
10
class
11
macro
average
Standard
deviation
precision
0.20
0
0.24
3
0.25
0
0.30
0
0.28
3
0.13
3
0.28
6
0.69
0
0.57
6
0.34
2
0.13
3 0.312
0.173
recall
0.18
0
0.18
0
0.12
0
0.30
0
0.26
0
0.08
0
0.24
0
0.80
0
0.68
0
0.52
0
0.22
0 0.325
0.236
F-Score
0.18
9
0.20
7
0.16
2
0.30
0
0.27
1
0.10
0
0.26
1
0.74
1
0.62
4
0.41
3
0.16
5 0.312
0.203
Driver mental state monitoring: opportunities
• Driver assistance systems• Real-time detection of mental states at risk: overload, fatigue, daydreaming…
• Testing and training drivers• Assessing driver reactions, ensure appropriate level of proficiency
• Leverage current sensors• Dashboard camera, seat/wheel sensors, body sensors
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 55
OpportunitiesUser interaction design
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 56
Opportunities
• Internet of things• Personal monitoring trend
• Car sensors
• Smart buildings
• Smart fabrics
• Analytics technology is mature• Multimodal
• Real-time
• Simply need to connect
© 2015 NICTA Joint Electrical Institutions Sydney - Engineers Australia, IEEE, IET 57 © 2015 Emotiv
Recommended