A Toolkit for Managing User Attention in Peripheral Displays Tara Matthews, Anind K. Dey*٭, Jennifer Mankoff٭, Scott Carter, Tye Rattenbury EECS Department,

A Toolkit for Managing User Attention in Peripheral

DisplaysTara Matthews, Anind K. Dey*٭, Jennifer Mankoff٭, Scott Carter, Tye Rattenbury

EECS Department, UC Berkeley*Intel-Berkeley Research

currently of HCII, Carnegie Mellon٭

10/27/2004 Peripheral Display Toolkit 2

Problematic Scenario• Bus schedule

– Requires a task switch to use– Doesn’t afford continuous monitoring

RTE LAWR GRIZ EUCL CNTRNUM HALL SR. GRIZ SHAT65 605a 616a 630a 65 635a 646a 700a 65 703a 716a 730a 65 735a 746a 800a 65 803a 816a 830a 65 835a 846a 900a 65 903a 916a 930a 65 935a 946a 1000a 65 1003a 1016a 1030a 65 1035a 1046a 1100a 65 1103a 1116a 1130a 65 1135a 1146a 1200p 65 1203p 1216p 1230p 65 1235p 1246p 100p 65 103p 116p 130p 65 135p 146p 200p 65 203p 216p 230p 65 235p 246p 300p65 303p 316p 330p 65 335p 346p 400p


Solution: Peripheral Displays• Provide awareness with min attention

• Separate from primary task

• Bus LED Display+ Get bus info at a glance+ Dynamically manages attention it attracts– Hard to accomplish


Why is creating PDs hard?• Need to abstract info to be glance-able

• Need mechanisms for dynamically managing attention PDs attract:– Deciding attention levels to attract

(notification levels)– Displaying info appropriately (transitions)

• Our contribution:– Peripheral Display Toolkit (PTK) supports

these key issues in PD creation


Example PTK Applications• Remote Activity

– Social Guitar– Audio Monitor– Motion Monitor– Remote Awareness Display

• Bus Displays– Bus Mobile– Bus LED

• Instant Messenger Status

IM Picture FrameSocial Guitar Bus LED BusMobile

Orb showing remote activity























Example PTK Applications+ Manage attention with

abstraction, notification levels, & transitions

+ Glance-able+ Few lines of code+ Simplified code design

& code re-use




Simple Example: Motion Monitor

no/low motion

medium motion

high motion


Outline• PTK: Motivation & Examples

• Managing attention– Abstraction– Notification levels– Transitions

• Toolkit Architecture & Library

• Conclusion


Managing User Attention

• Attention is a crucial issue in design– Significant impact on success– More info can be monitored w/o overburdening

• Attention management is difficult– Used to designing apps that attract full attention– Attention attracted should match info urgency

• Literature surveys led to key characteristics of attention management

info urgency user attention

Attentionnone focal


Attention Management Characteristics

info urgency user attention

From survey of peripheral displays:1. Abstraction

• enables lower-attention info consumption

2. Notification levels• help determine how much attention to

attract based on urgency of info

3. Transitions• enable display of info to match urgency


Abstraction• Enables lower-attention consumption of info• Extracting features or reducing fidelity of info• Motion Monitor: replaces video images with

general motion amount (high, medium, none)

video abstraction amount of

images motion

• high motion

• medium motion

• no/low motion


Notification Levels• Indicate the urgency of info

– Low levels = non-urgent info– High levels = more urgent info

• Motion Monitor: doesn’t show urgent info, so uses low notification levels


More on Notification Levels

• Attention is (roughly) scalar – PD designers must decide where on the scale displayed info should fall

0% 100%attention

awareness

unconscious



Attention Type1. None

0% 100%attention

awareness

unconscious



Attention Type1. None

Notification LevelIgnore

0% 100%attention

awareness

unconscious



Attention Type1. None2. Inattention

Notification LevelIgnore

0% 100%attention

awareness

unconscious



Attention Type1. None2. Inattention

Notification LevelIgnoreChange blind

0% 100%attention

awareness

unconscious



Attention Type1. None2. Inattention3. Divided

Notification LevelIgnoreChange blind

0% 100%attention

awareness

unconscious



Attention Type1. None2. Inattention3. Divided

Notification LevelIgnoreChange blindMake aware

0% 100%attention

awareness

unconscious



Attention Type1. None2. Inattention3. Divided4. Focused

Notification LevelIgnoreChange blindMake aware

0% 100%attention

awareness

unconscious



Notification LevelIgnoreChange blindMake awareInterrupt

Attention Type1. None2. Inattention3. Divided4. Focused

0% 100%attention

awareness

unconscious



Notification LevelIgnoreChange blindMake awareInterruptDemand action

Attention Type1. None2. Inattention3. Divided4. Focused5. Focused + interaction

0% 100%attention

awareness

unconscious


Transitions

• Enable display of info to match urgency

• Depend on notification level– Ignore: no changes– Change blind: fading, slow motion, tiny

updates– Make aware: discrete, abrupt updates– Interrupt: flashing, beeping, vibration– Demand action: interrupt until user does action

• Motion Monitor: varies pulse speeds for different notification levels


Outline• PTK: Motivation & Examples

• Managing attention– Abstraction– Notification levels– Transitions

• Toolkit Architecture & Library

• Conclusion


PTK Architecture1. Support for managing impact on human

attention using abstraction, notification levels, and transitions

2. Simplified code design and code re-use

3. Library of common PD components

Output

Transition

Input

NotificationMap

Abstractor


Motion Monitor Architecture

Input-Side Output-Side

Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans


Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans OrbWidget

Trans




Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans


Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans OrbWidget

Trans




Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans


Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans OrbWidget

Trans

motion = medium




Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans


Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans OrbWidget

Trans

motion = medium

notification level = change blind




Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans


Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans OrbWidget

Trans

motion = medium

notification level = change blind




Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans


Motion MonitorPTK

DiscoveryServer

DiscoveryServer

CameraInput

MotionAbstractor

NotificationMap

OrbWidget

Trans OrbWidget

Trans

5 lines of logic code


Remote Awareness Display

Count # people,How much motion?

Phone ring?Voices?

Add notification level


Motion Monitor Remote Awareness Display


Motion Monitor

DiscoveryServer

CameraInput

PTKDiscovery

Server

MotionAbstractor

Exact Match(Motion) Orb

WidgetTrans


Motion Monitor

DiscoveryServer

CameraInput

PTKDiscovery

Server

MotionAbstractor


WidgetTrans



Input-Side Output-SideDiscoveryServer

CameraInput

PTKDiscovery

Server

MotionAbstractor


WidgetTrans

Motion Monitor


CameraInput

PTKDiscovery

Server

MotionAbstractor


WidgetTrans Orb

WidgetTrans

Motion Monitor




CameraInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor


WidgetTrans

Motion Monitor


CameraInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor


WidgetTrans Orb

WidgetTrans

Motion Monitor




CameraInput

AudioInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor


WidgetTrans

Motion Monitor


CameraInput

AudioInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor


WidgetTrans Orb

WidgetTrans

Motion Monitor




CameraInput

AudioInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor

Exact Match(Motion)

Exact Match(Audio)

Threshold(# People)

OrbWidget

Trans

Motion Monitor


CameraInput

AudioInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor

Exact Match(Motion)

Exact Match(Audio)

Threshold(# People)

OrbWidget

Trans OrbWidget

Trans

Motion Monitor




CameraInput

AudioInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor

Exact Match(Motion)

Exact Match(Audio)

Threshold(# People)

OrbWidget

Trans

Orb Wrapper

Motion Monitor


CameraInput

AudioInput

PTKDiscovery

Server

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor

Exact Match(Motion)

Exact Match(Audio)

Threshold(# People)

OrbWidget

Trans OrbWidget

Trans

Orb Wrapper

Motion Monitor



54 – total lines of code


CameraInput

AudioInput

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor

Exact Match(Motion)

Exact Match(Audio)

Threshold(# People)

OrbWidget

Trans

Orb Wrapper

TickerWidget

Trans

Tikr. Wrapper

PTKDiscovery

Server



CameraInput

AudioInput

MotionAbstractor

# PeopleAbstractor

PhoneAbstractor

VoiceAbstractor

Exact Match(Motion)

Exact Match(Audio)

Threshold(# People)

OrbWidget

Trans OrbWidget

Trans

Orb Wrapper

TickerWidget

Trans TickerWidget

Trans

Tikr. Wrapper

PTKDiscovery

Server



General Architecture Diagram

Input-Side DiscoveryServer

Output-Side

Peripheral Display 1

(Abstractors) (NotificationMaps)

(Output Widgets w/ optional Transition)


Input 1

Input 2

Input N

Abs. 1

Abs. NPTK

DiscoveryServer

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 1Trans

Out 2

Out N

.

.

. …

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 2Trans

Out 1

Out N

…

Input-Side DiscoveryServer

Output-Side


(Abstractors) (NotificationMaps)

(Output Widgets w/ optional Transition)


Input 1

Input 2

Input N

Abs. 1

Abs. NPTK

DiscoveryServer

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 1Trans Out 1Trans

Out 2

Out N

.

.

. …

Abs. 2

Abs. N

Abs. 1 N.M. 1

N.M. 2

N.M. N

Out 2Trans Out 2Trans

Out 1

Out N

…


Library Components• Input

– audio, camera, Phidgets, Context Toolkit, online calendars, news, stocks, weather,Web page parser, serial port communication




• Output– ticker text, Ambient Orb, Phidgets

http://images.google.com/imgres?imgurl=http://www.solderjoint.com/catalog/product_images/leds.jpg&imgrefurl=http://www.solderjoint.com/catalog/solid_state.php&h=345&w=337&sz=9&tbnid=qy7n-LVaxYYJ:&tbnh=115&tbnw=113&start=1&prev=/images%3Fq%3Dleds%26hl%3Den%26lr%3D





• Abstractors– motion, people counting, voices, phone ringing





• Abstractors– motion, people counting, voices, phone ringing

• Notification– exact match, threshold, contains,

degree of change


Transitions• PTK provides architectural support

for implementing as animations• Timing• Number of steps• Flashing (single, repeated)

• Tied to output– Depend on output modality– Example transitions: color changing and

pulsing (orb); fading, motion, & flashing (ticker text)


Future Work & Conclusion• PTK supports the creation of PDs• Focuses on managing user attention• To do this, uses abstraction,

notification levels, transitions• Validated with 7 example applications

– Supports attention management– Facilitates code re-use

• Future work– Incorporate interruptibility and

local context sensing– Cross-modality animations– PD evaluation support


Questions?

• For more information & downloads:

www.cs.berkeley.edu/projects/io/ptk/

[email protected]

Thanks to…

NSF, Ed De Guzman, Jeff Heer,Scott Klemmer, Berkeley & CMU


What about user context?• PDs must 2 sets of attention issues:

– User context• interruptibility, primary activity, focus of attention

– Attention management• balancing info urgency and user attention

• PTK focuses on attention management– User context invasive to sense, not practical to

assume PDs have access– Management issues are independent of user

context

• Context input can be considered on a library level (in notification maps)


Related Work• AROMA: architectural support for abstraction

• Context Toolkit: infrastructure for context input and some abstraction

• Phidgets: library of hardware widgets for input and output

• Real World Interfaces Toolkit: library of X10 output widgets

• iStuff: infrastructure for distributed inputs and outputs


Focus Attention

• Peripheral display + attracting focused attention = contradiction? Nope!– Notifications enable displays to be peripheral

when not displaying urgent info– Attention is scalar: PDs should cover entire range

0% 100%attention

awareness

unconscious


PTK ValidationCreated 7 example applications that illustrate:• Supports attention management

• 3 used abstraction

• 5 used notification levels and transitions

• Easier to create• 5-55 lines of logic code

• Facilitates code re-use• used same input for 4 displays




Notification Maps

If event value equal to: Set notification level to:

no motion ignore

med motion change blind

high motion make aware

(null) interrupt

(null) demand action

Exact Match Notification Map


Notification Maps• Exact match

• checks event for equality w/specified values

• Threshold• determines if event within specified ranges

• Contains• checks if event contains specified strings

• Degree of change• determines if event has changed by a

specified amount since last event


Introduction• Peripheral displays (PDs) enable you

to monitor many info sources while maintaining a calm environment– But, only calm if designed to manage the

attention they attract

• PDs are hard to create– No existing tools or guidelines for

managing human attention

• Peripheral Display Toolkit (PTK):– Enables easier creation of PDs that

manage human attention


Why Peripheral Displays?• Provide awareness w/ minimal attention

• Separate from user’s primary activity

• Non-focal display not used unless providing peripheral info

• Important to ubicomp vision of many devices to one user


Peripheral Display Toolkit (PTK)

• Supports the creation of PDs1. Focuses on managing user attention

2. Simplifies code

3. Provides library of common components

• Managers user attention with– Abstraction– Notification levels– Transitions


Characteristics of Attention Management• Abstraction of info to be glance-able

• Dynamically managed attention– Notification levels for deciding how

much attention to attract– Transitions for displaying info

appropriately


Peripheral Display Toolkit (PTK)

• Supports the creation of PDs

• Focuses on managing user attention– Balance info urgency and user attention

• To do this, uses– Abstraction– Notification levels– Transitions



x = attention 100% 0%

y = awareness indirect cognitive

direct cognitive

influence

divided attention

focused attention

pre -

attention

setting (layout, color, objects, names)

inattention

conscious perception (x > 0)

change blind

ambient

alerting

influence

Documents

A Toolkit for Managing User Attention in Peripheral Displays Tara Matthews, Anind K. Dey*٭, Jennifer Mankoff٭, Scott Carter, Tye Rattenbury EECS Department,