Novel Visualization and Interaction for Small to

Jumbo Displays

Novel Visualization and Interaction for Small to

Jumbo Displays

Mary CzerwinskiMicrosoft Research

Mary CzerwinskiMicrosoft Research

Who Contributed: VIBE TeamWho Contributed: VIBE Team

Large Display Surfaces Are HereLarge Display Surfaces Are Here

Workstation in the Workstation in the realreal world world

OverviewOverview

Initial large display researchPrototypes around usability issues observed

Visualization and interaction New user experiences have to scale the wide continuum of displays

Future directions

Initial large display researchPrototypes around usability issues observed

Visualization and interaction New user experiences have to scale the wide continuum of displays

Future directions

Harris Poll Responses (7/02, N=1197)Harris Poll Responses (7/02, N=1197)

Mutiple PCs and Displays

0%

10%

20%

30%

40%

50%

60%

70%

80%

None Multiple monitorsattached to

multiplecomputers.

Laptop anddesktop monitor

connectedtogether.

Dualmon or higher

Config

Pe

rce

nt

Re

sp

on

da

nts

All

Mutiple PCs and Displays

0%

10%

20%

30%

40%

50%

60%

70%

80%

None Multiple monitorsattached to

multiplecomputers.

Laptop anddesktop monitor

connectedtogether.

Dualmon or higher

Config

Pe

rce

nt

Re

sp

on

da

nts

All

no multimonno multimon30%30%

plan to useplan to use multimonmultimon

38%38%

use multimonuse multimon32%32%

Peddie Peddie Research, Research, 2001 (N>6000)2001 (N>6000)

Multimon Usage TrendsMultimon Usage Trends

Why A Larger Display Surface?Why A Larger Display Surface?

Productivity benefits 10-30% (despite sw usability issues)Users prefer more display surface

Prices droppingFootprints getting smaller

Productivity benefits 10-30% (despite sw usability issues)Users prefer more display surface

Prices droppingFootprints getting smaller

Projected LCD Pricing 2002-2005

$437$378 $327 $283

$699$597

$510$436

$1,089

$905

$752$625

$0

$200

$400

$600

$800

$1,000

$1,200

2002 2003 2004 2005$

US

15" -13.5%

17" -14.6%

18" -16.9%

1st Prototype--dSharp Display1st Prototype--dSharp Display

Triple projectionmatrox parhelia card3028 x764 resolution42 in. acrossSlightly curved120 degree FOV

Triple projectionmatrox parhelia card3028 x764 resolution42 in. acrossSlightly curved120 degree FOV

Task Times – SignificantTask Times – Significant

Effects of Display Size on Task Times

0

20

40

60

80

100

120

140

160

DISPLAY

Ave

rage

Task

Tim

e (S

econ

ds)

Small

Large

Effects of Display Size on Task Times

0

20

40

60

80

100

120

140

160

DISPLAY

Ave

rage

Task

Tim

e (S

econ

ds)

Small

Large

User Satisfaction - SignificantUser Satisfaction - Significantthe tasks were easy to perform

0

1

2

3

4

5

Small Large

Display Size

Ave

rage

Rating

(1=D

isag

ree,

5=Agr

ee)

the tasks were easy to perform

0

1

2

3

4

5

Small Large

Display Size

Ave

rage

Rating

(1=D

isag

ree,

5=Agr

ee)

Windows Layout - SignificantWindows Layout - Significant

I was satisfied with the ease of windows layout

012345

Display Size

Ave

rage

Rat

ing

(1=D

isag

ree,

5=Agr

ee)

Small

Large

I was satisfied with the ease of windows layout

012345

Display Size

Ave

rage

Rat

ing

(1=D

isag

ree,

5=Agr

ee)

Small

Large

Tenets--Large Display UXTenets--Large Display UX

User studies show large display surfaces fundamentally change user interaction

Designed tools to better understand/complement how work practice changes

Large display surfaces provide non-linear productivity increases

Additional space has different utilitye.g. focal/peripheral displays provide different cues

User studies show large display surfaces fundamentally change user interaction

Designed tools to better understand/complement how work practice changes

Large display surfaces provide non-linear productivity increases

Additional space has different utilitye.g. focal/peripheral displays provide different cues

But…Usability IssuesBut…Usability Issues

Why click to bring a clearly visible window into focus? caused many errorsWhere is my cursor?Where is my start button?Where is my taskbar?Where are my dialogs?The software doesn’t know where the bezel is…

Why click to bring a clearly visible window into focus? caused many errorsWhere is my cursor?Where is my start button?Where is my taskbar?Where are my dialogs?The software doesn’t know where the bezel is…

VibeloggerVibelogger

1st activity repository for studying windows usage in aggregate

can profile users based on display sizecan be extended to visualize workflow and capture context

single user: capture task contexts to surface pertinent ui or provide reminders

1st activity repository for studying windows usage in aggregate

can profile users based on display sizecan be extended to visualize workflow and capture context

single user: capture task contexts to surface pertinent ui or provide reminders

Multitask VisualizationMultitask Visualization

colored block for each time point and appamount of shading indicates percentage of visibility of the window taskssubtasks

colored block for each time point and appamount of shading indicates percentage of visibility of the window taskssubtasks

Task Switching VisualizationTask Switching Visualization

switching tasks (red to blue)

how are email windows arranged and used?

compare to...

switching tasks (red to blue)

how are email windows arranged and used?

compare to...

Windows and Task Management Issues Emerge

Windows and Task Management Issues EmergeLarger displays = more open windows

Multimon users arrange windows spatially

Taskbar does not scale:

aggregation model not task-based

users can’t operate on groups of related windows

Larger displays = more open windows

Multimon users arrange windows spatially

Taskbar does not scale:

aggregation model not task-based

users can’t operate on groups of related windows

Relationship between # of Monitors and # of Windows Left Open

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

18.00

No. of Monitors

Avg

. #

of

Win

do

ws

Lef

t O

pen

Single Monitor

DualMon

TripleMon

Relationship between # of Monitors and # of Windows Left Open

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

18.00

No. of Monitors

Avg

. #

of

Win

do

ws

Lef

t O

pen

Single Monitor

DualMon

TripleMon

Changes in Window Access Patterns

Changes in Window Access Patterns

0

10

20

30

40

50

60

70

80

90

100

1 2 3

Number of Monitors

Per

cen

tag

e o

f A

cces

s T

ech

niq

ue

Win

Taskbar

Input: Drag-and-PopInput: Drag-and-Popproblem

large displays create long distance mouse movement

touch & pen input has problems moving between screen units

solution

drag-and-pop brings proxies of targets to the user from across display surfaces

the user can complete drag interactions locally—no need to deal with distances or to cross display borders

problem

large displays create long distance mouse movement

touch & pen input has problems moving between screen units

solution

drag-and-pop brings proxies of targets to the user from across display surfaces

the user can complete drag interactions locally—no need to deal with distances or to cross display borders

Table ClothTable ClothProblem:User wants to access content physically far away

Solution:Pan the desktop to user

Compress content to the right of focus

Grab content you need and snap back

Problem:User wants to access content physically far away

Solution:Pan the desktop to user

Compress content to the right of focus

Grab content you need and snap back

Multitasking SupportMultitasking Support

Projectbar, layoutbar, groupbarScalable fabricTask flasherSecret passwordsTable clothWincuts

Projectbar, layoutbar, groupbarScalable fabricTask flasherSecret passwordsTable clothWincuts

Task Management: GroupbarTask Management: Groupbar

Taskbar for lightweight grouping of windows

Allows for multiple bars, spatial placement of bars

Desktop snapshotting; task snapshots

Taskbar for lightweight grouping of windows

Allows for multiple bars, spatial placement of bars

Desktop snapshotting; task snapshots

Task management: Scalable Fabric

Task management: Scalable Fabric

Configurable central focus + periphery

Easy task switch from periphery to focus

Leverages human spatial memory

Configurable central focus + periphery

Easy task switch from periphery to focus

Leverages human spatial memory

Task FlasherTask Flasher

A more visual alt + tab

Uses 3d scaling and selection animation

Windows stay on the monitor on which they are positioned

A more visual alt + tab

Uses 3d scaling and selection animation

Windows stay on the monitor on which they are positioned

Secret PasswordsSecret Passwords

Problem: many touchscreen systems (eg. SmartBoard, TabletPC) have no keyboard

Use software virtual keyboard

Hard to hide password on a virtual keyboardPrevious research showed users think anything on large display is considered public…they watch (Tan et al, CHI ‘03)

Problem: many touchscreen systems (eg. SmartBoard, TabletPC) have no keyboard

Use software virtual keyboard

Hard to hide password on a virtual keyboardPrevious research showed users think anything on large display is considered public…they watch (Tan et al, CHI ‘03)

Solution: Secret PasswordsSolution: Secret Passwords

Meeting Support: WincutsMeeting Support: WincutsWhat about co-located collaborative work?

People bring different expertise and information on personal devices to meetings and need to share

today’s model is broken—only one person gets to display entire desktop at a time (for better or worse)

or, must share applications and files with otherswhat about private information?

What about UI that should be scaled for the task at hand?

What about co-located collaborative work?People bring different expertise and information on personal devices to meetings and need to share

today’s model is broken—only one person gets to display entire desktop at a time (for better or worse)

or, must share applications and files with otherswhat about private information?

What about UI that should be scaled for the task at hand?

Visualization ResearchVisualization Research

DateLens with Ben Bederson

Facetmap

Team Tracks (SW Vis)

DateLens with Ben Bederson

Facetmap

Team Tracks (SW Vis)

Datelens with Ben BedersonDatelens with Ben BedersonFisheye representation of dates

Compact overviews

User control

Integrated search

Integrated with outlook

Pen-enabled

Fisheye representation of dates

Compact overviews

User control

Integrated search

Integrated with outlook

Pen-enabled

FacetMapFacetMap

FacetMapFacetMap

Scalable Visualization for “all of your stuff”Queries MyLifeBits SQL database interactively

Domain includes web pages, digital images, documents, email, SenseCam physical activity data, VibeLog PC activity dataHeterogeneous data, metadata, annotations, and relationships organized into “facets”Browsing and searching accomplished by iterative selection from among available item attributes (and/or full text search)

Uses Piccolo.Net (on top of GDI+) for graphics & animation

Scalable Visualization for “all of your stuff”Queries MyLifeBits SQL database interactively

Domain includes web pages, digital images, documents, email, SenseCam physical activity data, VibeLog PC activity dataHeterogeneous data, metadata, annotations, and relationships organized into “facets”Browsing and searching accomplished by iterative selection from among available item attributes (and/or full text search)

Uses Piccolo.Net (on top of GDI+) for graphics & animation

ScalabilityScalability

Scales in three ways:Space: Recursive, space-filling algorithm generates useful views for any size screenItems: Large numbers of items aggregated/abstracted into groups and counts at multiple levelsFacets: New item attributes can be added to the visualization dynamically

Fixed minimum node size to guarantee readability

Consistent information densityGreater levels of detail are collapsed until they are feasible to present, given display constraints

Scales in three ways:Space: Recursive, space-filling algorithm generates useful views for any size screenItems: Large numbers of items aggregated/abstracted into groups and counts at multiple levelsFacets: New item attributes can be added to the visualization dynamically

Fixed minimum node size to guarantee readability

Consistent information densityGreater levels of detail are collapsed until they are feasible to present, given display constraints

Medium Size (No Filters)Medium Size (No Filters)

Medium Size (Filters Active)Medium Size (Filters Active)

Small SizeSmall Size

Large Size (Wall Display)Large Size (Wall Display)

FacetMap for Mobile Phones

FacetMap for Mobile Phones

Quick searching through structured data

Scalable UI

Half visual and half text list

Prototyping stage (desktop)

Multiple visualizations (pick the best for a given results set)

Quick searching through structured data

Scalable UI

Half visual and half text list

Prototyping stage (desktop)

Multiple visualizations (pick the best for a given results set)

SW Visualization: Team TracksSW Visualization: Team Tracks

Goal: help developers new to a code base familiarize themselves quickly

Based on logs of teams using the code base

Assumption: most frequently visited areas of the code are more important

We empirically verified this

Show developers most related areas of code based on where they currently are

Also give them previews of that code

Goal: help developers new to a code base familiarize themselves quickly

Based on logs of teams using the code base

Assumption: most frequently visited areas of the code are more important

We empirically verified this

Show developers most related areas of code based on where they currently are

Also give them previews of that code

Screenshot of Team TracksScreenshot of Team Tracks

Evaluation ResultsEvaluation Results

Better task completion rates9 / 9 completed tasks 1 and 2 (same)

3 / 9 completed task 3 (versus 1 / 7)dominated by algorithmic detail

7 / 9 completed task 4 (versus 1 / 7)dominated by finding relevant code fragments

Better quiz scores (t(16)=-2.04, p<.03, one-tailed)

Same importance ratings (r=0.45, p=.02)

Better task completion rates9 / 9 completed tasks 1 and 2 (same)

3 / 9 completed task 3 (versus 1 / 7)dominated by algorithmic detail

7 / 9 completed task 4 (versus 1 / 7)dominated by finding relevant code fragments

Better quiz scores (t(16)=-2.04, p<.03, one-tailed)

Same importance ratings (r=0.45, p=.02)

Future VIBE DirectionsFuture VIBE Directions

Novel interaction and visualization techniques that scale from small to very large displays

Continued evaluation and iteration of designs from a user-centered perspective

Automatic task identification

For more information: http://research.microsoft.com/research/vibe

Novel interaction and visualization techniques that scale from small to very large displays

Continued evaluation and iteration of designs from a user-centered perspective

Automatic task identification

For more information: http://research.microsoft.com/research/vibe

Thank You!

http://research.microsoft.com/research/vibe

Thank You!

http://research.microsoft.com/research/vibe