large displaysare like regular sized displays, only larger, right?

patrick baudisch

microsoft researchvisualization and interaction research

large screens and multimon

... are coming

[Jon Peddie ResearchDec, 2002 N=6652]

No Multimon30%

Plan to Use Multimon

38%

Use Multimon32%

• information mural[Guimbretière, Winograd]

• on large screens optical flow helps navigation [Tan 2001]

• large screens help productivity tasks [Czerwinski 2003]

• multi-monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001]

building a large display

focus plus context screens

help—I can’t find my mouse cursor

high-density cursor

help—I can’t reach my stuff anymore

drag-and-pop

context where there is no room for context

halo

mouse

motionmouse

motion

fill-in cursors

current framefill-in cursors

current framefill-in cursors

previous frame

fill-in cursors

previous frame

• Hardware– At least one hi-res

display– At least one larger low-

res display

• Software– scaling of the display

content is preserved– resolution varies

setup

application scenarios

video

field study: users & tasksSubject’s task Document/view

Sta

tic do

cum

en

ts

Web designer Page: 800 pixel

Mechanical engineer Polybot segment: 5cm

Graphic designer Poster: 1m

Architect in remodeling Building: 50m

Photogrammetry (2) Highway 2 miles

Geographic info. system County: 80km

Chip designers (2) Wafer: 12cm

Dyn

am

ic

Air traffic ctrl. tool builder Zone: 50km

Ego shooter gamer Surrounding: 360º

Submarine ROV op. Surrounding: 360º

Strategy gamers (2) Map: 30k pixel

Smallest detail Ratio

Table detail: 1 pixel 800

Clearance: 0.03mm 2,000

Align: 0.5mm 2,000

Accuracy: 1cm 5,000

Accuracy: 1 inch 100,000

Land boundaries: 0.5m 160,000

Grid: 0.5m 240,000

Plane distance in 25m steps 2,000

Aiming: 0.1º 3,000

Use arms: 1mm/0.05º 8,000

Aiming: 1 pixel 30,000

focus plus context screen VisualizationSame # of pixels

fisheye

55

overview plus detail

44

Display technology

homogeneous resolution

44wall-size, hi-res display

44

What participants

used

What participants

used

Available to½ of participants

Available to½ of participants

… andcurrentsolutions

experiment 1:• 3 interfaces:

• focus plus context screen• overview + detail• homogeneous

• 2 tasks

• 12 subjects from Xerox PARC• Within subjects, counter-balanced• Same number of pixels

task 1: closest hotel

8 maps per interface

F+C screen and O+D use same magnification factor

task 2: verify connections

Verify a different set of 24 connections on the board

0

100

200

300

400

500

600

700

Map task Board task

zooming+panning

overview+detail

focus+context screen

resultsA

vera

ge

ta

sk c

om

ple

tion

tim

es

in s

eco

nd

s

21% faster21% faster 36% faster36% faster

manually zooming

takes time

manually zooming

takes time

visually switching reorientation

visually switching reorientation

visually more ambiguous

visually more ambiguous

experiment 2:driving simulation

120 sec sequence100 fields of nails; 30 rocks; tradeoff

resultsM

ea

n n

um

be

r o

f co

llisi

on

s su

bje

cts

cau

sed

• Sweet spot:flight simulation, unmanned vehicles…

0

5

10

15

20

25

run-over nails rocks hit

overview+detailfocus+context screen

Error rate only 1/3 of two-

monitor setup

Error rate only 1/3 of two-

monitor setup

Subjects preferred thef+c interface

Subjects preferred thef+c interface

• so it worked really well withcontent that already wasfocus & context…

• …but what about the computer desktop?• how to view peripheral content in high-res• problem only because periphery is low-res?• not really… how to view a detail on a huge display wall?• it is just hard to see detail if located far away

the distinction of screen space into focusand context regions is always there(focus plus context screens only emphasize it)

ktop?ktop?but how about the computer desktop?

keeping the mouse working

• on a large screen, cursor isfurther away from user

• longer distances higher mouse acceleration

• temporal aliasing: 500 pixels jumps

• lack of visual continuity & weak stimulus users lose track of the cursor

the problem will get worse

• “yes, but won’t faster computers make this problem go away?”

NO: cursor update is limited by screen refresh rate

• screen refresh rate has actually decreased (LCDs)• larger screens + lower refresh rate status quo• future: even larger screens problem will get worse

how it works

previous cursorposition

current cursorposition

mouse

motion fill-in cursors

current framefill-in cursors

previous frame

inserts cursor image between actual cursor positions the mouse cursor appear more continuous

this is not the mouse trailthe windows

mouse trail…• makes mouse

trail last longer• drawback: cursor

images lag behind

...is not high-density cursor

• hd cursor makes mouse trail denser

• lag-free: mouse stops => cursor stops

video

benefitsprevious cursor position

mouse

motion

current cursor position

fill-in cursors

current framefill-in cursors

previous frame

mouse

motion

1. mouse cursor appear more continuous easier to track the cursor

2. higher “visual weight” easier to re-acquire the cursor

designs alternatives

acceleration

• reference: exponential acceleration

a

b

d

e

f

c

h

g

frame

designs alternativesa

b

d

e

f

c

h

g

frame acceleration

• motion blur with higher weight

designs alternativesa

b

d

e

f

c

h

g

frame acceleration

• temporal super-sampling vs. motion blur

chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive

chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• density = detectability vs. intrusiveness

transfer function

mouse speed

distancebetweencursorimages

onset threshold(configurable)

cursor trail provides no speed cues

hd c

urso

r has

no

effe

ct

transfer function(configurable)

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• optional cursor growth

user study

• conducted pre-study to define interface candidates• interfaces: control vs.

high-density cursor (conservative, tripleDensity, plusScaling) • fitts’ law task• triple-mon: button located at 5” to 40” distance• participants: 7 external participants, 5 coworkers

• hypotheses• high-density cursor faster• the greater the distance the

greater the effect• tripleDensity and plusScaling

faster than conservative

resultsti

me %

rela

tive t

o r

eg

ula

r cu

rsor

90

92

94

96

98

100

102

target distance (mm)125 250 500 750 1000

regular mouse cursor

high-density cursors

speedupup to 7%

conservative

+ scale

+3-dense

short distance

subjective satisfaction

Condition Liked Most Liked Least

control 0 7

HD_conservative 2 3

HD_tripleDensity 2 0

HD_plusScaling 6 0

• most participants did not notice that cursor was different!“did that condition use a different mouse acceleration?”…

• lesson we learned:display frame rate is not a hard limit

but how to see details?

• so this gets the mouse to the periphery—nice

• …but what if the user uses touch for input?• … or if user needs to see content in detail?

let’s focus on a specificcase for a moment:extend basic actionsdrag-and-drop and picking

scenario 1: long distances

dragging is designed for small screens…… but becomes time-consuming on large screens

scenario 2: dragging + bezels

dragging across bezels in display wall is no problem for the mouse…

…but a big problem when using pen/touch input

drag-and-pop: demo• users starts

dragging icon towards a distant folder or application

• icons of compatible type come towards mouse cursor

• user drops icon with minimal motion

• targets retractdrag-and-pop works across bezels

• drag-and-pop generalizes direct manipulation• bring content to the user• let the user interact with it• send content back

scenario 1: long distances

dragging on large screens

scenario 2: dragging + bezelsdragging acrossbezels in display wall

the displays we used…

design

selecting candidates

• initialize• all icons are candidates

• filter• eliminate icons with non-matching file types• eliminate icons that are too close• eliminate icons outside target angle• if necessary, restrict to some hard limit

preserving layout

• snap to grid• eliminate empty

rows and columns

• translate back• place center of

bounding box in front of user

• closer for experts

the rubber band

• animationdid not work

• “frozen”motion blur

• narrow midriff• suggests elasticity• clue for distance

• simplified version

getting it out of the way

• to rearrange icons on the desktop (overloaded):• any mouse motion moving away from the

“popped-up” icons de-activates drag-and-pop• introduce flick gesture into mouse motion

pre-study

• 3 layouts for study: sparse (11), frame (28), cluttered (35)

• 15 single, 6 dual, and4 triple monitor users

• overall resolutions 800,000 pixels to 3,900,000 pixels

• (= 66% more than the display wall used in the experiment).

user study

• participants: 2 female, 5 male• dynaWall

• 3 Smartboard• 15’ long (4.5m)• 3 x 1024x768 pixels

• native code not stable enough Macromedia Flash version

• task: drag icons into matching folder• highlighting disappeared when started• each desktop: 11-35 icons + 10 icons to be filed

results

• faster with drag-and-pop• error rate higher with drag-and-pop• most of the effect caused by the bezels

0 1 20

2

4

6

8

10

12

14

Number of Bezels Crossed

Drop

Pop

Control

Drag-and-pop

3.7 times3.7 timesspeedupspeedup

0

10

20

30

40

0 500 1000 1500 2000 2500Target Distance (pixels)

Drop

Pop

Control

Drag-and-pop

subjective satisfaction

• > 6 (out of 7)• “I liked using drag-and-pop”• “I always understood what was happening when drag-and-pop

was on”,• “I would use drag-and-pop for large displays.”

• < 3 for• “It took a long time to get used to drag-and-pop”• “It was hard to control what the targets did when drag-and-pop

was on.”

• drag-and-pop interface causes less manual stress and fatigue than the control interface

• lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)

drag-and-pick

problem• launch app or open file

drag-and-pick• user drags “background”• all icons in that direction

move to the cursor• user drags % releases

mouse over it• target is activated

• lesson we learned:screen space andinteraction don’t haveto be the same

• this was only for moving files and launching apps. need to solve this for the general case

can we push this further?

• (unless the app has fixed focus of attention) bring content to the user on demand

• that’s great—so maybe the periphery does not need to be full resolution after all

• can we push this even further? can we create a visual periphery without any peripheral screen space?

a first attempt… city lights

+

the problem

halo <demo>

cinematography

1. entry and exit points

2. point of viewarrow-based techniques

3. partially out of the frame halo

rings are familiar, graceful degradation

streetlamps

• aura visible from distance• aura is round• overlapping auras aggregate• fading of aura indicates distance

what we changed• smooth transition sharp edge• disks rings• dark background light background

app designers can use

• color• texture• arc thickness

user study

interfacesarc/arrow fading offscale 110-300m/cmmap as backdropreadability oksame selectable

size

hypothesis:

halo faster

halo ring distance from display border

legend

pre-study to define tasks

• 8 participants (6 GPS users, 2 PDA users)• informal interviews 10-40 minutes

4 tasks to be used in study

1. locate task

click at expected location of off-screen targets

had tosimulate on PC

2. closest task

click arrow/arc or off-screen location closest to car

3. traverse task

click all five targets so as to form shortest path

4. avoid task

click on hospital farthest away from traffic jams

procedure

• 12 participants• within subject design, counterbalanced• four training maps per interface/task,

then eight timed maps• questionnaire

task completion time

Task Arrow interface Halo interface

Locate 20.1 (7.3) 16.8 (6.7)

Closest 9.9 (10.1) 6.6 (5.3)

Traverse 20.6 (14.1) 16.8 (8.7)

Avoid 9.2 (4.7) 7.7 (5.8)

0

5

10

15

20

25

Locate Closest Traverse Avoid

Arrow interface

Halo interface

33%

16%

error rateTask Arrow interface Halo interface

Locate 23.5 pixels (21.6) 28.4 pixels (33.8)

Closest 22% (42%) 21% (41%)

Traverse 97.4 pixels (94.7) 81.0 pixels (96.7)

Avoid 15% (35%) 14% (34%)

0

5

10

15

20

25

30

Locate Closest Traverse Avoid

Arrow interface

Halo interface

participants underestimated distances by 26% participants saw ovals (gestalt laws?) we can compensate for that: width += 35%

0

1

2

3

4

5

6

7

8

9

Locate Closest Traverse Avoid

Arrow interface

Halo interface

subjective preference

results

• halo 16%-33% faster than arrows• no split attention• distortion-free space• scale independent• no need to annotate distance• perceive all rings at once

[treisman & gormican]

• limitation: max number or rings

access off-screen content

• make halo arcs (or city lights) clickable to pan display to that location

vs.

• preserve users’ spatial memory by using drag-and-pop instead: bring peripheral content to PDA screen

conclusions

• distance to peripheral space and human peripheral vision make peripheral content hard to read

• making periphery all hi-res does not solve this problem

• we need to support users using appropriate visualization and interaction techniques

mouse

motionmouse

motion

fill-in cursors

current framefill-in cursors

current framefill-in cursors

previous frame

fill-in cursors

previous frame

read more about it

focus plus context screens:UIST 2001, CHI 2002SIGGRAPH 2002 demo

high-density cursorInteract 2003

drag-and-popInteract 2003

city lightsCHI 2003

haloCHI 2003, UIST 2003 demo

thanks!• try it out www.patrickbaudisch.com

• thanks to: • focus plus context screens: victoria

bellotti, nathan good, paul stewart, pamela schraedley, michael brueckner, rich gold,

• high-density cursor: ed cutrell, george robertson, & VIBE

• drag-and-pop: & ed cutrell, dan robbins, mary czerwinski, peter tandler, ben bederson, and alex zierlinger

• halo: ruth rosenholtz, polle zellweger, jock mackinlay, lance good, and mark stefik

Seamless integration of displays

a b

Context

No reflections on focus screen

Focus

If I had to commercialize today…

• Build all-analog immersive video link• Immersive telepresence,• Remote operated vehicles, drones• Immersive VR• Remote medicine

ContextWall

Extra

(a) locate (b) closest

(d) avoid(c) traverse

How does it work?

The scaling software

• Display image on two display units of different resolution• Similar to two-headed display• but display units are overlapping• and one of them has to be scaled down

• (Related work “Flux capacitor”[Dr. Emmett Brown, 1985])

Linux/VNC

app

focus

contextinput

server

clip

scale

viewer

viewer

Image viewer

context

Photoshop

scale

ACDsee

ACDsee

.gif

.gif

mousefork

input

focus

related work• acquiring distant targets

• move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al., 1999)• flick snaps cursor to target (Dulberg et al. (1999)• sticky icons capture cursor (Swaminathan and Sato, 1997)• throwing gets across long distances (Geißler, 1998) • expanding targets save space on screen (McGuffin and Balakrishnan, 2002)• drag-and-pop (baudisch et al 2003)

• enhance detectability of the mouse cursor• <ctrl> for radar animation (Microsoft, Steve Bathiche)• cursor growth (Kensington Mouseworks 2001)• mouse trail for slow response LCDs (e.g. MS Windows)• liveCursor points in the direction of its motion (Ben Bederson)

• motion blur and temporal supersampling• reduce temporal aliasing, such as stroboscope, e.g. wheel spokes• rendering a scene multiple times (Dachille and Kaufman, 2000)• improve the perceived responsiveness of graphics apps (Conner and Holden, 1997)• help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)

design goals

• for users who track the cursorenhance the predictability of the cursor path• enhanced trail density/continuous blur• smooth interpolation of the cursor path• preservation of trail density as a cue for cursor speed.

• for users who reacquire the cursorincrease the detectability of the cursor (visual weight)• enhanced trail density• enhanced cursor opacity• and cursor scaling.

• preserve responsiveness

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• distance between cursor images as cue for mouse speed

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• smooth interpolation

bezier interpolation

cursor position

1. linear interpolation

2. attraction point

3. interpolate

pre-study

• goal: define interfaces for user study• participants: 14 coworkers

• informal procedure• try out high-density cursor• try out different settings (density, onset…)• choose “favorite” setting

• resulting interface parameters• 12-17 pixels/frame vs. 35 pixels/frame• distance = sqrt(n)• cursor growth on or off

touch/pen input breaks

touch/pen-input + multimon

touch and pen input renaissance• PDAs• Tablets• Liveboards /

Smartboards

multi-display systems• DynaWall, iRoom

Smartboard wall• connect tablet to

external screen• …

scenario 1: tables + screen

tablet users scribble with pen… but filing iconsinto folder on external monitor requires mouse

scenario 1: tablet + screen

filing icons into folder on external monitor

related work• techniques for transferring information

• drag-and-drop avoids hidden clipboard (e.g. Xerox Star)• hyper dragging (Rekimoto, 1999)• pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001)

• overcome large distances• magic pointing (Zhai et al., 1999) requires an indirect input device• gesture input techniques (Rubine, 1991)• throwing (Geißler, 1998) for reliable target acquisition?• laser pointers to acquire targets on a Smartboard (Myers et al. 2002)

• mouse-based interaction techniques• lodestones and lay lines (Jul, 2002)• flick (Dulberg et al., 1999)• sticky icons (Swaminathan and Sato 1997)

related work

• driving directionsvs. route planning aids

• overview-plus-detail• focus-plus-context

• pointing into off-screen space

inside applications…

• drag-and-pop workseven if target is• occluded• clipped• closed (folder)

• use the concept tofile emails?

intrusion border

handle

space for arcs…

and for corner arcs

reserve space for content

arc length = distance

handling many objects

• find best (restaurant): relevance cut-off

• see all (dangers): merge arcs