Large Display Research Overview Mary Czerwinski, George Robertson, Brian Meyers, Greg Smith, Daniel Robbins & Desney Tan

Large Display Research Overview

Mary Czerwinski, George Robertson, Brian Meyers, Greg Smith, Daniel Robbins & Desney Tan

Microsoft Research 2

Introduction

The increasing graphical processing power of the PC has fueled a powerful demand for larger displays

Despite the increasing affordability and availability of larger displays, most users’ display space represents less than 10% of their physical workspace area


Introduction continued

Current interfaces are designed around the assumption of a relatively small display providing access to a larger virtual world

How might users cope with and benefit from display devices that provide 25% to 35% of their physical desk area or perhaps one day cover entire office walls?

We evaluated usability issues for large displays and developed a series of research prototypes that address various issues we discovered

Also, large displays should and can present beautiful visualizations


Harris Poll responses (7/02, N=1197)

Mutiple PCs and Displays

0%

10%

20%

30%

40%

50%

60%

70%

80%

None Multiple monitorsattached to

multiplecomputers.

Laptop anddesktop monitor

connectedtogether.

Dualmon or higher

Config

Pe

rce

nt

Re

sp

on

da

nts

All


Why a Larger Display Surface?

Productivity benefits 10-30% (despite sw usability issues)

Users prefer more display surface

Prices dropping fast Footprints getting

smaller

Projected LCD Pricing 2002-2005

$437$378 $327 $283

$699$597

$510$436

$1,089

$905

$752$625

$0

$200

$400

$600

$800

$1,000

$1,200

2002 2003 2004 2005

$U

S

15" -13.5%

17" -14.6%

18" -16.9%


Productivity Study w/dSharp Display

Triple projection

Matrox parhelia card

3028 x764resolution

42 in. across Slightly

curved 120 degree

FOV


Task Times – Significant BenefitsEffects of Display Size on Task Times

0

20

40

60

80

100

120

140

160

DISPLAY

Ave

rage

Task

Tim

e (S

econ

ds)

Small

Large


User Satisfaction - Significant

the tasks were easy to perform

0

1

2

3

4

5

Small Large

Display Size

Ave

rage

Rat

ing

(1=D

isagr

ee,

5=Agr

ee)


Windows Layout - Significant

I was satisfied with the ease of windows layout

012345

Display Size

Ave

rage

Rat

ing

(1=D

isag

ree,

5=Agr

ee)

Small

Large


Cognitive Benefits of Large Displays

Czerwinski et al. document results showing that larger displays lead to improved recognition memory and peripheral awareness

Tan et al. demonstrate the advantages of large displays on 3D navigation in virtual worlds. Wider fields of view lead to increased ability to

process optical flow cues during navigation, cues that females are more reliant upon than males

Tan et al. also found that large displays provide for a more immersive experience when performing spatial tasks, building better cognitive maps of the virtual world


But…Usability Issues

Why click to bring a clearly visible window into focus? caused many errors

Where is my cursor? Where is my start

button? Where is my taskbar? Where are my dialogs? The software doesn’t

know where the bezel is…


Vibelog: How Users Interact with Displays

1st activity repository for studying windows usage in aggregate can’t fix what you can’t

measure can profile users can be extended

Single user: capture task contexts to surface pertinent ui or provide reminders


Multitasking Visualization

Colored block for each time point and app Amount of shading indicates percentage of

visibility of the window Tasks Subtasks


Task Switching Visualization

Switching tasks (red to blue) How are email windows arranged and

used? compare to...


Windows and Task Management Issues Emerge

More open windows

Users arrange windows spatially

Taskbar does not scale: aggregation model

not task-based users can’t operate

on groups of related windows

Relationship between # of Monitors and # of Windows Left Open

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

18.00

No. of Monitors

Avg

. #

of

Win

do

ws

Lef

t O

pen

Single Monitor

DualMon

TripleMon


Changes in Window Access Patterns

0

10

20

30

40

50

60

70

80

90

100

1 2 3

Number of Monitors

Per

cen

tag

e o

f A

cces

s T

ech

niq

ue

Win

Taskbar


Basic Usability Issues

Seven broad categories:1. Input: Losing track of the cursor 2. Input: Distal access to information 3. Window management problems4. Task management problems5. Configuration problems6. Failure to leverage the periphery7. Failure to use displays artistically

In this overview, research prototypes will be described that address many of these problems across the research community


Input: High Density Cursor (Baudisch et al.)

fill-in cursors

current framefill-in cursors

previous frame

mouse

motion

mouse

motion

solution• high-density cursor inserts additional

cursor images between actual cursor positions

• the mouse cursor appear more continuous

problem• at high mouse speeds, the mouse cursor

seems to jump from one position to the next

the windows mouse trail…• makes mouse trail last longer• drawback: cursor images lag behind

...is not high-density cursor• hd cursor makes mouse trail denser• lag-free: mouse stops=>cursor stops

regular mouse cursor

high-density cursor

Windows mouse trail

high-density cursor


Input: Drag-and-Pop (Baudisch et al.)

Problem Large displays create long

distance mouse movement Touch & pen input has problems

moving between screen units

Solution Drag-and-pop brings proxies of

targets to the user from across display surfaces

The user can complete drag interactions locally—no need to deal with distances or to cross display borders


Input: Continued 1 Vacuum (Bezerianos

& Balakrishnan) Vision-tracked multi-

finger gestural input (Malik, Ranjan & Balakrishnan)

Handheld projector (Forlines et al.)


Input: Continued 2

Vogel & Balakrishnan--Distant freehand pointing and clicking Hand controls pointer position and makes

click selection with finger or thumb


Input: Continued 3 Kahn et al.—”Frisbee”, a

remote control UI Grossman et al.—3D

modeling techniques


FlowMenus and Zoomscapes

Gruimbretiere et al. Zoomscapes allowed currently unused windows to hang

around, but at 25% of their normal size Flowmenus were pen-based, fluid interaction techniques for

invoking commands at the user’s point of interest


Input: TableTop Interaction

MERL’s DiamondTouch system and two-handed touch gestures

Hinrichs et al. “Interface Currents” for collaboration


Window & Task Management Support

Table ClothTask flasherLiveBoardTask ZonesGroupBarScalable FabricKimuraWinCuts


Window Management Challenges

What to do when you have your information spread out in physically large display surfaces?

Real display at Georgia Tech (Hutchings et al.)


Window Management: Task Flasher

A more visual alt + tab Uses 3d scaling and zooming animation to

show selected window Windows stay on the monitor on which

they are positioned


Window Management: Table Cloth

Problem: User wants to

access content physically far away

Solution: Pan the desktop

to user Compress

content to the right of focus

Grab content you need and snap back


Task Management: LiveBoard (Elrod et al.)

Pen-based group interaction around a large surface

Supported drawing, pop-up menus, selection and annotations

Boardwalk software provides “planks” or tasks, from which the user would choose

A plank automatically opened up a set of applications (e.g., meeting, scoreboard, slideshow, games, etc.)


TaskZones: Virtual Multimon Desktops (Hutchings et al.)

Problem: user has multiple desktops wall of

monitors how to switch

focus? might be using

a phone or remote control

Solution: TaskZones


Task Management: GroupBar

Taskbar for lightweight grouping of windows into tasks

Can have multiple bars for large displays

Download at http://research.microsoft.com/research/downloads/default.aspx (search for GroupBar)

~40,000 downloads Desktop snapshots


GroupBar Usage Study


Task Management: Scalable Fabric Scaled down versions of grouped windows in periphery Supports task switching and task reacquisition http://research.microsoft.com/research/downloads/

default.aspx (search for fabric); over 20,000 downloads


Scalable Fabric Usage StudyTask Times

0.00

5.00

10.00

15.00

20.00

Task Management Tool

Tool

Aver

age

Task

Tim

e (S

ec)

TaskBar

Scalable Fabric

Figure 9: Average task times +/- one standard error of the mean for TaskBar and Scalable Fabric.

Survey Question(1=Disagree, 5=Agree)

TaskBar Scalable Fabric

Task switching was easy to perform using the…

2.95 4.26

It was hard to go back and forth between my various windows and applications using…..

3.32 1.84

I was satisfied with the functionality of the ….

2.68 3.78

The TaskBar/Scalable Fabric is an attractive innovation for Windows.

3.16 4.47

Table 1: Average satisfaction ratings for the TaskBar and Scalable Fabric. All ratings were significantly in favour of Scalable Fabric at the p<.05 level.


Task Management: Kimura (MacIntyre et al.)

Supported multitasking and background awareness using interactive peripheral displays

“Montages” or activities based on desktop interaction


WinCuts: Initial Motivation

Problems: Sharing live windows/information is hard Screen space is scarce and laying out information

optimally is hard


WinCuts Video


Specify Region of Interest

Animatedadvertisement

Seldom usedinterface buttons

Scrolling ticker

Region of interest


Organize Content

Relevant content

Rescale WinCut so graph scales are comparable


Reconfigure Interfaces

Relevant interface elements

Relevant content


Sharing WinCuts across Machines

1. Click on “Share”2. Specify destination (also running WinCuts)3. WinCut appears on destination machine

Remote WinCuts work just like local WinCuts

Except input redirection disabled


Share Content when Collaborating


Desktop

Laptop orTabletPC

PDA

Cell Phone

Now we’re Thinking… With remote input redirection working:

Create ad hoc remote controls and interfaces Work across displays and devices


Huang & Mynatt’s Design Space for Peripheral Awareness Display Research


Interaction based on Distance

Vogel & Balakrishnan’s notion of an interactive, ambient, public display

Different functionality based on distance Ambient, Implicit,

Subtle and Personal spaces


Large Displays as Peripheral Awareness Surfaces

Brignull and Rogers’ Opinionizer

McCarthy’s et al.’s Unicast, Outcast & GroupCast

Izadi et al’s Dynamo ….etc.


And, Large Displays as Art and Info

Blinkenlights 2.0 in Berlin

Interactive waterfall display in children’s hospital

Weather patterns window in an art gallery at night

Etc….


Conclusions There is a clear trend toward larger displays Large displays increase user productivity, aid user

recognition memory, and in some cases can eliminate gender bias

User studies have identified numerous usability problems

Research prototypes were presented that outline techniques for solving many of these problems

The work of integrating these prototype solutions into one system remains to be done

Correcting these problems significantly improves the user experience on large displays

Large displays also useful tools for peripheral awareness and can be aesthetically pleasing


Acknowledgements

Gary Starkweather Patrick Baudisch Ed Cutrell Eric Horvitz Jonathan Grudin All of our colleagues doing large

display research and kindly granted me permission to show their work


Thanks for your Attention!

Questions? More information at:http://research.microsoft.com/research/vibe


References

Ballagas, R., Rohs, M. & Sheridan, J. (2005). Sweep and Point & Shoot: Phonecam-based interactions for large, public displays. In Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems, pp. 1-4.

Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B., and Zierlinger, A. Drag-and-Pop and Drag-and-Pick: techniques for accessing remote screen content on touch- and pen-operated systems. In Proceedings of Interact 2003, pp.57-64.

Baudisch, P. Cutrell, E, and Robertson, G. High-density cursor: a visualization technique that helps users keep track of fast-moving mouse cursors. In Proceedings of Interact 2003, pp. 236-243.

Bezerianos, A.& Balakrishan, R. (2005). Canvas Portals: View and space management on large displays. IEEE Computer Graphics and Applications, 25(4). pp. 34-43

Bezerianos, A. & Balakrishnan, R. (2005). The Vacuum: Facilitating the manipulation of distant objects. Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems. pp. 361-370.

Biehl, J.T. & Bailey, B.P. (2004). ARIS: An interface for application relocation in an interactive space. In Proceedings of Graphics Interface 2004, pp. 107-116.

Brignull, H. & Rogers, Y. (2003). Enticing people to interact with large public displays in public places. In Proceedings of Interact 2003, pp. 17-24.

Buxton, W., Fitzmaurice, G., Balakrishnan, R., Kurtenbach, G. (2000). Large displays in automotive design. IEEE Computer Graphics & Applications, July 2000.

Cao, X. & Blakrishnan, R. (2003). VisionWand: Interaction techniques for large displays using a passive wand tracked in 3D. Proceedings of UIST 2003 – the ACM Symposium on User Interface Software and Technology. pp. 173-182.

Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G. and Starkweather, G. (2003). Toward characterizing the productivity benefits of very large displays. In Proceedings of Interact 2003, pp. 9-16.

Elrod, S., Bruce, R., Gold, R., Goldberg, D., Halasz, F., Janssen, W., Lee, D., McCall, K., Pedersen, E., Pier, K., Tang, J. & Welch, B. (1992). LiveBoard: A large interactive display supporting group meetings, presentations and remote collaboration. In Proceedings of CHI 1992 – the ACM Conference on Human Factors in Computing Systems, pp. 599-607.


References Continued 2 Grossman, T., Balakrishnan, R., Kurtenbach, G., Fitzmaurice, G., Khan, A. & Buxton, W.. (2001). Interaction techniques for 3D

modeling on large displays. Proceedings of the ACM Symposium on Interactive 3D Graphics (I3DG2001), pp. 17-23. New York: ACM.

Grudin, J. (2002). Partitioning digital worlds: Focal and peripheral awareness in multiple monitor use. In Proceedings of CHI 2002 – the ACM Conference on Human Factors in Computing Systems, pp. 458-465.

Guimbretiere, F., Stone, M., and Winograd, T. (2001). Fluid interaction with high-resolution wall-size displays. In Proceedings of UIST 2001, pp. 21-30.

Huang, E.M., Russell, D.M. & Sue, A.E. (2004). IM Here: Public instant messaging on large, shared displays for workgroup interactions. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp. 279-286.

Hutchings, D., Czerwinski, M., Smith, G., Meyers, B., Robertson, G Display space usage and window management operation comparisons between single monitor and multiple monitor users. In Proceedings of AVI 2004, pp. 32-39.

Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N. & Kurtenbach, G. (2003). A remote control interface for large displays. ACM UIST 2003 Symposium on User Interface Software & Technology, pp. 127-136.

Khan, A. Matejka, J. Fitzmaurice, G. & Kurtenbach, G. (2005). Spotlight: Directing users' visual attention on large displays. Proceedings of CHI 2005 – the ACM Conference on Human Factors in Computing Systems pp. 791 – 798.

MacIntyre, B., Mynatt, E., Voida, S., Hansen, K., Tullio, J., Corso, G. (2001). Support for multitasking and background awareness using interactive peripheral displays. In ACM UIST 2001 Symposium on User Interface Software & Technology, pp. 41-50.

Malik, S., Ranjan, A. & Balakrishnan, R. (2005). Interacting with large displays from a distance with vision-tracked multi-finger gestural input. Proceedings of UIST 2005 - the ACM Symposium on User Interface Software and Technology. pp. 43-52.

McCarthy, J.F., Costa, T.J. & Liongosari, E.S. (2001). UniCast, OutCast & GroupCast: Three steps toward ubiquitous, peripheral displays. In Lecture Notes in Computer Science, Vol. 2201, pp. 332-*, Germany: Springer-Verlag.

Mynatt, E., Igarashi, T., Edwards, W., and LaMarca, A. (1999). Flatland: new dimensions in office whiteboards. In Proceedings of CHI 1999 – the ACM Conference on Human Factors in Computing Systems, pp. 346-353.

Paradiso, J.A., Leo, C.K., Checka, N. & Hsiao, K. (2002). Passive acoustic sensing for tracking knocks atop large interactive displays. In Proceedings of the IEEE Sensors 2002 Conference, pp. 1-6.

Pedersen, E., McCall, K., Moran, T., Halasz, F.G. (1993). Tivoli: An electronic whiteboard for informal workgroup meetings. In Proceedings of CHI 1993 – the ACM Conference on Human Factors in Computing Systems, pp. 391-398.

Robertson, G.G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., and Tan, D. (2005). Large display user experience. In IEEE CG&A special issue on large displays, 25(4), pp. 44-51.

Robertson, G., Horvitz, E., Czerwinski, M., Baudisch, P., Hutchings, D., Meyers, B., Robbins, D., and Smith, G. (2004), Scalable Fabric: Flexible Task Management. In Proceedings of AVI 2004 pp. 85-89.


References Continued 3 Russell, D. M., Drews, C. & Sue, A. (2002). Social aspects of using large interactive displays for

collaboration. In Proceedings of UbiComp 2002: Ubiquitous Computing: 4th International Conference, pp. 229-236.

Smith, G., Baudisch, P., Robertson, G., Czerwinski, M., Meyers, B., Robbins, D., and Andrews, D. (2003). GroupBar: The TaskBar Evolved. In Proceedings of OZCHI 2003, pp. 34-43.

Streitz, N.A., Geiler, J. & Holmer, T. (1998). Roomware for cooperative buildings: Integrated design of architectural spaces and information spaces. In Lecture Notes in Computer Science, Vol. 1370, pp. 4-*, Germany: Springer-Verlag.

Swaminathan, K. & Sato, K. (1997). Interaction design for large displays. In ACM’s Interactions (4), pp. 15-24.

Tan, D., Czerwinski, M., and Robertson, G. (2003). Women Go With the (Optical) Flow. In Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp. 209-215.

Tan, D.S. and Czerwinski, M. (2003). Information Voyeurism: Social impact of physically large displays on information privacy. Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems. pp. 748-749.

Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2003). With similar viewing angles, larger displays improve spatial performance. In Proceedings of CHI 2003 – the ACM Conference on Human Factors in Computing Systems, pp. 217-224.

Tan, D.S., Gergle, D., Scupelli, P. & Pausch, R. (2004). Physically large displays improve path integration in 3D virtual navigation tasks. In Proceedings of CHI 2004 – the ACM Conference on Human Factors in Computing Systems, pp. 439-446.

Tan, D.S., Meyers, B. & Czerwinski, M. (2004). WinCuts: Manipulating arbitrary window regions for more effective use of screen space. In Extended Abstracts of Proceedings of CHI 2004, pp. 1525-1528.

Vogel, D. & Balakrishnan, R. (2005). Distant freehand pointing and clicking on very large high resolution displays. Proceedings of UIST 2005 - the ACM Symposium on User Interface Software and Technology. pp. 33-42.

Vogel, D. & Balakrishnan, R. (2004). Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. Proceedings of UIST 2004 – the ACM Symposium on User Interface Software and Technology. pp. 137-146.

Documents

Large Display Research Overview Mary Czerwinski, George Robertson, Brian Meyers, Greg Smith, Daniel Robbins & Desney Tan