Collaborative Sketching with Distributed

Displays and Multimodal Interfaces

Florian Geyer, Hans-Christian Jetter, Ulrike Pfeil, Harald Reiterer

University of Konstanz

Human-Computer Interaction Group

Box D73, 78457 Konstanz, Germany

{firstname.lastname}@uni-konstanz.de

ABSTRACT

In this paper we describe a system design for supporting

creative group activities using distributed displays and mul-

timodal interaction. We describe the rationale behind our

approach and proposed interaction techniques for support-

ing collaborative sketching. Our goal is to understand how

the interface metaphor, display space and interaction mod-

alities may influence exploration and communication in

collaborative settings.

ACM Classification: H5.2 [Information interfaces and

presentation]: User Interfaces. - Graphical user interfaces.

General terms: Design, Experimentation

Keywords: Creativity support, Design tools, Collaboration,

Multi-display environments, Multimodal interaction

INTRODUCTION

When designers are creating and revising ideas during early

design stages, they often harness the spatial properties of

the environment [1]. Design sketches are shared on large

whiteboards or spread out on tables and walls for compari-

son and discussion with other designers. In design studio

environments, the room itself is an important tool that al-

lows organizing information in an informal and spatial way.

Especially during group activities, accessibility and visibili-

ty of artifacts is crucial for efficient communication. This is

one of the reasons why designers often prefer to work with

pen and paper during creative group activities [2]. Paper

provides a private workspace while sharing is as simple as

pinning the paper on the wall or spreading it out on the ta-

ble. However, when using desktop-based electronic tools,

the artifacts that are created are often hidden in file systems.

At the same time, multiple devices are required to work

simultaneously in group settings. Even when using electron-

ic whiteboard systems, exchanging sketches to other dis-

plays is awkward and can lead to breakdowns of the crea-

tive flow. In the following, we examine existing approaches

to these issues before describing the rationale and proposed

interaction techniques of our alternative approach.

EXISTING APPROACHES

i-LAND [3] is a smart environment supporting creative

problem solving activities within groups. It allows access-

ing and transferring digital artifacts between different de-

vices, like whiteboards, tables and mobile displays. Trans-

ferring objects is realized by drag & drop gestures across

devices or by saving digital artifacts onto a physical token

that can be accessed from retrieval stations. In Team Storm

[4], designers can cooperatively sketch ideas with a system

that combines multiple tablet devices (focus displays) with

an electronic whiteboard (context display). Individual

sketches can be transferred from the tablets to the white-

board for discussion via a shared workspace window.

Sketches can also be transferred back to the tablets for an-

notation by individuals. Both approaches support efficient

collaboration for sketching in early design phases. Never-

theless, we aim to extend these approaches by investigating

the effects of the overall interface metaphor, display space

as well as interaction modalities on exploration and com-

munication during different group activities.

RESEARCH GOALS

We think that the spatial and physical properties of the

workspace are crucial for effective collaboration in creative

group work. We therefore emphasize on supporting (a) in-

formal spatial information management by providing flexi-

ble display space for different situations and activities. This

includes both the (b) design of the overall interface meta-

phor as well as the (c) design of the physical work envi-

ronment. As we think that the properties of physical arti-

facts contribute to creative thinking, we also seek to

(d) improve ad-hoc accessibility of the shared information

by making the digital representations more tangible. Never-

theless, instead of the need to transfer content between de-

vices and displays, we think that (e) one common work-

space across multiple distributed displays may provide ben-

efits for improving group awareness. Finally, we aim on

(f) supporting natural interaction modalities, like touch and

pen on different display sizes and display types in order to

support a variety of creative work styles suitable for flexi-

ble group compositions and alternating activities like colla-

borative sketching, discussions and presentations, private

reflective work or distributed feedback sessions.

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that cop-

ies bear this notice and the full citation on the first page. To copy other-

wise, to republish, to post on servers or to redistribute to lists, requires

prior specific permission and/or a fee.

ITS’10, November 7–10, 2010, Saarbrücken, Germany.

Copyright 2009 ACM 978-1-60558-745-5/09/10...$10.00.

TOOL DESIGN

Since our approach emphasizes on supporting the spatial

properties of creative design practice, we decided to em-

ploy a zoomable pin board as the overall interface metaphor

[5]. Sketches on the pin board can be scaled, rotated and

moved. This allows designers to arrange the digital repre-

sentations in a similar way as on physical display walls. The

pin board can also be navigated by zooming manipulations.

Thereby, the view onto the contents of the whiteboard can

be changed according to the current point of interest.

Zooming actions can also be triggered remotely across dif-

ferent devices. Manipulations and annotations however are

synchronized over a central database repository to keep all

contents consistent. Additionally, we design our tool around

a physical environment that resembles a digital design stu-

dio by providing extensive display space on one large inter-

active wall with pen interaction, two multi-touch tabletops

and multiple tablet PCs (see Figure 1, a). The pin board

interface spreads across all these displays providing inde-

pendent views on the same content. These views can also be

used as focus and context displays (see Figure 1, b).

We think that different activities should be supported by

suitable modalities. Collaborative tasks like sorting existing

sketches and focusing on sketches during discussions work

well on tabletops because multi-touch interaction provides

an efficient modality for this task. However, cooperatively

creating content is better done at the interactive whiteboard

as it offers more display space and more precise pen input.

However, for individual reasoning, private displays are

preferred. This is also the reason why we decided to inte-

grate mobile tablets and digital pen and paper into our sys-

tem (see Figure 1, b,c). Tablets can be used for individual

sketching, browsing and annotation. Digital pen & paper

adds the natural modality of physical paper to our work-

space. New sketches can be created by simply grabbing a

blank sheet of paper while existing digital sketches can be

printed out for individual activities like reflection, distri-

buted reviews or annotation. Sketches made on paper are

instantly transferred via Bluetooth connections to the digital

representations on the pin board interface. Synchronization

then updates all distributed displays. By supporting multiple

digital pens at the same time, we can extend the capacity for

parallel input. This allows for real-time collaboration

awareness, even when designers work on regular paper.

Thus, our system enables parallel work and supports differ-

ent modalities, by still providing the same basic interface

metaphor. This allows users to flexibly choose an adequate

modality for the current activity without the need to transfer

content from one screen to another.

FUTURE WORK

We are currently extending the system with groupware fea-

tures. This includes an automatic locking mechanism of

objects that are currently manipulated to cope with concur-

rency problems. In addition, we are also adding a location

tracking system for identification of users and their display

association. In future work, we want to integrate this meta-

data for analytical visualizations of design sessions and for

presenting passive information on displays that are not ac-

tively used by the group.

REFERENCES

1. Vyas D., Veer G.V., Heylen D., Nijholt A. Space as a

Resource in Creative Design Practices. In Proceedings

of INTERACT’09, LNCS, Springer, 2009, pp. 169-172

2. Cook D., Bailey B. Designers' Use of Paper and the

Implications for Informal Tools. In Proceedings of

OZCHI’05, CHISIG, AU, 2005. pp. 1-10

3. Streitz N. A., Geißler J., Holmer T., et al. i-LAND: An

Interactive Landscape for Creativity and Innovation. In

Proceedings of CHI’99, SIGCHI, ACM Press, NY, US.

1999. pp. 120-127

4. Hailpern J., Hinterbichler E., Leppert C., et al. TEAM

STORM: Demonstrating an Interaction Model for

Working with Multiple Ideas During Creative Group

Work. In Proceedings of C&C’07, ACM Press, NY, US.

2007. pp. 193-202

5. Jetter, H-C., Gerken, J., Zöllner, M., Reiterer, H. Mod-

el-based Design and Prototyping of Interactive Spaces

for Information Interaction. To appear in Proceedings of

HCSE'10, Springer. 2010.

Figure 1: Design studio setting (a), focus and context navigation by remote zooming actions (b), digital pen and paper (c)