October 2005 111

I N V I S I B L E C O M P U T I N G

T oday, physical and electronicmedia coexist like ships pass-ing in the night: proximatebut unaware of each other.Contemporary design studios,

offices, and labs are filled with bothphysical and electronic artifacts, butthe two exist separately, and the infra-structure for moving between mediarepresentations—scanning and print-ing—is heavyweight and cumbersome,at odds with the freewheeling, organicnature of creative work.

For the past six years at UC Berkeleyand Stanford University, my colleaguesand I have conducted research intouser interfaces that bind physical andelectronic representations of artifactsfor integrated interaction: Manipu-lation in one medium effects a corre-sponding change in the artifact’s dualmedium. Based on fieldwork withdesigners, office workers, scientists,and engineers, we’ve created and eval-uated integrated interfaces rangingfrom whiteboards to oral history tran-scripts to field notebooks.

DESIGNERS’ OUTPOSTWe initially became interested in

integrated interaction techniques whilestudying Web site designers. MarkNewman, James Landay, and I learnedthat, even in this high-tech industry,users preferred working with stickynotes and marking on walls and white-boards in a project’s early design

stages. This observation led us to cre-ate the Designers’ Outpost (http://hci.stanford.edu/research/outpost).

Interactive whiteboardWith this system, shown in Figure 1,

users collaboratively author Web sitearchitectures on a touch-sensitive elec-tronic whiteboard augmented withcameras that capture documentsplaced on the wall. Designers createnew pages by writing with standardpens on sticky notes and organize aWeb site by physically moving thenotes around on the board.

The Designers’ Outpost vision sys-tem uses two cameras to provide inter-active feedback and near-interactivecapture. A 640 × 480 pixel firewirevideo camera mounted inside thewhiteboard offers interactive ratetracking of physical objects placed onthe board. A high-resolution digitalstill camera captures the objects’ con-tents, albeit at a lower frame rate—oneframe every few seconds.

Users can link notes by drawing aline with an electronic stylus as well as

add ink annotations to a design to con-vey design rationale. Electronic captureenables replacing physical documentswith their electronic images. Usershave the same capabilities when con-tent is electronic as when it’s physical,thereby retaining physical directmanipulation.

The main timeline is a visually nav-igable time-ordered set of designthumbnails. The thumbnails aren’t justa visualization of the changes in theboard—they’re also a direct manipu-lation interface: Users can touch athumbnail to return the board to thecorresponding state.

Dual media representationsWith Designers’ Outpost, users

interact with physical and electronicmedia in a georeferenced manner—thetwo representations occupy the samephysical space and appear to overlayeach another. Users can perform oper-ations on physical objects by tappingon them to invoke an electronic menu.

Documents that are physical in onespace are electronic in the other spaceand vice versa. Thus, a user can placea document on a wall in San Francisco,and colleagues in Seattle can synchro-nously see the electronic dual of thatdocument. This interface retains thewall-size scale and physicality of thedesign process, extending it withadvantages that electronic capture pro-vides: fluid movement between designtools, design versioning and history,and remote collaboration.

Papier-Mâché toolkitDeveloping vision-based interfaces

for the Designers’ Outpost was difficultand time-consuming, and it required adetailed understanding of the underly-

Integrating Physical and Digital Interactions Scott R. Klemmer, Stanford University

Integrated userinterfaces enablericher forms of collaboration.

112 Computer

I n v i s i b l e C o m p u t i n g

user experience. Today’s desktop inter-faces are largely designed to occupy theuser’s entire attention. Our experiencesshow that, as we enter the age of ubiq-uitous computing, the most successfultechnologies honor the calm interactionstheir analog-only predecessors afforded.

BOOKS WITH VOICESAfter three years working on the

Designers’ Outpost, I was curious toexplore another domain. Long fasci-nated by documentary work, I beganlooking at technology support optionsfor oral history. Through UC Berkeley’sRegional Oral History Office (http://bancroft.berkeley.edu/ROHO), the sec-ond-oldest and one of the largest oralhistory offices in the country, I attendeda training workshop and conductedtwo oral history interviews with DavidPatterson and Carlo Sequin.

Our contextual inquiry into oral his-torians’ practices unearthed a curiousincongruity. While oral historians con-sider interview recordings a central his-

ing technologies and algorithms. Thisrealization inspired us to create thePapier-Mâché toolkit, which intro-duced a software architecture foracquiring and abstracting physicalinput (http://hci.stanford.edu/research/papiermache).

Calm user experienceDuring the project’s three years, iter-

ative evaluation of the Designer’sOutpost by 27 professional designersdemonstrated substantial merit in a sys-tem that is simultaneously tangible andvirtual. The integrated interactions thatdesigners found most valuable were

• the fluid transition from artifactson walls to single-user tools forfleshing out design details;

• lightweight support for design his-tory; and

• richer distributed collaboration.

The evaluations also consistentlyunderscored the importance of a calm

torical artifact, they prefer using thewritten transcripts of these recordings.We hypothesized that this was becausepaper books have a superior “inter-face”: Users can read, skim, highlight,photocopy, and share them with fargreater ease than audio tapes or MP3files.

This insight inspired me and my col-leagues at Ricoh Innovations to createBooks with Voices, which providesbarcode-augmented paper transcriptsfor fast, random access to digital video interviews on a PDA (http://hci.stanford.edu/research/oral-history). Inone study, oral historians found thatthis lightweight, structured access tovideo recordings offered a level of emotion not available in printed tran-scripts alone. The video also helpedobservers clarify the text and observenonverbal cues.

BUTTERFLYNETWhen I arrived at Stanford in 2004,

Andreas Paepcke and Hector Garcia-Molina asked me if I would like to col-laborate with the university’s InfoLab(http://www-db.stanford.edu) on theNational Science Foundation-fundedBioACT! project (http://www-db.stanford.edu/bioact).

Ron Yeh and I conducted structuredinterviews with field biologists, whichrevealed the field notebook to be theirresearch’s central organizing artifact.These notebooks contain everythingimportant: narrative writing, tabulardata, photographs, pasted-in experi-mental results, photocopied pagesfrom colleagues’ notebooks, even plantspecimens. The contents are so valu-able that participants would only let usphotograph pages from their note-books in their presence.

Field notebooks are robust, easy tobrowse, readable outdoors, and easilyportable. Electronic tools offer a dif-ferent set of equally powerful benefits:dynamic organization, textual search,remote sharing, and easy backup. Toexplore integration of these benefits,we developed ButterflyNet (http://hci.stanford.edu/research/biology).

Figure 1. Designers’ Outpost system. Users collaboratively author Web site architectureson a touch-sensitive electronic whiteboard augmented with cameras that capture documents placed on the wall.

ing of design students’ paper “idealogs,” photographs, wall-based ideas,and captured information from studiocritiques. We’re deploying the systemin an interaction design course this falland anticipate that rich, low-overheadcapture of design ideas will facilitatestudents’ ability to share their workwith teammates and staff as well as tocreate electronic portfolios after com-pleting a project.

T he next decade promises sweep-ing transformations in how weinteract on the move. Looking

forward, we’re exploring ways to com-bine an electronic notebook with aflexible, high-resolution, input surface

Electronic field notebookField biologists input data in the

ButterflyNet system with an Anoto penthat converts their handwriting intodigital images. These pens are an orderof magnitude cheaper than tablet PCsand offer an order of magnitudegreater battery life. Another advantageis that the system reverts to traditionalpen and paper when the Anoto pen’sbattery runs out.

ButterflyNet integrates field re-searchers’ notes with other digitalinformation that they explicitly cap-ture as well as environmental data suchas digital camera photographs, sensornetwork data, and Global PositioningSystem data. The system timestampsand correlates this heterogeneous fielddata and then presents it through thebrowser shown in Figure 2, enablingbiologists to “leaf through” their notesalong with related media and othercontextual information.

Chunyuan Liao and FrançoisGuimbretière at the University ofMaryland, College Park, collaboratedwith my group to create pen gesturetechniques for users to explicitly des-ignate relationships between notes andphotographs.

Just as the Designers’ Outpost trans-fers data from walls to personal designtools for later refinement, ButterflyNettransfers notebook contents to spread-sheets for data processing. It also auto-matically maintains the data’s lineageback to its original form, a key benefitof integrated interaction. In addition,users can share their work with othermembers of their group.

We’re currently evaluating Butterfly-Net at the Jasper Ridge BiologicalPreserve (http://jasper1.stanford.edu).We believe that the system will offerfield biologists a greater understandingof their data with minimal overheadand that these benefits will foster evenricher collaboration.

Beyond biologyBrian Lee and I have been extending

the ButterflyNet infrastructure toenable the electronic capture and shar-

and a smart phone that provides com-putation, interactivity, and networkaccess. These complementary tech-nologies afford integrated interactionsthat speak to a model of nearly invisi-ble computing: ready-made for every-day use but capable of consciousmanipulation for richer interactions. �

SSccootttt RR.. KKlleemmmmeerr is an assistant pro-fessor of computer science at StanfordUniversity, where he codirects theHuman-Computer Interaction Group.Contact him at srk@.cs.stanford.edu.

Editor: Bill Schilit, Intel Research Seattle; bill.schilit@intel.com

Figure 2. ButterflyNet system. Biologists can “leaf through” their field notes, relatedmedia, and other contextual data.

Get accessto individual IEEE Computer Society documents online.

More than 100,000 articles and conference papers available!

$9US per article for members • $19US for nonmembers

www.computer.org/publications/dlib