People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu...

People, Pens, and Computers

François GuimbretièreHCIL, University of Maryland

francois@cs.umd.edu

Ken HinckleyMSR

kenh@microsoft.com

New devices, old tasks

Middle picture from Sellen et al.

People, Pens, and Tablet PC

The New Yorker

Illustration from Ken Hinckley presentation at Stanford

Typical setting for today’s interface

• Fixed stable environment, with a keyboard,

• Indirect interaction,

• High precision pointing system

Typical Tablet PC use

• Portable, unstable environment, without a keyboard

• Direct interaction,

• Low precision aiming

Pen based interface for Tablet PC

Document areaHow to create a more fluid interface?

Interface frameworkHow to make it more pen-friendly?

Pen based interface for Tablet PC

Interface frameworkHow to make it more pen-friendly?

• Empirical foundations• Use of strokes to cross target is more pen friendly

• Crossing is as efficient as point-and-click [Accot & Zhai, 2002]

• The basic interactor

• How expressive is it?

CrossY: crossing-based GUI[Apitz & Guimbretière 04]

CrossY

video

Previous Work

• Theoretical basis• Steering Law, Trajectory-Based Tasks [Accot & Zhai 97-02]

• Limited scope examples• Toggle Map [Baudish 98]

• Lotus Notes: multiple e-mail selection

• Conceptual design• Visual Instruments: [Winograd & Guimbretière 98]

• Overloading• Gedrics: [Geißler 95]

Command composition

• From stroke-by-stroke interaction• Borders are used to validate/cancel

Command composition

• From stroke-by-stroke interaction• Borders are used to validate/cancel

• To multi-command stroke

Scrolling

Line by line area

Page by page area

Absolute area

CrossY scrollbar

• Overloading simplify interactions• Shorter distances to issue commands

• Not as much precision necessary

CrossY scrollbar

• Overloading simplify interactions• Shorter distances to issue commands

• Not as much precision necessary

• Extending stroke for repeat• No need to wait for a timeout

Cursor control

• Cross to jump to an absolute position

Cursor control

• Cross to jump to an absolute position

Cursor control

• Cross to jump to an absolute position

• Near drag for coarse adjustment

Cursor control

• Cross to jump to an absolute position

• Near drag for coarse adjustment

Cursor control

• Cross to jump to an absolute position

• Near drag for coarse adjustment

• Far drag for fine adjustment• Similar to FineSlider [Masui 95]

• But one single stroke

Use of directionality

• Continuous find and replace

Use of directionality

• Continuous find and replace• Reverse direction for undo

What we learned

• Very well received by users• HCIL open house

• UIST

• Space requirements• Similar to point-and-click

• Trade-off with command combination due to sloppiness

• Overloading vs. easy discovery• Consistency helps with adoption

• Known in Windowing systems

Pen based interface for Tablet PC

Document areaHow to create a more fluid interface?

Scriboli and Stitching

Ken’s presentation

My desk

Affordances of paper documents[Sellen 01]

• Easy to navigate• Two-handed interactions and tactile feedback

• Reading across more than one document at once

• Easy to annotate• Directly on the document or on a nearby pad

• Well accepted during meetings• Socially accepted conventions

• Very difficult to modify• Printed documents are created and edited as digital documents

• Expensive to distribute and archive

Bridging the gap: previous work

• Digital emulation• FreeStyle system [Wang 89]

• MATE [Hardock 93]

• XLibris [Schilit 98], [Golovchinsky 02]

• Tight coupling• DigitalDesk [Wellner 93], Ariel [Mackay 95]

• A-Book [Mackay 02]

• PaperLink [Arai 97]

• Intelligent Paper [Dymetman 98]

• Paper as input device• Xax [Johnson 93]

• Anoto

• Paper PDA [Heiner 99], [Avrahami 01]

Cohabitation[Guimbretière 03]

Stroke capture

From Anoto documentation

• Requirement• Stroke coordinates on the page

• Page ID

• Large address space

• Possible technologies• Anoto

• DataGlyphs [Hecht 94]

• MEMO pen [Nabeshima 95]

System architecture

My Desk with PADD

PapierCraft: command system for PADD[Liao, Guimbretiere and Hinckley 05]

PADD document

Anoto pen

PADD notepad

PapierCraft: using PADD as proxies

Copy Paste

Synchronization On your Tablet PC

PapierCraft

• Gesture/Ink• Use the command button

• Scopes

• Command selection• Marking menu

• Writing down an unambiguous prefix of a command

• Batch mode processing

Early feedback and future work

• Small scale evaluation• Commands with written words were very popular

• Higher recognition rate needed for strokes only feedback

• Contextual information might provide enough feedback during pastes

• Future work• More reliable recognition engine

• Pen-based feedback• Haptic feedback, LED

• Real time processing : streaming stroke

Conclusions

• Four pen-based interfaces• CrossY explores crossing-based interfaces

• Scriboli explores high performance direct manipulation interfaces

• Stitching explores multi-devices interactions

• PapierCraft uses PADD as document proxies

• One goal• Bring the ease of use of pen and paper to digital interfaces

• Future• Seamless integration of both media

Acknowledgments

• Collaborators• Jim Hollan

• Students• Georg Apitz, Nicholas Chen (CrossY/ScribolY)

• Kevin Conroy, Dave Levin, Chunyuan Liao (PADD/ProofRite)

• Sponsors• Microsoft

• HP

• Anoto, Logitech, Maxell

• Colleagues and friends• Corinna Löckenhoff

• Samrat (Bobby) Bhattacharjee

• Ben Bederson