Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory

Dialog Design - Gesture & Pen Interfaces

This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to [email protected] are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: November 2010.

Dialog Styles1. Command languages 2. WIMP - Window, Icon, Menu, Pointer3. Direct manipulation4. Speech/natural language5. Gesture & pen

UI Design - Georgia Tech 2

Many Types of Gesture-based UIs• Purpose

Text input vs. selecting objects and actions on objects• Gestures constrained to 2D surface vs. not constrained

(i.e., 3D)• Single vs. multiple points

One finger vs. multi-touch• Sensed via something in/on user’s hand vs. not

Stylus on tablet vs. finger on touchpad Data glove vs. video for hand gestures

• Size of sensing area iPhone – iPad – tablet computer – table-top – wall - room

• Type/scale of user motor control Finger movements vs. hand movements vs. body

movements• Work area used by one user vs. more than one


Agenda• Look at several points in the gesture

design space PDAs such as iPhone, iPad, Palm Pilot

Text input with some more general usesScroll, resize, rotate

Stylus or fingers (for multi-touch) Multi-touch, shared work space 3d Gestures, video recognition


PDAs – the New Swiss Army Knife


What is a Personal Digital Assistant?• A multi-function, portable device supporting

activities such as Communications by voice, text, video Listen, read, watch – audio, news, books, video Web browsing Calendaring Gaming Social networking Navigating from point A to point B Managing personal information of all sorts Etc etc etc Aka iPhone, iPad – a small but powerful computer

with pen or gesture input


Personal Digital Asst. (PDA)


Palm VII

HP Jornada Handspring VisorCirca 2000

Palm IIIc

Apple Newton (1993)

Been around for a longggg time

Contemporary PDAs


iPhone

Acer Android Tablet

Motorola

Nokia BlackBerry

iPad

Interacting with a PDA• Text input methods

Soft keyboard Free-form ink – no recognition Stylized characters to facilitate recognition Superimposed on keyboard Block printing recognition Script recognition

• Stylus or finger


Soft KeyboardsPros/cons of soft keyboards vs. hard keyboards?Finger vs. stylus soft keyboards?


Free-form Ink• Ink is the data, take as is

• Human is responsible forunderstanding andinterpretation

• Like a sketch pad


Examples• Digital Ink - CMU

Video, CHI ‘98 View it at www. ……..

• Flatland - Xerox PARC Video, CHI ‘99 View it at www. ….


Gesture Recognition - Graffiti• Stylized characters to facilitate

recognition• Graffiti - Unistroke alphabet on Palm

PDA Unistroke = one stroke


Gesture on Keyboard - Cirrin• Developed by Jen Mankoff (GT Ph.D -

> Berkeley -> CMU)• Word-level unistroke technique• UIST ‘98 paper• Use stylus to go

from one letterto the next ->


Gesture on Keyboard - Quikwriting • Developed by Ken Perlin• UIST ‘98 paper


Quikwriting


p l

http://mrl.nyu.edu/projects/quikwriting/

e

Said to be as fast as graffiti, but have to learn more

Gesture Recognition - Shapewriter

• Aka Shark• Live demo on iPad


Gesture Recognition – ShapeWriter/Shark Video


Recognizing Block Printing and Script• Recognizing letters and numbers and

special symbols • Lots of commercial systems• English, kanji, etc.• Not perfect, but people aren’t either!

People - 96% handprinted single characters Computer - >97% is really good

• OCR (Optical Character Recognition)


Recognition Issues• Off-line vs. On-line

Off-line: After all writing is done, speed not an issue, only quality.

Work with either a bit map or vector sequence On-line: Must respond in real-time - but have

richer set of features - acceleration, velocity, pressure

• Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity and to correct errors 1 I l On-line: show choices or best guess


More Issues• Boxed vs. Free-Form input

Sometimes encounter boxes on forms• Printed vs. Cursive

Cursive is much more difficult to impossible• Letters vs. Words

Cursive is easier to do in words vs individual letters, as words create more context

Similar to typo correction on phones and tablets


Recognizing Printed Characters


Beyond Text: Gestures – single point


-Might mean delete

-Insert

-Paragraph

Define a series of (hopefully) simple drawing gesturesthat mean different commands in a system

Brown’s Line-o-grammer


Beyond Text: Gestures – Multiple Points


Mac Trackpad: up to four points of input

For power users

Pen Use Modes• Inputting text vs. selecting vs. action

On iPhone/iPadEnter text mode by selecting a text entry box

Single touch => selectSingle touch + depress => move actionMulti touch => other actions

• Other ways to switch modes? Mode icon(s) – for instance to draw lines

vs. print text Others with which you are familiar??


Creative Uses of Multi-touch Surfaces

• Liquid Text – Craig Tashman On my Mac

LiquidText | Multitouch Document Manipulation.flv

• Jeff Han http://www.ted.com/talks/

jeff_han_demos_his_breakthrough_touchscreen.html

• BrailleTouchUI Design - Georgia Tech 27

Wall-size Single Touch => Single User


Complete withpie menu

Multi-touch => Multi-User• Multi-touch permits multi-users

Two users can simultaneously operate on two areas or display or on two objects

• But can’t determine which user is doing what


Wall-size, Multi-Touch => Multi-User


Table-size, Multi-Touch: MS Surface


• Simultaneous user multi-touch

• No user ID• No pressure• Modest price

Microsoft Surface•


Table-size, Multi-Touch: Mitsubishi Diamond Touch• Simultaneous

user multi-touch

• Pressure!!• User ID!!• Expensive!!!!


Diamond Touch Video• On my Mac

Diamond Touch CeBit 2011 – YouTube.flv


3D Gestures, Video Recognition

• Imaginary• On my Mac

interactive ui in minority report.flv


3D Gestures, Video Recognition• This one is real• On my Mac

Hands Free 3D_ Second Life Object Editing Demo.flv


A Different Application• Signature verification

• But not with a mouse :)


Pen-less Gestures• As in iPhone & iPad – 2D• Minority Report

http://www.youtube.com/watch?v=NwVBzx0LMNQ&feature=related

• Toshiba http://www.youtube.com/watch?v=MDUN01U--jE&feature=fvsr

• Multi-touch http://www.youtube.com/watch?v=ok1n0lwY1ZA&feature=related

• Put that there – gesture & speech http://www.youtube.com/watch?v=RyBEUyEtxQo


The End


Documents

Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory