Gesture Recognition - York University€¦ · Graffiti Research 1 Enns, N., &MacKenzie,I.S.(1998).Touch ‐basedremote control devices.ExtAbstracts of CHI'98, 229‐230. New York:

Scott MacKenzieYork University

Gesture Recognition

CSE4443 – Mobile User Interfaces

© Scott MacKenzie

Map

• Gestures are everywhere• Android gestures• Unistrokes / Graffiti recognition• Android (revisited)

2© Scott MacKenzie

Map


3© Scott MacKenzie

in life

ASL (American Sign Language)

© Scott MacKenzie

Copy Editing

© Scott MacKenzie

Traffic Control

© Scott MacKenzie

Piazza Venezia

© Scott MacKenzie

Semaphore Signaling

© Scott MacKenzie

N U J V

© Scott MacKenzie

Map

• Gestures are everywhere• Android gestures• Unistrokes / Graffiti• Android (revisited)

10© Scott MacKenzie

in computing

Lightpen on CRT1

1 Sutherland, I. E. (1963). Sketchpad: A man‐machine graphical communication system. Proceedings of the AFIPS Spring Joint Computer Conference, 329‐346. New York: ACM.

© Scott MacKenzie

Finger(s) on CRT1

1 Herot, C. F., & Weinzapfel, G. (1978). One‐point touch input of vector information for computer displays. Proceedings of SIGGRAPH 1978, 210‐216. New York: ACM.

© Scott MacKenzie

Pointing1

1 Bolt, R. (1980). Put‐that‐there: Voice and gesture at the graphics interface. Computer Graphics, 14(3), 262‐270.

© Scott MacKenzie

Finger Gestures1

1 Minsky, M. R. (1984). Manipulating simulated objects with real‐world gestures using a force and position sensitive screen. Proceedings of SIGGRAPH 1984, 195‐203. New York: ACM.

© Scott MacKenzie

Virtual Devices and Templates1

1 Buxton, W., Hill, R., & Rowley, P. (1985). Issues and techniques in touch‐sensitive tablet input.Proceedings of SIGGRAPH '85, 215‐224. New York: ACM.

© Scott MacKenzie

Hand Gestures1

1 Zimmerman, T. G., Lanier, J., Blanchard, C., Bryson, S., & Harvill, Y. (1987). A hand gesture interface device. Proceedings of CHI+GI 1987, 189‐192.

© Scott MacKenzie

Pie Menus1

1 Callahan, J., Hopkins, D., Weiser, M., & Shneiderman, B. (1988). An empirical comparison of pie vs. linear menus. Proceedings of CHI '88, 95‐100. New York: ACM.

© Scott MacKenzie

Pen‐based Computing (early 1990s)

GRiDPad (Grid Systems)

MessagePad 2100(Apple)

325 Point (Fujitsu)

Pilot (Palm)

© Scott MacKenzie

© Scott MacKenzie 19

EO Personal Communicator

1993

Marking Menus1

1 Kurtenbach, G., & Buxton, W. (1993). The limits of expert performance using hierarchic marking menus. Proceedings of the INTERCHI '93, 482‐487. New York: ACM.

MeatBreadFruit & Veg Staples

MiscJunk

Groceries

Auto House

Clothing

© Scott MacKenzie

Marking Menus1

1 Kurtenbach, G., & Buxton, W. (1993). The limits of expert performance using hierarchic marking menus. Proceedings of the INTERCHI '93, 482‐487. New York: ACM.

© Scott MacKenzie

GEdit1

1 Kurtenbach, G., & Buxton, W. (1991). GEdit: A testbed for editing by contiguous gesture. SIGCHI Bulletin, 23(2), 22‐26.

© Scott MacKenzie

Today

© Scott MacKenzie


Gestures for text entry…

Unistrokes1

1 Goldberg, D., & Richardson, C. (1993). Touch‐typing with a stylus. Proceedings INTERCHI '93, 80‐87. New York: ACM.

© Scott MacKenzie

Graffiti

96% accuracy after five minutes1

1 MacKenzie, I. S., & Zhang, S. (1997). The immediate usability of Graffiti. Proceedings of Graphics Interface '97, pp. 129‐137. Toronto: Canadian Information Processing Society.

© Scott MacKenzie

1995

Graffiti Research

1 Enns, N., & MacKenzie, I. S. (1998). Touch‐based remote control devices. Ext Abstracts of CHI '98, 229‐230. New York: ACM.2 MacKenzie, I. S., Chen, J., & Oniszczak, A. (2006). Unipad: Single‐stroke text entry with language‐based acceleration. Proceedings of NordiCHI 2006, 78‐85. New York: ACM.3 Tinwala, H., & MacKenzie, I. S. (2009). Eyes‐free text entry on a touchscreen phone. Proceedings of TIC‐STH 2009, 83‐88. New York: IEEE.

1 2

3

© Scott MacKenzie

RollPad1

1 Oniszczak, A., & MacKenzie, I. S. (2004). A comparison of two input methods for keypads on mobile devices. Proceedings of NordiCHI 2004, 101‐104. New York: ACM..

© Scott MacKenzie

T‐Cube1

1 Venolia, D., & Neiberg, F. (1994). T‐Cube: A fast, self‐disclosing pen‐based alphabet. Proceedings of CHI '94, 265‐270. New York: ACM. © Scott MacKenzie

QuikWriting1

1 Perlin, K. (1998). Quikwriting: Continuous stylus‐based text entry. Proceedings of UIST '98, 215‐216. New York: ACM. © Scott MacKenzie

Cirrin1

1 Mankoff, J., & Abowd, G. A. (1998). Cirrin: A word‐level unistroke keyboard for pen input, Proceedings of UIST ‘98 (pp. 213‐214): New York: ACM.

© Scott MacKenzie

Edge Keyboard1

1 Wobbrock, J. O., Myers, B. A., & Hudson, S. E. (2003). Exploring edge‐based input techniques for handheld text entry. Proceedings of ICDCS 2003, 280‐282. New York: IEEE.

© Scott MacKenzie

MDITIM1

1 Isokoski, P., & Raisamo, R. (2000). Device independent text input: A rationale and an example. Proc of the Working Conference on Advanced Visual Interfaces ‐ AVI 2000, 76‐83. New York: ACM.

Four strokes

• North

• South

• East

• West

c

E

S

W

© Scott MacKenzie

EdgeWrite1

1 Wobbrock, J. O., Myers, B. A., & Kembel, J. A. (2003). EdgeWrite: A stylus‐based text entry method designed for high accuracy and stability of motion. Proc UIST 2003, 61‐70. New York: ACM.

© Scott MacKenzie

Shape Writing1

1 Zhai, S., & Kristensson, P.‐O. (2007). Learning shape writing by game playing. Ext Abstracts of CHI 2007, 1971‐1976. New York: ACM. © Scott MacKenzie

Map



Android Gestures (Review)

Touch Longpress

Swipe Drag Doubletouch

Pinchopen

Pinchclosed

http://developer.android.com/design/patterns/gestures.html

1‐finger Gestures 2‐finger Gestures


GestureDetector.SimpleOnGestureListener“a convenience class”

1‐finger Gestures


ScaleGestureDetector.SimpleOnScaleGestureListener“a convenience class”

ScaleGestureDetector


2‐finger Gestures

Demo_Scale


(see PaintPanel class in Demo_Scale)

Map



Background: Handwriting Recognition

• Advantages– considerable existing skill– good for annotating, sketching, etc.– handwriting is natural

• Disadvantages– handwriting recognition is hard!– in early products, goods delivered did not live up

to promises (next slide)


Unistrokes

• What is a Unistroke?– A single stylus or finger gesture– Begins on touch– Ends on lift– Each “symbol” is one gesture

• Benefits– Recognition is simple– Gestures made on top of one another– Heads‐up typing– Eyes‐free typing


Unistrokes vs. Graffiti

• Unistrokes– Invented by David Goldberg, Xerox PARC – First presented at INTERCHI ’93 a

– Patent held by Xerox (issued Jan. 1997)• Graffiti

– An “example” of Unistrokes– Commercial product– 3COM PalmPilot, HP Omnigo, etc.

a Goldberg, D., & Richardson, C. (1993). Touch‐typing with a stylus. Proceedings of the INTERCHI '93 Conference on Human Factors in Computing Systems, 80‐87. New York: ACM. 46

© Scott MacKenzie

Palm Pilot

Graffiti strokes are made here

Recognized results appear here


1997

Goldberg’s Unistrokes

A B C D E F G H I J K L M

N O P Q R S T U V W X Y Z

Goal: Fast for experts


Palm's Graffiti

A B C D E F G H I J K L M

N O P Q R S T U V W X Y Z

Goal: Easy for novices


Demo_Graffiti


Steps for Gesture Recognition

0. Input points for a gesture (aka stroke)1. Normalize the points to fit in a unit bounding

box2. Look for straight‐line strokes (N, S, E, W, NE,

SE, SW, NW)3. Compute stroke features4. Use stroke features to look for a match in the

dictionary



Step #0(input points for stroke)

private ArrayList gesture;...public boolean onTouchEvent(MotionEvent me){

...int x = Math.round(me.getX());int y = Math.round(me.getY());gesture.add(new Point(x, y));

...

String raw = u.recognize(gesture);

GraffitiPanel.java

An instance of the Unistroke class

Step #1(normalize points to fit in a unit bounding box)


012

3(0,0) (1,0)

(0,1) (1,1)


Unistrokes.java

Step #2a(look for straight‐line strokes)


Unistrokes.java


Unistrokes.java

Step #2b(look for straight‐line strokes)

Step #3a(compute stroke features)

• Cumulative motion– cumulative xmotion (e.g., A = 1, Z = 3)– cumulative ymotion (e.g., A = 2, Z = 1)


Step #3a(compute stroke features)


Unistrokes.java

Steps #3b(compute stroke features)

• Quadrants– 1st quadrant– 2nd quadrant– 2nd last quadrant– last quadrant


Unistrokes.java

Steps #3b(compute stroke features)


Step #3c(compute stroke features)


• Start/stop directions– start x direction (+ve, ‐ve)– start y direction (+ve, ‐ve)– stop x direction (+ve, ‐ve)– stop y direction (+ve, ‐ve)

Note:X 0 = left 1 = rightY 0 = down 1 = up

Step #3c(compute stroke features)


Unistrokes.java

Unistrokes.java

Step #4(use stroke features to look for match in dictionary)


Dictionary Entries (1 of 2)

• Each entry contains…– symbol (the ‘output’)– 1st quadrant– 2nd quadrant– 2nd last quadrant– last quadrant– min cumulative x– max cumulative x

0, 1, 2, 3, don’t care

Need min and max because the user’s strokes are “sloppy”


Dictionary Entries (2 of 2)

– min cumulative y– max cumulative y– start xmotion – start ymotion– stop xmotion– stop ymotion

• Note: can have multiple entries for each symbol (e.g. cw “0”, ccw “0”)

Need min and max because the user’s strokes are “sloppy”

+ve, ‐ve, don’t care


Dictionary Example (“B”)

Symbol B1st quad 22nd quad 3n‐1 quad 1n quad 2kx min 2.2kx max 4ky min 1.5ky max 2.5start x don’t carestart y +stop x ‐stop y don’t care

0

12

3


Gesture Set (Graffiti)


Can you imagine what the dictionary entry would look like for each of these gestures?

Graffiti Dictionary


Unistrokes.java

etc.

Map



Graffiti for Android


DemoGraffiti


Graffiti(experiment software)

See also…MacKenzie, I. S., & Castellucci, S. J. (2012). Reducing visual demand for gestural text input on touchscreen devices. Extended Abstracts CHI 2012, pp. 2585‐2590. New York: ACM. 72

© Scott MacKenzie

Graffiti vs. Soft Keyboard

• Frame Model of Visual Attention

PointFrame

TargetFrame

SurfaceFrame

EnvironmentFrame

SoftKeyboard Graffiti


Graffiti IME



Other possibilities…

Air Gestures


(Graffiti)

Shape Writing


Thank You


Documents

Gesture Recognition - York University€¦ · Graffiti Research 1 Enns, N., &MacKenzie,I.S.(1998).Touch ‐basedremote control devices.ExtAbstracts of CHI'98, 229‐230. New York: