Talk v1.0Gopi FlahertyApril 15, 2003

Talk v1.0Gopi FlahertyApril 15, 2003

Gesture ToolkitsGesture Toolkits

Graffiti on PalmExtensions for X11Windows for Pen Computing

Graffiti on PalmExtensions for X11Windows for Pen Computing

Boring Gesture SystemsBoring Gesture Systems

Keyboard-like behaviorSimple input model with streams of charactersMost applications behave as if they have a keyboardSome Palm devices have keyboards instead of graffiti - one even comes in a graffiti or keyboard version - and apps don’t notice

Keyboard-like behaviorSimple input model with streams of charactersMost applications behave as if they have a keyboardSome Palm devices have keyboards instead of graffiti - one even comes in a graffiti or keyboard version - and apps don’t notice

GraffitiGraffiti

Graffiti 2.0: different alphabet, more two stroke charactersWhen the user enters a double stroke gesture, the app receives a letter, a backspace, and the next letterAPI documentation suggests hardcoding the list of these gestures, and implementing a delay if this behavior is a problem

Graffiti 2.0: different alphabet, more two stroke charactersWhen the user enters a double stroke gesture, the app receives a letter, a backspace, and the next letterAPI documentation suggests hardcoding the list of these gestures, and implementing a delay if this behavior is a problem

GraffitiGraffiti

Simple extension to X11Has a configuration file with a 3x3 grid. Gestures are defined based on the order you hit squares in the gridBehaves similarly to GraffitiToolkit?

Simple extension to X11Has a configuration file with a 3x3 grid. Gestures are defined based on the order you hit squares in the gridBehaves similarly to GraffitiToolkit?

xstrokexstroke

Gestures for simple commandsSome simple gesture systems - possibly graffiti - let you have gestures for commandsGenerally very limited, also requires that gesture system knows about commands, and most simple gesture systems don’t get very involved

Gestures for simple commandsSome simple gesture systems - possibly graffiti - let you have gestures for commandsGenerally very limited, also requires that gesture system knows about commands, and most simple gesture systems don’t get very involved

Slightly more advancedSlightly more advanced

Existing apps - console UNIX apps via graffiti or xstroke is better than trying to make input fieldsRetrofitting an app such that half of it is gesture-based frequently has problemsGood pen/gesture input requires fundamentally different ways of interacting and programming

Existing apps - console UNIX apps via graffiti or xstroke is better than trying to make input fieldsRetrofitting an app such that half of it is gesture-based frequently has problemsGood pen/gesture input requires fundamentally different ways of interacting and programming

Are these approaches ever useful?Are these approaches ever useful?

Systems that provide some gesture resources but only at a more basic levelUseful widgets, but with only limited configurabilityUsually a fixed set of gestures that are recognizable

Systems that provide some gesture resources but only at a more basic levelUseful widgets, but with only limited configurabilityUsually a fixed set of gestures that are recognizable

Middle of the roadMiddle of the road

Text entry widget is very configurablePhone, Date, Time, Address and Name are field types

Text entry widget is very configurablePhone, Date, Time, Address and Name are field types

Newton WidgetsNewton Widgets

Text entry widgets have many callback methods that can be utilized for more configurabilityviewWordScript is called when a word is recognizedviewChangedScript is called to indicate that the text in the entry has changedviewGestureScript indicates that a gesture has been entered - this would be a raw, unrecognized gesture

Text entry widgets have many callback methods that can be utilized for more configurabilityviewWordScript is called when a word is recognizedviewChangedScript is called to indicate that the text in the entry has changedviewGestureScript indicates that a gesture has been entered - this would be a raw, unrecognized gesture

Gesture widgets are primarily forms-basedInternal implementation is opaqueApplication isn’t involved in recognition normallyBecoming slightly more involved in gesture processing appears to be possible through some reverse engineering

Gesture widgets are primarily forms-basedInternal implementation is opaqueApplication isn’t involved in recognition normallyBecoming slightly more involved in gesture processing appears to be possible through some reverse engineering

Why do we need toolkit support?There are hundreds of useful apps for the NewtonMost apps seem quite content to use the standard widgetsNewton widgets are clearly optimized for pen and gesture input, despite not being very involved in the gesture parsing

Why do we need toolkit support?There are hundreds of useful apps for the NewtonMost apps seem quite content to use the standard widgetsNewton widgets are clearly optimized for pen and gesture input, despite not being very involved in the gesture parsing

The Interesting StuffThe Interesting Stuff

Fundamentally, the widgets are still very much forms-basedIntelligent dynamic forms can be extremely interactive, but they are still forms and still limitedHacking widgets to do things they weren’t intended to do can get tedious and slow

Fundamentally, the widgets are still very much forms-basedIntelligent dynamic forms can be extremely interactive, but they are still forms and still limitedHacking widgets to do things they weren’t intended to do can get tedious and slow

What is the cursor?Write gestures centered on cellEdit line on the topEarly newton app: viewGestureScript -> viewWordScript?

What is the cursor?Write gestures centered on cellEdit line on the topEarly newton app: viewGestureScript -> viewWordScript?

Common apps: spreadsheet

Common apps: spreadsheet

Standard text entry fields format text themselves, with limited flexibilityEven basic word processors have quite complex custom formatting needs - justified text, tab editing, headers and footersIt can be hard to combine custom rendering code with standard gesture widgets

Standard text entry fields format text themselves, with limited flexibilityEven basic word processors have quite complex custom formatting needs - justified text, tab editing, headers and footersIt can be hard to combine custom rendering code with standard gesture widgets

Common apps:Word ProcessorCommon apps:Word Processor

If a toolkit is to provide easy support for complex apps, it needs to be extremely transparentApplications must be able to override as much or as little as is necessary for them to achieve their UI goals

If a toolkit is to provide easy support for complex apps, it needs to be extremely transparentApplications must be able to override as much or as little as is necessary for them to achieve their UI goals

Results:Results:

Hybrid system, designed to allow handwriting in apps without modificationMany basic recognition features can be used in apps that work with or without the penProvides more advanced features and tighter pen integration that is harder to integrate with non-pen interaction

Hybrid system, designed to allow handwriting in apps without modificationMany basic recognition features can be used in apps that work with or without the penProvides more advanced features and tighter pen integration that is harder to integrate with non-pen interaction

Microsoft TabletPCMicrosoft TabletPC

PenInputPanelPenInputPanel

PenInputPanelPenInputPanel

Object that can be attached to any input field in your appAllows app to override default behaviors

Object that can be attached to any input field in your appAllows app to override default behaviors

Superclass of RichEdit or RichTexBoxProvides far more controlApplications can register controls to receive various predefined gestures

Superclass of RichEdit or RichTexBoxProvides far more controlApplications can register controls to receive various predefined gestures

InkEditInkEdit

Pre-Defined GesturesPre-Defined Gesturesenum ApplicationGesture { AllGestures = 0, NoGesture = 0xf000, Scratchout = 0xf001, Triangle = 0xf002, Square = 0xf003, Star = 0xf004, Check = 0xf005, Curlicue =0xf010, DoubleCurlicue = 0xf011, Circle = 0xf020, DoubleCircle = 0xf021, SemiCircleLeft = 0xf028, SemiCircleRight = 0xf029, ChevronUp = 0xf030, ChevronDown = 0xf031, ChevronLeft = 0xf032, ChevronRight = 0xf033, ArrowUp = 0xf038, ArrowDown = 0xf039, ArrowLeft = 0xf03a, ArrowRight = 0xf03b,

Up = 0xf058, Down = 0xf059, Left = 0xf05a, Right = 0xf05b, UpDown = 0xf060, DownUp = 0xf061, LeftRight = 0xf062, RightLeft = 0xf063, UpLeftLong = 0xf064, UpRightLong = 0xf065, DownLeftLong = 0xf066, DownRightLong = 0xf067, UpLeft = 0xf068, UpRight = 0xf069, DownLeft = 0xf06a, DownRight = 0xf06b, LeftUp = 0xf06c, LeftDown = 0xf06d, RightUp = 0xf06e, RightDown = 0xf06f, Exclamation = 0xf0a4, Tap = 0xf0f0, DoubleTap = 0xf0f1,};

Future Gestures 1Future Gestures 1Infinity Switch in and out of gesture mode Cross Delete Paragraph mark New paragraph Section New section Bullet Application-specific Bullet-cross Application-specific Squiggle Bold Swap Exchange content Openup Open up space between words Closeup Close up extra space Rectangle Selects enclosed content Circle-tap Application-specific Circle-circle Application-specific Circle-cross Application-specific Circle-line-vertical Application-specific Circle-line-horizontal Application-specific Plus Paste Double-up Scroll up Double-down Scroll down

Future Gestures 2Future Gestures 2

Double-left Scroll left Double-right Scroll right Triple-up Page up Triple-down Page down Triple-left Application-specific Triple-right Application-specific Bracket-over Application-specific Bracket-under Application-specific Bracket-left Start of selection Bracket-right End of selection Brace-over Application-specific Brace-under Application-specific Brace-left Start of discontinuous selection Brace-right End of discontinuous selection Triple-tap Application-specific Quadruple-tap Application-specific

Custom RecognizersCustom Recognizers

Custom Recognizer objects can be registered to Ink objectsAn InkCollector object collects gestures, which can be passed to a custom recognizer, or to the original Microsoft one, or both in parallel

Custom Recognizer objects can be registered to Ink objectsAn InkCollector object collects gestures, which can be passed to a custom recognizer, or to the original Microsoft one, or both in parallel

For each recognizer:For each recognizer:

enum RecognizerCapabilities { DontCare = 1, Object = 2, FreeInput = 4, LinedInput = 8, BoxedInput = 16, CharacterAutoCompletionInput = 32, RightAndDown = 64, LeftAndDown = 128, DownAndLeft = 256, DownAndRight = 512, ArbitraryAngle = 1024, Lattice = 2048};

Egg Freckles?One of the largest challenges with computer recognition of any typeDealing with recognition ambiguity can be the difference between a pen system working or not working

Egg Freckles?One of the largest challenges with computer recognition of any typeDealing with recognition ambiguity can be the difference between a pen system working or not working

Recognition AmbiguityRecognition Ambiguity

Sidesteps the issue completelyYou write the stroke the way you’re supposed toIf it mis-recognizes, the assumption is that it was your faultBlaming the user is argued by some to be the cause of its success

Sidesteps the issue completelyYou write the stroke the way you’re supposed toIf it mis-recognizes, the assumption is that it was your faultBlaming the user is argued by some to be the cause of its success

Graffiti:What ambiguity?

Graffiti:What ambiguity?

Basic input: What letter or word did the user type?One of the easier types of ambiguity to deal withThe recognizer provides an n-best list, select the top, and you provide a UI to making alternate selections from the list

Basic input: What letter or word did the user type?One of the easier types of ambiguity to deal withThe recognizer provides an n-best list, select the top, and you provide a UI to making alternate selections from the list

Different kinds of ambiguityDifferent kinds of ambiguity

This can be somewhat more challenging with multiple recognizers: how do you intermingle their respective n-best lists?If gestures can perform actions, the UI can be difficult: how do you present a choice of “cut”, “copy” or the word “hello” to the user?Newton had interesting choice in n-best: do letter by letter, ignoring dictionary weighting. Produced a new n-best list

This can be somewhat more challenging with multiple recognizers: how do you intermingle their respective n-best lists?If gestures can perform actions, the UI can be difficult: how do you present a choice of “cut”, “copy” or the word “hello” to the user?Newton had interesting choice in n-best: do letter by letter, ignoring dictionary weighting. Produced a new n-best list

Simple AmbiguitySimple Ambiguity

If gesture recognition is being used in a time-constrained environment, it brings up other interesting challengesResolving ambiguities is distracting and slow, but retaining n-best information indefinitely can increase storage space immenselyLazy ambiguity resolution can’t be used for action gestures without extreme penalties

If gesture recognition is being used in a time-constrained environment, it brings up other interesting challengesResolving ambiguities is distracting and slow, but retaining n-best information indefinitely can increase storage space immenselyLazy ambiguity resolution can’t be used for action gestures without extreme penalties

Ambiguity and timeAmbiguity and time

Later versions of the Newton OS let you do lazy recognitionWhen text is written, it’s retained as the original glyphs, segmented into words and reflowed with word wrap as normal textAt any stage in the future, the user can request that text be recognizedRetain as glyphs is also an option in the n-best correction widget

Later versions of the Newton OS let you do lazy recognitionWhen text is written, it’s retained as the original glyphs, segmented into words and reflowed with word wrap as normal textAt any stage in the future, the user can request that text be recognizedRetain as glyphs is also an option in the n-best correction widget

The lazy approachThe lazy approach

Very lazy recognitionSomewhat similar to the Newton’s “keep it as glyphs”, but significantly more structured than simple text editingOne of the design goals is to deal with the problem of recognition systems demanding immediate ambiguity resolution - very distracting

Very lazy recognitionSomewhat similar to the Newton’s “keep it as glyphs”, but significantly more structured than simple text editingOne of the design goals is to deal with the problem of recognition systems demanding immediate ambiguity resolution - very distracting

DENIM and SILKDENIM and SILK

What about segmentation?Most shipping recognition systems do segmentation of strokes into individual collections, and do not allow any future re-analysis of the decisionAllowing segmentation ambiguity is difficult from a UI perspective - double tapping a word is easy, but what about trying to resolve a pair of words

What about segmentation?Most shipping recognition systems do segmentation of strokes into individual collections, and do not allow any future re-analysis of the decisionAllowing segmentation ambiguity is difficult from a UI perspective - double tapping a word is easy, but what about trying to resolve a pair of words

Meta-ambiguityMeta-ambiguity

Another aspect - who gets the gesture?On a screen with forms, how do you resolve which text entry field gets the text?Approaches I have seen are the simple ones - there’s a bounding box, strokes in the box are for that widgetThis reduces the density of widgets and requires careful design

Another aspect - who gets the gesture?On a screen with forms, how do you resolve which text entry field gets the text?Approaches I have seen are the simple ones - there’s a bounding box, strokes in the box are for that widgetThis reduces the density of widgets and requires careful design

SegmentationSegmentation

A common problem in poorly designed Newton apps: text entry box right next to control widgetsWrite a stroke - start too close to the window drag area and the window moves insteadNearly impossible to resolve, since control widgets demand immediate feedback, but a serious problem on small screens

A common problem in poorly designed Newton apps: text entry box right next to control widgetsWrite a stroke - start too close to the window drag area and the window moves insteadNearly impossible to resolve, since control widgets demand immediate feedback, but a serious problem on small screens

Was it a gesture?Was it a gesture?

How much support is needed in the app for dealing with ambiguity?Cut and paste on the Newton retains n-best objectsIn theory, nearly infinite knowledge possibilitiesWhich text field wants the input?

How much support is needed in the app for dealing with ambiguity?Cut and paste on the Newton retains n-best objectsIn theory, nearly infinite knowledge possibilitiesWhich text field wants the input?

Application SupportApplication Support

Imagine two text fields: One is a city, the other is a countyIt’s generally impractical to have a complete list of all of these, but a possible list can existIf there is ambiguity about which field an entry is for, the application’s knowledge about likely values can add increased weighting knowledge

Imagine two text fields: One is a city, the other is a countyIt’s generally impractical to have a complete list of all of these, but a possible list can existIf there is ambiguity about which field an entry is for, the application’s knowledge about likely values can add increased weighting knowledge

Application IntelligenceApplication Intelligence

Dean RubineDean Rubine

Specifying Gestures By Example

Specifying Gestures By Example

Provide a series of samples for each gestureWorks exclusively with single strokesEager: recognizes during the stroke to increase responsivenessDrawing ink on screen is easy - so process during the strokeBattery systems slow processor during strokes

Provide a series of samples for each gestureWorks exclusively with single strokesEager: recognizes during the stroke to increase responsivenessDrawing ink on screen is easy - so process during the strokeBattery systems slow processor during strokes

Basic outline:Basic outline:

Jennifer Mankoff, Scott E. Hudson, Gregory D. Abowd

Jennifer Mankoff, Scott E. Hudson, Gregory D. Abowd

Providing Integrated Toolkit Level Support for Ambiguity in

Recognition Based Interfaces

Providing Integrated Toolkit Level Support for Ambiguity in

Recognition Based Interfaces

Basic ambiguityBasic ambiguity

Segmentation AmbiguitySegmentation Ambiguity

Shortcomings in other systems

Shortcomings in other systems

Nobody seems to deal with target ambiguitySegmentation ambiguity is also rarely tackledApplication-specific knowledge is rarely used

Nobody seems to deal with target ambiguitySegmentation ambiguity is also rarely tackledApplication-specific knowledge is rarely used

Toolkit identifies ambiguous events and sends them to the mediation subsystemEach mediator receives the event in turnMediators can ignore, partially resolve, or defer their decision to the futureSome mediators have a UI, some do notAlso allows much lazier mediation

Toolkit identifies ambiguous events and sends them to the mediation subsystemEach mediator receives the event in turnMediators can ignore, partially resolve, or defer their decision to the futureSome mediators have a UI, some do notAlso allows much lazier mediation

Mediation ToolkitMediation Toolkit