Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals A Hierarchical Spatial Backoff Model Approach

Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals A Hierarchical Spatial Backoff Model ApproachYing Yin1,2, Tom Ouyang1, Kurt Partridge1, and Shumin Zhai11 Google Logo here

2MIT Logo here

Foundations to current methodsLanguage modelingvocabulary1-gram, 2-gram N gram frequenciesSpatial modelsconverting input touch points into probabilities of lettersEdit distance correctionassigning cost to insertion, deletion, and other spelling errorsUser and posture independentSmart touch keyboards have made a great amount of progress in the last few years2Research questionsOne promising area for improvement is by making them adapt to the userWhat types of adaption are possible? How do they affect performance?ContributionsA novel hierarchical adaptive modelShow benefits of posture and user adaptationOnline posture classification method13.2% reduction in character error ratecompared to base modelwithout language model

Types of adaptationIndividual differences (cf. Findlater & Wobbrock, 2012)Furthermore, people use different hand postures to type (cf. Azenkot and Zhai, 2012)A single model cannot account for the variability 5This page can be a good place to show the videoDifferent typing postures: two thumbs, one finger, or one thumb

6this vieo is very good. Teh main point seems to show differemt hand postures. So perhaps it is better to show in "however sections"?

Types of adaptationDifferent postures different touch patternsTouch patterns also depend on letter keys

(Azenkot & Zhai, 2012) Need adaptationA single model cannot account for the variability 7this slide seemed overlapping with the last a lot. Also it is more effective to copy paste a figure or two from Azenkot & Zhai here, better than just wordsChallenges of adaptationComplexitythree adaptive factors: key, posture, individuallarge number of submodelsneed sufficient data to build each submodelModel selectionwrong selection may hurt keyboard qualityuncertainty in posture classification Hierarchical spatial backoff model (SBM)

Focused on spatial model9Hierarchical spatial backoff model (SBM)Combinatorial and fine grained adaptationConservativeDoes not require an extra training phase Updates the model continuously onlineFocused on spatial modelInstantaneous adaptation 10Research methodPepper dataset (Azenkot & Zhai, 2012)30 right-handed participantsgiven random phrases to typebetween-subject: each person uses one posture84,292 touch points in total10-fold cross validationWe compared the effectiveness of different adaptive spatial models without language models.11Is this Pepper? If so:

Dataset: the "Pepper dataset" (Azenkot & Zhai, Mobile HCI 2012)Comparison of spatial models

Without language modelThe first one is the baseline method where the key decoding process is simply checking which keys visual bounding box the touch point lies in.With mixture of Gaussian models, the key detection process is finding the key that has the highest likelihood given the the touch point and the models.Posture & key adaptive model and the individual & key adaptive model give significant reduction in error rate which are about 12% and 14.7% respectively.

12make sure you clearly state verbally how these were generated "if we use models in decoding touch input data, without language modeling ...Two-thumb

One-fingerEffective key areasagain you need to say quite a bit to be clear about what these figures meanPosture classificationSVM-based classifierBased on correlation between time and distance between consecutive touch pointsno additional sensors requiredspeed independent86.4% accuracyReal-timeA variety of sensor, signals, and algorithms could be used, but optimal posture classification is not the primary goal of this paperCorrelation -> speed independentNo sensor required14It will be nice to have some sort of visual graphical support to this slide.Posture adaptation

94%15Like other high information figures, it is important to go slow and explain the vertical, horizontal and each of the three lines.

Individual adaptationPrototype implementation of SBM13.2% reduction in character error ratecompared to base modelwithout language modelIntegrated with real keyboardcombined with language modelruns on Android phone in real-time

Future workWeighted average of submodels instead of making binary decisionsMore data: real-use logging and game playingUser studies validate the accuracy and speed improvementhow users adapt their behavior to SBMCombine spatial and language models ContributionsA novel hierarchical adaptive modelShow benefits of posture and user adaptationOnline posture classification methodOpens up many more interesting HCI questions

Q & A

Prototype implementation of SBMPosture & key adaptation modelssupervised and batch learningIndividual adaptation modelsunsupervised and online learningBacks-off to more basic models when posture estimation is uncertain (conf. < 0.94)there is insufficient user data (< 50 data points)Labeled data from Pepper dataset22