Interaction Techniques with Mobile Devices
Jingtao Wang
March 6th, 2006 Guest Lecture for CS160
Agenda
Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile
ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile DevicesInteract With Other Devices
Why Mobile Devices Matters
6.5 billion people in the world 1.5 billion cell phones worldwide 500 million PCs (?) 46 million PDAs 1 million TabletPCs
Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned
6,500
1,500
50046 1
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
Peopl
e
Cell P
hones PCs
PDAs
Table
tPCs
Mill
ion
s
Agenda
Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile
ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile DevicesInteract With Other Devices
Mark Weiser (1952 – 1999)
Introduced the idea of “ubiquitous computing”
Weiser’s Vision
“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it”
Weiser’s Vision: Pervasive
Mainframemany people share 1 computer
PC1 person with 1 computer
Ubicompmany computers server each person
Agenda
Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile
ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile DevicesInteract With Other Devices
Key Challenges in Making Mobile Applications
Limited Physical Resources CPU, Memory, Screen Size, Input Devices,
Battery Life etcDiversified Context of UseDifferent ActivitiesLimited Attention
Limited Physical Resources
A mobile device usually has 1/100 CPU power, 1/30 Screen resources, 1/20 Memory, and extremely limited input devices when compared with desktops in the same era.
Small Screen Geography is different
a. Large Screenb. Small Screen
Diversified Context of Use
Different Activities
People use small-screen devices for different activities than desktops; don’t assume you understand these activities already
Limited Attention
Don’t assume your applications have people’s full attention; they’re doing something else while using your device.
Context, Activity, Attention
There is more opportunity for purpose-specific or context-specific devices than for general-purpose solutions that try to work for everyone in any situation.
One Sentence Summary
Mobilize, Don’t Miniaturize !
There is no silver bullet in designing mobile applications, but there is one sentence you should remember -
Agenda
Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile
ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile Devices
Input Techniques for Mobile Devices
PointingText Input
(Virtual) Keyboard InputHandwriting InputSpeech Input
Marker Based Input
Common Pointing/Navigation Techniques
iPod Dialpad
TrackPoint
JogDial
Touch ScreenFour-directional keypad
TinyMotion – Camera Phone Based Pointing
Detecting the movements of cell phones in real time by analyzing image sequences captured by the built-in camera.
Typical movements include - horizontal and vertical movements, rotational movements and tilt movements.
Input Techniques for Mobile Devices
PointingSensor Augmented Input
Text Input(Virtual) Keyboard InputHandwriting InputSpeech Input
Marker Based Input
(Virtual) Keyboard Input
How to Make QWERTY Keyboards Portable ?
Break
Making QWERTY Keyboards Portable
Reducing the Absolute SizeReducing the Number of KeysMaking Keyboards FoldableProjective Keyboard
Projective Keyboard
From http://www.vkb-tech.com
Projective Keyboard – Working Mechanisms
1. Template creation
2. Reference plane illumination
3. Map reflection coordinates
4. Interpretation and
communication
Can We Perform Better Than QWERTY?
Originally QWERTY layout is manually optimized for two handed, alternative typing to minimize mechanical jam
OPTI ATOMIK
OPTI II FITALY
Theories Behind Quantitative Keyboard Layout Optimization
Fitt’s LawDigraph Distribution Model in a Language
Can We Use Less Buttons than a Full QWERTY?
12-button Keypad15-button Keyboard
Half Keyboard
Disambiguation Methods for Reduced Size Keyboard
The QWERTY keyboard itself is ambiguous! ( A vs. a, 3 vs. #)
Pressing Several Keys together (shift key) Multiple Key Press
Multi-Tap (22.5 wpm*) Two-Key Input (25.0wpm*)
Dictionary/Statistics Based Disambiguation Methods T9/T15 (45.7 wpm*) LetterWise
*Performance Upper Bound Estimation from Silfverberg 2000
FingerSense – Button Disambiguation by Fingertip Identification
Differentiating a pressing action by identifying the actual finger involved
Can be Faster than Regular Tapping When the Adjacent Tapping Involves Different Fingers and Different Buttons (59% on a phone keypad)
Input Techniques for Mobile Devices
PointingSensor Augmented Input
Text Input(Virtual) Keyboard InputHandwriting InputSpeech Input
Marker Based Input
Handwriting Input
1938 George Hansel, U.S. Patent 2,143,875, machine recognition of handwriting
1957 T. L. Dimond's stylator - the first on-line handwriting recognizer prototype
Newton, Palm Pilot, PocketPC, CrossPad, TabletPC
Handwriting Recognition - Terminology
Printed Character Recognition (OCR) Relatively mature these days, key challenges – layout
analysis, fonts recovery, robust recognition for low quality, low resolution input
Major Usage – Digital Library Handwritten Character Recognition
Online HWR (With Temporal info) Character, Word, Sentence Level
Offline HWR (Using raster image as input, no temporal info)
Major Usage : Bank Check Recognition, Postal Automation
Word/Sentence Level Recognizer
Build on Top of Character Recognizer General Strategy
Over Segmentation Call Character/Component Recognizer, Get a List
of Candidates with Scores Apply Geometry Spatial Information ( size,
component gap ) Language Information (Dictionary, Language Model etc) to Each Sub Path
Use Hypnosis Search (Dynamic Programming, A* etc) to Determine the Best Possible Path
Challenges in Online HandwritingRecognition
Character Set/Dictionary Size (Especially Asian Languages!)
Cursive Writing Styles/Broken Strokes/Duplicate Strokes/Omitted Components
Stroke Order VariationsLimited memory and CPU Power in Small
Devices
Some Prototype Recognizers from IBM
New Form Factors - Anoto Pen
Commercial Product is Available In the U.S. Market (Logitech IO Pen)
Uses A Camera Mounted Beside the Tip of the Pen and Preprinted Dot Patterns to Detect Pen Movment
SHARK – Shorthand Writing on Stylus Keyboard
A Combination of Virtual Keyboard and Handwriting Recognition
Writing Shape of a Word (Shorthand) is Defined By the on Screen Location of Characters in the Word
EdgeWrite Input
An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area
Faster and More Reliable Than Regular Graffiti
Especial Useful for People with Motor and Muscle Disabilities
Input Techniques for Mobile Devices
PointingSensor Augmented Input
Text Input(Virtual) Keyboard InputHandwriting InputSpeech Input
Marker Based Input
Input Techniques for Mobile Devices
PointingSensor Augmented Input
Text Input (Virtual) Keyboard InputHandwriting InputSpeech Input
Marker Based Input
Emerging Marker Based Interactions on Camera Phones
Towards More Sensitive Mobile Devices
Agenda
Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile
ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile DevicesInteract With Other Devices
Peephole Displays (With Demo)
Zoomable Interface on Mobile Devices
ZoneZoom By Microsoft Take advantage of spatial
memory
VS.
Halo - A Virtual Periphery for Mobile Devices
Provding Visual Cue for Objects Located Out of the Small Screen
Agenda
Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile
ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile DevicesInteract With Other Devices
Using Mobile Devices with Desktop Computers
Pebbles Project at CMU Using a PDA as additional keypad, touch pad, scroll
wheel and controller of PointPoint slides for desktop Applications
http://www.pebbles.hcii.cmu.edu/
Using Mobile Devices with Laptops
Wang 2002
Using Mobile Devices with Large Displays
Ballagas 2005
Question and Answer
Backup Slides
Electromyographic (EMG) Keyboard
NASA Virtual Keyboard SenseBoardKeyboard
The General Flow of Handwriting Recognition
Major Players in this Area (Embedded)
English ART - ART Recognizer CIT - Jot IBM - Derived from Multi-lingual version Microsoft - Transcriber ( Licensed version of Calligrapher) & Own Sin
gle character recognizer Motorola Paragraph - Calligrapher
Chinese/Japanese FineArt - GoGoPen Hanwang - more than 70% PDA market share in mainland China IBM Embedded HWR Motorola Lexicus - DragonPen PenPower - most influencial in Taiwan
Sensor Augmented Pointing