Embed Size (px)
Citation preview
PhonePoint Pen: Using Mobile Phones to Write
in Air
Sandip Agrawal, Ionut Constandache, Shravan Gaonkar,
Romit Roy Choudhury
ACM MobiHeld 2009
Noting small pieces of information,
quickly and effortlesslycan be useful
But aren’t there several solutions ...
Buy Milk
State of the Art
Sticky notes … Difficult to organize electronicallyDifficult to organize electronically Not always handyNot always handy
Typing on phone keyboards
Tiny keyboard sizesTiny keyboard sizes Small inter-key spacingSmall inter-key spacing
Audio recording Cannot sketch diagramsCannot sketch diagrams Time consuming to browse through voice messagesTime consuming to browse through voice messages
So, need a solution that is
Easy to use
Always-with-me
Allows sketching
Online/Searchable QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Proposed Approach: PhonePoint Pen
User holds phone like a penUser holds phone like a pen Writes messages in air
Built-in accelerometer Captures hand movementCaptures hand movement Converts to image / textConverts to image / text
Display text on screen Also sent to user’s email addressAlso sent to user’s email address
Use Cases
Mashing with cameras Superimposing text on picturesSuperimposing text on pictures
Sketching Giving directions to your home Giving directions to your home Exchanging figures on the flyExchanging figures on the fly
One handed use E.g., carrying a suitcaseE.g., carrying a suitcase
Duke
University
Use Cases
Applications in healthcare Hand movement disabilitiesHand movement disabilities Speech problems after surgerySpeech problems after surgery
Military and emergency response
Quick and silent updatesQuick and silent updates
Prototype
Current prototype on Nokia N95 Equipped with 3-axes accelerometerProgrammed usign Python + MATLAB
several design challenges emerge …
Design Challenges (1)
Hands rotate while writing Accelerometers only measure linear Accelerometers only measure linear acceleration acceleration Rotation injects ambiguity Rotation injects ambiguity (Wii uses gyroscope)(Wii uses gyroscope)
Proposed Approach: • Hold phone in non-rotating gripHold phone in non-rotating grip• Determine angular orientation during the Determine angular orientation during the
pausepause
Design Challenges (2)
Background Vibration (Noise) Sensitive accelerometersSensitive accelerometers Jitter from hand vibrationsJitter from hand vibrations
Proposed Approach: • Smooth the accelerometer readings with Smooth the accelerometer readings with
moving averagemoving average• Suppress acceleration values < thresholdSuppress acceleration values < threshold
Design Challenges (3)
Computing Displacement Ambiguity when acceleration is zeroAmbiguity when acceleration is zero Can be constant velocity or stationary Can be constant velocity or stationary Worse with noiseWorse with noise
Proposed Approach: • Detect change of direction Detect change of direction • Reset velocity between strokesReset velocity between strokes Reset
Design Challenges (4)
Lifting Pen from the Paper e.g., dotting the “i” and crossing the “t”e.g., dotting the “i” and crossing the “t”
Proposed Approach: • Impulse on the Z axis during the liftImpulse on the Z axis during the lift• Off the Record – On the RecordOff the Record – On the RecordP
Z-axisspikes
Design Challenges (5)
Character transition e.g., B vs 13e.g., B vs 13
Proposed Approach: • Use delimited such as “.”Use delimited such as “.”• 1.3 for 13 1.3 for 13
Drawing a Rectangle
RawRaw Accelerometer ReadingAccelerometer Reading
Noise Smoothing using Noise Smoothing using Moving AverageMoving Average
Background Noise Background Noise SuppressionSuppression
Velocity Plot after Avoiding Velocity Plot after Avoiding Velocity DriftsVelocity Drifts
Final RectangleFinal Rectangle
Raw Accelerometer Data for M and iRaw Accelerometer Data for M and i
Character Recognition
More PhonePen words …
More PhonePen words …
20 words tested with online OCR web service.78% accuracy
20 words tested with online OCR web service.78% accuracy
Results
Video Demo
QuickTime™ and a decompressor
are needed to see this picture.
Of Course, Not a Product Yet
Lowercase character recognition Cursive handwriting more complicated
Need smaller hand movements in air
Youtube comment: looking crazy waving hands Slashdot: Nearby people can get injured
Need increased robustness 20% failure when writing in air 5% failure when writing on a surface
Of Course, Not a Product Yet
What about longer messages Sentences? Email replies?
Erasing alphabets Shake your hands like using a eraser
Ongoing work is pursuing
these directions …
Ongoing work is pursuing
these directions …
Swipe your credit card and sign in air …
The vision is:
Thanks
Visit Systems Networking Research Group (SyNRG)
@ Duke University
Google “synrg duke”
Noting small pieces of information,
quickly and effortlesslycan be useful
So, what are the options ...
CallJohn @ 6658
Buy Milk
Right on3rd
traffic light
Velocity Plot after Velocity Plot after Avoiding Velocity DriftsAvoiding Velocity Drifts
Final RectangleFinal Rectangle
