PhonePoint Pen: Using Mobile Phones to Write in Air Romit Roy Choudhury Assistant Professor

PhonePoint Pen: Using Mobile Phones to Write

in Air

Romit Roy ChoudhuryAssistant Professor

Noting small pieces of information,

quickly and effortlesslycan be useful

So, what are the options ...

Buy Milk

State of the Art

Sticky notes … organization is a nightmareorganization is a nightmare not handynot handy

Typing on keyboard

tiny keyboard sizestiny keyboard sizes small inter-key spacingsmall inter-key spacing

Audio recording cannot sketch informationcannot sketch information time consuming to browse through voice messagestime consuming to browse through voice messages

So, need a solution that is

Easy to use

Always-with-me

Allows sketching

Searchable

Proposed Approach: PhonePoint Pen

User can write messages in air holding the phone like a penholding the phone like a pen

Use built-in accelerometer

capture hand movementcapture hand movement

Decode text and image sent to user’s email address for future use sent to user’s email address for future use

Use Cases

Mashing with cameras superimposing text on picturessuperimposing text on pictures

Sketching exchanging figures between caller/calleeexchanging figures between caller/callee

One handed use e.g., carrying a suitcase e.g., carrying a suitcase

Duke

University

Prototype

Current prototype implemented on Nokia N95 mobile phones

several design challenges emerge …

Design Challenges (1)

Lack of a Gyroscope Accelerometers only measure linear Accelerometers only measure linear acceleration acceleration Linear Movements – Rotation Ambiguity Linear Movements – Rotation Ambiguity

Proposed Approach: • Hold Phone in Non-rotating GripHold Phone in Non-rotating Grip• Determine Angular Orientation during Determine Angular Orientation during the the

pause pause

P

Design Challenges (2)

Background Vibration (Noise) Sensitive AccelerometersSensitive Accelerometers Significant Jitter by hand vibrationsSignificant Jitter by hand vibrations

Proposed Approach: • Smooth the accelerometer readings with Smooth the accelerometer readings with

moving average over last n (=7) readingsmoving average over last n (=7) readings• Suppress acceleration values < 0.5 Suppress acceleration values < 0.5

P

2/ sm

Design Challenges (3)

Computing Displacement Erroneous Acceleration ReadingErroneous Acceleration Reading Ambiguity when acceleration is zeroAmbiguity when acceleration is zero

Proposed Approach: • Detect Pause Using Moving WindowDetect Pause Using Moving Window• Reset Velocity to Zero in Between StrokesReset Velocity to Zero in Between Strokes

P

Design Challenges (4)

Lifting Pen from the Paper e.g., dotting the “i” and crossing the “t”e.g., dotting the “i” and crossing the “t”

Proposed Approach: • Impulse on the Z axis during the liftImpulse on the Z axis during the lift• Off the Record – On the RecordOff the Record – On the Record

P

Drawing a Rectangle

RawRaw Accelerometer ReadingAccelerometer Reading

Noise Smoothing using Noise Smoothing using Moving AverageMoving Average

Background Noise Background Noise SuppressionSuppression

Velocity Plot after Velocity Plot after Avoiding Velocity DriftsAvoiding Velocity Drifts

Final RectangleFinal Rectangle

Raw Accelerometer Data for M and iRaw Accelerometer Data for M and i

More PhonePen words …

Ongoing Work

Lowercase character recognition

Smaller hand movements in air

Increased robustness and reliability

Swipe your credit card and sign in air …

Our vision:

Thank You!

email: romit@ee.duke.edu

Web: http://synrg.ee.duke.edu/