deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,

deWristified: Handwriting Inference Using Wrist-Based

Motion Sensors Revisited

Raveen [email protected]

Anindya [email protected]

Murtuza [email protected]

University of Texas at San Antonio

Wrist Wearables

• Extends the functionality of traditional wristwatches beyond timekeeping.

• Captures rich contextual information about the wearer.• Enables several novel context-based applications.

2019-05-16 SPriTELab @ UTSA 2

Motion Sensors

• Two main types of motion or inertial sensors:• Accelerometer: records device acceleration.• Gyroscope: records device angular rotation.

• Accessing motion sensors on wearable devices:

• All applications have access to motion sensors by default (also referred to as zero-permission sensors) on most wearable OSs.

• Applications’ access to motion sensors cannot be regulated on most wearable OSs –we can’t turn them off!

• Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand?


Inferring Private User Inputs (Using Wrist Wearables)

2019-05-16 4SPriTELab @ UTSA

State-of-the-Art in Handwriting Recognition (Using Wrist Wearables)


Airwriting (Amma et al.) Whiteboard writing (Arduser et al.)

Finger writing (Xu et al.) Pen(cil) writing (Xia et al.)

Adversary Model

• Adversary has knowledge of the type of handwriting.

• Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors.• Could employ a Trojan app for this!

• Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors.


Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables)

• Airwriting (Amma et al.)• Custom-designed hand glove with very

high precision sensors.• Our adversary relies on target user’s

smartwatch or fitness band.

• Only uppercase words.

• Whiteboard writing (Arduser et al.)• Not generalized (training and testing

data not from different participants).

• Only uppercase alphabets.

• No handwriting activity detection.

• Finger writing (Xu et al.)• Use of Shimmer, a specialized sensing

device intended for lab studies.

• Not generalized (training and testing data not from different participants).

• Pen(cil) writing (Xia et al.)• Only lowercase alphabets.

• Controlled data collection.

• No handwriting activity detection.


Our Research

• How practical is handwriting inference when• Using consumer-grade wrist

wearables,

• Using generalized training and testing,

• Writing in a uncontrolled and unconstrained manner, and

• Both upper and lowercase alphabets are modeled ?


New Uncontrolled and Unconstrained

Writing Data

Existing Models

Handwriting Inference Framework


Experimental Setup

• 28 participants for the four writing scenarios.• 18 to 30 years of age• 13 male, 15 female

• Two different wrist-wearables.• Sony Smartwatch 3, LG Watch Urbane

• Accelerometer and gyroscope recorded at 200Hz.

• Participants provided with appropriate writing apparatus.


Writing Tasks (In-Lab)

• Alphabets. • Individual alphabets one at a time.

• Covered all 26 English alphabets in random order.

• Each alphabet was written 10 times.

• Both upper and lower cases.

• Words.• 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012).

• Each participant wrote 20 words, in both upper and lower cases.

• Sentence.• "the five boxing wizards jump quickly" in both upper and lower cases.


Writing Activity Recognition (Out of Lab)

• 2 participants.

• Wore a smartwatch for an entire day.

• Performed the four writing scenarios at random times.

• Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text.


Replicated Inference Frameworks

• Airwriting • Hidden Markov Model (HMM)

• Whiteboard writing • Dynamic Time Warping (DTW)

• Finger writing • Naive Bayes, Logistic Regression and Decision Tree classifiers

• Pen(cil) writing• Random Forest classifier


Personalized Inference Accuracy


Writing Activity Detection:56% recall and 57% precision for air and finger writing39% recall and 47% precision for pencil writing23% recall and 34% precision for whiteboard writing

Personalized Inference Accuracy(Whiteboard Writing)


Lowercase Uppercase

Generalized Inference Accuracy


Writing Activity Detection:35-40% recall for airwriting, whiteboard writing and pencil writingOnly 8% recall for finger writing

Factors Affecting Inference Accuracy

• Number of Strokes.




Number of strokes for the same letter for different participants (lowercase).

Number of strokes for the same letter for different participants (uppercase).



Lowercase Uppercase

Variance in number of strokes per alphabet per participant, averaged for all participants



• Order of Strokes.

• Direction of Strokes.






• Order of Strokes.

• Direction of Strokes.

• Uppercase vs Lowercase.

• Specialized Devices.


Airwriting (Amma et al.)

Conclusion

• We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations.

• Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands.• Primarily due to highly varying nature of handwriting.

• Replicable artifacts: https://sprite.utsa.edu/art/dewristified


Documents

deWristified: Handwriting Inference Using Wrist …...2019-05-16 SPriTELab @ UTSA 7 Our Research •How practical is handwriting inference when •Using consumer-grade wrist wearables,