CHI 2011 – May 9, 2011

Your Noise is My Command

Sensing Gestures

Using the Body as

an Antenna

Gabe Cohn1,2

Dan Morris1

Shwetak N. Patel1,2

Desney S. Tan1

1Microsoft Research 2University of Washington

ubicomp lab

2

Beyond the Keyboard and Mouse

3

Speech Recognition

4

Capacitive Touch Screen

5

Skinput Harrison et al., CHI 2010

6

Muscle-Computer Interfaces Saponas et al., CHI 2010

Saponas et al., UIST 2009

Saponas et al., CHI 2008

7

Computer Vision and Depth Cameras

8

9

10

11

Sensing Gestures

Using the Body as

an Antenna

no instrumentation to environment

touch gestures anywhere in the home

minimal instrumentation to user

Typical

“bunny ears”

TV antenna

12

Typical

“teenager”

human antenna

dielectric with

complex geometry

40 Hz – 400 MHz

13

“body antenna effect”

- body area networks (BAN)

- analyzing electrical activity on body

human body antenna

< 200 kHz

electromagnetic (EM) noise

from powerlines and appliances

14

15

Powerline Noise

Energy Monitoring and Activity

Recognition

Gupta et al., Ubicomp 2010

Patel at al., Ubicomp 2007

16

Hypothesis

Powerline noise differs at different

- locations in the home

- positions on the wall

Using the human body antenna to

receive this noise, we can determine

- location in the home

- position on the wall

of a user’s touch gestures

17

Apparatus measure voltage

sample with NI-DAQ

400 kS/s

16-bit resolution

store on laptop hard drive

Core Experiment

10 Homes

• single-family and townhouses

• 1 – 3 floors

• Area: 120 – 290 sq m (µ = 215)

1300 – 3100 sq ft (µ = 2310)

• Built: 1948 – 2006 (µ = 1981)

18

10 Participants

• 5 male / 5 female

• Age: 28 – 61 (µ = 38)

• Weight: 52 – 82 kg (µ = 64)

115 – 180 lbs (µ = 141)

• Height: 150 – 188 cm (µ = 169)

59 – 74 in (µ = 67)

19

Locations

6 locations in each home

5 light switches

2 in same room

1 blank wall

above an outlet

Procedure

6 gestures per location

hold each for 6 seconds

guided by computer

commands

6 locations

6 gestures per location

4 “rounds” (repetitions)

144 total gestures per

participant 20

21

Analysis 5 sec

Segmentation

22

Analysis 5 sec

82 ms Segmentation Windowing

23

Analysis Feature Extraction

Frequency (kHz)

Frequency (kHz)

Time (s)

Am

pli

tud

e (

V)

Po

wer

(dB

) P

ow

er

(dB

)

DC

RMS

Time Domain FFT 0 – 2 kHz

2 Features 332 Features

FFT 0 – 25 kHz FFT 0 – 200 kHz High Freq. Peaks

250 Features 400 Features 18 Features

1002 Total Features per 82 ms window

24

Analysis Machine Learning

1002 features per 82 ms time

window

classification using the Weka

SVM (SMO implementation)

4-fold cross validation

fold by “round”

training/testing sets in different

“rounds”

avoids over-fitting

25

Results Wall Touch

2-class classification

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

Chance Accuracy

98.5%

26

Results Touch Position on Wall

0% 20% 40% 60% 80% 100%

Chance Accuracy

87.4%

74.3%

All Walls

Blank Walls Only

5-position classification

27

Results Location in Home

0% 20% 40% 60% 80% 100%

Chance Accuracy

Touch Walls

Stand Near Walls

99.1%

99.5%

6-location classification

2 walls in same room

28

Results Location in Home and Position on Wall

30-class problem

0% 20% 40% 60% 80% 100%

Chance Accuracy

79.8%

un-optimized classification on each 82 ms window

29

Summary Core Experiment:

• Location in home - near 100%

• Position on around switch – 87%

Additional Exploration:

• Differentiate right/left hand

• Differentiate appliance touched

• Estimate proximity to wall

• Estimate continuous position on wall

30

Limitations

Improved feature set - relative features

- temporal features

Signal injection - on-body

- into power line

Real-time system

Smaller system (no laptop)

Contact point to body - contactless coupling to body

Explore gesture set

Explore application space

Generalizability of noise signals - Training procedure

- Locations with poor signals

- Strong signals overwhelm others

Changing state of home - Appliances switched on/off

Future Work

31

Your Noise is My Command

Sensing Gestures

Using the Body as an

Antenna

Gabe Cohn

Dan Morris

Shwetak N. Patel

Desney S. Tan

ubicomp lab

Thank You!

www.gabeacohn.com

gabecohn@uw.edu

32

Backup Slides

Applications

Long-term: • Home automation • Gaming

Short-term: • Impromptu interaction session • Meetings/presentations

33