SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

Written by Martin Azizyan, Ionut Constandache, & Romit Choudhury

Presented by Craig McIlwee

Motivation

• Provide logical localization• Using GPS only isn’t good enough– Doesn’t work well indoors– Doesn’t account for dividing walls

• Dedicated hardware is not scalable

Approach

• Create an ambience fingerprint using sound, light, color, and user movement– Noise signatures specific to type of location/store– Chain stores have color themes– User movement indicative of store type

Architecture/Algorithm

Architecture/Algorithm

• Data is recorded on the phone, preprocessed, and sent to a server

• Filter module– Subsets the candidates– Wifi, movement, sound

• Match module– Selects the best candidate– Color/sound, Wifi

Architecture/Algorithm

• No single module needs to be perfect– If each module is ‘good enough’ then all modules

combined are sufficient– Being simple reasonably accurate instead of

sophisticated and perfect reduces resources required for processing

Sound Module

• Filter– Sound varies over time

• Fingerprints captured from various times of day• Similarity of fingerprints is used to create a

threshold for a potential match• Match if within the threshold, discard otherwise– Threshold is generous– More false positives is better than false negatives

Motion Module

• Filter– Variations in user behavior

• Record 4 samples/second, use moving average over last 10 samples

• Minor variations suppressed

Motion Module

• User movement is classified as stationary or mobile

• 3 profiles defined– Long stationary – restaurant– Frequent movement with longer stationary –

browsing– Frequent movement with shorter stationary –

shopping• Some logical locations fit multiple profiles

Motion Module

Color/Light Module

• Match• Images captured from camera while facing

downward– Floor themes are consistent– Other orientations introduce noise– Common orientation when checking email, text

messages, etc

Color/Light Module

• Analyze patterns in the image• First attempt was to convert pixels to RGB

values– Failed due to shadow and reflection influences

• Second attempt was to convert to HSL values– Isolates light on its own axis

Color/Light Module

Color/Light Module

• Same/similar colors result in clusters when graphed

• Dominant colors generate larger clusters• Similarity calculated as distance between

cluster centroids and size of the clusters• Most similar candidate is the match

Wifi Module

• Normally a filter, match if camera is not available

• Capture MAC address of available access points every 5 seconds

• Compare occurrence ratio of currently available access points to known access points

Known Issues

• Sound varies over time– Split day into 2 hour windows, capture fingerprints during

each window– No mention of day of week, time of year

• Camera in pocket– All testing done with phone in hand– Expected rise in wearable devices

• Mimicking user behavior– Initial data showed artificial behavior– Subsequent attempts shadowed real customers

Known Issues

• Resource (energy) intensive• Accelerometer fingerprint takes time to

capture• Non-business locations may not exhibit

enough diversity– Offices, airports, libraries

Evaluation

• Recorded fingerprints of 51 locations– “War-sensed” by students– 2 different groups during different times of day

• Group A’s fingerprints used as database while Group B was at the location collecting their own fingerprints

• Accuracy analysis was done on various combinations of sensors types

• All sensor types combined yielded 87% accuracy