1

SurroundSense: Mobile Phone Localization

via Ambience Fingerprinting

Ionut Constandache

Co-authors:

Martin Azizyan and Romit Roy Choudhury

2

Context

Pervasive wireless connectivity

+

Localization technology

=

Location-based applications

3

Location-Based Applications (LBAs)

For Example:

GeoLife shows grocery list when near Walmart

MicroBlog queries users at a museum

Location-based ad: Phone gets coupon at Starbucks

iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4

Location-Based Applications (LBAs)

For Example:

GeoLife shows grocery list when near Walmart

MicroBlog queries users at a museum

Location-based ad: Phone gets coupon at Starbucks

iPhone AppStore: 3000 LBAs, Android: 500 LBAs

Location expresses context of user

Facilitates content delivery

5

Location is an IP addressAs if for content delivery

6

Thinking about Localization

from an application perspective…

7

Emerging location based apps need

place of user, not physical location

Starbucks, RadioShack,

Museum, Library

Latitude, Longitude

8

Emerging location based apps need

place of user, not physical location

Starbucks, RadioShack,

Museum, Library

Latitude, Longitude

We call this Logical Localization …

9

Can we convert from

Physical to Logical Localization?

10

Can we convert from

Physical to Logical Localization?

State of the Art in Physical Localization:

1. GPS Accuracy: 10m

2. GSM Accuracy: 100m

3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

11

Widely-deployable localization technologies

have errors in the range of several meters

Can we convert from

Physical to Logical Localization?

State of the Art in Physical Localization:

1. GPS Accuracy: 10m

2. GSM Accuracy: 100m

3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-100m

12

Several meters of error is inadequate

to logically localize a phone

Physical Location

Error

13

Several meters of error is inadequate

to logically localize a phone

RadioShackStarbucks

Physical Location

Error

The dividing-wall problem

14

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

15

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

16

It is possible to localize phones by

sensing the ambience

Hypothesis

such as sound, light, color, movement, WiFi …

17

Sensing over multiple dimensions extracts more

information from the ambience

Each dimension may not be unique,

but put together, they may provide a

unique fingerprint

18

SurroundSense

Multi-dimensional fingerprint

Based on ambient sound/light/color/movement/WiFi

Starbucks

Wall

RadioShack

19

B AC DE

Should Ambiences be Unique Worldwide?

FG

HJ

I

LMN

O

P

Q

Q

R

K

20

Should Ambiences be Unique Worldwide?

B AC DE

FG

HJ

I

K

LMN

O

P

Q

Q

R

GSM provides macro location (strip mall)

SurroundSense refines to Starbucks

21

+

Ambience Fingerprinting

Test Fingerprint

Sound

Acc.

Color/Light

WiFi

LogicalLocation

Matching

Fingerprint

Database

=

Candidate Fingerprints

GSM Macro Location

SurroundSense Architecture

22

Fingerprints

Sound:

(via phone

microphone)

Color:

(via phone

camera)

Amplitude Values

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

No

rma

lize

d C

ou

nt

0.14

0.12

0.1

0.08

0.06

0.04

0.02

0

Acoustic fingerprint

(amplitude distribution)

Color and light fingerprints on HSL space

Lig

htn

es

s1

0.5

0

Hue

0

0.5

1 00.2

0.40.6

0.81

Saturation

23

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Moving

24

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Moving

Queuing

25

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Queuing Seated

Moving

26

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Pause for product

browsing

Moving

27

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Pause for product

browsing

Short walks between

product browsing

Moving

28

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Walk more

Moving

29

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Walk more Quicker stops

Moving

30

Fingerprints

Movement: (via phone accelerometer)

WiFi: (via phone wireless card)

Cafeteria Clothes Store Grocery Store

Static

ƒ(overheard WiFi APs)

Moving

31

Discussion

Time varying ambience

Collect ambience fingerprints over different time windows

What if phones are in pockets?

Use sound/WiFi/movement

Opportunistically take pictures

Fingerprint Database

War-sensing

32

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

33

Evaluation Methodology

51 business locations

46 in Durham, NC

5 in India

Data collected by 4 people

12 tests per location

Mimicked customer behavior

34

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

Ac

cu

rac

y (

%)

Cluster

Localization accuracy per cluster

35

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

Ac

cu

rac

y (

%)

Cluster

Localization accuracy per cluster

Multidimensional

sensing

36

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

Fault tolerance

Ac

cu

rac

y (

%)

Cluster

Localization accuracy per cluster

37

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

Ac

cu

rac

y (

%)

Cluster

Localization accuracy per clusterSparse WiFi APs

38

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

No WiFi APsAc

cu

rac

y (

%)

Cluster

Localization accuracy per cluster

39

Evaluation: Per-Scheme Accuracy

Mode WiFi Snd-Acc-WiFi Snd-Acc-Lt-Clr SS

Accuracy 70% 74% 76% 87%

40

Evaluation: User Experience

Random Person Accuracy

Average Accuracy (%)

0 10 20 30 40 50 60 70 80 90 100

1

0.9

0.8

0.7

0.6

0.5

CD

F

0.4

0.3

0.2

0.1

0

WiFI

Snd-Acc-WiFi

Snd-Acc-Clr-Lt

SurroundSense

41

Economics forces nearby businesses to be different

Not profitable to have 3 coffee shops

with same lighting, music, color, layout, etc.

SurroundSense exploits this ambience diversity

Why does it work?

The Intuition:

42

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

43

Limitations and Future Work

Energy-Efficiency

Localization in Real Time

Non-business locations

44

Limitations and Future Work

Energy-Efficiency

Continuous sensing likely to have a large energy draw

Localization in Real Time

Non-business locations

45

Limitations and Future Work

Energy-Efficiency

Continuous sensing likely to have a large energy draw

Localization in Real Time

User’s movement requires time to converge

Non-business locations

46

Limitations and Future Work

Energy-Efficiency

Continuous sensing likely to have a large energy draw

Localization in Real Time

User’s movement requires time to converge

Non-business locations

Ambiences may be less diverse

47

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

48

SurroundSense

Today’s technologies cannot provide logical localization

Ambience contains information for logical localization

Mobile Phones can harness the ambience through sensors

Evaluation results:

51 business locations,

87% accuracy

SurroundSense can scale to any part of the world

49

Questions?

Thank You!

Visit the SyNRG research group @

http://synrg.ee.duke.edu/