41
DARWIN PHONES: THE EVOLUTION OF SENSING AND INFERENCE ON MOBILE PHONES PRESENTED BY: BRANDON OCHS Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy, Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 5-20.

Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

  • Upload
    faxon

  • View
    42

  • Download
    0

Embed Size (px)

DESCRIPTION

Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs. - PowerPoint PPT Presentation

Citation preview

Page 1: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

DARWIN PHONES: THE EVOLUTION OF SENSING AND INFERENCE ON MOBILE PHONESPRESENTED BY: BRANDON OCHS

Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy, Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 5-20.

Page 2: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

What does Darwin do? A Smartphone platform for urban

sensing Proof of concept model uses microphone Communicates with other local devices

to improve inference accuracy (collaborative inference)

Framework can be expanded to gatherinformation using a range of sensor data

Page 3: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

What about battery life? Communicates with backend server to

do the CPU-intensive machine learning algorithms

Local devices share models rather than re-computing them

Sensing is enabled/disabled as the system sees fit

Page 4: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Common Urban Sensing Challenges Human burden of training classifiers Ability to perform reliably in different

environments (indoor vs outdoor) The ability to scale to a large number of

phones without hurting usability and battery life.

Darwin overcomes all of these through classifier/model evolution, model pooling, and collaborative inference

Page 5: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Types of Learning Supervised: Given a fully-labeled training

set

Semi-Supervised: Given a small training set that is evolved

Unsupervised: No training set is given

Page 6: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Darwin Steps Evolution, Pooling, and Collaborative Inference

These represent Darwin’s novel evolve-pool-collaborate model implemented on mobile phones

Page 7: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Classifier Evolution Automated approach to

updating models over time

Needs to account for variability in sensing conditions and settings

Variability in background noise and phone location require separate models

Page 8: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Model Pooling Reuses models that have already

been built and evolved on other phones

Exchange classification models whenever the model is available from another phone

Classifiers do not need to be retrained, which increases scalability

Can pool models from backend servers

Page 9: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Collaborative Inference Combines results from multiple

phones Run inference algorithms in

parallel on the same classifiers System is more robust to

degradation in sensing quality Increases accuracy

Page 10: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Darwin Design: Computation Reduces the on-the-phone computation

by offloading some of the work to backend servers

Backend server uses a machine learning algorithm to compute a Gaussian Mixture Model (2 hours for 15 seconds of audio)

Feature vectors are computedlocally

Page 11: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Darwin Design: Context

Context (in/out of pocket, in/out of bag) will impact the sensing and inference capability

Classifier evolution makes sure the classifier of an event is robust across different environments

Page 12: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Darwin Design: Co-location Accounts for a group of co-located phones

running the same classification algorithm and sensing the same event but computing different inference results

Phones pool classification models when collocated or from backend servers

Compares against its own model and the co-located model

Drastically reduces classification latency Exploits diversity of different phone sensing

context viewpoints

Page 13: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Speaker Recognition Attempts to identify a speaker by

analyzing the microphone’s audio stream

Suppresses silence, low amplitude audio, and chunks that do not contain human voice

Reduce false positives by pre-processing in 32ms blocks

Page 14: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Speaker Modeling

Feature vector consisting ofMel Frequency Cepstral Coefficients

Each speaker is modeled with 20 Gaussians

An initial speaker model is built by collecting a short training sample

Page 15: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Classifier Evolution: Training Step Short training phase (30 seconds) used

to build a model which is later evolved First 15 seconds used as the training set Last 15 seconds used as baseline for

evolution

Page 16: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Classifier Evolution: Evolution Step Semi-supervised learning strategy If the likelihood of the incoming audio

stream is much lower than any of the baselines then a new model is evolved

Page 17: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs
Page 18: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Collaborative Inference Local inference phase can be broken into

three steps: Local inference operated by each individual

phone Propagation of the result of the local

inference to the neighboring phones Final inference based on the neighboring

mobile phones local inference results Each node individually operates inference

on the sensed event Results and confidence broadcasted

Page 19: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs
Page 20: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Privacy and Trust Raw sensor data is not stored on or

leaves the mobile phone

The content of a conversation or raw audio data is never disclosed

Users can choose to opt out of Darwin

Page 21: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experimental Results Tested using a mixture of five N97 and

iPhones used by eight people over a period of two weeks

Audio recorded in different locations

Classifier trained indoors

Page 22: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 1 Parameters Three people walk along a sidewalk of a

busy road and engage in conversation The speaker recognition application

without the Darwin components runs on each of the phones carried by the people

Page 23: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 1 Results: Without Evolution

Page 24: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 2 Parameters Meeting setting in an office environment

where 8 people are involved in conversation

The phones are located at different distances from people in the meeting, some on the table and some in people’s pockets

Page 25: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 2 Results

Page 26: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 2 Results

Page 27: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 3 Parameters Five phones in a noisy restaurant Three of the five people are engaged in

conversation Two of the five phones are placed on the

table Phone 4 Is the closest phone to speaker

4 and also the closest phone to another group of people having a loud conversation

Page 28: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 3 Results

Page 29: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 3 Results

Page 30: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 3 Results

Page 31: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 3 Results

Page 32: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 4 Parameters Five people walk along a sidewalk and

three of them are talking The greatest improvement is observed

by speaker 1, whose phone is clipped to their belt

Page 33: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 4 Results

Page 34: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Experiment 4 Results

Page 35: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Time and Energy Measurements Baselines for power use determined Measurements performed using the

Nokia Energy Profiler tool No data gathered for the iPhone Smart duty cycling required later to save

battery life

Page 36: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Time and Energy Measurements

Page 37: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Possible Applications Virtual square application

Social application for a group of friends Place discovery application

Use collaborative inference to determine location

Friend Tagging application Exploit face recognition to tag friends on

pictures

Page 38: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Future Work

Duty cycling for improved battery life

Simplified classification techniques

Page 39: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Improvements On The Paper Studies don’t show conclusive evidence;

there should be separate control models for each of the scenarios

Page 40: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

Conclusion The Darwin system combines classifier

evolution, model pooling, and collaborative inference

Results indicate that the performance boost offered by Darwin off sets problems with sensing context

The Darwin system provides a scalable framework that can be used for other urban sensing applications

Page 41: Darwin Phones: The Evolution of Sensing and Inference on Mobile Phones Presented By: Brandon Ochs

References [1] Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy,

Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 5-20. 

[2] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM Models for Speaker Identification systems. In Electrical and Computer Engineering, 2004. Canadian Conference on, volume 1, 2004

[3] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM Models for Speaker Identification systems. In Electrical and Computer Engineering, 2004. Canadian Conference on, volume 1, 2004.