Mobile Phone Sensing
CMPE 58C – Wireless Sensor Networks
• Why Mobile Phone Sensing? – Enabling Factors
• Sensors and Potentials
• Applications
• Mobile Phone Sensing Architecture – Sense
– Learn
– Share, Inform
• Open Topics/Challenges
• Inter-disciplinary collaboration potentials
Contents
1. Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, Andrew T. Campbell. "A Survey of Mobile Phone Sensing", In IEEE Communications Magazine, September, 2010.
2. Andrew T. Campbell, Shane B. Eisenman, Nicholas D. Lane, Emiliano Miluzzo, Ronald Peterson, Hong Lu, Xiao Zheng, Mirco Musolesi, Kristof Fodor, Gahng-Seop Ahn,"The Rise of People-Centric Sensing", In IEEE Internet Computing Special Issue on Sensor Networks, 2008.
3. Deborah Estrin, “Participatory sensing: applications and architecture”, in Proceedings of MobiSys 2010.
References
what is the future of sensing?
Evolution of Sensing Prototype
accelerometer
digital compass
microphone
WiFi/bluetooth GPS
….
light sensor/camera
sensing
gyroscope
density
programmability
- free SDK
- multitasking
application distribution
collect huge amount of data for research
purposes
cloud infrastructure
cloud - backend support
store and crunch big data (fusion)
run machine learning algorithms (learning)
- sensing
- run machine learning algorithms locally (feature extraction + inference)
societal scale sensing
global mobile sensor network
• Acceloremeter: Orientation/User Experience – Characterizing the Physical Movements of the User:
Running, Walking, Standing (CenceMe Application)
• Microphone – Classifying distinct sounds: Being in a conversation,
using ATM, listening to music, driving (SoundSense Application), identifying a speaker
• Camera – Photo Blogging, Traffic monitoring, Pollution
monitoring
Sensors and Potentials
E. Miluzzo et al., “Sensing meets Mobile Social Networks: The Design, Implementation,
and Evaluation of the CenceMe Application,” Proc. 6th ACM SenSys, 2008, pp. 337–50.
H. Lu et al., “Sound-Sense: Scalable Sound Sensing for People-Centric Applications
on Mobile Phones,” Proc. 7th ACM MobiSys, 2009, pp. 165–78.
• GPS: – Tracking users
– +acceloremeter: mode of transportation (walking, biking, using a vehicle)
• Bluetooth: Identifying Social Interactions (Reality Mining Project at MIT)
• GSM, WiFi: Using traces for localization, even tracking of mode of transportation
Sensors and Potentials
Reddyet al., “Using mobile phones to determine transportation modes” , ACM
Transactions on Sensor Networks, 2010.
Sohn et al., “Mobility Detection Using Everyday GSM Traces” , Ubicomp, 2006.
Reality Mining Project: Machine Perception and Learning of Complex Social Systems,
http://reality.media.mit.edu/
• Transportation – Large-scale network to provide fine-grained
traffic information (e.g., accurate travel time estimation)
• UC Berkeley/Nokia/NAVTEQ, “Mobile Millennium”; http://traffic.berkeley.edu/
• A. Thiagarajan et al., “VTrack: Accurate, Energy-Aware Traffic Delay Estimation Using Mobile Phones,” Proc. 7th ACM SenSys, 2009.
• Social Networking – Sharing activity and presence information through Social
Networks • CenceMe: Available on AppStore and compatible with Facebook
Application Domains
• Environmental Monitoring – Environmental Impact Report, pollution:
• UCLA PEIR Project: M. Mun et al., “Peir, the Personal Environmental Impact Report, as a Platform for Participatory Sensing Systems Research,” Proc. 7th ACM MobiSys, 2009.
– Sound, Air Pollution, etc. (Can be used for city planning) • UCLA Urban Sensing Project (Weather, air quality) • EarPhone: An End-to-End Participatory Urban Noise
Mapping System • Health and Well-Being
– Physical Activity Monitoring • UbiFit Garden: S. Consolvo et al., “Activity Sensing in the Wild: A
Field Trial of Ubifit Garden,” Proc. 26th Annual ACM SIGCHI Conf. Human Factors Comp. Sys., 2008
• Mobile Habits, Towards a Physical Diary • BikeNet: Sharing biking experience
– Spread of a Disease in Urban Areas, Chronic Disease Management – Nutrition, Medication, Interaction Monitoring
Application Domains
Sensing Scale
"A Survey of Mobile Phone Sensing", In IEEE Communications Magazine, September, 2010.
• Challenge – Impact of scaling sensing applications from
personal to population scale is unknown • Information sharing (trust)
• Privacy
• Data Mining on large scale data
• Low commitment of users
Sensing Scale
• Opportunistic Sensing: Automated Data Collection, no user involvement – Lower burden on the user
– Good for community sensing
– Phone Context Problem
• Participatory Sensing: An individual actively participates in the data collection – Higher burden on the user
– Depends on the enthusiasm of the user
Sensing Paradigms
•Develop models to best understand the usability and performance issues
of these schemes
•Hybrid sensing schemes?
An Example Architecture
•Sense •Interpret/Learn •Inform/Share/Persuasion
"A Survey of Mobile Phone Sensing", In IEEE Communications
Magazine, September, 2010.
• Programmability: – Open, Offer SDK’s, existing libraries (machine
learning – Weka) can be cross-compiled. – Vendors did not think that 3rd parties would use
continuous sensing to develop applications – Unpredictable, variable sampling: Acceloremeter
sampling on Nokia Symbian Phones range 25-38 Hz (depending on CPU load)
– GPS using a varying amount of power – Vendors: Will/Should improve as sensing
applications become popular – Research Topics: Propose sensing abstractions
and API’s that can be standardized and adopted by vendors
Sense: The Mobile Phone as a Sensor
• Continuous Sensing: – Multi-tasking not supported by most mobile
phones (except Android and Nokia Maemo)
– Still a resource-limited device for complex signal processing and inference
– Still battery limited for real-time applications
– Open topics:
Sense: The Mobile Phone as a Sensor
study the trade-off of continuous sensing with the goal of
minimizing the energy cost while offering sufficient accuracy
and real-time responsiveness
• Phone Context – Different use of mobile phones (on the go,
etc) by different people, background environment or actions (phone in the pocket), etc.
– Affects the success of machine learning algorithms to interpret data (dynamic & unexpected conditions)
– Open Topics:
Sense: The Mobile Phone as a Sensor
Sharing learning models between phones
Model Evolution: Semi-supervised learning, active learning
Collaborative Sensing
• Training Models – Supervised Learning requires labeling of
training data: Feasible for small scale sensing but cannot handle wide range of behaviors and context
– Open Topics: • Use of unsupervised, semi-supervised learning?
Learn: Interpreting Sensor Data
"A Survey of Mobile Phone Sensing", In IEEE Communications Magazine, September, 2010.
• Multi-modal Sensing – Better recognition of user activities with multiple
sensor streams (GPS+microphone = more than location, instead of at home -> in a conversation at home)
– Open topics: Using sensor fusion techniques for multi-modal sensing
• Scaling Models – Different people with different usage of phones? – Open Topics:
• Incorporate people in the process using active learning and community guided learning?
• Adapting models on the fly? • Common machine learning toolkit that allows researchers to
build and share models
Learn: Interpreting Sensor Data
• Sharing – Visualization using a web portal
– Community awareness: Social networks
– Personalized Sensing: Human Phone Interfacing
• Persuasion: Persuade users to make positive behavioral changes – Largely unexplored: Which metaphors and
feedback are most effective? (requires inter-disciplinary research)
• Privacy
Inform/Share/Persuasion
• Machine Learning: Application/Development of Inference/Classification Models
• Security/Privacy/Trust: Most fundamental requirement of a sensing system
• Wireless Networking: Opportunistic Networking, Vertical Handover Decisions to collect data, Localization/Tracking with Multi-modal sensing
• Data Mining: Large-scale information processing?
• Social Networking: How to share/inform data?
• HCI and Persuasive Computing: How to design feedback systems, user studies?
Potential Inter-Disciplinary Research
• Mobile Phone Sensing is an emerging field • Challenges/Open Issues:
– Privacy and Resource-Sensitive Reasoning with Noisy Data and Noisy Fields
– Adapt machine learning algorithms into the field – How to collect ground-truth data to assess the
accuracy of algorithms that interpret sensor data?
– How to validate experiments? – How to deal with the potentially massive amount
of data made available? – How to run such large-scale studies?
To Wrap Up
Mobile Phone Sensing @ CMPE
• Activity Recognition on Mobile Phones – How to recognize the activities of a user using the
accelerometers available on the phones?
• Efficient Feature Extraction on Mobile Phones – Which features are more efficient in order to
recognize the activities of a user?
• Movement Based Group Sensing – Can we identify groups of people by analyzing their
acceleration signals?
27
• Offline; Daily routine of a person
• Online; “what the user is currently doing”
Recruitment for
Participatory
Sensing
Applications
Offline vs. Online Sensing
Collect sensor data from the users that are “currently walking” in downtown area
Fittness coaching, rehabilitation
Why Online Training?
Drawback of Training Phase
• Costly training phase vs. Limited memory on phones
Hot topic
• Activity Recognition systems with no training data or requires limited training by end user
Target • Reduce the training burden on the user
• What is the performance of classifiers when executed online with online training in smartphones? – Will smartphones replace the PCs?
• Limited Training set which is collected in few minutes
• Application tested with different Android Phones instead of focusing on one model
Motivation
Activity Logger & Classifier Application
• Activity Logger;
– Collecting raw data from accelerometer
– Logging collected data
Activity Classifier; ◦ Classification with selected classifier ◦ Extracts features from training sets ◦ System parameters;
◦ Window size ◦ Sampling rate ◦ Order interval
• Naive Bayes
• Min Distance
• Clustered KNN;
– Decreased computation burden
– Preprocessing
– K sets for max, min, average, and std deviation per activity
– Classification process
Classifiers
Read all raw data from training log file
Calculate average of magnitudes for each
data value
Select K nearest points to the average magnitude by comparing their euclidean
distances
Store set of K points in memory to be used at
classification step
Collect raw sensor data during a
window duration
Calculate average of magnitudes for the data points in the
window
Compare average mag. with each point in
previously generated 4*K sets in terms of euclidean
distances
Select K nearest neighbor
Check activity label of each data in the
final K set
Label the window for the average feature with the
activity which is voted the most
• 5 Test subjects
– Average age 27, 1 female & 4 male
• Phone located in front pocket of pants
• Phone models
• System Parameters
– Window size
– Sampling Rate
– K Value
Test Design – System Parameters
Test Design – System Parameter
•Test cycle (60 sec) •Labelled Data
•Voice commands •Performance metrics
•Precision •Recall •Accuracy •F-measure
Running
Walking
Sitting
Standing
Results
Clustered KNN Naïve Bayes
Accuracy 92.13% 47.61%
Precision 92.45% 51.15%
Recall 92.09% 42.30%
F-measure 92.27% 46.19%
Classification Running Walking Standing Sitting
Tru
th Running 1020 46 14 7
Walking 178 729 146 7 Standing 121 12 1063 18 Sitting 18 2 30 1084
Comparison of Clustered KNN and Naive Bayes
Overall Confusion Matrix
Results
Window size (sec)
0.5 1 2
Sampling Interval (msec)
10 50 100 10 50 100 10 50 100
K
10 87.9 87.8 87.7 88.4 90.3 89.5 88.6 87.8 89.3
50 91.1 90.0 91.4 91.9 92.1 90.8 88.9 89.4 91.0
Average Accuracy Rates (%)
• Clustered KNN Results Based on System Parameters
Results
Activity Classifier CPU Usage Memory Usage
KNN Clustered 29% 21.9 MB Naïve Bayes 42% 12.6 MB Min Distance 27% 19.6 MB TTS Service 13% 12.4 MB
Benchmark Applications CPU Usage Memory Usage
System 10% 28.8 MB Norton Mobile 4% 19.4 MB
Internet 2% 37.0 MB Google Maps 1% 31.2 MB
CPU & Memory Usage for Activity Classifier & Other Applications on Samsung Galaxy Gio
Paper Discussion
• K. K. Rachuri, C. Mascolo, M. Musolesi, and P. J. Rentfrow, SociableSense: Exploring the Trade-offs of Adaptive Sampling and Computation Offloading for Social Sensing, ACM MobiCom '11, Las Vegas, USA, 2011.
• Emiliano Miluzzo, Nicholas D. Lane, Kristóf Fodor, Ronald A. Peterson,Hong Lu, Mirco Musolesi, Shane. B. Eisenman, Xiao Zheng, Andrew T. Campbell, "Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application", In Proc. of 6th ACM Conference on Embedded Networked Sensor Systems (SenSys '08), Rale igh, NC, USA, Nov. 5-7, 2008.
38