Context Aware Mobile Devices

Applied Research & Technology CenterMotorola, Inc.

Keshu Zhang

December 3rd, 2008

Context-Aware Applications

• Capture context information from multiple sources and learn associations of context cues with personal preferences and behaviors in order to adapt the configuration of devices, the behavior of interfaces, or to offer personalized access to services.

• With the increasing computing power of mobile devices and the increase in bandwidth for wireless communications, mobile devices become a logical platform for context aware applications.

Context-aware applications for mobile devices

• Adaptive applications alter the state of the mobile device depending upon context.

When Cindy goes into a meeting, her phone senses that she is in a meeting room. Her device then captures her recurring team meeting every Wednesday from 10 AM to 12 PM, so it knows how to filter her calls.

When Tom, her boyfriend calls, Cindy’s phone displays options ---- she is in a meeting and will call him back later. Or, she can delay the answer to step out of the meeting and take care of the call.

Context-aware applications for mobile devices

• Sharing applications allow multiple users to share context (e.g., location and presence), media, and/or other information with each other.

Two friends meet up during a night out and decide they will share any photos that they take for the rest of the evening. They access camera mode on each of their devices and touch their devices together, which automatically starts a photo sharing session between them.

For the rest of the evening, whenever they take a photo it is automatically sent to their friends at the club via Bluetooth.

Everyone in the sharing session is able to view the photos then and there, or at another time if they prefer. The photos are automatically tagged with information about where the photos were taken and who was there. At the end of the night the connection is dropped … until the next time they meet up!

• Context aware search applications allow users to search for more information, which can include looking up a specific topic on the Internet, finding relevant points of interest, and finding locations of friends.

Context-aware applications for mobile devices

Cathy and Jen are visiting Seattle for the first time. During a tour, they find a fancy restaurant that they might like to try. The device locates a restaurant by location and picture. A review site is linked directly to a map.

Cathy and Jen decide to try the restaurant since the reviews look very good.

After the dinner, Cathy posts her own personal rating of four and half stars, and also adds a short comment – “Great steak – Huge!!”.

Context-aware applications for mobile devices

• Assistive applications aid the user in completing a task more efficiently and smoothly.

Michelle has just flown to Florence without any definite plans as a ‘get-away’ vacation. Her Fuzzy Navigation on her device creates a spur-of-the- moment with points of interest.

Her device also notices where she is and suggests the sightseeing places and restaurants around her current location that relate to her interests.

Michelle posts her itinerary on the web. Fernando and Vicky are planning to visit Florence soon. They find Michelle’s travel path and pictures and really like her trip. They think they might like to take the same journey.

ARTC Context Aware ResearchArea: Driving context Immediate device handling

contextLifestyle and usage context

Context data • Driving simulator sensors

or

• Sensor information form a real car bus

• Touch sensors• IR proximity• accelerometers

• GPS, GSM Cell Id• Bluetooth devices around the phone• Phone profile• Active phone application• Phone idle/active time• Charger status• Incoming/Outgoing calls, messages• User interaction with the Phonebook and call log• Media captured with the device (photos, audio, video, text)

Application • Polite phone • Antenna power optimization

• Audio optimization, adaptive volume, smart ringing,

• hand-gestures as an interface, pedometer applications

• Dynamic profile• Route prediction• Application prioritization

Abstract task Learn situation classification from context data

Learn sensor reading classification from context data

Learn sensor reading classification / clustering / on-line adaptation from context dataFind significant patterns in data (data mining)

Database collection

Naturalistic driving data (400 variables X 36 drivers x 10 hours)

100 users x 1 hour phone handling and gestures

30 users X 2 months of phone sensor and activity logging

• Modeling and classification of driver activity

– Comes from our attempt to develop context aware intelligent driver assistance systems

• To keep the human in control at all time.

• To focus on applications, such as appropriate timing and modality of information presentation to the driver.

• To optimize driver cognitive workload.

• To warn the driver in case of unnoticed imminent danger.

– Construct the context aware driver assistance system

• Heuristic coding rules

• Statistical machine learning

Driving Context

• Driving sensor data collection

– Collect data in a driving simulation lab which is an instrumented car in a surround video virtual world with full visual and audio simulation of various roads, traffic and pedestrian activity.

– 425 separate variables describing an extensive scope of driving data

– 36 subjects X 10 hours of driving

Driving context collection and annotation

• Database annotation

– is to label the sensor data with meaningful classes, supervised learning and modeling techniques then become available.

– provides a basis for research in characterizing driver behavior in different contexts.

– uses semi-automatic annotation tool

• Characteristics of the driving sensor data

– hundreds of variables(sensors)

– millions of observations

– mixed type data

• Characteristics of the driving domain

– 29 driving maneuver classes

– Driving classes are not mutually exclusive (“SlowMoving” and “TurningRight”)

• Apply the Random Forests for instantaneous classification

– Because the maneuver labels may be overlapping, we train a separate Random Forest for each maneuver treating it as a binary classification problem.

– For new sensor data, each forest produces a probability of the class it belongs to.

– Convert the detector signals into discrete and possibly overlapping labels, and to assign a confidence value to each label.

Driving Data Classification

Driving Data Classification

Probabilities for maneuver classes graphed (from a Random Forest), the “detector” signals from each of the forests are graphed.

LaneDepartureLeft

CurvingRight

TurningRight

SlowMoving

Parking

Driving Data Classification

Segmentation into overlapping maneuver classes by median filtering, low-pass filtering

Sensor Selection

• Four levels of answers for maneuver classification (or any other driving data classification problem)

– Can it be done with ideal sensors?

– What is the best subset of sensors?

– What sensors are available in cars in near future?

– What sensors are available in cars now? Size

of t

he s

enso

r set

Qua

lity

of th

e se

nsor

s

Mining Naturalistic Driving Data

Important raw variables for each maneuver class

Classifier (Random Forest) produces importance measures

Variables in 6 clusters, maneuvers in 4 clusters

Sensor Selection

Hypervariate experiment (~400 variables)36 subjects X 10 hours of drivingCollect, annotate naturalistic driving dataClassify to 29 maneuver classesTools and methods to handle hundreds of variables and

millions of samples

If the driver’s workload is high

then direct the incoming call to voicemail

Polite Phone

There is no direct indicator of driver workload!

Indirect measure: “maneuver”

How to detect maneuvers? From what?

Car-device communicationsGadget protocol

Immediate Device Handling Context• Phone state recognition

– Determine if the phone is held against the head (for RF optimization)

– if the phone is being held or worn, or on a table

– Gesture and activity recognition•Shaking, tapping, rotation, tilting ……•Running, walking, driving ……

• Mobile Devices platform with possible following sensors:– Accelerometer– Capacitive touch sensors– Infrared proximity sensor

• Context recognition– Phone state can be distinguished at 10% error

rate using PCA + KNN– Gesture can be recognized with 5% error rate

using DTW+ Hierarchical Clustering• Further applications:

– Audio optimization, adaptive volume, smart ringing,

– hand-gestures as an interface, pedometer applications

Lifestyle and Usage Context• Task

– Understand what can be inferred about the user with context easily available at a device.

– By inferring the user and usage context, the mobile device UI can recognize and adapt to the current situation and user needs and predict future user goals to offer proactive assistance

• Context data logging– GPS, GSM Cell Id– Bluetooth devices around the phone– Phone profile, active phone application– Phone idle/active time, charger status– Incoming/outgoing calls, messages– User interaction with the phonebook and call log– Media captured with the device

• Personal Lifestyle Learning– Automatic discovery of prominent locations (home/work/gym/…/away

detector) online based on adaptive mixture model – Behavior patterns learning based on association rule/frequent itemsets

mining.– Lifestyle clustering/classification, predictability

• Applications– Location based services

•Adapt application (such as phonebook) to location•Association of profiles to locations (dynamic profile)•Location-based search

– Route prediction and proactive traffic advisory– Interaction with calendar

east

nort

h

east

time

north

Events:CallApplication

activeProfile change

Location + device usage

Location + time + device usage

• Pilot subject over 10 months5 college student subjects over 2 months 5 employee subjects over 3 months4 mobile employees or business owners

over 3 months

“Semantic” Locations

• Definition of Semantic Locations– The locations that user spend most of time at.

• Home• Office

– The locations that user mostly engaged in some activities.• The location that frequently visits but stay for a brief time.

• Learning Semantic Locations is the basis for– learning routes between the locations– designing context aware applications.

Existing Location Learning Methods

• GIS (geographic information system) based method– Landmark-based location system uses cell towers of a GSM phone

network to learn important places.

• Extract locations where GPS signal disappears and then reappears about a small region.

• Use driving acceleration and turning speed to help identify important locations.

• Machine learning techniques– K-mean– Density based clustering algorithm (DBSCAN)

Existing Location Learning Methods

• GIS (geographic information system) based method– Landmark-based location system uses cell towers of a GSM phone

network to learn important places.

• Extract locations where GPS signal disappears and then reappears about a small region.

• Use driving acceleration and turning speed to help identify important locations.

• Machine learning techniques– K-mean– Density based clustering algorithm (DBSCAN)

Batched Methods

Adaptive Location Learning

• does not require the data to be collected in advance.

• is a sequential method that process only a small amount of data each time.

• continuously adapts the learned model with new coming data.

• Benefits– is time efficient.– is memory economical.– can adapt to the users’ daily life.– is highly desired in real applications.

Semantic Locations Learning Database

• Continuous stream of GPS data points.• The real collected GPS data is not always continuous. The GPS data

need to be re-sampled.– Why?

• GPS signal may get lost due to– Entering into some buildings– Concrete canyons in urban areas

• GPS device may get power off due to– Long time loss connection – Low battery level

– How?• Use Cell Id information • Make up the void GPS data by repeating the last valid GPS reading, if

the area where GPS signal disappeared and reappeared, is covered by the same group of Cell Ids.

Adaptive Location Clustering

• Model the GPS data as a mixture of locations with noise, where each location follows a Gaussian distribution.

• represents the number of significant location candidates• denotes the significance of the learned locations• specifies the location center and size

tϖ

∑=

∑=tN

tttttt ggp1

),;()|(l

lll μφϖθ

tg

},,,,{ 11 tt Nt

Nttt ΣΣ μμ K

tN

Adaptive Location Clustering

• Given new GPS readings, an on-line algorithm is employed to update the model, i.e. adding/deleting/merging locations, updating the parameters of Gaussian mixture model, etc.

• The mixture model’s parameters as well as the structures will be adapted with the new coming observations.

– Update Rule

– Creating Rule

– Pruning Rule

– Merging Rule

Semantic Location Identification • Why?

– The location clusters are represented by the major components in the mixture model. – The adjustment weights in the mixture model denote the significance of the locations. – Not all location clusters are significant since some location clusters are recently added to the model and need time to be justified.– The location clusters only provide the candidates of the important locations.

• How?– Pick the top N corresponding location clusters which covers 95% of location significance weights.

Location Clustering Result

Location Clustering for the first day

Location Clustering Result

Location Clustering for the first week

Location Clustering Result

Location Clustering for the first month

Location Clustering Result

BusinessmanCollege studentOffice employee

Location Clustering for Different Types of Users

Location Clustering Result

Number of phone callsDuration of phone calls

Location Clustering in terms of Different Activities

Location Clustering Result

Location Clustering in terms of Different time durations

Transportation Routes Extraction

• Route detection: find the GPS strings between semantic location pairs by detecting that the user is leaving from one semantic location boundary to another semantic location boundary.

• Detect frequent route: find similar routes; get the frequency of each route between semantic location pairs and find the most frequent ones.

Unique Route Discovery

• Goal–Compare the GPS sequences between locations and extract the unique ones.

• Method– MinMax criterion to compare two routes

Route Extraction Result

10 am from home to location 37 am from home to office 4 pm from office back to home

Frequent routes in different location pairs and time

Route Frequency Table

Route ID 4 8 25 26 27 28 29 30

frequency 0.6 0.1429 0.0286 0.1143 0.0286 0.0286 0.0286 0.0286

TABLE I. FREQUENCIES FOR ROUTES FROM HOME TO OFFICE

Route ID 8 28 35 36 37 38 39 40 41

frequency 0.4643 0.2143 0.0357 0.0357 0.0357 0.0714 0.0714 0.0357 0.0357

TABLE II. FREQUENCIES FOR ROUTES FROM OFFICE TO HOME

Automatic Traffic Advisory System

Objective• Develop a location-based traffic advisory system for mobile

device that provides the user with unobtrusive assistance on traffic problems along user’s predicted route.

Benefits• Unobtrusive service• No user settings or user action needed• Provides users with better overall value and experiences• Increasing brand and carrier loyalty

Automatic Traffic Advisory System

ContextSniffer

SharedMemory

Location/Route

Learner

Cache

SharedFile

for Java

Profile

WatchDog

TrafficAdvisory

Automatic Traffic Advisory

Conclusions

• The context aware mobile devices can recognize the current situation and user’s needs, and predict user’s future goals to offer proactive assistance.

• Context aware and personalization features will become essential in supporting new applications.

• It can connect consumers with media and services through multiple devices in the office, car, or home.

Questions?

keshu.zhang@motorola.com