Upload
ula
View
34
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Recognizing Human Activity from Sensor Data. Henry Kautz University of Washington Computer Science & Engineering graduate students : Don Patterson, Lin Liao CSE faculty : Dieter Fox, Gaetano Borriello UW School of Medicine : Kurt Johnson - PowerPoint PPT Presentation
Citation preview
Recognizing Human Activity from Sensor Data
Recognizing Human Activity from Sensor Data
Henry Kautz
University of WashingtonComputer Science & Engineering
graduate students: Don Patterson, Lin Liao
CSE faculty: Dieter Fox, Gaetano Borriello
UW School of Medicine: Kurt Johnson
Intel Research: Matthai Philipose, Tanzeem Choudhury
Converging Trends…Converging Trends…
Pervasive sensing infrastructureGPS enabled phonesRFID tags on all consumer productsWireless motes
Breakthroughs in core artificial intelligenceAfter “AI boom” fizzled, basic science went on…Advances in algorithms for probabilistic reasoning and machine learning
Bayesian networksStochastic sampling
Last decade: 10 variables 1,000,000 variablesHealthcare crisis
Epidemic of Alzheimer’s Disease Deinstitutionalization of the cognitively disabledNationwide shortage of caretaking professionals
...An Opportunity...An Opportunity
Develop technology toSupport independent living by people with cognitive disabilities
At homeAt workThroughout the community
Improve health careLong term monitoring of activities of daily living (ADL’s)Intervention before a health crisis
The University of Washington Assisted
Cognition Project
The University of Washington Assisted
Cognition ProjectSynthesis of work in
Ubiquitous computingArtificial intelligenceHuman-computer interaction
ACCESSSupport use of public transit
CAREADL monitoring and assistance
This TalkThis TalkBuilding models of everyday plans and goals
From sensor dataBy mining textual descriptionBy engineering commonsense knowledge
Tracking and predicting a user’s behavior
Noisy and incomplete sensor dataRecognizing user errors
First steps toward proactive assistive technology
ACCESSAssisted Cognition in Community, Employment, & Support Settings
Supported by The National Institute on Disability &
Rehabilitation Research (NIDDR)The National Science Foundation (NSF)
ACCESSAssisted Cognition in Community, Employment, & Support Settings
Supported by The National Institute on Disability &
Rehabilitation Research (NIDDR)The National Science Foundation (NSF)
Learning & Reasoning About Transportation
Routines
TaskTask
Given a data stream from a wearable GPS unit...
Infer the user’s location and mode of transportation (foot, car, bus, bike, ...)Predict where user will goDetect novel behavior
User errors?Opportunities for learning?
Why Inference Is Not Trivial
Why Inference Is Not Trivial
People don’t have wheelsSystematic GPS error
We are not in the woodsDead and semi-dead zonesLots of multi-path propagationInside of vehiclesInside of buildings
Not just location trackingMode, Prediction, Novelty
GPS Receivers We UsedGPS Receivers We Used
Nokia 6600 Java Cell Phone with Bluetooth GPS
unit
GeoStats wearable GPS
logger
Geographic Information Systems
Geographic Information Systems
Bus routes and bus stopsData source: Metro GIS
Street mapData source: Census 2000
Tiger/line data
ArchitectureArchitecture
Learning Engine
Inference Engine
GIS
Database
Goals Paths Modes Errors
Probabilistic ReasoningProbabilistic Reasoning
Graphical model: Dynamic Bayesian network
Inference engine: Rao-Blackwellised particle filters
Learning engine: Expectation-Maximization (EM) algorithm
Graphical Model (Version 1)
Graphical Model (Version 1)
Transportation ModeVelocityLocation
BlockPosition along blockAt bus stop, parking lot, ...?
GPS Offset ErrorGPS signal
Rao-Blackwellised Particle Filtering
Rao-Blackwellised Particle Filtering
Inference: estimate current state distribution given all past readingsParticle filtering
Evolve approximation to state distribution using samples (particles)Supports multi-modal distributionsSupports discrete variables (e.g.: mode)
Rao-BlackwellisationEach particle includes a Kalman filter to represent distribution over positionsImproved accuracy with fewer particles
TrackingTracking
blue = foot
green = bus
red = car
LearningLearning
User model = DBN parametersTransitions between blocksTransitions between modes
Learning: Monte-Carlo EMUnlabeled data30 days of one user, logged at 2 second intervals (when outdoors)3-fold cross validation
ResultsResults
ModelMode Prediction
Accuracy
Decision Tree(supervised)
55%
Prior w/o bus info 60%
Prior with bus info 78%
Learned 84%
Pro
babili
ty o
f co
rrect
ly
pre
dic
ting t
he f
utu
rePro
babili
ty o
f co
rrect
ly
pre
dic
ting t
he f
utu
re
City BlocksCity Blocks
Prediction AccuracyPrediction Accuracy
How can we improve
predictive power?
Transportation Routines Transportation Routines
BA
Goalswork, home, friends, restaurant, doctor’s, ...
Trip segmentsHome to Bus stop A on FootBus stop A to Bus stop B on BusBus stop B to workplace on Foot
Work
“Learning & Inferring Transportation Routines”, Lin Liao, Dieter Fox, & Henry Kautz, AAAI-2004 Best Paper Award
Hierarchical ModelHierarchical Model
Transportation mode
x=<Location, Velocity>
GPS reading
Goal
Trip segment
xk-1
zk-1 zk
xk
mk-1 mk
tk-1 tk
gk-1 gk
Hierarchical LearningHierarchical Learning
Learn flat modelInfer goals
Locations where user is often motionlessInfer trip segment begin / end points
Locations with high mode transition probability
Infer trips segmentsHigh-probability single-mode block transition sequences between segment begin / end points
Perform hierarchical EM learning
Inferring GoalsInferring Goals
Inferring Trip SegmentsInferring Trip Segments
Going to work Going home
Correct goal and route predicted
100 blocks away
Novelty & Error DetectionNovelty & Error Detection
Approach: model-selectionRun several trackers in parallel
Tracker 1: learned hierarchical modelTracker 2: untrained flat modelTracker 3: learned model with clamped final goalEstimate the likelihood of each tracker given the observations
Detect User Errors
Untrained Trained Instantiated
Application:
Opportunity Knocks
Application:
Opportunity Knocks
Demonstration (by Don Patterson) at AAHA Future of Aging Services, Washington, DC, March, 2004
CARECognitive Assistance in Real-world
Environments
supported by the Intel Research Council
CARECognitive Assistance in Real-world
Environments
supported by the Intel Research Council
Learning & Inferring Activities of Daily Living
Research HypothesisResearch Hypothesis
Observation: activities of daily living involve the manipulation of many physical objects
Cooking, cleaning, eating, personal hygiene, exercise, hobbies, ...
Hypothesis: can recognize activities from a time-sequence of object “touches”
Such models are robust and easily learned or engineered
Sensing Object Manipulation
Sensing Object Manipulation
RFID: Radio-frequency ID tagsSmallSemi-passiveDurableCheap
Where Can We Put Tags?Where Can We Put Tags?
How Can We Sense Them?How Can We Sense Them?
coming... wall-mounted “sparkle reader”
Example Data StreamExample Data Stream
Making TeaMaking Tea
Building ModelsBuilding Models
Core ADL’s amenable to classic knowledge engineeringOpen-ended, fine-grained models: infer from natural language texts?
Perkowitz et al., “Mining Models of Human Activities from the Web”, WWW-2004
Experimental SetupExperimental Setup
Hand-built library of 14 ADL’s17 test subjectsEach asked to perform 12 of the ADL’sData not segmentedNo training on individual test subjects
Activity Prior Work
CAREAccuracy/Recall
Personal Appearance 92/92
Oral Hygiene 70/78
Toileting 73/73
Washing up 100/33
Appliance Use 100/75
Use of Heating 84/78
Care of clothes and linen 100/73
Making a snack 100/78
Making a drink 75/60
Use of phone 64/64
Leisure Activity 100/79
Infant Care 100/58
Medication Taking 100/93
Housework 100/82
95/84
General SolutionQuantitative Results
Point SolutionQuantitative Results
Point SolutionAnecdotal Results
General SolutionAnecdotal Results
Pervasive Computing, Oct-Dec 2004
Current DirectionsCurrent Directions
Affective & physiological stateagitated, calm, attentive, ...hungry, tired, dizzy, ...
Interactions between peopleHuman Social Dynamics
Principled human-computer interaction
Decision-theoretic control of interventions
Why Now?Why Now?
A goal of much work of AI in the 1970’s was to create programs that could understand the narrative of ordinary human experienceThis area pretty much disappeared
Missing probabilistic toolsSystems not able to experience worldLacked focus – “understand” to what end?
Today: tools, grounding, motivation
Challenge to Nanotechnology
Community
Challenge to Nanotechnology
CommunityCurrent sensors detect physical or physiological state: user mental state must be indirectly inferredTo what can extend can nanotechnology afford direct access to a person’s emotions and intentions?