Vision
Understanding human behavior from sensor dataActively prompting, warning,and advisingAlerting caregivers as necessary
Computer systems that improve the independence and safety of people suffering from cognitive limitations by…
Technical Foundations
Inexpensive, easily deployed sensors
GPS cell phones – locationRFID tags – object manipulationWearables – motion, sound, pulse, ...
Advances in algorithms for high-level analysis of sensor data
General Approach
userprofile
generalknowledge
wearablesensors
environmentalsensors
decisionmaking
userinterface
caregiveralerts
behavior
cognitive state
intentions
activity
ExamplesMaintaining a daily schedule
Compensating for memory problemsCompensating for lowered self-initiative
NavigationIndoors or outdoors
Performing activities of daily livingStep-by-step prompting
Behavior regulationImproving self-awareness
Safety and wellnessNeed for immediate helpLong term health trends
Where is the Work Being Done?
Field is coming together from many communities
Computer science – artificial intelligence, robotics, ubiquitous computingRehabilitation engineeringRehabilitation medicineAssisted-living & nursing homesGerontology
Problems in Using Public Transportation
Learning bus routes and numbersTransfers, complex plansRecovering from mistakes
Current GPS Navigation Devices
Designed for drivers, not bus riders!Should I get on this bus?Is my stop next?What do I do if I miss my stop?
Destination manually enteredHigh cognitive overhead
Device decides which route is “best”Familiar route better than shorter one
Catastrophic failure when signal is lost
Frequent “dead zones” in urban areas
Idea User carries GPS cell phoneSystem infers user’s state
Walking? Getting on a bus?
System learns about userImportant places, routes
Breaks from routine = user may be confused or lost
Offer helpCall caregiver
Transportation Routines
BA
Goal: intended destinationWorkplace, home, friends, restaurants, …
Trip segments: <start, end, mode>
Home to Bus stop A on Foot
Bus stop A to Bus stop B on Bus
Bus stop B to workplace on Foot
WorkplaceHome
GPS readingzk-1 zk
Edge, velocity, positionxk-1 xk
k-1 k Data (edge) association
Time k-1 Time k
mk-1 mk Transportation mode
tk-1 tk Trip segment
gk-1 gk Goal
Cognitive mode { routine, novel, error }
Dynamic Bayesian Network
ck-1 ck
GPS camera-phone“Knocks” when there is an opportunity to help
Can I guide you to a likely destination?I think you made a mistake!This place seems important – would you photograph it?
Prototype: Opportunity Knocks
GoalA home monitoring system that
Assists user in performing activities of daily livingTracks activities, and provides prompts and warnings as neededCan be deployed in an ordinary home
Initial Application
Accurate, automated ADL logs
Changes in routine often precursor to illness, accidentsHuman monitoring intrusive & inaccurate
Image Courtesy Intel Research
Technical RequirementsSensor hardware that can be practically deployed in a ordinary homeMethods for activity tracking from sensor data Methods for automated prompting that consider
Probability of user errorsProbability of system errorsCost / benefit tradeoffs
Object-Based Activity Recognition
Activities of daily living involve the manipulation of many physical objects
Kitchen: stove, pans, dishes, …Bathroom: toothbrush, shampoo, towel, …Bedroom: linen, dresser, clock, clothing, …
We can recognize activities from a time-sequence of object touches
Sensing Object Manipulation
RFID: Radio-frequency identification tags
SmallSemi-passiveDurableCheap
Near future: use products’ own tags
Improving RobustnessTracking fails if novel (but reasonable) objects are used
Solution: smooth parameters over abstraction hierarchy of object types
Experiment: ADL Form Filling
Tagged real home with 108 tags14 subjects each performed 12 of 14 ADLs in arbitrary orderGiven trace, recreate activities
Results: Detecting ADLs
Activity Prior Work
SHARP
Personal Appearance 92/92
Oral Hygiene 70/78
Toileting 73/73
Washing up 100/33
Appliance Use 100/75
Use of Heating 84/78
Care of clothes and linen 100/73
Making a snack 100/78
Making a drink 75/60
Use of phone 64/64
Leisure Activity 100/79
Infant Care 100/58
Medication Taking 100/93
Housework 100/82
Legend
Point solution
General solution
Inferring ADLs from Interactions with Objects
Philipose, Fishkin, Perkowitz, Patterson, Hähnel, Fox, and Kautz
IEEE Pervasive Computing, 4(3), 2004
RFID
Current / Future DirectionsMore detailed sensor data
Machine vision fused with direct sensing
More detailed models of cognitive stateAffect recognition
Automated inventionsPrompting & guidanceInteraction: Audio? Visual? Tactile?
Interactive design/testing with various target populations
TBIAlzheimersAutism