Technical Solutions Underlying Wireless Health Systems

Technical Solutions

Underlying Wireless Health

Systems

John A. StankovicBP America Professor

Department of Computer ScienceUniversity of Virginia

http://www.cs.virginia.edu/wsn/medical/http://wirelesshealth.virginia.edu/

http://www.cs.virginia.edu/wsn/medical/

What’s Wrong With Wires

And we don’t want a patient tethered to a bed or fixed medical device

Realisms and Main Points

• Realisms– Humans and their behaviors are not simple

• Example: Sleep

– Environments are not simple• Example: Acoustics

Realities – Sounds Encountered

Physiological: Sneezing, nose blowing, sniffling, clearing throat, hiccup, eating, burp, humming, laughter, drinking, snoring

Objects: phone vibrating or ringing, typing, mouse wheel, unwrapping food, papers rustling, clothes rustling, television, piano, moving furniture, doors opening and closing, objects dropping or moving, footsteps, pouring liquid, coffee percolation, dishwasher, cleaning sounds

Ambient: truck backing up, siren, birds chirping, passing airplane, traffic, motorized tools (lawnmower, etc)

Main Point

• Many current solutions work ONLY when humans and environments are (assumed to be) very constrained

Outline

• Motivation

• Architecture

• ADLs and Semantic Anomaly Detection

• Control Heart Rate

• Vision: Aggression in Dementia Patients (not today)

• Acoustics: Reverberation (not today)

Motivation - The Problems

• Aging Populations• High Cost of Medical Care• Lack of Facilities• Quality of Life Issues

• Solution: Home Health Care CCRC Assisted Living

Vision - Smart Living Space• Humans-in-Loop• Heterogeneous• Evolution• Open• Privacy

Reality: NOT SIMPLE

Research Q: Architecture

• Alarmnet • Web based (Empath)

InferenceDB

Med Records

WebServer

Our Solution: Empath

• Realisms– Off-the-shelf

– Non-expert user

– Easy installation and maintenance

– Remote monitoring

– Extensible (HW)

– Evolve (Behavior)

– Patients are very different than grad students

• Main Points– Web-like architecture

is quite flexible– Single hop

Cloud Controller

Inference Modules

Monitoring Modules

MongoDB

Web Server

CDMA Modem

4G

MQTT Broker

Home Controller

System Repository

Monitoring Modules

Collectors Sensors

Base Station

Architecture

Applications – Versions of Architecture

• Depression and General Anomalies

• Dementia and Incontinence

• Exercise and Epilepsy

Depression Detection andGeneral Anomalies

• Multi-modal

• Passive

• Combines Objective and Subjective Measures

Empath: Depression Monitoring

Motion and Contact Sleep Data PHQ-9 Acoustic Weight

DB

Eating Sleep Quality Movement Mood Weight Gain/Loss

Patient DisplayCaregivers

Display

Depression Inference

Cloud

Caregivers Display

Dementia and Incontinence

• Bedwetting and sleep agitation relationship

• Interventions on sleep

• Sleep labs impractical

• Goal: 50 patients

Dementia and Incontinence(subsystem of Empath)

• Passive Sleep system uva built

• UVA Tempo device uva built

• Wetness sensors – Dry Buddy off-shelf

• Acoustic sensors – Screaming off-shelf

ID Age Sex No. of Nights

No. of Nights Wetness Detected

No. of Nights Wetness Reported

Who Deoplyed

Deployment Time (Minutes)

No. of Intermediate Visits

1 65 Female 6 2 5 Grad. Student

20 0

2 87 Male 8 5 5 Grad. Student

20 0

3 71 Male 5 4 5 Grad. Student

20 0

4 73 Female 5 5 5 Nurse 30 0

5 79 Male 5 0 0 Nurse 30 1

6 88 Male 5 4 5 Nurse 30 1

7 89 Female 5 5 5 Nurse 30 0

Example Correlations

Epilepsy Study(subsystem of Empath)

• Linkage: between exercise, stress, sleep and epilepsy seizures

• Measure: HR, BP (CareTaker), sleep quality, frequency of seizures

• Intervention: qigong (Chinese healing art)

Many Seizures at Night

Status

• Depression – 2 homes

• Dementia – 11 real patients (on-going)

• Epilepsy – 2 real patients

Research Q: (Robust) Activities of Daily Living

• Monitor eating, sleeping, exercising, …

• What’s normal

• Detect anomalies

• Intervene: send alarms, inform someone, prevent, …

Realisms

• Activity Recognition (AR) of ADLs– Higher accuracy required– Overlapped activities– Across room activities– Many realities (missing data) ….

• Anomaly detection– Accurate AR and what’s normal support good

anomaly detection

Main Point

• Normal behavior is very complex– Per day– On Wednesdays– Two times per week– Every other month– In summer when condition X exists– Grouping of activities– Context dependent– …

Big Picture

28

Motion Sensors

Contact Sensors

Bed Sensors

Body Sensors

Pre-Processing

Activity Recognition

Sleep Eat Toilet Shower Exercise

Learn Regular Behavior

Entertainment

Behavioral Anomaly Detection

AALO (Active Learning based Activity recognition in the presence of Overlapped activities)

29

Motion Sensors

Contact Sensors

Bed Sensors

Body Sensors

Pre-Processing




Entertainment


AA

LO

Overla

pped

Acti

vities

Holmes: A comprehensive anomaly detection system

30

Motion Sensors

Contact Sensors

Bed Sensors

Body Sensors

Pre-Processing




Entertainment


Hol

mes

Context-basedMultiple Models

Temporal & CausalCorrelations

Semantic RulesReduce False

Positives & Negatives

31

Robust Activity Recognition

• Accurate detection and summary of daily activities are essential

• Proper training and periodic retraining are necessary

• Necessary to ensure user comfort

31PervasiveHealth 2012

Overlapped Activities

• Interrupted by room changes

• Concurrent in the same room

• Across rooms (not done yet)

PervasiveHealth 2012 32

33

Room Level Segmentation

• First consider spatial regularity

• In real life:– Within one episode, multiple activities may take place

• We use Itemset identification

– An activity may not complete within one episode

PervasiveHealth 201233

Segmentation into Occupancy Episodes

(<timestamp, sensor-firing> pairs)(<roomID, entranceTime, duration, usedSensors>)

34

Room Level Segmentation

• Successive Occupancy Episode Merging (SOEM)– to identify overlapped activities interleaved across

multiple rooms

• We construct a new occupancy episode by merging two occupancy episodes in the same room that are separated by less than a time interval threshold


35

Example Evaluation

Public Dataset

• Single resident home• Activity annotation by the

resident using Bluetooth headset

• 26 days of data• 4 rooms (Bedroom,

Kitchen, Toilet, Shower)• ‘Out of house’ considered

as another room

List of Sensors


kitchen_microwave kitchen_groceries

kitchen_cups kitchen_fridge

kitchen_plates toilet_door

kitchen_dishwasher toilet_flush

kitchen_freezer shower_door

kitchen_pans bedroom_door

kitchen_washingmach front_door

36

Effectiveness of SOEM


• Accuracy of ‘Sleep’ & ‘Prepare Dinner’ detection improved• SOEM does not reduce the accuracy for any activity

37

Comparison with Supervised Algorithms


Semantic Anomaly Detection

• Sensor Level Anomalies

• Activity Level Anomalies– Point– Context– Collective

• Semantics

Sensor Level Anomalies

• May be due to different types of sensor failures– Fail-stop failure (may be detected by Ping)– Stuck-at failure– Transient– Obstructed field-of-view– Sensor movement

Sensor Level Anomalies

• How to detect:– Use temporal and spatial correlation among

different sensors

– Use multiple classifiers

– Many systems assume no sensor failures!

K. Kapitanova, E. Hoque, D. Alessandrelli, J. Stankovic, S, Son and K. Whitehouse, Being SMART About Failures: Assessing Repairs in Activity Detection, Ubicomp, Sept. 2012.

Activity Level Anomalies

• Point Anomalies– For each activity, one or more models based on

training data

– Different set of features for different activities• Sleep: sleep onset time, duration, number of

interruptions, movement level• Showering: Start time, duration, water usage

– Use statistical anomaly detection

– Too many false alarms if used alone!

Context Anomalies

• For each activity, one or more models based on training data– Daily, specific day / days of week, specific

time periods in a day

42

Collective Anomalies

• Monitor instances of same activity for N days collectively– Example: exercise twice per week

• Monitor whether other activities happen before/after/during that activity in each time period– Example: have coffee before breakfast

• Monitor instances of all activities for N days collectively– Example: every week shop, exercise, see doctor, …

Semantics

• Watching a pet for a few days

• Entertaining visitors

• Power outage

• Recovering from major medical operation

• Sensors died

• Extreme weather: cold/heat wave

• Social security check did not arrive: cannot purchase food or medication

• Major medication change

• Life changing events: sibling etc died, new grandchild etc.

Holmes Framework

45

Training Data

Semantics: Expert Knowledge & User Feedback

Explainable Scenarios


Calculate Anomaly Score and Initial Filter

Testing Data

Regular Behavior Models

True Anomalies

SemanticAnomaly Filters

MusicalHeart

• Music non-invasive recommendation system to control heart rate– Bio-feedback (human-in-the-loop)– Context aware

• Sitting, lying, walking, jogging, exercising• Home, traveling

– Personalized– Learns

Contexts: jogging, relaxing, traveling, etc.

Wellness - Musical Heart

Realisms

• Movement (rustling) of ear buds

• Music reaction is personal

• Max HR per person• Context dependent• Impact changes over time

Main Points

• Human in the Loop• PI Control matches well

Calm Your Heart

Exercise Your Heart

3 Parts to Musical Heart

Sensor EquippedEarphones

Smartphone AppCloud

Musical Heart: Hardware (SEPTIMU)

• SEPTIMU v2 • Sensors: IMU Microphone IR Reflective Sensor Thermometer

• Computation: Tiny OS

• Communication: Audio Jack Bluetooth

• Power: Li-Polymer battery

Beijing, China

Musical Heart: Software Architecture

In-phone SensorsIn-phone Sensors

Heart RateDetector

Heart RateDetector

Activity LevelDetector

Activity LevelDetector

ContextDetectorContextDetector

Sensor Data DispatcherSensor Data Dispatcher

Septimu SensorsSeptimu Sensors

SeptimuMic.

SeptimuMic.

SeptimuAcc.

SeptimuAcc.

In-phoneGPS

In-phoneGPS

In-phoneWiFi

In-phoneWiFi

CombinerCombiner

MusicPlayerMusicPlayer ProxyProxy

Music Recommender

Sensing

Heart Rate and Activity Detector

MusicRecommender

Algorithm 1: Heart Rate Detection• Filtering: A low pass filter to remove non-

heart beat signals.

Signals from Ear(music + heart beats)

After filtering(heart beats)

music

Detect Heart Rate

• V1: Microphone– Movement of ear phone, ambient sounds, music,

…– Dynamic programming solution– Accuracy: (1-7 bpm error)

• V2: LED– Accuracy: 1-2 bpm

Algorithm 2: Activity Level Inference

Threshold1

Threshold2

We use k-means clustering to learn the thresholds

Result: Activity Level Inference

L1 L2 L3

L1 0.9998 0.0002 0

L2 0 0.9997 0.0003

L3 0 0.0280 0.9720

Predicted level

Act

ual le

vel L1 L2 L3

L1 0.989 0.011 0

L2 0 0.951 0.049

L3 0 0.037 0.963

Predicted level

Act

ual le

vel

• Single Activity • Sequence of Activities

99.1% 96.8%

Algorithm 3: Music Recommendation

I

PProcessSet Point

Proportional

Integral


I

PSet Point

Proportional

Integral

Process: Human Heart

CurrentHeart Rate


I

P

Desired Heart Rate

Chart

CurrentHeart Rate

Set-point: Desired Heart Rate

Proportional

Integral


I

P

Desired Heart Rate

Chart

CurrentHeart Rate

Find a matching song

Reflex

History

u

Deployment: Goal directed jogging • Typical Cardio Exercise Plan:

Time Desired Intensity (%)

5 min 60% - 70%

3 min 70% - 80%

2 min 80% - 90%

3 min 70% - 80%

2 min 80% - 90%

5 min 60% - 70%

Result: Achieving Desired Heart Rate


5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%



5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity

Control Input(music feature, u)

u



5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity




5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity




5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity




5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity




5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity




5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity




5 min 65%

3 min 75%

2 min 85%

3 min 75%

2 min 85%

5 min 65%

Desired Intensity

Achieved Intensity

Avg. Deviation: 11.4%Control Input

(music feature, u)

Summary

• 1000s of deployments for WH

• Technical Solutions– Architecture– AR and Anomaly Detection– MusicalHeart– Vision: Kintense– Acoustics: in the home (reverberation)

Summary

• (Personalized) Wireless Health– Body Sensor Networks– Environmental and AR Networks

• Easy to Modify over Time– Incorporate new technology as it becomes

available– Adapt as medical conditions change

Kintense – Detecting Aggressive Behavior

• Main Point– Cohen-Mansfield metrics for dementia patients

• Hitting• Kicking• Throwing• Punching• …

– Usable in other contexts, e.g., reception areas, …

Our Solution: Kintense

• Kinect

• Array of Supervised Learners

• Unsupervised Learner• Human input

• Robust – People– Orientation– Speed– In-home obstacles

• Performance– DTW vs Gesture

Recognition vs Kintense

Kintensehttp://www.youtube.com/watch?v=LUJLXcPFMpg

http://www.youtube.com/watch?v=LUJLXcPFMpg

http://www.youtube.com/watch?v=LUJLXcPFMpg

Resonate• Get 1 clean speech sample in each room for

each person• Use reverberation impulse response simulator

– Generate many training samples at different locations

• Create suite of classifiers• Tested

– 6 different types of rooms and long term deployments in home and office

– Close to ideal performance

PervasiveHealth 2012 78

Reducing User Labeled Data

• User needs to label only 19 clusters

• Any supervised algorithm requires labeling of 291 activity instances

• With supervised algorithms, amount of user labeling proportionally increases with number of training days

Documents

Technical Solutions Underlying Wireless Health Systems