Programmer friendly abstractions for in- and on-body sensing Thomas Plagemann

Programmer friendly abstractions for in- and on-body sensing

Thomas Plagemann

Background

22. april 2023 2

Applications

OperatingSystems

CommunicationsData Management

DMMS

Green electronics

Power efficientrobust nano circuits

Wireless sensor networks and radio, data conversion, low power/low voltage digital signal processing

NANO

Background (cont.)

3

GlucoSense: an implantable blood sugar sensor for continuous monitoring

[Source: http://www.apollon.uio.no/vis/art/2011_2/artikler/innovasjon_radar][Source: http://heim.ifi.uio.no/~hafliger/GlucoSense/index.html]

Short range radar with millimeter precision

The talks message…

22. april 2023 44A/D or D/A conversion

Processing & communication

A/D or D/A conversion


A/D or D/A conversion


Application 1Application 1 Application 2Application 2 Application nApplication nChallenges:How to integrate heterogeneous sensors?How to model their abilities … (semantics)How to discover & locate sensors?How to achieve seamless extensibility

Application developers, e.g., home care:Domain knowledgeDifferent environmentsDifferent installations

Hand-made hard-coded solutions for each application and each installation will not work!

Core Problem:

We miss appropriate abstractions and models!

But can high-level abstractions fulfill “real-time” requirements?

Two studies:

Online Analysis of Myocardial Ischemia From Medical Sensor Data Streams with Esper

CommonSense: Complex Event and Deviation Detection for Ambient Assisted Living

Heart Attack Prediction – Study 1

• Recreate an off-line technique of Elle et al. [1] conducted in MATLAB

• Early recognition of regional cardiac ischemia• 3-way accelerometer placed on the left ventricle of the heart• Data set from surgery performed on pigs

22. april 2023 5

Figure show a perfect overlap, thus the technique by Elle et al [1] can be recreated online using Esper


Occlusion occurs after 80 secondsOcclusion occurs after 80 seconds

Perfusion after 170 seconds


SELECT edv(y)FROM Accelerometer WINDOW LENGTH(512)


Easier than MATLABEasier than MATLAB

Heart Attack Prediction – Study 2

• Improve results by adding beat-to-beat detection using a QRS detection algorithm on ECG signals– Each ECG trace of a normal heartbeat typically contains a QRS

event– A good reference for separating heartbeats

• Problem: Heartbeat duration is not a static pre-known size. DSMS window techniques only describe static time-based or tuple-based windows.

• Solution: Introduce variable length triggered tumbling windows

22. april 2023 6

Heart Attack Prediction – Study 2 (cont.)

22. april 20237

Plot shows fixed sliding window (512 samples) and dynamic triggered window (based on QRS detection) => less variance!


SELECT edv(y) FROM Accelerometer TRIGGER WINDOW BY QRSD(ECG.value)


Sudden drop caused by ultra sound probe


OcclusionOcclusion PerfusionPerfusion

Heart Attack Prediction - Insights

• Real-time analysis is possible• Declarative CQL API very popular for medical researchers

• Adapting windows size to heart beat rate yields better results

22. april 2023 8

CommonSens Foundations Complex event detection and deviation detection in

Ambient Assisted Living Abstract away the differences Models

Events and queries Sensor

Capabilities: state variables the sensor can observe Coverage area

Environments Objects Permeability affects coverage areas Locations of interest (LoIs)

LoI

Sensor model• Tuple sources:

• A/D conversion physical sensors

• Data aggregation logical sensors

• Stored data external sources

• Capabilities describe what a tuple source can provide (identified by labels)

• A tuple source can provide one to several labels

• Different types of tuple sources, e.g. a camera can provide images, videos, and motion detection

• A label can have one to several tuple sources that provides it

• Same tuple source types but different manufacturers

Capabilities and Multimodality

Sensor capabilities are abstractions Allow multimodality

One capability provided by many different sensors, e.g. DetectMotion

Motion detectors Cameras with motion detector software

One sensor can provide many capabilities, e.g. cameras combined with software

DetectMotion DetectObject DetectPerson RecogniseFace ...

CommonSense: Core Idea

22. april 2023 12

Event Model:- Atomic- Complex- Temporal- Spatial


Sensor Model:- Physical & logical- Capabilities- Signal Types- Coverage area


Environment Model:- Objects - Permeability- Sensors


CommonSense: Core Idea (cont.)

22. april 2023 13







Statements (LoIs & Capabilities)

Sensor placement

Statement instantiation

State machine creation

A priori

CommonSense: Core Idea (cont.)

22. april 2023 14








Sensor placement



Data gathering

Evaluation

A priori

Runtime, i.e., event processing

Conclusions

• CommonSense provides new abstractions:• Logical & physical sensors• Capabilities & Locations of Interest

• Support for sensor placement• Re-use and simplified personalization• Non-optimized prototype works in real-time

• More general principles:• Declarative APIs• Late binding• Do not differentiate in the system between physical and

logical sensors

• But what about actuators?22. april 2023 15

Acknowledgements

• This research was funded by the Norwegian Research Council in the DT-Stream and SIRIUS project, the Intervention Centre, and the University of Oslo

• Special thanks for pictures and transparencies:

• Philip Haeflinger• Tor Sverre Lande• Morten Lindeberg• Stig Støa• Jarle Søberg

22. april 2023 16

Questions?

22. april 2023 17

To read further• Lindeberg, Morten Gunnar Bjørner; Goebel, Vera Hermine & Plagemann, Thomas Peter (2010). Adaptive sized

windows to improve real-time health monitoring: a case study on heart attack prediction , In James Z., Wang (ed.), Proceedings of the international conference on Multimedia information retrieval. Association for Computing Machinery (ACM). ISBN 978-1-60558-815-5. Special poster session 1: processing data streams from body sensor networks. s 459 - 468

• Lindeberg, Morten Gunnar Bjørner; Kristiansen, Stein; Plagemann, Thomas Peter & Goebel, Vera Hermine (2010). Challenges and techniques for video streaming over mobile ad hoc networks. Multimedia Systems. ISSN 0942-4962. . doi: 10.1007/s00530-010-0187-8

• Støa, Stig; Lindeberg, Morten Gunnar Bjørner & Goebel, Vera Hermine (2008). Online Analysis of Myocardial Ischemia From Medical Sensor Data Streams with Esper , In Simone, Frattasi (ed.), First International Symposium on Applied Sciences on Biomedical and Communication Technologies, 2008. ISABEL '08.. IEEE conference proceedings. ISBN 978-1-4244-2647-8. Paper. s 1 - 5

• Søberg, Jarle; Goebel, Vera Hermine & Plagemann, Thomas Peter (2010). Detection of spatial events in CommonSens , In Ansgar, Scherp (ed.), Proceedings of the 2nd ACM international workshop on Events in multimedia. Association for Computing Machinery (ACM). ISBN 978-1-4503-0176-3. 10. s 53 - 58

• Søberg, Jarle; Goebel, Vera Hermine & Plagemann, Thomas Peter (2010). CommonSens: Personalisation of Complex Event Processing in Automated Homecare, In Slaven, Marusic (ed.), Proceedings of the 2010 Sixth International Conference on Intelligent Sensors, Sensor Networks and Information Processing. IEEE conference proceedings. ISBN 978-1-4244-7176-8. 4. s 275 - 280

• Søberg, Jarle; Goebel, Vera Hermine & Plagemann, Thomas Peter (2008). To happen or not to happen: towards an open distributed complex event processing system , In Sam, Michiels (ed.), Proceedings of the Fifth Middleware Doctoral Symposium. Association for Computing Machinery (ACM). ISBN 978-1-60558-361-7. Artikkel nummer 5. s 25 - 30

• Skjelsvik, Katrine S; Søberg, Jarle; Goebel, Vera Hermine & Plagemann, Thomas Peter (2007). Using Continuous Queries for Event Filtering and Routing in Sparse MANETs , In Patrick, Kellenberger (ed.), 11th IEEE International Workshop on Future Trends of Distributed Computing Systems. IEEE Computer Society. ISBN 0-7695-2810-4. Session V: Wireless and Mobile Ad Hoc Networks. s 138 - 145

22. april 2023 18

Back-up Transparencies

22. april 2023 19

Online Analysis of Myocardial Ischemia From Medical Sensor Data Streams with EsperStig Støa1, Morten Lindeberg2 and Vera Goebel2

1 The Interventional Centre (IVS), Rikshospitalet University Hospital, Oslo, Norway2 Distributed Multimedia Systems (DMMS), Department of Informatics, University of Oslo, Norway

Background

• Data Stream Management Systems (DSMSs) from database community [4][5] => Complex Event Processing (CEP)[6]

• Provide versatile and generic query interfaces for managing streams (query and analysis) in real-time

• Essential components:• Stream data model• Stream operations• Query language• Window techniques

We will use Esper [3]22.04.23 21ISABEL 08, Morten Lindeberg, [email protected]

Experiment Goal #1• Recreate an off-line technique of Elle et al. [1] conducted in

MATLAB• Early recognition of regional cardiac ischemia• 3-way accelerometer placed on the left ventricle of the heart

• Single metric: • Fast Fourier Transformation (FFT) is used to examine the

accelerometer signal in the frequency-domain• Euclidian distance vector (EDV(i)) between reference vector

RV(0) and current vector CV(j), where j is the latest sample number

• CV(j) : FFT over sliding window (size 512 over y-axis)• RV(0) : FFT over baseline window (first 512 samples)

• Data set from surgery performed on pigs at the Interventional Centre

• We can conduct experiments with the same data set (referred to as data set 1)

22.04.23 ISABEL 08, Morten Lindeberg, [email protected] 22

Experiment Goal #2


• Improve results by adding beat-to-beat detection using a QRS detection algorithm on ECG signals– Each ECG trace of a normal heartbeat typically

contains a QRS event– A good reference for separating heartbeats

• We need a to perform FFT over a sliding window of variable size!

• Cannot use the same data, use new data set that include ECG (data set 2)

[ image from http://www.wikipedia.org ]

http://www.wikipedia.org/

Challenges1. Incorporate signal processing operations

• Problem: Not supported in the query language• Fast Fourier Transformation of the accelerometer signals• Euclidian distance vector from baseline window• QRS detection for detecting the heartbeats from the ECG

signals• Solution: Custom aggregate functions

2. Static sized windows are not feasible for beat-to-beat detection• Problem: Heartbeat duration is not a static pre-known size.

DSMS window techniques only describe static time-based or tuple-based windows.

• Solution: Introduce variable length triggered tumbling windows

3. Synchronize the two streams• Problem: QRS detection introduces variable delay (approx.

91 samples)• Solution: Introduce variable buffer, that “slows” the

accelerometer stream down22.04.23 24ISABEL 08, Morten Lindeberg, [email protected]

Signal processing operations

• Implement as custom aggregate functions• Use defined Java interface and simply add to query engine

• Implemented methods:• QRSD(v): QRS detection based upon algorithm

from Hamilton et al. [2], source code is also public available

• edv(v): Euclidian distance from baseline

22.04.23 25ISABEL 08, Morten Lindeberg, [email protected]

Variable length triggered tumbling windows

• The ECG stream is aggregated into a stream consisting of QRS events Sb

• This stream (Sb) triggers the flushing of the sliding window w(t) where the custom aggregation over the stream Sa is performed

• The window is not supported by Esper => We implemented a “workaround” exploiting functionality of externally timed windows


Stream Synchronization

• The QRS detection algorithm over the ECG stream introduce a variable delay Δt

• Introduce the same delay to the accelerometer stream• Accelerometer stream is sent through a FIFO queue,

with dynamic size• QRS detection function sets the dynamic size of the

FIFO queue (also triggers the flushing of the aggregate window, in order to obtain dynamic windows, as already seen)


Results #1 (data set 1)




Occlusion occurs after 80 secondsOcclusion occurs after 80 seconds





Easier than MATLABEasier than MATLAB









OcclusionOcclusion PerfusionPerfusion



The bottom plot represents local minimum value for the accelerometer streamThe bottom plot represents local minimum value for the accelerometer stream

SELECT edv(y), min(y) FROM Accelerometer TRIGGER WINDOW BY QRSD(ECG.value)

SELECT edv(y), min(y) FROM Accelerometer TRIGGER WINDOW BY QRSD(ECG.value)

Query with added local minimum value => easy to change! Query with added local minimum value => easy to change!

Conclusion

• DSMSs can be used for real-time analysis => easy for medical practitioners to investigate novel methods

• Illustrated a method of online analysis of medical sensor data focusing on detection of myocardial ischemia

• Added beat-to-beat detection by using ECG • Results with less variance

• Introduced a new type of window for DSMSs: Variable length triggered tumbling windows


Future Work

• Further study / integration of variable length triggered tumbling windows

• Introduce operation for stream synchronization• Delay Tolerant Streaming Services (DT-STREAM)

• Motivation: Allow audio visual services (AV) and medical sensor streaming in emergency scenarios with partial or non-existing communication infrastructure

• Network model: Sparse MANET (Mobile Ad-hoc Network), in worst case a Delay Tolerant Network (DTN)

• Handle disruptions / high delays, exploit / predict mobility

• Funding: Norwegian Research Council (Proj. 183312/S10)


References[1] Elle, O. J., Halvorsen, S., Gulbrandsen, M. G., Aurdal, L., Bakken, A., Samset, E.,

Dugstad H. and Fosse, E. ”Early recognition of regional cardiac ischemia using a 3-axis accelerometer sensor.“ Institute of Physics Publishing. Physiological Measurement, Volume 26, Number 4, August 2005, pp. 429-440

[2] Hamilton, P. S. and Tompkins, W. J. “Quantitive Investigation of QRS Detection Rules Using the MIT/BIH Arrhythmia Database.” IEEE Transactions on Biomedical Engineering, Volume BME-33, Number 12, December 1986, pp. 1157-1165

[3] “Esper.” Available online: http://esper.codehaus.org/[4] Brian Babcock, Shivnath Babu, Mayur Datar, Rajeev Motwani, Jennifer

Widom: "Models and issues in data stream systems" i PODS '02: Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, 2002. ISBN: 1-58113-507-6

[5] Lukasz Golab, M. Tamer Ozsu: "Issues in data stream management" i SIGMOD Rec., Volume 32, No. 2, 2003. ISBN: 0163-5808.

[6] “Complex Event Processing.” Web page: http://www.complexevents.com/


http://esper.codehaus.org/

http://www.complexevents.com/

Deviation Detection in Automated Home Care using CommonSens

Jarle Søberg, Vera Goebel, Thomas PlagemannDepartment of Informatics, University of Oslo

Automated Home Care

Multimodal heterogeneous sensors deployed in a home

Real-time detection of activities of daily living (ADLs)

Behavioural patterns of monitored person in given locations in the home

Taking medications Preparing meals

Interpret ADLs as spatial complex events Con: Requires substantial domain knowledge

Possibly many rules Pro: Many similarities

Only differences are structures of homes, sensor placement and persons

Challenges & Requirements

• How to integrate the different sensors?• How to model their abilities … (semantics)• Application developers, e.g., home care:

• Domain knowledge• Different environments• Different installations

• Requirements:• Support for placement and coverage area

calculation• Re-use & easy personalization• Avoid to address particular sensors

• Capabilities• Locations-of-Interests (LoI)

22. april 2023 36

Outline

• Part 1: Core concepts in CommonSens• Event model • Environment model• Sensor model• Summary

• Part 2: Deviations• What are deviations wrt. sensor readings• Query processing• Evaluation

• Conclusions22. april 2023 37

CommonSens Foundations Abstract away the differences Models

Events and queries Sensor

Capabilities: state variables the sensor can observe Coverage area

Environments Objects Permeability affects coverage areas Locations of interest (LoIs)

LoI

Event model

• An event is something that happens• Related work splits events into several different types of events

• Atomic events (e.g. Atrey 2009)• The smallest type of event• Can have one timestamp (transient)• Can have duration• Are modeled as tuples

• Compound/composite events• A composition of two or more atomic events (Atrey et

al. 2006)• Can be consecutive and concurrent• Can have causality properties

Depends on the level of abstraction

Events: a Core Concept for Applications

22. april 2023 40

Real World:States & state transitions

Real World:States & state transitions

SensorsSensors

EventsEvents

QueriesQueries

Describe events

Detect events

Event: (stove == ON) AND (monitored person leaves home)

Deviation: (take medicine; 8:00 – 10:00)

Event model (cont.)

• Area of Interest• A place in the home where interesting events

happen, or should happen• Granularity can vary

• Room• In front of the cooker• At the toilet• The light switch in the living room

• Also decides what sort of information we are interested in

Environment model

• We want to model objects that make up the environment

• Introducing building blocks• Related work is 2D/3D modeling of

environments• We have extracted the elements we need to match

our requirements

• Shape• A set of triplets that are

coordinates• A function that describes the shape, if possible

),,( aaa zyx

Environment model (cont.)

• Permeability• A value between 0 and 1• What signal type this permeability is valid for

• Object• A set of permeability tuples• A shape• Identifiers

• Which sensors can detect it

Environment model (cont.)

• Environment• A set of objects and how they are related to

each other

Sensor model• Tuple sources:

• A/D conversion physical sensors

• Data aggregation logical sensors

• Stored data external sources

• Capabilities describe what a tuple source can provide (identified by labels)

• A tuple source can provide one to several labels

• Different types of tuple sources, e.g. a camera can provide images, videos, and motion detection

• A label can have one to several tuple sources that provides it

• Same tuple source types but different manufacturers

Capabilities and Multimodality

Sensor capabilities are abstractions Allow multimodality

One capability provided by many different sensors, e.g. DetectMotion

Motion detectors Cameras with motion detector software

One sensor can provide many capabilities, e.g. cameras combined with software

DetectMotion DetectObject DetectPerson RecogniseFace ...

Sensor Placement and Query Instantiation Queries define spatial events using capabilities

and locations of interest Use multiple sensors to approximate the

locations of interest Two sets created during query instantiation: Isec(loi)

and NoIsec(loi)

FPProb(loi) gives probability for false positives based on Isec and NoIsec




and NoIsec(loi)


LoI




and NoIsec(loi)


LoI




and NoIsec(loi)


LoI




and NoIsec(loi)


LoI




and NoIsec(loi)


LoI




and NoIsec(loi)


LoI

Core Idea

22. april 2023 54







Core Idea (cont.)

22. april 2023 55








Sensor placement



A priori

Core Idea (cont.)

22. april 2023 56








Sensor placement



Data gathering

Evaluation

A priori

Runtime, i.e., event processing

Outline

• Part 1: Core concepts in CommonSens• Event model • Environment model• Sensor model• Summary

• Part 2: Deviations• What are deviations wrt. sensor readings• Query processing• Evaluation

• Conclusions22. april 2023 57

What are deviations in the system?

22. april 2023 58

• Set of queries instantiated in CommonSens• State variables that need to be monitored• Deviation of a query := N – E

Query processing in CommonSens

• IqC2 = IqA1 → during(IqA2 → IqA4, Iqa3 → IqA5) → IqA6

22. april 2023 59

Evaluation

• Functionality test with synthetic traces

22. april 2023 60

[(DetectPerson==Person1,LoI1,10,max50%)->(DetectPerson==Person1,LoI2,10,min50%)->(DetectPerson==Person1,LoI3,21,30,max50%)->(DetectPerson==Person1,LoI4,31,40,min50%)]

[during([(DetectPerson==Person1,LoI5)],[(DetectPerson==Person1,LoI6)])]

Evaluation (cont.)

• Functionality test with real world traces from Cook and Schmitter-Edgecombe [3]:

• Several sensors are placed inside a home, subjects are told to follow given patterns for phone use: look up a number in a phone book that is on the shelf, call the number, and write down the cooking directions given on the recorded message, put the phone book back.

22. april 2023 61

[during([(DetectPhoneBookPresent==ABSENT)->(PhoneUsage==START)->(PhoneUsage==END)->(DetectPhoneBookPresent==PRESENT)],[(DetectMotion==ON,ByTheTelephone,119s,min 10%)])]

Conclusions

• CommonSens:• Sensor model, physical & logical sensors• Environment model, permeability• Event model, location of interest

• Deviation ≠ “everything else”

• Future work:• Apply it in real settings• Quality of Information• Adding control and action

22. april 2023 62

Questions?

22. april 2023 63

Coverage Area Calculation

22. april 2023 64

Scalability (#Sensors)

22. april 2023 65

2 sensors22 sensors42 sensors62 sensors82 sensors

Scalability (#Concurrent Queries)

22. april 2023 66

Documents

Programmer friendly abstractions for in- and on-body sensing Thomas Plagemann