View
306
Download
2
Category
Preview:
DESCRIPTION
(PhD Thesis presentation) The use of wearable or on-body sensors to monitor the human behavior is now on the forefront of human activity recognition. Nevertheless, the actual results for human activity recognition are fairly constrained and generally restricted to ideal or laboratory scenarios. Activity recognition systems are designed to comply with ideal conditions and are of limited utility in realistic domains. To become real-world applicable, activity recognition systems must satisfy operational and quality requirements that pose complex challenges, most of which have been sparsely and vaguely investigated to date. Classic activity recognition systems assume that the sensor setup remains identical during the lifelong use of the system. However, in users' daily life, sensors may fail, run out of battery, be misplaced or experience topological variations. These changes may lead to significant variations in the sensor measurements with respect to the default case. Consequently, activity recognition systems devised for ideal conditions may react in an undesired manner to imperfect, unknown or anomalous sensor data. This potentially translates into a partial or total malfunctioning of the activity recognition system. In this thesis, novel expert systems are proposed to address the challenges of making activity recognition systems functional in real-world scenarios. An innovative methodology, the hierarchical weighted classifier, that leverages the potential of multi-sensor configurations, is defined to overcome the effects of sensor failures and faults. This approach proves to be as valid as other standard activity recognition models in ideal conditions while outperforming them in terms of robustness to sensor failure and fault-tolerance. This methodology also shows outstanding capabilities to assimilate sensor deployment anomalies motivated by the user self-placement of the sensors. Furthermore, a novel multimodal transfer learning method that operates at runtime, with low overhead and without user or system designer intervention is developed. This approach serves to automatically translate activity recognition capabilities from an existing system to an untrained system even for different sensor modalities. This is of key interest to support sensor replacements as part of equipment maintenance, sensor additions in system upgrades and to benefit from sensors that happen to be available in the user environment. The potential of these advanced expert models leads to new research directions such as autonomous systems self-configuration, auto-adaptation and evolvability in activity recognition. Thus, this thesis opens-up a new range of opportunities for activity recognition systems to operate in real-world scenarios. Work described in the following dissertation: Banos, O.: Robust Expert Systems for more Flexible Real-World Activity Recognition. Ph.D. Thesis, University of Granada, Granada (SPAIN) (2014)
Citation preview
Robust Expert Systems for more Flexible Real-World
Activity Recognition
Granada, Friday, April 25, 2014
Presented by: Oresti Baños
Supervised by: Miguel Damas, Héctor Pomares and Ignacio Rojas Department of Computer Architecture and Computer Technology,
CITIC-UGR, University of Granada, SPAIN
Human Activity
2
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Health
Abnormal behavior detection
Proactive Assistance
Labour risk prevention
Wellness
Sports
Gaming
Human Activity
• Why is identifying human activity interesting?
3
REPORT (AWARENESS, MOTIVATIONAL,…) - USER - THIRD PARTIES ACT ON OUR BEHALF (TRIGGER ALERTS, TURN THE LIGHTS ON,…)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Activity Recognition (AR)
• Activity recognition concept
“Recognize the actions and goals of one or more agents from a series of observations on the agents' actions and the environmental conditions”
• Activity recognition process
4
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Ambient camera y micro pero predominan los wearables: se pueden usar outdoor, directamente en el cuerpo, privacidad… Hay varias formas, fuera indoor
Phenomena
Human activity (body motion)
Measurement
Sensing (ambient/wearables)
Processing
Data adequation and knowledge
inference
Recognized Activity
Wearable Activity Recognition
5
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
The major limitation within the use of wearable computing has been obtrusiveness… Technology miniaturization has helped to surpass this issue as we are withnessing during the last months. Each day we see a new gadget which further promises full recognition capabilities… but is it true?
• Wearable activity recognition systems are ready!
The first system capable of fully
recognize your daily routine.
AtlasWearables (2014)
The simplest way to understand
your day and night.
Jawbone Up (2014)
The best activity tracker on the
market. Fitbit Force (2014)
The device that tracks your active
life and measures all kind of
activities. Nike Fuel (2014)
Wearable Activity Recognition
6
Is wearable computing mature enough? News: - http://mobihealthnews.com/31697/sur
vey-one-third-of-wearable-device-owners-stopped-using-them-within-six-months/
- http://nothingtosay.com/2013/11/wearable-gadgets-of-various-and-dubious-value/
- http://www.technologyreview.com/news/521931/fitness-trackers-still-need-to-work-out-kinks/
- http://www.elmundo.es/blogs/elmundo/el-gadgetoblog/2014/03/12/pulseras-inteligentes-no-tanto.html
- http://well.blogs.nytimes.com/2014/03/10/activity-trackers-dont-sense-everything/?_php=true&_type=blogs&smid=tw-nytimes&_r=0
- http://www.crowdfundinsider.com/2014/04/35002-indiegogos-reputation-may-hinge-outcome-gobe-campaign/
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
The major limitation within the use of wearable computing has been obtrusiveness… Technology miniaturization has helped to surpass this issue as we are withnessing during the last months. Each day we see a new gadget which further promises full recognition capabilities… but is it true?
• But… do wearable activity recognition systems meet people’s expectance?
SO YOU’RE TELLING ME
A BRACELET CAN TRACK MY COMPLETE ACTIVITY
LONGLIFE?
Challenges for Real-World Activity Recognition
• Actively investigated:
– Reliability
– Simplicity
– Latency
• Barely addressed:
– Privacy
– Fault-tolerance
– Usability
– Unobtrusiveness
– Fashionability
– Self-configuration
– Auto-adaptation
– Evolvability
7
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Challenges for Real-World Activity Recognition
8
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Buscar imagen de fault-tolerance
• Actively investigated:
– Reliability
– Simplicity
– Latency
• Barely addressed:
– Privacy
– Fault-tolerance
– Usability
– Unobtrusiveness
– Fashionability
– Self-configuration
– Auto-adaptation
– Evolvability
Thesis Motivation and Objectives
• Motivation:
“Create more advanced systems capable of handling real-world AR issues as well as to incorporate more intelligent capabilities to transform experimental prototypes into actual usable applications”
• Objectives:
– O1: “Investigate the tolerance of standard AR systems to unforeseen sensor failures and faults, as well as contribute with an alternate approach to cope with these technological anomalies” Fault-tolerance
– O2: “Research the robustness of standard AR systems to unforeseen variations in the sensor deployment, as well as contribute with an alternate approach to cope with these practical anomalies” Usability, Unobtrusiveness
– O3: “Study the capacity of standard AR systems to support unforeseen changes in the sensor network, as well as contribute with an alternate approach to cope with these topological variations” Self-configuration, auto-adaptation, evolvability
9
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Activity Recognition Process
10
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Phenomena
Human activity (body motion)
Measurement
Sensing (ambient/wearables)
Processing
Data adequation and knowledge
inference
Recognized Activity
How does it work exactly?
Activity Recognition Process
11
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
The Activity Recognition Chain (ARC)
Phenomena
Human activity (body motion)
Measurement
Sensing (ambient/wearables)
Processing
Data adequation and knowledge
inference
Recognized Activity
Activity Recognition Chain (ARC)
12
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Activity Recognition Chain (ARC)
13
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
During the acquisition maybe explain accelerometers and how they work (video)
Activity Recognition Chain (ARC)
14
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Activity Recognition Chain (ARC)
15
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Colador y pan, OUT! Cambiar por otro…
Activity Recognition Chain (ARC)
16
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Activity Recognition Chain (ARC)
17
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Activity Recognition Chain (ARC)
18
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Tolerance of AR systems to sensor faults and failures
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Objective: “Investigate the tolerance of standard AR systems to unforeseen sensor failures and faults, as well as contribute with an alternate approach
to cope with these technological anomalies”
Problem Statement
20
SENSOR ERRORS
Are standard activity recognition systems prepared to cope with sensor technological anomalies?
Is it possible to keep the systems functioning under the effects of sensor errors?
Activity recognition process
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Body motion sensing
Signal processing and
reasoning
Recognition of activities
Signal effects
Sensor Technological Anomalies
21
• Faults (overheating, environmental changes, decalibration)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
• Failures (outages, breakdowns, disconnection, battery depletion)
Necesidad de SotA?
Sensor Technological Anomalies in AR: Related Work
• Detection of sensor anomalies
– Sensor query (Rost06)
– Neighborhood data correlation
• Signal level (Yao10)
• Feature level (Ramanathan09)
• Reasoning (Rajasegarar07, Ganeriwal08)
• Counteraction of sensor anomalies
– Data imputation (Uchida13)
– Sensor fusion (Sagha13)
S. Rost and H. Balakrishnan. Memento: A health monitoring system for wireless sensor networks. In 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, volume 2, pp. 575-584, 2006. Y. Yao, A. Sharma, L. Golubchik, and R. Govindan. Online anomaly detection for sensor systems: A simple and efficient approach. Performance Evaluation, 67(11):1059-1075, November 2010. N. Ramanathan, T. Schoellhammer, E. Kohler, K. Whitehouse, T. Harmon, and D. Estrin. Suelo: human-assisted sensing for exploratory soil monitoring studies. In Proceedings of the 7th ACM Conference on Embedded Networked Sensor Systems, pp. 197-210, 2009. S. Rajasegarar, C. Leckie, M. Palaniswami, and J. C. Bezdek. Quarter sphere based distributed anomaly detection in wireless sensor networks. In IEEE International Conference on Communications, pp. 3864-3869, June 2007. S. Ganeriwal, L. K. Balzano, and M. B. Srivastava. Reputationbased framework for high integrity sensor networks. ACM Transaction on Sensor Networks, 4(3):1-37, June 2008. R. Uchida, H. Horino, and R. Ohmura. Improving fault tolerance of wearable wearable sensor-based activity recognition techniques. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 633-644, 2013. H. Sagha, H. Bayati, J. del R. Millan, and R. Chavarriaga. On-line anomaly detection and resilience in classifier ensembles. Pattern Recognition Letters, 34(15):1916-1927, 2013
22
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor Failures in Classic AR Systems
• Single-sensor ARC (SARC)
23
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
• Single-sensor ARC (SARC)
24
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
If the sensor fails, the complete system fails
Solution Use more sensors for redundancy
(multi-sensor ARC or MARC)
Sensor Failures in Standard AR Systems
• Feature fusion multi-sensor ARC (FFMARC)
25
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor Failures in Standard AR Systems
• Feature fusion multi-sensor ARC (FFMARC)
26
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
If a sensor fails, the complete system fails
Solution Independent ARCs + decision fusion
Sensor Failures in Standard AR Systems
• Decision fusion multi-sensor ARC (DFMARC)
27
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor Failures in Standard AR Systems
• Decision fusion multi-sensor ARC (DFMARC)
28
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
If a sensor fails, the system is still capable of functioning
but…
Is it capable of recognition?
Sensor Failures in Standard AR Systems
• Hierarchical decision (HD)
– Information from some sensors more valuable than from others (e.g., body part for a certain activity) Ranking of decisions
– Decisions mainly made on top (recognition relies on a sensor or few sensors) Problem when top-ranked sensors get unavailable
• Majority voting (MV)
– Equality scheme (all sensors have the same importance) Fairness, decisiveness
– A plurality of weak decisors may prevail over the rest Tyranny of the majority
29
Sensor Failures in Standard AR Systems
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
…
c11,c12,…,c1k
c1=φ(c11,c21,…, cM1), c2=φ(c12,c22,…, cM2),
… ck=φ(c1k,c2k,…, cMk)
c21,c22,…, c2k
cM1,cM2,…,cMk
…
…
…
…
…
…
c11,c12,…,c1k
c1=φ(c11,c21,…, cM1), c2=φ(c12,c22,…, cM2),
… ck=φ(c1k,c2k,…, cMk)
c21,c22,…, c2k
cM1,cM2,…,cMk
…
A Novel Method: Hierarchical Weighted Classifier
30
Sensor M
Sensor 2
SM
S2
S1 α11
Ψ
C12
C1N
C11
Ψ
C21
C22
C2N
Ψ
CM1
CM2
CMN
Ψ
Decisio
n
Activity level
(base classifier)
Sensor level
(sensor classifier)
Network level
(sensor fusion)
β11
α12 β12
α1N β1N
α21 β21
α22 β22
α2N β2N
αM1 βM1
αM2 βM2
αMN βMN
γ11,…,1N δ11,…,1N
γ21,…,2N δ21,…,2N
γM1,…,MN δM1,…,MN
Sensor 1
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
31
Sensor M
Sensor 2
SM
S2
S1 α11
Ψ
C12
C1N
C11
Ψ
C21
C22
C2N
Ψ
CM1
CM2
CMN
Ψ
Decisio
n
Activity level
(base classifier)
Sensor level
(sensor classifier)
Network level
(sensor fusion)
β11
α12 β12
α1N β1N
α21 β21
α22 β22
α2N β2N
αM1 βM1
αM2 βM2
αMN βMN
γ11,…,1N δ11,…,1N
γ21,…,2N δ21,…,2N
γM1,…,MN δM1,…,MN
Sensor 1
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
A Novel Method: Hierarchical Weighted Classifier N activities & M sensors
32
Sensor M
Sensor 2
SM
S2
S1 α11
Ψ
C12
C1N
C11
Ψ
C21
C22
C2N
Ψ
CM1
CM2
CMN
Ψ
Decisio
n
Activity level
(base classifier)
Sensor level
(sensor classifier)
Network level
(sensor fusion)
β11
α12 β12
α1N β1N
α21 β21
α22 β22
α2N β2N
αM1 βM1
αM2 βM2
αMN βMN
γ11,…,1N δ11,…,1N
γ21,…,2N δ21,…,2N
γM1,…,MN δM1,…,MN
Sensor 1
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
A Novel Method: Hierarchical Weighted Classifier N activities & M sensors
S1 α11
Ψ
C12
C1N
C11 β11
α12 β12
α1N β1N
𝑞𝑚 𝑥𝑚𝑘= 𝑎𝑟𝑔𝑚𝑎𝑥
𝑞
𝑂𝑚 𝑥𝑚𝑘
𝑂𝑚 𝑥𝑚𝑘= 𝑊𝐷𝑚𝑛 𝑥𝑚𝑘
𝑁
𝑛=1
𝛼𝑚𝑛 =𝑇𝑃𝑚𝑛
𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛
𝛽𝑚𝑛 =𝑇𝑁𝑚𝑛
𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛
𝑊𝐷𝑚𝑛 𝑥𝑚𝑘=
𝛼𝑚𝑛, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
0, 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
∀𝑞 = 𝑛
𝛽𝑚𝑛 , 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
0, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
∀𝑞 ≠ 𝑛
33
Sensor M
Sensor 2
SM
S2
S1 α11
Ψ
C12
C1N
C11
Ψ
C21
C22
C2N
Ψ
CM1
CM2
CMN
Ψ
Decisio
n
Activity level
(base classifier)
Sensor level
(sensor classifier)
Network level
(sensor fusion)
β11
α12 β12
α1N β1N
α21 β21
α22 β22
α2N β2N
αM1 βM1
αM2 βM2
αMN βMN
γ11,…,1N δ11,…,1N
γ21,…,2N δ21,…,2N
γM1,…,MN δM1,…,MN
Sensor 1
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
A Novel Method: Hierarchical Weighted Classifier N activities & M sensors
S1 α11
Ψ
C12
C1N
C11 β11
α12 β12
α1N β1N
S1 γ11,…,1N δ11,…,1N
𝑊𝐷𝑚𝑛 𝑞𝑚 𝑥𝑚𝑘=
𝛾𝑚𝑛, 𝑞𝑚 𝑥𝑚𝑘= 𝑛
𝛿𝑚𝑛 , 𝑞𝑚 𝑥𝑚𝑘≠ 𝑛
𝛾𝑚 = 𝛾𝑚1, 𝛾𝑚2, … , 𝛾𝑚𝑛 =𝑇𝑃𝑚1
𝑇𝑃𝑚1 + 𝐹𝑁𝑚1,
𝑇𝑃𝑚2
𝑇𝑃𝑚2 + 𝐹𝑁𝑚2, … ,
𝑇𝑃𝑚𝑛
𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛
𝑞𝑚 𝑥𝑚𝑘= 𝑎𝑟𝑔𝑚𝑎𝑥
𝑞
𝑂𝑚 𝑥𝑚𝑘
𝑂𝑚 𝑥𝑚𝑘= 𝑊𝐷𝑚𝑛 𝑥𝑚𝑘
𝑁
𝑛=1
𝛼𝑚𝑛 =𝑇𝑃𝑚𝑛
𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛
𝛽𝑚𝑛 =𝑇𝑁𝑚𝑛
𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛
𝑊𝐷𝑚𝑛 𝑥𝑚𝑘=
𝛼𝑚𝑛, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
0, 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
∀𝑞 = 𝑛
𝛽𝑚𝑛 , 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
0, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
∀𝑞 ≠ 𝑛
𝛿𝑚 = 𝛿𝑚1, 𝛿𝑚2, … , 𝛿𝑚𝑛 =𝑇𝑁𝑚1
𝑇𝑁𝑚1 + 𝐹𝑃𝑚1,
𝑇𝑁𝑚2
𝑇𝑁𝑚2 + 𝐹𝑃𝑚2, … ,
𝑇𝑁𝑚𝑛
𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛
∀𝑛 = 1, … , 𝑁
34
Sensor M
Sensor 2
SM
S2
S1 α11
Ψ
C12
C1N
C11
Ψ
C21
C22
C2N
Ψ
CM1
CM2
CMN
Ψ
Decisio
n
Activity level
(base classifier)
Sensor level
(sensor classifier)
Network level
(sensor fusion)
β11
α12 β12
α1N β1N
α21 β21
α22 β22
α2N β2N
αM1 βM1
αM2 βM2
αMN βMN
γ11,…,1N δ11,…,1N
γ21,…,2N δ21,…,2N
γM1,…,MN δM1,…,MN
Sensor 1
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
A Novel Method: Hierarchical Weighted Classifier
S1 α11
Ψ
C12
C1N
C11 β11
α12 β12
α1N β1N
S1 γ11,…,1N δ11,…,1N
Ψ
Decisio
n
N activities & M sensors
𝑞 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑞
𝑂 𝑥𝑚
𝑂 𝑥𝑚 = 𝑂 𝑥1𝑘, 𝑥2𝑘, … , 𝑥𝑀𝑘= 𝑊𝐷𝑝 𝑞𝑝 𝑥𝑝𝑘
𝑀
𝑝=1
𝑊𝐷𝑚𝑛 𝑞𝑚 𝑥𝑚𝑘=
𝛾𝑚𝑛, 𝑞𝑚 𝑥𝑚𝑘= 𝑛
𝛿𝑚𝑛 , 𝑞𝑚 𝑥𝑚𝑘≠ 𝑛
𝛾𝑚 = 𝛾𝑚1, 𝛾𝑚2, … , 𝛾𝑚𝑛 =𝑇𝑃𝑚1
𝑇𝑃𝑚1 + 𝐹𝑁𝑚1,
𝑇𝑃𝑚2
𝑇𝑃𝑚2 + 𝐹𝑁𝑚2, … ,
𝑇𝑃𝑚𝑛
𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛
𝑞𝑚 𝑥𝑚𝑘= 𝑎𝑟𝑔𝑚𝑎𝑥
𝑞
𝑂𝑚 𝑥𝑚𝑘
𝑂𝑚 𝑥𝑚𝑘= 𝑊𝐷𝑚𝑛 𝑥𝑚𝑘
𝑁
𝑛=1
𝛼𝑚𝑛 =𝑇𝑃𝑚𝑛
𝑇𝑃𝑚𝑛 + 𝐹𝑁𝑚𝑛
𝛽𝑚𝑛 =𝑇𝑁𝑚𝑛
𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛
𝑊𝐷𝑚𝑛 𝑥𝑚𝑘=
𝛼𝑚𝑛, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
0, 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
∀𝑞 = 𝑛
𝛽𝑚𝑛 , 𝑥𝑚𝑘 𝑛𝑜𝑡 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
0, 𝑥𝑚𝑘 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑 𝑎𝑠 𝑞
∀𝑞 ≠ 𝑛
𝛿𝑚 = 𝛿𝑚1, 𝛿𝑚2, … , 𝛿𝑚𝑛 =𝑇𝑁𝑚1
𝑇𝑁𝑚1 + 𝐹𝑃𝑚1,
𝑇𝑁𝑚2
𝑇𝑁𝑚2 + 𝐹𝑃𝑚2, … ,
𝑇𝑁𝑚𝑛
𝑇𝑁𝑚𝑛 + 𝐹𝑃𝑚𝑛
∀𝑛 = 1, … , 𝑁
Evaluation of the Tolerance to Sensor Technological Anomalies
• Model validation
– Performance in ideal circumstances
– Tolerance to sensor failures
– Tolerance to sensor faults
35
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
• Benchmark dataset:
– MIT Activities of Daily Living Dataset*
• 9 acts
• 5 biaxial accelerometers
• 20 subjects (17-48 years old)
• Out-of-lab
• Experimental setup:
36
Five
acc
eler
om
eter
s
Walking Sitting and relaxing Standing still Running Bicycling Lying down Brushing teeth Climbing stairs
36
Eig
ht
act
ivit
ies
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
5 biaxial accelerometers
(limbs and trunk)
LP Elliptic Filter (Fc = 20Hz)
6 seconds sliding window
Mean, STD, kurtosis, MCR,
(...)
DT, NB, KNN, SVM (as base
classifiers)
Evaluation of the Tolerance to Sensor Technological Anomalies
* http://architecture.mit.edu/house_n/data/Accelerometer/BaoIntille.htm
• Performance in ideal circumstances:
37
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
Evaluated models: - SARC (≡ S) - FFMARC (≡ FF) - HWC
Parameters: - Feature sets: 1, 5,
10, 20 feat. - Base classifiers: DT,
NB, KNN, SVM Evaluation procedure: - 10-fold CV - 100 iterations
Las figuras irán a color
• Performance in ideal circumstances:
38
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
Evaluated models: - SARC (≡ S) - FFMARC (≡ FF) - HWC
Parameters: - Feature sets: 1, 5,
10, 20 feat. - Base classifiers: DT,
NB, KNN, SVM Evaluation procedure: - 10-fold CV - 100 iterations
Las figuras irán a color
• Performance in ideal circumstances:
39
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
Evaluated models: - SARC (≡ S) - FFMARC (≡ FF) - HWC
Parameters: - Feature sets: 1, 5,
10, 20 feat. - Base classifiers: DT,
NB, KNN, SVM Evaluation procedure: - 10-fold CV - 100 iterations
Las figuras irán a color
• Tolerance to sensor failures:
40
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
Evaluated model: - HWC
Parameters: - Feature set: 10
feat. - Classifier: KNN
Evaluation procedure: - 10-fold CV - 100 iterations
Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)
Baseline Accuracy (Ideal conditions) =
96.34%
• Tolerance to sensor failures:
41
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
1 missing sensor
Evaluated model: - HWC
Parameters: - Feature set: 10
feat. - Classifier: KNN
Evaluation procedure: - 10-fold CV - 100 iterations
Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)
Baseline Accuracy (Ideal conditions) =
96.34%
• Tolerance to sensor failures:
42
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
1 missing sensor
2 missing sensors
Evaluated model: - HWC
Parameters: - Feature set: 10
feat. - Classifier: KNN
Evaluation procedure: - 10-fold CV - 100 iterations
Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)
Baseline Accuracy (Ideal conditions) =
96.34%
• Tolerance to sensor failures:
43
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
1 missing sensor
2 missing sensors
3 missing sensors
4 missing sensors
Evaluated model: - HWC
Parameters: - Feature set: 10
feat. - Classifier: KNN
Evaluation procedure: - 10-fold CV - 100 iterations
Legend: H (hip), W (wrist), A (arm), K (ankle), T (thigh)
Baseline Accuracy (Ideal conditions) =
96.34%
• Tolerance to sensor faults:
44
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
Ideal case Dynamic range shortening
• Tolerance to sensor faults:
45
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Evaluation of the Tolerance to Sensor Technological Anomalies
Evaluated models: - SARC - FFMARC - HWC
Parameters: - Feature set: 10 feat. - Classifier: KNN Evaluation procedure: - 10-fold CV - 100 iterations
AR model/ #faulty sensors 0 1 2 3 4 5
New dynamic range = 30% original dynamic range [-3g,3g]
SARC (hip) 82±5 66±4 - - - - SARC (wrist) 88±5 54±6 - - - - SARC (arm) 80±3 58±7 - - - -
SARC (ankle) 83±4 58±8 - - - - SARC (thigh) 89±2 72±4 - - - -
FFMARC 97±2 88±4 76±5 61±8 42±11 39±13 HD 90±3 85±4 80±9 68±13 59±16 53±20 MV 82±6 79±5 67±7 43±10 36±14 31±19
HWC 96±2 96±2 93±3 86±5 73±8 65±14 New dynamic range = 10% original dynamic range [-1g,1g]
SARC (hip) 82±5 21±11 - - - - SARC (wrist) 88±5 18±9 - - - - SARC (arm) 80±3 26±14 - - - -
SARC (ankle) 83±4 21±7 - - - - SARC (thigh) 89±2 20±6 - - - -
FFMARC 97±2 70±5 41±8 17±15 21±11 18±9 HD 90±3 80±6 59±13 42±12 30±17 21±16 MV 82±6 77±6 46±11 38±10 27±13 26±8
HWC 96±2 94±2 87±6 53±2 27±17 25±19
Conclusions
• Assuming a lifelong invariant sensor setup is unrealistic and may lead to a malfunctioning of the activity recognition system
• Body-worn sensors are subject to faults (signal degradation) and failures (absence of signal) normally unforeseen at design and runtime
• Classic activity recognition approaches (SARC, FFMARC) are not capable of dealing with sensor failures and are of limited utility under the effect of sensor faults
• The proposed alternate model (HWC) renders similar performance to standard activity recognition models in ideal conditions, proves to be robust to sensor failures and a relevant tolerance to sensor faults
46
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Robustness of AR systems to sensor deployment variations
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Objective: “Research the robustness of standard AR systems to unforeseen variations in the sensor deployment, as well as contribute with an alternate
approach to cope with these practical anomalies”
Problem Statement
48
SENSOR DEPLOYMENT CHANGES
Are activity recognition systems flexible enough to
allow users to wear the sensors on their own?
Is it possible to keep the systems functioning under the effects of sensor displacement?
Activity recognition process
Body motion sensing
Signal processing and
reasoning
Recognition of activities
Body motion sensing
Signal processing and
reasoning
Recognition of activities
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor Displacement
• Categories of sensor displacement
– Static: position changes can remain static across the execution of many activity instances, e.g. when sensors are attached with a displacement each day
– Dynamic: effect of loose fitting of the sensors, e.g. when embedded into clothes
• Sensor displacement new sensor position signal space change
• Sensor displacement effects depends on
– Original/end position and body part
– Activity/gestures/movements performed
– Sensor modality
49
Sensor displacement = rotation + translation (angular displacement) (linear displacement)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor Displacement Effects
Changes in the signal space propagates through the activity recognition chain (e.g., variations in the feature space)
RCIDEAL LCIDEAL= LCSELF
50
RCSELF ≠ RCIDEAL
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor Displacement in AR: Related Work
• Features invariant to sensor displacement
– Heuristics (Kunze08)
– Genetic algorithm for feature selection (Förster09a)
• Feature distribution adaptation
– Covariate shift unsupervised adaptation (Bayati09)
– Online-supervised user-based calibration (Förster09b)
• Classification (dis)similarity
– Output classifiers correlation (Sagha11)
K. Kunze and P. Lukowicz. Dealing with sensor displacement in motion-based onbody activity recognition systems. In 10th international conference on Ubiquitous computing, pp. 20–29, 2008.
K. Förster, P. Brem, D. Roggen, and G. Tröster. Evolving discriminative features robust to sensor displacement for activity recognition in body area sensor networks. In Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP), 2009 5th International Conference on, pp. 43–48, 2009.
H. Bayati, J. del R Millan, and R. Chavarriaga. Unsupervised adaptation to on-body sensor displacement in acceleration-based activity recognition. In Wearable Computers (ISWC), 2011 15th Annual International Symposium on, pp. 71–78, June 2011.
K. Förster, D. Roggen, and G. Tröster. Unsupervised classifier self-calibration through repeated context occurrences: Is there robustness against sensor displacement to gain? In Proc. 13th IEEE Int. Symposium on Wearable Computers (ISWC), pp. 77–84, 2009.
H. Sagha, J. R. del Millán, and R. Chavarriaga. Detecting and rectifying anomalies in Opportunistic sensor networks 8th Int. Conf. on Networked Sensing Systems, pp. 162-–167, 2011
51
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Approaches to Investigate on Sensor Displacement
El enfoque debería ser más bien… ahora vamos a comprobar la validez de nuestro modelo para sensor displacement, de modo que las dos siguientes transparencias se dejarían fuera…
Synthetically Modeled
Sensor Displacement
Realistic Sensor
Displacement
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Cambiar el título a «Sensor Displacement Study» o algo así…
52
Approaches to Investigate on Sensor Displacement
El enfoque debería ser más bien… ahora vamos a comprobar la validez de nuestro modelo para sensor displacement, de modo que las dos siguientes transparencias se dejarían fuera…
Synthetically Modeled
Sensor Displacement
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
53
Synthetically Modeled Sensor Displacement
• Sensor rotation Rotational noise (RN)
• Sensor translation Additive noise (AN)
• Examples:
54
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
𝑀𝑅𝑁 =
𝑐 𝜃 𝑐 𝜓 −𝑐 𝜙 𝑠 𝜓 + 𝑠 𝜙 𝑠 𝜃 𝑐 (𝜓) 𝑠 𝜙 𝑠 𝜓 + 𝑐 𝜙 𝑠 𝜃 𝑐 (𝜓)
𝑐 𝜃 𝑠 𝜓 𝑐 𝜙 𝑐 𝜓 + 𝑠 𝜙 𝑠 𝜃 𝑠 (𝜓) −𝑠 𝜙 𝑐 𝜓 + 𝑐 𝜙 𝑠 𝜃 𝑠(𝜓)
− 𝑠 𝜃 𝑠 𝜙 𝑐 𝜃 𝑐 𝜙 𝑐 𝜃
𝑥𝑟𝑜𝑡𝑦𝑟𝑜𝑡𝑧𝑟𝑜𝑡
= 𝑀𝑅𝑁 ×
𝑥𝑟𝑎𝑤𝑦𝑟𝑎𝑤𝑧𝑟𝑎𝑤
𝑥𝑡𝑟𝑦𝑡𝑟𝑧𝑡𝑟
= 𝑇𝐴𝑁 +
𝑥𝑟𝑎𝑤𝑦𝑟𝑎𝑤𝑧𝑟𝑎𝑤
𝑇𝐴𝑁 = 𝜇𝐴𝑁 + 𝜎𝐴𝑁2 + 𝑟𝑎𝑛𝑑_𝑛𝑜𝑟𝑚𝑎𝑙_𝑑𝑖𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛 (𝜇𝐴𝑁=0)
0 1 2 3
-2
0
2
Acce
lera
tio
n (
g)
Time (s)
Original
0 1 2 3
-2
0
2
Time (s)
RN
=15º
0 1 2 3
-2
0
2
Time (s)
RN
=90º
0 1 2 3
-2
0
2
Time (s)
AN
=0.1g
0 1 2 3
-2
0
2
Time (s)
AN
=0.5g
0 1 2 3
-2
0
2
Acce
lera
tio
n (
g)
Time (s)
Original
0 1 2 3
-2
0
2
Time (s)
RN
=15º
0 1 2 3
-2
0
2
Time (s)
RN
=90º
0 1 2 3
-2
0
2
Time (s)
AN
=0.1g
0 1 2 3
-2
0
2
Time (s)
AN
=0.5g
0 1 2 3
-2
0
2
Acce
lera
tio
n (
g)
Time (s)
Original
0 1 2 3
-2
0
2
Time (s)
RN
=15º
0 1 2 3
-2
0
2
Time (s)
RN
=90º
0 1 2 3
-2
0
2
Time (s)
AN
=0.1g
0 1 2 3
-2
0
2
Time (s)
AN
=0.5g
0 1 2 3
-2
0
2
Acce
lera
tio
n (
g)
Time (s)
Original
0 1 2 3
-2
0
2
Time (s)
RN
=15º
0 1 2 3
-2
0
2
Time (s)
RN
=90º
0 1 2 3
-2
0
2
Time (s)
AN
=0.1g
0 1 2 3
-2
0
2
Time (s)
AN
=0.5g
Walking Sitting
Proposed in: H. Sagha, J. R. del Millán, and R. Chavarriaga. Detecting and rectifying anomalies in Opportunistic sensor networks. 8th Int. Conf. on Networked Sensing Systems, pp. 162 – 167, 2011
• Benchmark dataset:
– MIT Activities of Daily Living Dataset*
• 9 acts
• 5 biaxial accelerometers
• 20 subjects (17-48 years old)
• Out-of-lab
• Experimental setup:
55
Five
acc
eler
om
eter
s
Walking Sitting and relaxing Standing still Running Bicycling Lying down Brushing teeth Climbing stairs
55
Eig
ht
act
ivit
ies
Evaluation of the Robustness to Sensor Displacement (Synthetic)
* http://architecture.mit.edu/house_n/data/Accelerometer/BaoIntille.htm INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
5 biaxial accelerometers
(limbs and trunk)
LP Elliptic Filter (Fc = 20Hz)
6 seconds sliding window
10 feat. set KNN (as base
classifier)
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
• Performance drop under the effects of sensor rotation and translation:
56
Evaluated model: - SARC Parameters: - Feature set: 10 feat. - Base classifiers: KNN Evaluation procedure: - 10-fold CV - 100 iterations
Evaluation of the Robustness to Sensor Displacement (Synthetic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Ro
tati
on
Tr
ansl
atio
n
• Performance drop under the effects of sensor rotation and translation:
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
57
Evaluated model: - SARC Parameters: - Feature set: 10 feat. - Base classifiers: KNN Evaluation procedure: - 10-fold CV - 100 iterations
Evaluation of the Robustness to Sensor Displacement (Synthetic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
30%
Ro
tati
on
Tr
ansl
atio
n
23% 3%
8%
8%
20%
15% 25% 5%
15%
1%
4%
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
• Performance drop under the effects of sensor rotation and translation:
58
Evaluated model: - SARC Parameters: - Feature set: 10 feat. - Base classifiers: KNN Evaluation procedure: - 10-fold CV - 100 iterations
Evaluation of the Robustness to Sensor Displacement (Synthetic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
70% Ro
tati
on
Tr
ansl
atio
n
50% 20%
55%
6%
15% 3%
8%
8%
20%
15% 45%
75%
5%
15% 43%
30% 1%
4%
4%
10% 25%
30%
23%
Approaches to Investigate on Sensor Displacement
Realistic Sensor
Displacement
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
59
• No dataset for studying the effects of sensor displacement!
• Observe
– Variability introduced with respect to the ideal setup when the sensors are self-placed by the users
– Effects of large sensor displacements (extreme de-positioning)
• Scenarios
– Ideal-placement
– Self-placement
– Induced-displacement
Implementing Realistic Sensor Displacement
Ideal Self Induced
60
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
NEW DATASET* (REALDISP)
*Freely available at: www.ugr.es/~oresti/datasets
REALDISP Dataset: Study Setup
• Cardio-fitness room
• 9 IMUs (9DoF) ACC, GYR, MAG
• Laptop data storage and labeling*
• Camera offline data validation
• 17 volunteers (22-37 years old)
*Annotation tool: http://crnt.sourceforge.net/CRN_Toolbox/Home.html 61
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
REALDISP Dataset: Activity Set
• Activities intended for:
– Body-general motion: Translation | Jumps | Fitness
– Body-part-specific motion: Trunk | Upper-extremities | Lower-extremities
62
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
• Experimental setup:
• Studies:
– AR systems: SARC, FFMARC, HWC
– Settings: Ideal-placement, Self-placement, Induced-displacement
– Scenarios: 10 activities, 20 activities, 33 activities (all)
• Evaluation procedure
– 10-fold CV, 100 iterations
63
Evaluation of the Robustness to Sensor Displacement (Realistic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
9 triaxial accelerometers (all limbs and
trunk)
No preprocessing
(raw data)
6 seconds sliding window
FS1=mean
FS2=mean,std
FS3=mean,std,max,min,mcr
DT, KNN, NB (as base classifiers)
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
64
Evaluation of the Robustness to Sensor Displacement (Realistic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Idea
l Se
lf
Ind
uce
d
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
65
Evaluation of the Robustness to Sensor Displacement (Realistic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Idea
l Se
lf
Ind
uce
d
13%
25%
25%
45%
3%
13%
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
66
Evaluation of the Robustness to Sensor Displacement (Realistic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Idea
l Se
lf
Ind
uce
d
13%
25%
15%
50%
25%
45%
30%
45%
3%
13%
5%
15%
HWC (multi-sensor) FFMARC (multi-sensor) SARC (single sensor)
67
Evaluation of the Robustness to Sensor Displacement (Realistic)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Idea
l Se
lf
Ind
uce
d
13%
25%
15%
50% 50%
15% 25%
45%
30%
45% 45%
30%
3%
13%
5%
15%
5%
25%
Conclusions
• Classic activity-aware systems assume a predefined sensor deployment that further remains unchanged during runtime, which are not lifelike assumptions
• Body-worn inertial sensors are subject to deployment changes (displacement) in real-world contexts, potentially leading to signal variations with respect to ideal patterns
• Activity recognition systems proves to be more sensitive to sensor rotations than translations, specially when located on body parts of reduced mobility
• Standard models (SARC,FFMARC) suffer from a critical performance worsening when the sensors are largely depositioned or self-placed by the users
• The HWC significantly outperforms the tolerance of standard activity recognition models (up to 30%), effectively showing outstanding capabilities to assimilate the changes introduced during the self-placement of the sensors and to moderately overcome the situation of largely depositioned sensors
68
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Hablar de realistic, novel, innovatively… Use text below…
Supporting AR systems network changes: instruction of
newcomer sensors
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Objective: “Study the capacity of standard AR systems to support unforeseen changes in the sensor network, as well as contribute with an
alternate approach to cope with these topological variations”
Problem Statement
70
SENSOR INFRASTRUCTURE CHANGES
- Sensor replacement, buys a new gadget, uses a different device depending on the context
- System without recognition capabilities, or at least not the ones desired onwards…
- New dataset must be collected, user involvement,… ?
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Collect a training dataset
Train and test the model
The AR system is “ready”
Collect a training dataset
Train and test the model
The AR system is “ready”
Do we need to collect a new dataset each time
the sensor topology changes?
Is it possible to leverage the knowledge of a functional system to instruct a system to
operate on a newcomer sensor?
Activity recognition system design
Infraestructure Changes: Newcomer Sensors
71
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor replacement (repair/upgrade)
Sensor addition (redundancy)
Sensor discovery (opportunistic use)
Transfer learning
Instruction of Newcomer Sensors
72
Teacher
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Classic approach
Limitations: - Predefined setup and deployments - System designer involvement - User/s involvement
Learner
“Mechanism, ability or means to recognize and apply knowledge and skills learned in previous tasks or domains to novel tasks or domains”
Collection of a new dataset for each possible scenario
Transfer Learning in AR: Related Work
• Transfer between wearable sensors
– Translation of locomotion recognition capabilities (Calatroni11)
• Model parameters
• Labels
• Transfer between ambient sensors
– Translation among smart homes through meta-featuring (van Kasteren10)
• Common meta-feature space
• Limitations
– Long time scales operation
– Incomplete transfer
– Difficult transfer across modalities
A. Calatroni, D. Roggen, and G. Tröster, “Automatic transfer of activity recognition capabilities between body-worn motion sensors: Training newcomers to recognize locomotion,” in Proc. 8th Int Conf on Networked Sensing Systems, 2011. T. van Kasteren, G. Englebienne, and B. Kröse, “Transferring knowledge of activity recognition across sensor networks,” in Proc. 8th Int. Conf on Pervasive Computing, 2010.
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
73
Multimodal Transfer Methods
• System identification (signal level)
• Transfer methods (reasoning level)
Ψ𝐴→𝐵 𝑡 Sensor Domain
A
Sensor Domain
B
Activity Templates
Activity Models
0 1 2 3-0.5
0
0.5
1
1.5
Time (s)
Accele
ration (
G)
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of activity models (features + labels, classification models)
Transfer of activity templates (patterns + labels)
74
Transfer of Activity Templates
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
• Transfer of the recognition capabilities of an existing source system (S) that operates on activity templates (patterns) to an untrained target system (T) that lacks from these capabilities
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
75
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
Coexistence… (T)
0 20 40-1
0
1
2
Time (s)
Positio
n (
m)
0 20 40-1
0
1
2
Time (s)A
ccele
ration (
G)
Transfer of Activity Templates
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
(1) Both systems coexists during a certain period of time
76
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
(2) A mapping function between source and target domains is discovered through system identification (MIMO model)
Transfer of Activity Templates
Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)
77
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
(3) The activity templates are translated from source to target domain
Transfer of Activity Templates
78
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Templates
(3) The activity templates are translated from source to target domain
System S (source domain) System T (target domain)
L3
79
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)
0 1 2 3-0.5
0
0.5
1
1.5
Time (s)
Accele
ration (
G)
X
Y
Z
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Templates
(3) The activity templates are translated from source to target domain
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
L3 L3
80
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
L1 L2 L3
Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Templates
• System identification
– 3) The activity templates are translated from source to target domain
0 1 2 3
0
0.5
1
1.5
Time (s)
Accele
ration (
G)
XYZ
X
Y
ZFitness.. Esto es lo que idealmente busco…
81
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
Ψ𝑆→𝑇 𝑡 : 𝑋𝑆(𝑡) → 𝑋 𝑇(𝑡) ≈ 𝑋𝑇(𝑡)
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
L1 L2 L3 0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Templates
(4) Once the templates have been translated, the target system is ready for activity detection
82
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
L1 L2 L3 0 2 4
-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
L1 L2 L3 0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Templates
(4) Once the templates have been translated, the target system is ready for activity detection
Instruction completed!
83
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
• Transfer of the recognition capabilities of an existing source system (S) that operates on activity models (features + classification model) to an untrained target system (T) that lacks from these capabilities
84
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
Coexistence… (T)
0 20 40-1
0
1
2
Time (s)P
ositio
n (
m)
0 20 40-1
0
1
2
Time (s)
Accele
ration (
G)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
(1) Both systems coexists during a certain period of time
85
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
(2) A mapping function between target and source domains is discovered through system identification (MIMO model)
86
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
(3) The source activity models are translated to the target domain so both use the same activity models
87
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
(3) The source activity models are translated to the target domain so both use the same activity models; these activity models also define the target activity recognition system
88
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
(4) The target system continuously translate its signals into the source domain to operate on the transferred recognition system
0 1 2 3 4-0.5
0
0.5
1
1.5
XYZ
89
𝑋𝑆(𝑡)
𝑋𝑇(𝑡)
System S (source domain) System T (target domain)
Sig
na
l le
vel
Rea
son
ing
le
vel
Ψ𝑇→𝑆 𝑡 : 𝑋𝑇(𝑡) → 𝑋 𝑆(𝑡) ≈ 𝑋𝑆(𝑡)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Transfer of Activity Models
(4) The target system continuously translate its signals into the source domain to operate on the transferred recognition system; since then it is ready for activity detection
Instruction completed!
0 1 2 3 4-1
0
1
2
X
Y
Z
90
Evaluation of Multimodal Transfer
• Models validation
– Transfer between IMU and IMU (Identical Domain Transfer)
– Transfer between Kinect and IMU (Cross Domain Transfer)
91
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
L1 L2 L3 0 2 4 6
-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4 6-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 2 4-1
0
1
2
Time (s)
Positio
n (
m)
XYZ
0 1 2 3-1
0
1
2
Time (s)
Positio
n (
m)
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
L1 L2 L3 0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 5 10-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
0 2 4-1
0
1
2
Time (s)
Accele
ration (
G)
X
Y
Z
Transfer of Activity Templates
Transfer of Activity Models
0 1 2 3 4-1
0
1
2
X
Y
Z
Multimodal Kinect-IMU Dataset: Study Setup
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
*Freely available at: www.ugr.es/~oresti/datasets 92
Multimodal Kinect-IMU Dataset: Study Setup
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
MTx XSENS IMUs
- 3D ACC + (3D GYR, 3D MAG, 4D QUA) - Sampling rate 30Hz
Applications
93 Xsens data logger http://crnt.sourceforge.net/CRN_Toolbox/References.html
Multimodal Kinect-IMU Dataset: Study Setup
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
MICROSOFT KINECT
- RGB cam + IR cam + IR led - Depth map (0.5-6m)
- 15 joints skeleton tracking - 3D position - Tracking range (1.2-3.5m) - Sampling rate 30Hz
Applications
94 Kinect data logger http://code.google.com/p/qtkinectwrapper/
Multimodal Kinect-IMU Dataset: Scenarios
Geometric Gestures (HCI)
48 instances per gesture
Other scenarios were also collected as part of this dataset (more info at www.ugr.es/~oresti/datasets) 95
Multimodal Kinect-IMU Dataset: Scenarios
Geometric Gestures (HCI) Idle (Background)
~5 min of data 48 instances per gesture
Other scenarios were also collected as part of this dataset (more info at www.ugr.es/~oresti/datasets) 96
Transfer between IMU and IMU
• Analyzed transfers
– Transfer of Activity Templates and Activity Models from:
• RLA (3D acceleration) to RUA (3D acceleration)
• RUA (3D acceleration) to RLA (3D acceleration)
• RUA (3D acceleration) to BACK (3D acceleration)
• BACK (3D acceleration) to RUA (3D acceleration)
• RLA (3D acceleration) to BACK (3D acceleration)
• BACK (3D acceleration) to RLA (3D acceleration)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
97
Evaluation of Transfer between IMU and IMU
• Mapping:
– Model MIMO3x3 mapping with 10 tap delay
– Types
• Problem-domain mapping (PDM)
• Gesture-specific mapping (GSM)
• Unrelated-domain mapping (UDM)
– Learning 100 samples (~3.3s)
• Activity recognition model:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Triaxial acceleration
(IMU)
No preprocessing
(raw data)
Instance based segmentation
FS=max,min KNN (standard
classifier)
98
Evaluation of Transfer between IMU and IMU
• Transfer of Activity Templates:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
99 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
LA=lower arm UA=upper arm
B=back
Evaluation of Transfer between IMU and IMU
• Transfer of Activity Templates:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
100 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
LA=lower arm UA=upper arm
B=back
<1% <3%
Evaluation of Transfer between IMU and IMU
• Transfer of Activity Templates:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
101 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
LA=lower arm UA=upper arm
B=back
<1% <3%
35%
15% 17%
28% 35% 28%
10% 10%
Evaluation of Transfer between IMU and IMU
• Transfer of Activity Templates:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
102 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
LA=lower arm UA=upper arm
B=back
<1% <3%
12% 20%
35%
55%
15% 17%
28% 35%
60%
28%
30%
50%
10% 10%
Evaluation of Transfer between IMU and IMU
• Transfer of Activity Models:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
103 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
LA=lower arm UA=upper arm
B=back
Evaluation of Transfer between IMU and IMU
• Transfer of Activity Models:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
104 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
LA=lower arm UA=upper arm
B=back
Transfer between Kinect and IMU
• Analyzed transfers
– Transfer of Activity Templates (Kinect to IMU) :
• HAND (3D position) RLA (3D acceleration)
• HAND (3D position) RUA (3D acceleration)
• HAND (3D position) BACK (3D acceleration)
– Transfer of Activity Models (IMU to Kinect):
• RLA (3D acceleration) HAND (3D position)
• RUA (3D acceleration) HAND (3D position)
• BACK (3D acceleration) HAND (3D position)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
105
Evaluation of Transfer between Kinect and IMU
• Mapping:
– Model MIMO3x3 mapping with 10 tap delay
– Types
• Problem-domain mapping (PDM)
• Gesture-specific mapping (GSM)
• Unrelated-domain mapping (UDM)
– Learning 100 samples (~3.3s)
• Activity recognition model:
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
106
Triaxial acceleration
(IMU) / Triaxial position (KINECT)
No preprocessing
(raw data)
Instance based segmentation
FS=max,min KNN (standard
classifier)
Evaluation of Transfer between Kinect and IMU
• Transfer of Activity Templates (From Kinect to IMU)
• Transfer of Activity Models (From IMU to Kinect)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
107 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
RLA= right lower arm RUA= right upper arm
BACK=back KINECT=hand
Evaluation of Transfer between Kinect and IMU
• Transfer of Activity Templates (From Kinect to IMU)
• Transfer of Activity Models (From IMU to Kinect)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
108 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
RLA= right lower arm RUA= right upper arm
BACK=back KINECT=hand
<4% <4% <8% <6%
Evaluation of Transfer between Kinect and IMU
• Transfer of Activity Templates (From Kinect to IMU)
• Transfer of Activity Models (From IMU to Kinect)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
109 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
RLA= right lower arm RUA= right upper arm
BACK=back KINECT=hand
<4% <4% <8% <6%
30%
45%
35%
60%
Evaluation of Transfer between Kinect and IMU
• Transfer of Activity Templates (From Kinect to IMU)
• Transfer of Activity Models (From IMU to Kinect)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
110 BS=baseline source | BT=baseline target | PDM=problem-domain mapping | GSM=gesture-specific mapping | UDM=unrelated-domain mapping
RLA= right lower arm RUA= right upper arm
BACK=back KINECT=hand
<4%
35%
50%
<4% <8% <6%
30%
45%
35%
60% 55%
30% 35% 35%
Evaluation of Transfer between Kinect and IMU
• Transfer of Activity Templates (From Kinect to IMU)
• Transfer of Activity Models (From IMU to Kinect)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
From Kinect to IMU (RLA) From IMU (RLA) to Kinect
FS1=mean FS2=max,min
111 30 samples = 1 s
Evaluation of Transfer between Kinect and IMU
• Transfer of Activity Templates (From Kinect to IMU)
• Transfer of Activity Models (From IMU to Kinect)
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
From Kinect to IMU (RLA) From IMU (RLA) to Kinect
FS1=mean FS2=max,min
112 30 samples = 1 s
25% 20%
Conclusions
• Classical training procedures are not practical to instruct newcomer sensors in dynamically varying and evolvable activity recognition setups
• A novel multimodal transfer learning model is proposed to translate the recognition capabilities of an existing system to a new untrained system, at runtime and without expert or user intervention
• As few as a single gesture (≈3 seconds) of data is enough to learn a mapping model that captures the underlying relation between systems of identical or different modality
• The transfer between IMUs across close-by limbs achieves a recognition accuracy superior to 97% (>2% below baseline), and 95% (>4% below baseline) for the transfer between Kinect and IMU, independently of the direction of the transfer
• Low-variance data unrelated to the activities of interest can be also used to learn a mapping, albeit with more data
113
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Conclusions and future work
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Contributions
• Identification of the requirements and challenges posed by AR systems in real-world conditions
• Evaluation of the tolerance of standard AR systems to sensor technological anomalies, particularly sensor failures and faults
• Definition and development of a novel model, so-called HWC, to overcome the effects of sensor failures and faults. Evaluation of the robustness of the proposed HWC model to the effects of sensor failures and faults
• Evaluation of the tolerance of standard AR systems to sensor deployment variations, particularly static and dynamic sensor displacements
• Evaluation of the robustness of the proposed HWC model to the effects of sensor displacements
• Definition, development and validation of a novel multimodal transfer learning method that operates at runtime, with low overhead and without user or system designer intervention
115
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Contributions
• Collection and curation of an innovative benchmark dataset to investigate the effects of sensor displacement, introducing the concept of ideal-placement, self-placement and induced-displacement. This dataset includes a wide range of physical activities, sensor modalities and participants. Apart from investigating sensor displacement, the dataset lend itself for benchmarking activity recognition techniques in ideal conditions. The dataset is publicly available to the research community at http://www.ugr.es/~oresti/datasets
• Collection and curation of a novel multimodal dataset to investigate transfer learning among ambient sensing and wearable sensing systems. The dataset could be also used for gesture spotting and continuous activity recognition. The dataset is publicly available to the research community at http://www.ugr.es/~oresti/datasets
116
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Selected Publications
• International Journals (SCI-indexed) – Banos, O., Toth M. A., Damas, M., Pomares, H., Rojas, I. Dealing with the effects of
sensor displacement in wearable activity recognition. Sensors, MDPI (2014) [Under review]
– Banos, O., Damas, M., Guillen, A., Herrera, L.J., Pomares, H., Rojas, I. Multi-sensor fusion based on asymmetric decision weighting for robust activity recognition. Neural Processing Letters, Springer (2014) [Under review]
– Banos, O., Galvez, J. M., Damas, M., Pomares, H., Rojas, I. Window size impact in activity recognition. Sensors, MDPI, vol. 14, no. 4, pp. 6474-6499 (2014)
– Banos, O., Damas, M., Pomares, H., Rojas, F., Delgado-Marquez, B., Valenzuela, O. Human activity recognition based on a sensor weighting hierarchical classifier. Soft Computing, Springer, vol. 17, pp. 333-343 (2013)
– Banos, O., Damas, M., Pomares, H., Rojas, I. On the Use of Sensor Fusion to Reduce the Impact of Rotational and Additive Noise in Human Activity Recognition. Sensors, MDPI, vol. 12, no. 6, pp. 8039-8054 (2012)
– Banos, O., Damas, M., Pomares, H., Prieto, A., Rojas, I.: Daily Living Activity Recognition based on Statistical Feature Quality Group Selection. Expert Systems with Applications, Elsevier, vol. 39, no. 9, pp. 8013-8021 (2012)
117
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Selected Publications
• Book chapters – Banos, O., Toth M. A., Damas, M., Pomares, H., Rojas, I. Amft, O.: Evaluation of inertial sensor
displacement effects in activity recognition systems. Science and Supercomputing in Europe (Information & Communication Technologies), HPC-Europe 2 (2013) ISBN: 978-84-338-5400-1
• Conference papers – Banos, O., Damas, M., Pomares, H., Rojas, I.: Handling displacement effects in on-body sensor-
based activity recognition. In: Proceedings of the 5th International Work-conference on Ambient Assisted Living an Active Ageing (IWAAL 2013), San Jose, Costa Rica, December 2-6, (2013) [BEST PAPER AWARD]
– Banos, O., Damas, M., Pomares, H., Rojas, I.: Activity recognition based on a multi-sensor meta-classifier. In: Proceedings of the 2013 International Work Conference on Neural Networks (IWANN 2013), Tenerife, June 12-14, (2013)
– Banos, O., Toth, M. A., Damas, M., Pomares, H., Rojas, I., Amft, O.: A benchmark dataset to evaluate sensor displacement in activity recognition. In: Proceedings of the 14th International Conference on Ubiquitous Computing (Ubicomp 2012), Pittsburgh, USA, September 5-8, (2012)
118
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Selected Publications
• Conference papers (cont.) – Banos, O., Calatroni, A., Damas, M., Pomares, H., Rojas, I., Troester, G., Sagha, H., Millan, J. del
R., Chavarriaga, R., Roggen, D.: Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems Across Sensor Modalities. In: Proceedings of the 16th annual International Symposium on Wearable Computers (ISWC 2012), Newcastle, United Kingdom, June 18-22 (2012)
– Banos, O., Damas, M., Pomares, H., Rojas, I.: Human multisource activity recognition for AAL problems. In: Proceedings of the 5th International Symposium on Ubiquitous Computing and Ambient Intelligence (UCAmI 2011), Riviera Maya, Mexico, December 5-9, (2011)
– Banos, O., Damas, M., Pomares, H., Rojas, I.: Recognition of Human Physical Activity based on a novel Hierarchical Weighted Classification scheme. In: Proceedings of the 2011 International Joint Conference on Neural Networks (IJCNN 2011), IEEE, San Jose, California, July 31-August 5, (2011)
– Banos, O., Pomares, H., Rojas, I.: Ambient Living Activity Recognition based on Feature-set Ranking Using Intelligent Systems. In: Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN 2010), IEEE, Barcelona, July 18-23, (2010)
119
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Future Work
• Collection of new large standard datasets
• Dynamic reconfiguration of the HWC
• Self-adaptive HWC
• Tolerance to other sensor technological and topological anomalies
• Multiple trainers and complex modalities in transfer learning
• Integration in commercial systems and end-user applications
120
INTRODUCTION TECHNOLOGICAL ANOMALIES DEPLOYMENT VARIATIONS NETWORK CHANGES CONCLUSIONS
Sensor M
Sensor 2
SM
S2
S1 α11
Ψ
C12
C1N
C11
Ψ
C21
C22
C2N
Ψ
CM1
CM2
CMN
Ψ
Decisio
n
Activity level
(base classifier)
Sensor level
(sensor classifier)
Network level
(sensor fusion)
β11
α12 β12
α1N β1N
α21 β21
α22 β22
α2N β2N
αM1 βM1
αM2 βM2
αMN βMN
γ11,…,1N δ11,…,1N
γ21,…,2N δ21,…,2N
γM1,…,MN δM1,…,MN
Sensor 1
FEEDBACK
¡Gracias a todos!
121
Recommended