Embed Size (px)
Citation preview
UNIVERSITY OF CASTILLA-LA MANCHAESCUELA SUPERIOR DE INFORMÁTICA
COMPUTER SCIENCE DEGREE
FINAL DEGREE PROJECT
AglaeaStroke remote rehabilitation project
Pedro Manuel Gómez-Portillo López
July, 2017
AGLAEASTROKE REMOTE REHABILITATION PROJECT
UNIVERSITY OF CASTILLA-LA MANCHA
ESCUELA SUPERIOR DE INFORMÁTICA
Technologies and Information Systems Department
SPECIFIC TECHNOLOGY OF
SOFTWARE ENGINEERING
FINAL DEGREE PROJECT
AglaeaStroke remote rehabilitation project
Author: Pedro Manuel Gómez-Portillo López
Supervisor: Dr. D. Carlos González Morcillo
Supervisor: D. Santiago Sánchez Sobrino
July, 2017
Pedro Manuel Gómez-Portillo López
Ciudad Real – Spain
E-mail: [email protected] site: https://github.com/gomezportillo
© 2017 Pedro Manuel Gómez-Portillo López
This document and the information in it are provided in confidence, for the sole purpose of thetitled recipient to read and may not be disclosed to any third party or used for any other purposewithout the express written permission of Furious Koalas Interactive.
i
Abstract
Stroke is the result of stopping blood flow to the brain. It is usually caused by a bloodclot that blocks a cerebral artery and its effects range from physical to cognitive problems ofdifferent severity, being death the worst-case scenario.
This pathology is quite common and does not discriminate by race or sex. Thus, countriesall over the world struggle to reduce it and help those who suffer it and have to live with itseffects.
The present Final Degree Project comes as a result of the development of Aglaea, theprototype of a tool developed by the spin-off from the University of Castilla-La ManchaFurious Koalas Interactive for the British company Kyenom Limited in order to help strokepatients in their rehabilitation making use of paradigms such as Computer Vision, MixedReality and Serious Gamification.
To this end, techniques and devices from the Motion Tracking area are used to capturethe rehabilitation exercises performed by a patient. Then, Aglaea stores this exercises in astandardised file format and compares them using a temporal sequences analysis algorithmso as to evaluate the similarity degree between two exercises.
In addition, this project supports two different roles, clinician and patient, and will allowusers to store both video and motion capture information from the exercises they perform,allowing them to reproduce it.
Aglaea is presented as a tool capable of analysing the rehabilitation exercises performedby a patient at home and comparing them with a reference movement or gold standard,informing both the patient and their clinician about the obtained results.
iii
Resumen
Los derrames cerebrales o ataques cerebrovasculares son efecto de la detención del flujosanguíneo al cerebro. Normalmente son causados por un coágulo sanguíneo que bloqueauna arteria cerebral y sus efectos varían desde problemas físicos a cognitivos de diferentegravedad, y en el peor de los casos la muerte.
Esta patología es bastante común y no discrimina por raza o sexo, por lo que países detodo el mundo luchan por reducirla y ayudar a los que la sufren y tienen que vivir con susefectos.
El presente Trabajo fin de Carrera surge como resultado del desarrollo de Aglaea, el proto-tipo de una herramienta desarrollada por la spin-off de la Universidad de Castilla-La Mancha,Furious Koalas Interactive, para la compañía británica Kyenom Limited con el objetivo deayudar a pacientes de derrame cerebral en su rehabilitación haciendo uso de paradigmascomo la Visión por Computador, la Realidad Mixta y la Gamificación.
Para ello, utiliza técnicas y dispositivos del área de Motion Tracking para capturar los ejer-cicios de rehabilitación que los usuarios realicen, los almacena en un formato estandarizadoy los compara haciendo uso de un algoritmo de análisis de secuencias temporales con el finde poder evaluar el grado de similitud de dos ejercicios.
Además, soporta dos roles diferentes, médico y paciente, y dará soporte para que sususuarios almacenen tanto el vídeo como la información de captura de movimiento de losejercicios que realicen, permitiendo que sean reproducidos desde dentro de la aplicación.
Aglaea se presenta como una herramienta capaz de analizar los movimientos de los ejerci-cios de rehabilitación realizados por un un paciente en su casa y compararlos con movimien-tos de referencia correctos, informando tanto al paciente como a su médico de los resultadosobtenidos.
v
A mis padres, por tantos tuppers ytantas madrugadas acompañándome a Ciudad Real.
vii
Contents
Abstract iii
Resumen v
Contents ix
List of Tables xiii
List of Figures xv
List of Listings xvii
List of Acronyms xix
1 Introduction 1
1.1 Rehabilitation and gamification . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Socio-economic impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Project proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.6 Document structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Introducción 7
2.1 Rehabilitación y gamificación . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Contexto . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Entorno . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Impacto socio-económico . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5 Propuesta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.6 Estructura del documento . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Objectives 13
3.1 General objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
ix
3.2 Specific objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2.1 Comparative study of available hardware tracking devices . . . . . 14
3.2.2 Task delegation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2.3 Recording, playback and exercise comparison . . . . . . . . . . . . 14
3.2.4 Usable, complete and accessible user interface . . . . . . . . . . . 14
3.2.5 Gamification dynamics . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.6 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.7 Development philosophy . . . . . . . . . . . . . . . . . . . . . . . 16
4 State of the Art 17
4.1 History of medicine and rehabilitation . . . . . . . . . . . . . . . . . . . . 18
4.2 Motion Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.1 Available cameras . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.2 Exchange formats . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.3 Skeletal comparison . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 Graphical representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3.1 Web development . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3.2 Gamification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3.3 Three-dimensional representation . . . . . . . . . . . . . . . . . . 29
4.4 Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.4.1 Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.4.2 Websockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 Methodology 33
5.1 Working methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.1.1 Development process . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.1.2 Development methodology . . . . . . . . . . . . . . . . . . . . . . 35
5.1.3 Milestone planning and iterations . . . . . . . . . . . . . . . . . . 39
5.2 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2.1 Hardware resources . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2.2 Software resources . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6 Architecture 45
6.1 Architecture overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.2 Requirement analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3 Camera study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
x
6.3.1 Razer Stargazer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.3.2 Microsoft Kinect 1.8 . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.3.3 Microsoft Kinect 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.3.4 Comparison results . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.4 System communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.4.1 Communication protocol . . . . . . . . . . . . . . . . . . . . . . . 59
6.5 System development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.5.1 Capture module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.5.2 Processing module . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.5.3 Module overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.5.4 Displaying module . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6.6 Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7 Results 79
7.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.1.1 Clinician . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
7.1.2 Patient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.3 Resources and costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.3.1 Project budget . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.3.2 Code statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
8 Conclusions 93
8.1 Accomplished objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
8.1.1 Task delegation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
8.1.2 Comparative study of available hardware tracking devices . . . . . 94
8.1.3 Usable, complete and accessible user interface . . . . . . . . . . . 94
8.1.4 Recording, playback and exercise comparison . . . . . . . . . . . . 94
8.1.5 Gamification dynamics . . . . . . . . . . . . . . . . . . . . . . . . 95
8.1.6 Development philosophy . . . . . . . . . . . . . . . . . . . . . . . 95
8.2 Future working lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
8.2.1 Gamification dynamics expansion . . . . . . . . . . . . . . . . . . 95
8.2.2 Three-dimensional visualisation . . . . . . . . . . . . . . . . . . . 96
9 Conclusiones 97
9.1 Objetivos cumplidos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
xi
9.1.1 Delegación de tareas . . . . . . . . . . . . . . . . . . . . . . . . . 97
9.1.2 Estudio comparativo de los sistemas de captura de movimientos dispon-ibles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
9.1.3 Interfaz e usuaro usable, completa y accesible . . . . . . . . . . . . 98
9.1.4 Grabación, reproducción y comparación de ejercicios . . . . . . . . 98
9.1.5 Dinámicas de gamificación . . . . . . . . . . . . . . . . . . . . . . 99
9.1.6 Filosofía de desarrollo . . . . . . . . . . . . . . . . . . . . . . . . 99
9.2 Líneas de trabajo futuro . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
9.2.1 Expansión de las dinámicas de gamificación . . . . . . . . . . . . . 99
9.2.2 Visualización tridimensional desde el sitio web . . . . . . . . . . . 100
A Camera characteristics 105
A.1 Microsoft Kinect 1.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
A.2 Microsoft Kinect 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
A.3 Razer Stargazer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
B BVH file example 111
C Screen points persistance file example 113
D Speech Recognition Grammar File 115
E Source code 117
E.1 Camera communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
E.2 Data processing and displaying . . . . . . . . . . . . . . . . . . . . . . . . 117
F CD content 119
G Conclusión personal 121
References 123
xii
List of Tables
5.1 Iteration planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
7.1 Improvements in frame rate . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.2 Cost breakdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.3 Number of lines of source code by language . . . . . . . . . . . . . . . . . 90
A.1 Kinect 1.8 characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
A.2 Kinect 1.8 joint identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . 106
A.3 Kinect 2 characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
A.4 Kinect 2 joint identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
A.5 Razer Stargazer characteristics . . . . . . . . . . . . . . . . . . . . . . . . 109
A.6 Razer Stargazer joint identifiers . . . . . . . . . . . . . . . . . . . . . . . . 110
xiii
List of Figures
1.1 Kyenom logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Furious Koalas logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Environment diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Subsystems in Aglaea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1 Logo de Kyenom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Logo de Furious Koalas . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Diagrama de entorno . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Subsistemas de Aglaea . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.1 Conceptual map of the background of Aglaea . . . . . . . . . . . . . . . . 18
4.2 Rod of Asclepius, universal symbol of medicine . . . . . . . . . . . . . . . 19
4.3 Use of motion tracking with markers, www.lighthouse.org . . . . . . . . . . 21
4.4 Bone hierarchy in Aglaea . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.5 The Ninja Project loading page . . . . . . . . . . . . . . . . . . . . . . . . 27
4.6 Animating in Blender the main character of Arrow and the Wicked Ogre . . 30
4.7 Protocol encapsulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1 Dilbert facing up team roles, from http://dilbert.com . . . . . . . . . . . . 36
5.2 Scrum workflow, adapted from https://www.scrumalliance.org . . . . . . . 36
6.1 Detailed subsystem diagram . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.2 Use case diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3 Kinect v1.8 infrared pattern, from http://graphics.stanford.edu/ . . . . . 51
6.4 Kinect tracking modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.5 2D projection from 3D point, adapted from http://www.adobe.com . . . . . . 53
6.6 Razer Stargazer joint tracking . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.7 Razer Stargazer depth image . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.8 Communication sequence diagram . . . . . . . . . . . . . . . . . . . . . . 58
6.9 Capture module class diagram . . . . . . . . . . . . . . . . . . . . . . . . 61
xv
6.10 Capture module flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.11 Double buffer state #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.12 Double buffer state #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.13 Processing module class diagram . . . . . . . . . . . . . . . . . . . . . . . 66
6.14 BVH file being reproduced in Blender . . . . . . . . . . . . . . . . . . . . 69
6.15 JavaScript class diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.16 Video player overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.17 Levelling up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7.1 Role selector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.2 Clinician page overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
7.3 Clinician recording exercise . . . . . . . . . . . . . . . . . . . . . . . . . 81
7.4 Clinician playing exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
7.5 Clinician assigning exercises . . . . . . . . . . . . . . . . . . . . . . . . . 83
7.6 Clinician consulting results . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.7 Patient page overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.8 Patient performing exercise . . . . . . . . . . . . . . . . . . . . . . . . . . 86
7.9 Invalid exercise warning . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
7.10 Patient web application evolution . . . . . . . . . . . . . . . . . . . . . . . 87
7.11 Frame rate evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.12 C++ repository. Commits per week . . . . . . . . . . . . . . . . . . . . . . 90
7.13 C++ repository. Additions and modifications per week . . . . . . . . . . . 91
7.14 Python and HTML repository. Commits per week . . . . . . . . . . . . . . 91
7.15 Python and HTML repository. Additions and modifications per week . . . . 92
A.1 Kinect 1.8 components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
A.2 Kinect 2 components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
A.3 Razer Stargazer components . . . . . . . . . . . . . . . . . . . . . . . . . 109
xvi
List of Listings
6.1 Razer Stargazer retrieving tracking information . . . . . . . . . . . . . . . 48
6.2 Memory leak solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.3 Kinect 1.8 retreiving tracking information . . . . . . . . . . . . . . . . . . 50
6.4 Seated mode flag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.5 Smoothing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.6 Kinect 2 retrieving tracking information . . . . . . . . . . . . . . . . . . . 54
6.7 Image serialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.8 Processing module initiation . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.9 Synchronisation mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.10 Namespace declaration in JavaScript . . . . . . . . . . . . . . . . . . . . . 71
6.11 Namespaces on Python server . . . . . . . . . . . . . . . . . . . . . . . . 72
6.12 Drawing an image over a canvas in JavaScript . . . . . . . . . . . . . . . . 73
6.13 Drawing a single point in a canvas . . . . . . . . . . . . . . . . . . . . . . 73
6.14 Joint iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.15 Complementary joints draw . . . . . . . . . . . . . . . . . . . . . . . . . . 75
xvii
List of Acronyms
BVH Biovision Hierarchy
DTW Dynamic Time Warping
FDP Final Degree Project
FPS Frames Per Second
FTP File Transfer Protocol
GPU Graphical Processing Unit
GUI Graphical User Interface
HCI Human-Computer Interaction
HMM Hidden Markov Model
HTTP Hypertext Transfer Protocol
IDE Integrated Development Environment
IP Internet Protocol
JPEG Joint Photographic Experts Group
JSON JavaScript Object Notation
KISS Keep it simple, stupid!
MO-CAP Motion Capture
MVC Model View Controller
RUP Rational Unified Process
SDK Software Development Kit
TCP Transmission Control Protocol
TFG Trabajo Fin de Grado
UDP User Datagram Protocol
UPD Unified Process Development
xix
Chapter 1
Introduction
T HE Information Age has opened the doors to automation in incredible ways. It hasbeen possible for computers to analysed and perform repetitive tasks in a matter of
seconds that previously had to be performed by people. Furthermore, computers are usuallymore accurate than people.
The use of this automation is very common and the number of companies that dedicatepersonnel to perform monotonous tasks susceptible to such automation is quickly decreasing.Instead of that, these companies develop systems that achieve it in a more quick, economicaland effective way. These systems tend to be easily replicable and to allow high parallelismso that only a fraction of the initial staff is needed to monitor such system.
Thanks to the success of these systems and the advances in automation techniques, theyare being deployed in areas that have nothing to do with factory industrialisation, includingthe area of physical rehabilitation.
1.1 Rehabilitation and gamificationPhysical rehabilitation is part of the treatment a person must take to recover the state or part
of the state lost due to an accident or illness. According to the World Health Organisation1,rehabilitation is a process aimed at recovering and maintaining an optimal physical, sensory,
intellectual, psychological and social state. In addition, they add, rehabilitation provides
disabled people with the tools they need to attain independence and self-determination.
Therefore, it is easy to understand the importance of this process and the necessary it canbe for those whose disorders have generated more serious sequels.
However, rehabilitation can be a slow and frustrating treatment for several reasons; usuallyentails a significant physical effort for the patient and they may not accomplish the expected
1http://www.who.int/topics/rehabilitation/
1
therapeutic results. Also, there are times when the patient has to overcome psychologicalproblems such as learned helplessness [Mar04], where they genuinely think they cannotperform the exercises their doctor has proposed.
In an attempt to assist patients in their rehabilitation process, new techniques are beingapplied to this area. Among these techniques, the most famous one is the gamification.
Gamification aims to incorporate mechanisms obtained from games to a real scenariowhere it requires a certain effort on the people side, motivating people in order them to takethe initiative.
1.2 ContextThis section will address the context in which this FDP has been framed and that has
conditioned its development. It will begin by introducing the two companies involved in theproject to continue describing how the need for the development of Aglaea arose and whatits expectations are.
Kyenom Limited2 is a company located in Leeds, England, which in its own words isdedicated to providing intelligent technologies that improve the quality of life of people byproviding hardware and software solutions to monitor and understand activities taking placein a home.
Figure 1.1: Kyenom logo
On the other hand, Furious Koalas Interactive3 is a young spin-off from the Universityof Castilla-La Mancha with more than fifteen years of experience in R&D in the context ofthe design, development and deployment of interactive solutions that provides services inareas such as Gamification and Computer Vision.
Kyenom needed a prototype to show the potential of a tool to supervise remote rehabilita-tion focused on stroke patients. This prototype should be complete enough to be presented atinvestor meetings and allow this company to obtain sufficient funding to be able to developa project with full functionality.
At the end of 2016 Kyenom contacted Furious Koalas Interactive to offer them the contractfor the development of this prototype, since it fits perfectly with the services offered by
2http://www.kyenom.com/3https://www.furiouskoalas.com/
2
Figure 1.2: Furious Koalas logo
the second company. This prototype, in addition to meeting the functional requirements,should be scalable and modular enough to allow to easily continue its development in anear future.
So in this context Aglaea was born, a project that pretends to show the potential offeredby remote rehabilitation.
1.3 EnvironmentThe execution environment of Aglaea consists of a Microsoft Kinect 360 camera which
was selected after an appropriate market study (see chapter 6), a software system, and apersonal computer, so that both patient and clinician can use their own computer withouthaving to acquire a more powerful one.
This camera, which will be discussed in depth in further chapters, in addition to generatingcolour and depth images offers a motion tracking system that will be used to detect themovements performed by the users and a voice recognition system that allows controllingpart of the application by using voice commands.
The software system is light enough to be deployed on a personal computer and completeenough to meet the requirements imposed by Kyenom.
In addition, the application supports two interaction roles. On the one hand, the clini-cian, who can register new rehabilitation exercises, assign them to patients and consult theirresults. On the other hand it is the role of patient, who can perform assigned exercises andobtain the results in in real time.
Figure 1.3 reflects a typical execution scenario with a patient. As it can be seen, the patientis sitting in front of the camera performing exercises and getting feedback on the computerscreen.
1.4 Socio-economic impactThis project will have associated both social and economic impacts. Given its main object-
ive, Aglaea pursuits to automate the supervision of rehabilitation exercises, so it is expectedthat the price of rehabilitation therapies using this tool decrease and its number of usersincrease for several reasons.
3
Figure 1.3: Environment diagram
• A single clinician will be able to monitor a larger number of patients in parallel,as instead of having to spend a large time on each patient being physically with themalong the sessions, the clinician will be able to directly consult the results that a patientobtains.
• In addition, people who could not easily access to rehab sessions due to geographicalreasons may find it very interesting to do so from home.
• Likewise, patients will not need to travel so frequently to their clinician’s office, avoid-ing in this way to drive, use public transport or rely on someone else to accompanythem.
• On the other hand, this system could be installed in hospitals to assist clinicians intheir task, helping to reduce the congestion of waiting rooms.
• In addition, a gamification-based approach allows repetitive rehabilitation exercisesnot to be so mechanical for patients.
1.5 Project proposal
The development of this FDP has meant the deployment of a displaying system with RealTime graphics by means of different modules that must work synchronously and concur-rently to offer a correct result. In addition, it has to be adapted to personal computers, thusavoiding latency and performance issues.
4
Figure 1.4 the overview of the modules and subsystems that form the application and howthey relate and communicate with each other. These modules will be widely described inchapter 6.
Camera
Communication subsystem
Capturesubsystem
C++ module
Communication subsystem
Processingsubsystem
Python module
Communication subsystem
Displaysubsystem
Web module
Figure 1.4: Subsystems in Aglaea
• Capture subsystem. This subsystem is responsible for retrieving images and inform-ation regarding skeleton tracking and voice commands from the camera through itsSDK.
• Processing subsystem. This subsystem is responsible for working with data obtainedfrom the camera, as well as data generated by patients and clinicians.
• Communication subsystem. The development of this project has lead to work with alarge amount of generated data, which has meant to have intercommunicated correctlyand fluidly the system modules. In total, the system has three different communicationsubsystems, one for each module.
• Display subsystem. This module is devoted to the interaction of users with the system.It will display images, videos and information related to tracking movements and voicecommands in addition to other data in patient and doctor browsers.
Therefore, the development of this project involves an effort for integrating different tech-nologies from very different Areas of Knowledge, such as motion capture and temporalsequences analysis, and synchronise them in order to achieve the expected results.
Similarly, several threads will work concurrently to send the data through the system andtake it to those modules that need it while working with the lowest possible latency.
1.6 Document structureThis document has been structured following the FDP normative from the Escuela Superior
de Informática of the University of Castilla-La Mancha through the following chapters.
5
Chapter 3. ObjectivesThis chapter describes the objectives and sub-objectives for the Aglaea project.
Chapter 4. State of the ArtThis chapter presents a presentation of the current status of the areas covered in thisFDP, collecting and exposing relevant information related to MO-CAP, Mixed Realityand Gamification, among other areas.
Chapter 5. MethodologyThis chapter justifies and explains the working methodology chosen to be applied dur-ing the development of this project, as well as the hardware and software resourcesused.
Chapter 6. ArchitectureThis chapter details the design and implementation of this project and discusses therelevant problems that have arisen throughout the development and the solutions thathave been given to them.
Chapter 7. ResultsThis chapter presents the results of the project, as well as the efforts devoted to improveit, the costs derived from its development and a small statistical study about the sourcecode of the project.
Chapter 9. ConclusionesThis chapter presents the objectives achieved and propose future working lines thatcould be applied in a future development.
6
