A Mobile Application for Physical Activity Recognition usingAcceleration Data from Wearable Sensors for Cardiac Rehabilitation

M. Chaari1,2, M. Abid2,3, Y. Ouakrim2,3, M. Lahami1 and N. Mezghani2,31National School of Engineers of Sfax, Sfax University, Tunisia

2LICEF Research Center, TELUQ, Montreal, Canada3Laboratoire de Recherche en Imagerie et Orthopedie (LIO), CRCHUM, Montreal, Canada

Keywords: mHealth, Mobile Application, Cardiac Rehabilitation, Human Activity Recognition (HAR), WearableSensors, Classification.

Abstract: mHealth applications are an ever-expanding frontier in today’s use of technology. They allow a user to recordhealth data and contact their doctor from the convenience of a smartphone. This paper presents a first ver-sion release of a mobile application that aims to assess compliance of cardiovascular diseased patients withhome-based cardiac rehabilitation, by monitoring physical activities using wearable sensors. The applicationgenerates reports for both the patient and the doctor through an interactive dashboard, as initial proposal, thatprovides feedback of physical activities of daily living undertaken by the patient. The application integrates ahuman activity recognition system, which learns a support vector machine algorithm to identify 10 differentdaily activities, such as walking, going upstairs, sitting and lying, from accelerometer data using a connectedtextile including movement sensors. Our early deployment and execution results are promising since they areshowing good accuracy for recognizing all the ten daily living activities.

1 INTRODUCTION

Cardiac rehabilitation (CR) is a systematic model ofchronic vascular disease care that pro-actively moni-tors these conditions using a multi-faceted approach.This approach includes behavior change strategies re-lated to a sustainable lifestyle and adherence to phar-macological treatment as well as therapeutic exercisesand physical activity programs to improve secondaryprevention outcomes in patients with cardiovasculardisease or recovering from surgery. CR reduces to-tal mortality and cardiac mortality by 20 to 25% (Cyret al., 2018). It may also reduce the number ofhospitalizations related to heart disease and the needfor new revascularization procedures in patients withcoronary artery disease. However, only a minority ofeligible patients participate in CR programs. Home-based cardiac rehabilitation (HBCR) is definitely oneof the new urgently needed strategies to improve theparticipation rate. It uses remote coaching with indi-rect exercise supervision and helps limit hospital orclinic visits (Thomas et al., 2019). In recent decades,CR has evolved from simple surveillance aimed at asafe return to physical activity, to a multidisciplinaryapproach focused on patient education, personalizedphysical training, changing risk factors and the well-being of cardiac patients (Mampuya, 2012). More-

over, recent advances in information and communica-tion technologies have been used to enhance HBCRprograms (Varnfield et al., 2011). Besides improv-ing the quality of measures, wearable devices andportable medical sensors have also proven effectivein monitoring a greater number of patients in pre-vention and rehabilitation programs in a personalizedmanner. As a result, and thanks to recent tools, theuse of home-based mHealth programs has been in-creasing, achieving good control over vital signs andphysical activities (Medina et al., 2017). Recent re-search studies in human activity recognition (HAR)have focused on sensor-based home monitoring sys-tems. HAR is defined as the ability of an intelligentsystem to infer temporally contextualized knowledgeregarding the state of the user and to classify a set ofhuman activities on the basis of a set of sensor read-ings. Its role, extremely important in the burgeon-ing healthcare field, is to provide all the necessarydata on the patient’s health and well-being outside ahospital setting. Technological advances have madeHAR a rapidly growing area, thanks to the use of af-fordable mobile platforms such as smart phones andother personal tracking devices (Damasevicius et al.,2016), which, together with body-worn sensors, cansolve the cardiac patient monitoring problem. Thereare countless examples of applications that use hu-

man activity recognition to indicate the health statusof humans. Fitbit, for example, offers smart watchesand fitness bands to allow healthcare professionalsto track individuals and monitor their activity trendsover time (Paul et al., 2015). Tactio Health, SamsungHealth and many more are able to track the number ofsteps, sleep patterns, passive periods, etc (Voicu et al.,2019). They are designed to facilitate monitoring pa-tients efficiently so doctors can have better control oftheir chronic conditions. Although these apps recog-nize a very limited number of activities, they have ex-cellent results, being extremely accurate in detectingthe type of activity performed.

Our study aims to develop a mobile application formonitoring physical activities by using wearable sen-sors as part of a HBCR program which is designed forpatients who are unable to attend the traditional fa-cility based cardiac rehabilitation. The application’sgoal is to help clinicians perform remote follow-upon their patients. It displays their daily routine activi-ties on a simple dashboard, providing doctors with thenecessary information to check patient’s health condi-tion. So this allows patients to exercise in their ownhomes while being supervised by one of the cliniciansin cardiac rehabilitation all done using the mobile ap-plication. Patients also have access to the applicationso they can also check their improvement.

This work will be presented as follows: Section 2analyzes the methods used in this project. In Section3, the system architecture is explained, with specifica-tion of the technologies chosen for the application de-velopment. Section 4 focuses on presenting the mainresults, and, finally, the conclusion is given in Section5.

2 METHODS AND MATERIALS

The work presented here was carried out as a jointacademic-industry research project designed to mea-sure patient adherence to HBCR programs. In thepresent study, a mobile application is developed foractivity monitoring based on data acquired by theHexoskin intelligent textile shirt1, developed by CarreTechnologies (Montreal, Canada). The Hexoskin isan easily donned, comfortable stretch shirt that canbe used in any ambient environment. Data acquisi-tion by the Hexoskin is non-invasive and can be per-formed continuously without hampering the move-ments of the person wearning it. Hexoskin offers theonly clinically validated system that allows preciseECG cardiac monitoring with lung function and ac-

1https://www.hexoskin.com/

tivity monitoring over the long-term (Banerjee et al.,2015). Its measurements were found to be reliable inmany research studies. This system offers a recordingcapacity of over 42,000 data points of scientific infor-mation per minute. The sensors are placed away fromthe chest area so users can safely engage in contactsports (Cherif et al., 2018a). A cable running fromthe shirt is plugged into the companion device that issecured in a waist pocket (Figure 1).

Figure 1: Hexoskin shirt, smart device and USB cable.

Once reception of body metrics data begins, theHexoskin device can wirelessly stream the data to amobile device in real-time, or store the informationuntil the user is able to transfer it via the suppliedUSB cable. In practice, the Hexoskin enables real-time monitoring of cardiac, respiratory and 3D accel-eration data via cardiac, breathing and movement em-bedded sensors, respectively (Figure 2). In this paper,we focus on movement sensors that calculate 3-axisaccelerations.

(a) (b) (c)

Figure 2: Hexoskin embedded sensors : (a) cardiac, (b)breathing and (c) movement sensors. The cardiac sensorsallow to track the heart rate and display the electrocardio-gram (ECG) in real time. The breathing sensors indicate thebreathing capacity for a given activity. The movement sen-sors calculate 3-axis accelerations, activity levels and steps.

This study extends previous research (Cherif et al.,2018b), by implementing a HAR framework on a

mobile phone using 3D acceleration data. As illus-trated in Figure 3, the HAR framework consists oftwo phases: the offline data training and the onlineclassification.

Figure 3: The HAR framework includes an offline learningand an online classification.

2.1 Data Collection for Offline Learning

The data collection has been approved by institu-tional ethics committees and all subjects providedwritten informed consent before participating. Datacollection was performed at the research center ofthe University of Montreal Hospital Center (Mon-treal, Canada) and recruited participants were in goodhealth and didn’t have chronic pain or known loco-motor or heart problems. Fourteen more healthy vol-unteers have been included in the pre-established database with the same data collection protocol. A to-tal of 40 voluntary participants wore the smart textileHexoskin and performed 5 repeats of a sequence of10 tasks: going up the stairs, going down the stairs,walking, running, sitting, falling right, falling left,falling backward, falling forward and lying down.The choice of these selected activities was made torepresent the majority of everyday living activities(Attal et al., 2015). To ensure a good recording sig-nal, patients were instructed to moisten the three grayelectrodes inside the shirt, connect the Hexoskin de-vice to the shirt connector and insert it horizontallyinto the shirt pocket with the wire upward and thelight outward.

The accelerations were collected from the 3-axissensors integrated into the Hexoskin. In total, theacceleration data of 40 healthy volunteers perform-ing the same data collection protocol was collectedand considered in the classification model. A videorecording was made during the whole session. Its se-quences were used later to label the different classesof physical activities.

Figure 4: A scenario of data collection with a participantwearing the Hexoskin. Each sub-image corresponds to atask, i.e., an activity.

2.2 Data Transmission for OnlineClassification

When the shirt is unplugged, data recording ceasesand the device automatically shuts down after 60 sec-onds. The administrator logs in to the participant’sHexoskin account, connects the device to the com-puter with the USB cable and launches the HxSer-vices application. The data start to synchronize auto-matically and are saved on the Hexoskin server.

In a real-life context, the patient would performthe data transmission daily using his own computerto transfer his records from the smart device to theHexoskin server. Then, he can launch the mobile ap-plication. Once he does, a request is sent to the Hex-oskin server via the Application Programming Inter-face (API) 2 (Application Programming Interface) toget the patient’s data. Upon receiving the request, theHexoskin server sends a response.

2.3 Classification

The collected acceleration data are trained, during theoffline phase, using a machine learning classificationalgorithm which returns the resulting activities in la-bel form.

As shown in Figure 5, the machine learning clas-sification algorithm consists of the following steps:First, the acceleration norm is computed from thethree axis acceleration signal. The sequence of theacceleration vector norm is then segmented into shorttime sequences using a rectangular window aroundeach detected peak. More precisely, peaks are de-tected by thresholding the norm. Time-domain andfrequency domain features are computed on 1s-lengthwindows from the 3D acceleration data. The Relief-Falgorithm is then used to rank individual features ac-

2https://api.hexoskin.com/docs/page/oauth2/

cording to their relevance scores. After the selectionprocess, the top ten features were retained to be usedfor the classification. The next step is a physical ac-tivity classification system developed using Matlab.The classification uses the Support Vector Machine(SVM) classifier. It is a supervised machine learn-ing algorithm that aims to find the optimal separatingdecision hyperplanes between classes with the max-imum margin between patterns of each class. Thetests were done according to the process of Leave-one-subject-out cross validation for the SVM model.Tests of its performance, evaluated in terms of correctclassification rates, have demonstrated the reliabilityof the approach with an accuracy of 95.4%.There has been numerous studies to investigate dif-ferent machine learning models for physical humanactivity recognition using wearable sensors for accel-eration signals. In (Attal et al., 2015), the authorsproposed a classification methodology to recognize,using acceleration data, different classes of daily liv-ing human activities, by comparing different machinelearning techniques namely, k-Nearest Neighbor (k-NN), SVM, Gaussian Mixture Models (GMM), andRandom Forest (RF). In our paper, the choice of thefeature extraction, selection and classification meth-ods was made just for deploying the HAR frameworkon mobile devices. The next step, which is our con-cern, is the implementation of the mobile application,and the deployment of our own custom model forHAR. We’ll be detailing this step in the next section.

Figure 5: Block diagram of the proposed physical activityclassification system as proposed in (Cherif et al., 2018b).In their work, the authors tested an SVM and a KNN clas-sifier.

3 APPLICATIONARCHITECTURE ANDTECHNOLOGIES

The objective of this project part was to develop amobile application for physical activity classificationfrom data collected by the Hexoskin textile. The ap-plication will be used to implement a tool to measurecompliance with home rehabilitation treatments. Thephysiological classification and analysis systems as

Figure 6: System scenario: data reception, HAR classifi-cation and local visualization. The Data is located in theHexoskin server. The offline data training is performed inthe backend server. Only the results are transplanted ontothe mobile device dashboard.

well as the compliance measurement tool will be in-tegrated in the future into the Hexoskin platform.

The collected data, analysis and patient local inter-face of the project was implemented in a smartphoneapplication compatible with Android or iOS. This ap-plication as show in Figure 6 has the following mainfunctions:

• Data reception (steps 1, 2 and 3): After data trans-mission from the Hexoskin shirt to the server, theapplication has to receive Hexoskin informationpackages from the server using an API. Then, thesystem has to extract the acceleration, cardiac andrespiratory data.

• HAR classifier (step 4): The app uses a classifieralgorithm to build a recognition model to estimatethe current activity.

• Local visualization (step 5): All the latest sensordata can be seen in the mobile screen, in conjunc-tion with the latest recognized activity.

Figure 7 presents the different parts of our appli-cation. The frontend comprises the presentation layerbased on an Ionic 4 (Dunka et al., 2017), Angular73 and Cordova4 framework to produce a multiplat-form application imitating the native behavior, group-ing the Man-Machine interfaces and scripts that com-municate with the backend based on the protocol Hy-pertext Transfer Protocol (HTTP) and Java Script Ob-ject Notation (JSON) exchange format.

The backend, corresponding to the second part,comprises the application server developed usingPython5 technology, since the classification model

3https://angular.io/4https://cordova.apache.org/5https://www.python.org/

Figure 7: Global architecture of the application.

was converted from Matlab to Python in order to facil-itate the connection with the front end. We used theFlask6 framework provided with a lightweight Webserver, which requires minimal configuration, can becontrolled via Python code and can easily exchangeinformation with MongoDB7, in which the data isstored.

The third part, the Hexoskin Web server (Websteret al., 2017), contains the data for the patient wearing

6http://flask.palletsprojects.com/en/1.1.x/7https://www.mongodb.com/fr

the connected textile, and it is the Hexoskin Rest API8

and the classification system that make it possible tointeract and manipulate the Hexoskin data via HTTPrequests. This includes access to biometrics and userinformation, but also data annotation, advanced reportvisualization, metrics retrieval, and more.

4 RESULTS

This mobile application is developed for an mHealthcontext, that may be used for communication be-tween a patient undertaken a HBCR and a clinician.However, we will only present screenshots and func-tionalities to illustrate the dashboard dedicated to thedoctor. The patient’s perspective will be consideredin future work.

Authentication. Authentication is a process thatperforms identity verification. It is an importantphase in order to secure the application. To accessthe various features of the application, the user musttype his credential (login and password) into thecorresponding fields (Figure 8).

Patient’s Profile. The profile interface, as shownin Figure 8, is the first interface the user seesand interacts with. It provides a variety of pa-tient information such as name, birth date, heartrate, weight, height,etc. The user can then access themain menu to discover the features of our application.

Dashboard. The dashboard depicted in Figure 9shows the activities corresponding to the chosenrecord. Each activity is described by its name andduration. The patient’s advancement is also recordedin the form of progress bars, allowing the doctor tocheck whether the patient is following the exerciceprogram.

Medical Report Generation. After consulting thedashboard, the doctor can write his medical report.To do so, he clicks on ”Write a report” to accessthe report form. He indicates his patient’s state(improved, not improved or deteriorated), and thenwrites his notes concerning the records he has seen.

Medical Reports List. Each report is saved in thelist (Figure 8) where it is indexed by date, content,doctor’s name, record id and duration.

8https://api.hexoskin.com/docs/index.html#api-keys-and-oauth-requests

Figure 8: Login, patients profile and medical reports pages.

Patient’s Program. The doctor can set out an exer-cice program for his patients, setting the performancedurations of the individual activities as goals to reach.

This mHealth solution gathers information fromthe wearable sensors incorporated in an intelligenttextile and process it using a HAR approach accessi-ble via a mobile application. These kinds of applica-tions provide a novel approach to the health care sys-tem, resulting in new and better services such as con-stant monitoring of patients and remote consultationservices. Together, these services yield a faster andmore reliable health care industry with almost zerowaiting time, changing the classical approach of thehealth care system as a reactive service to that of apreemptive industry. In this project, the main focus isto improve the health care services for patients who,while their health status is not critical, still need con-stant monitoring. We have used the Hexoskin as aneasy-to-wear washable textile that tracks heart rate,breathing rate and acceleration and has been used inseveral research studies showing its efficiency (Villaret al., 2015). Moreover, having all the data recordedin the server allows it to be accessed rapidly when-ever needed, which is profitable in terms of energyand time. The classification model has an overall ac-curacy of 95.4% and a very short response time. Inaddition, we have developed an interactive interfacewhich allows the doctor to monitor his patient’s healtheasily from home.

The proposed HAR system is used for training andrecognition of human activities in daily living, and in-

tegrated in a mobile device as a first release version.An extensive work will be accomplished for train-ing and recognition of physical excercices in rehabil-itation process. The key requirements for the HARsystem designed to support the rehabilitation processwill be specified in conjunction with the rehabilitationteam.

5 CONCLUSIONS

In this paper, we have presented a multi-platformHAR-based mobile application for HBCR. We devel-oped an application that allows doctors to monitortargeted patients at their homes, using the Hexoskinintelligent-textile shirt. The shirt records accelera-tion signals which are then sent to our application foranalysis and classification, in order to recognize oneof the following activities: walking, running, fallingright, falling left, falling backward, falling forward,going up stairs, going down stairs, sitting and lying.The application allows the doctor to visualize the tex-tile recordings, each representing wearing of the Hex-oskin for one day, and to write reports about theserecords. The idea was to design an archive of medi-cal reports as a reference the doctor can use to betterjudge a patient’s condition. We aim to increase thepatients particpating rate in HBCR programs thanksto this system, so that they can lead a normal and safelife without having to worry about their health. Thework process involved several stages in order to ac-complish the development and implementation of the

Figure 9: The dashboard page.

application and to collect the data. The collection ofdata for the new database was also an important phasein developing the classification system. However, thisapplication is not limited to integrated functionalitiessuch as monitoring patient’s activity and writing re-ports.

In future work, we aim to add several types ofexercises that are included in a cardiac rehabilitationprogram according to the guidelines of the AmericanCollege of Sports Medicine (ACSM). In conjuctionwith a cardiac rehabilitation team, we will work ongathering empirical data from a set of patients withheart failure following a cardiac rehabilitation pro-gram. While, the activities we have examinated so farare quite simple, many of them could in fact be partof more complex routines or behaviors. For instance,swimming and cycling are composed of several in-stances of walking, running, and sitting (among otheractivities) characterized by a certain logical sequenceand duration. Moreover, integrating cardiac and res-piratory signals into the medical decision and the clas-sification system would be important for a better clas-sification rate. Future expanded datasets should alsoinclude more participants, especially patients in reha-bilitation. We intend also to conduct an extensive setof experiments to compare different machine learn-ing and deep learning approaches, especially ensem-ble methods, for human activity recognition using theacceleration data from the Hexoskin.

REFERENCES

Attal, F., Mohammed, S., Dedabrishvili, M., Chamroukhi,F., Oukhellou, L., and Amirat, Y. (2015). Physical Hu-man Activity Recognition Using Wearable Sensors.Sensors, 15(12):31314–31338.

Banerjee, T., Anantharam, P., Romine, W., Lawhorne, L.,and Sheth, A. (2015). Evaluating a potential com-mercial tool for healthcare application for people withdementia. In the International Conference on HealthInformatics and Medical Systems.

Cherif, N., Mezghani, N., Gaudreault, N., Ouakrim, Y.,Mouzoune, I., and Boulay, P. (2018a). Physiologi-cal data validation of the hexoskin smart textile. InBIODEVICES, pages 150–156.

Cherif, N., Ouakrim, Y., Benazza-Benyahia, A., andMezghani, N. (2018b). Physical activity classificationusing a smart textile. In 2018 IEEE Life Sciences Con-ference (LSC), pages 175–178.

Cyr, C., Robert, S., and Sylvie, P. (2018). Modeles dereadaptation cardiaque alternatifs utilisant les struc-tures de gestion de cas et des maladies chroniques.40:4–6.

Damasevicius, R., Vasiljevas, M., Salkevicius, J., andWozniak, M. (2016). Human activity recognition inaal environments using random projections. Com-putational and Mathematical Methods in Medicine,2016:1–17.

Dunka, B., Emmanuel, E., and Oyerinde, Y. (2017). Hybridmobile application based on ionic framework tech-nologies. International Journal of Recent Advancesin Multidisciplinary Research, 04:3121–3130.

Mampuya, W. (2012). Cardiac rehabilitation past, present

and future: An overview. Cardiovascular DiagnosisTherapeutics, 2:38.

Medina, J., Olmo, M., Aguilera, M., and Espinilla, M.(2017). Real-time monitoring in home-based car-diac rehabilitation using wrist-worn heart rate devices.Sensors, 17.

Paul, S. S., Tiedemann, A., Hassett, L. M., Ramsay, E.,Kirkham, C., Chagpar, S., and Sherrington, C. (2015).Validity of the fitbit activity tracker for measuringsteps in community-dwelling older adults. BMJ OpenSport & Exercise Medicine, 1(1).

Thomas, R., Beatty, A., Beckie, T., Brewer, L., Brown,T., Forman, D., Franklin, B., Keteyian, S., Kitzman,D., Regensteiner, J., Sanderson, B., and Whooley, M.(2019). Home-based cardiac rehabilitation: A scien-tific statement from the american association of car-diovascular and pulmonary rehabilitation, the ameri-can heart association, and the american college of car-diology. Journal of Cardiopulmonary Rehabilitationand Prevention, 39:1.

Varnfield, M., Karunanithi, M., Sarela, A., Garcia, E., Fair-full, A., Oldenburg, B., and Walters, D. (2011). Up-take of a technology-assisted home-care cardiac reha-bilitation program. The Medical journal of Australia,194:15–19.

Villar, R., Beltrame, T., and Hughson, R. (2015). Valida-tion of the hexoskin wearable vest during lying, sit-ting, standing and walking activities. Applied Physi-ology, Nutrition, and Metabolism, 40:1019–24.

Voicu, R.-A., Dobre, C., Bajenaru, L., and Ciobanu, R.-I. (2019). Human physical activity recognition usingsmartphone sensors. Sensors, 19:458.

Webster, E., Sukaviriya, N., Chang, H.-Y., and Kozloski,J. (2017). Predicting cognitive states from wearablerecordings of autonomic function. IBM Journal of Re-search and Development, 61:2:1–2:11.