Serious Games Application for Memory TrainingUsing Egocentric Images

Gabriel Oliveira-Barra1, Marc Bolanos1, Estefania Talavera1,2, AdrianDuenas1, Olga Gelonch3, and Maite Garolera3

1 Universitat de Barcelona2 University of Groningen

3 Consorci Sanitari de Terrassa

Abstract. Mild cognitive impairment is the early stage of several neu-rodegenerative diseases, such as Alzheimer’s. In this work, we addressthe use of lifelogging as a tool to obtain pictures from a patient’s dailylife from an egocentric point of view. We propose to use them in com-bination with serious games as a way to provide a non-pharmacologicaltreatment to improve their quality of life. To do so, we introduce a novelcomputer vision technique that classifies rich and non rich egocentric im-ages and uses them in serious games. We present results over a datasetcomposed by 10,997 images, recorded by 7 different users, achieving 79%of F1-score. Our model presents the first method used for automaticegocentric images selection applicable to serious games.

Keywords: lifelogging, serious games, egocentric vision, mild cognitiveimpairment, machine learning, computer vision

1 Introduction

Fig. 1: Person using the Narra-tive Clip camera.

Dementia can result from different causes,the most common being Alzheimers disease(AD) [10], and it is often preceded by a pre-dementia stage, known as Mild Cognitive Im-pairment (MCI), characterized by a cognitivedecline greater than expected by an individ-ual’s age, but which does not interfere notablywith their daily life activities [19,11]. Cur-rently, medical specialists design and applyspecial activities that could serve as a treat-ment tool for cognitive capabilities enhance-ment. Even though, these activities are notspecially designed for the patients, which lim-its their engagement in some cases.

A possible alternative to the application ofgeneric exercises would be the use of person-alized images of the daily life of the patients acquired by lifelogging devices.

arX

iv:1

707.

0882

1v1

[cs

.CV

] 2

7 Ju

l 201

7

II

Fig. 2: Examples of egocentric images recorded by the Narrative Clip camera.

Lifelogging consists of a user continuously recording their everyday experiences,typically via wearable sensors including accelerometers and cameras, among oth-ers. When the visual signal is the only one recorded, typically by a wearablecamera, it is referred to as visual lifelogging [4]. This is a trend that is rapidlyincreasing thanks to advances in wearable technologies over recent years. Nowa-days, wearable cameras are very small devices that can be worn all-day long andautomatically record the everyday activities of the wearer in a passive fashion,from a first-person point of view. As an example, Fig. 2 shows pictures taken bya person wearing such a camera.

Recent studies have described wearable cameras or lifelogging technologies asuseful devices for memory support for people with episodic memory impairment,such as the one present in MCI [15,8]. The design of new technologies to beapplied on this field requires to take into account people capabilities, limitations,needs and the acceptance of the wearable devices, since it can directly affect thetreatment. So far, some studies have deeply focus into the factors associated tothe use of these devices [24,13].

Lifelogging and privacy: In terms of privacy, in 2011, the European Unionagency ENISA evaluated the risks, threats and vulnerabilities of lifelogging ap-plications with respect to central topics as privacy and trust issues. In theirfinal report, they highlighted that lifelogging itself is still in its infancy butnevertheless will play an important role in the near future [3]. Therefore, theyrecommended further and extensive research in order to influence its evolution tobe better prepared to mitigate the risks and maximize the benefits of these tech-nologies. In addition, other researchers have also evaluated the possible ethicalrisks involved on using lifelogging devices on medical studies [7].

Serious games for MCI: Serious games (also known as games with a purpose)are digital applications specialized for purposes other than simply entertain-ing, such as informing, educating or enhancing physical and cognitive functions.Nowadays they are widely recognized as promising non-pharmacological toolsto help assess and evaluate functional impairments of patients, as well as to aidwith their treatment, stimulation, and rehabilitation [21]. Boosted by the publi-cation of a Nature letter showing that video game training can enhance cognitivecontrol in older adults [2], there is now a growing interest in developing seriousgames specifically adapted to people with AD and related disorders. Prelimi-nary evidence shows that serious games can successfully be employed to train

III

physical and cognitive abilities in people with AD, MCI, and related disorders[17]. [18] performed a literature review of the experimental studies conducted todate on the use of serious games in neurodegenerative disorders and [21] studiedrecommendations for the use of serious games in people with AD and relateddisorders, reporting positive effects on several health-related capabilities of MCIpatients such as voluntary motor control, cognitive functions like attention andmemory or social and emotional functions. For instance they can improve theirmood and increase their sociability, as well as reduce their depression.

Our contribution: Different studies have proven the benefits of directly stim-ulating the working memory. Our contribution in this paper consists in using asstimuli the autobiographical images of the MCI patients that was acquired bythe wearable cameras. By doing this, we intend to accomplish the goal of en-hancing their motivation and at the same time treat them in a more functionaland multimodal manner [9,16,1]. The application, which will allow the user toexercise either at the sanitary center or at home, will be composed by seriousgames where the patient has to observe a series of images and interact withthem.

Although the stimuli provided by egocentric images can be of greater impor-tance than non-personal images, it is important to note both, that egocentricimages are captured in an uncontrolled environment, and that wearable cam-eras usually have free motion that might cause most images to be blurry, darkor empty of semantic content. Considering this important limitations togetherwith the limited capabilities of MCI patients, we propose the development of anegocentric rich images detection system intended to select only images with se-mantic and relevant content. Our hypothesis is that, by using personal daily liferich images, the motivation of the patient will increase, and as a consequence,the health-related benefits provided by the treatment.

This paper is organized as follows. We describe the proposed serious gameand model for rich images selection in Section 2 and Section 3, respectively.In Section 4, we describes the experimental setup and show quantitative andqualitative evaluation. Finally, Section 5 draws conclusions and outlines futureworks.

2 Proposed Serious Game: ”Position Recall”

MCI patients experiment problems in their working memory [23], herefore, itis of high importance to do exercises for stimulating it. All this under the neu-roplasticity paradigm, which has proven that it is possible to modify the braincapabilities and the hypothesis of ”use it or lose it”, which are the basis of thestudies related to the cognitive stimulation of elderly people [22]. Thus, in thiswork, we introduce a serious game that we name as ”Position Recall”, which wasdesigned by neuropsychologist of Consorci Sanitari de Terrassa for improving theworking memory. The mechanics of this game follow this scheme:

IV

The first screen explains to the patient the instructions of the game and inthe second the patient is informed that, before starting the game, there willbe some practice examples that will serve to understand its logic. To start, thepatient must select his preferred level of difficulty (Level 1, 2 or 3).

– Level 1 shows 3 images of the patients’ day during 8 seconds and theyare asked to remember their positions. Immediately after they disappear, asingle ”target” image is shown and they are asked to select in what positionit was placed. After some trials the number of images displayed are increasedto 4 and then to 5.

– Level 2 follows the same procedure as the 1st level, but the timespan be-tween the moment where the images disappear and the target image is shownis increased. During this timespan, called latency time, a black screen isshown.

– Level 3 follows the same procedure as the 2nd level, but now a distrac-tor image is shown instead of a black screen during the latency time. Thedistractor image is also an image belonging to the patients’ day.

The reward system of the game are points that are given after each level,and are calculated as 100xnumber of correct answers. There are 10 trials perlevel translating into a maximum of 1000 points per level and maximum of 3000points per game. Figures 3a and 3b show the mechanics of the developed game.

(a) A predefined number of picturesof the patient is shown to him duringfew seconds at random positions in thescreen.

(b) After a certain time passed, the pa-tient is asked to recall in what posi-tion one of the pictures, picked up ran-domly, was placed before.

The images to be shown during the serious games should be significant for thepatient. We propose to use images that represent past moments of the user’s life,i.e. from the egocentric photostreams recorded by the patient. On the followingsection, we describe the proposed model for rich images selection.

V

3 What did I see? Rich images detection

The main factor for providing a meaningful image selection algorithm is thefact that the proposed serious games intend to work on cognitive and sentimentenhancement. Considering the free-motion and non-intentionality of the picturestaken by wearable cameras [4], it is very important to provide a robust methodfor images selection.

Two of the most important and basic factors that determine the memorabilityof an image [14,5] can be described as 1) the appearance of human faces, and2) the appearance of characteristic and recognizable objects. In this paper, wefocus on satisfying the second criterion by proposing an algorithm based oncomputer vision. Our proposal consists in a rich images detection algorithm,which intends to detect images with a high number of objects and variabilityand at the same time avoids images with low semantical content, understandingas rich any image that is neither blur, nor dark and that contains clearly visiblenon-occluded objects. In Fig. 4 we show the general pipeline of our proposal.

person (0.9)

glass (0.62)

glass (0.43)

table (0.42)

hand (0.79)

window (0.76)

Object Detection Pyramidal Division

Feature Vector Extraction

Level 1

Level 2

Level 3

1

23.4

Feature Vector Normalization

Random Forest Classification

No

#objects

colour var.

contains people?cell coverage

classconfidence 0.76

window80%

Level 3cell coverage

classconfidence

NoneNoneNone

.

.

.

Object 1

Object 2

. . .C

ell 1

Fig. 4: Scheme of the proposed rich images detection model.

Our algorithm for rich images detection (consists in 1) objects detection:where the neural network named YOLO9000 [20] is applied in order to detectany existent object in the images and their associated confidences ci. 2) the imageis divided in a pyramidal structure of cells, 3) a set of richness-related features

VI

are extracted, 4) the extracted features are normalized and 5) a Random ForestClassifier (RFC) [6] is trained to distinguish the differences between rich ornon-rich images. When extracting features, the image is divided in a pyramidalstructure of cells with different sizes at each level. The set of extracted featuresare:

– Numbers of objects the cell contains.– Variance of color in the cell.– Does the cell contain people?– Object Scale. Real number between 0 and 1.– Object Class. Class identifier that varies between 1 and 9418.– Object Confidence ci.

where all features are repeated for each cell and the last three kinds of featuresare repeated for each object appearing in the cells. The image cell divisionsapplied are 1x1, 2x2 and 3x3, the maximum of objects selected per cell are 5,3 and 2, respectively and all objects are sorted by their confidence ci beforeselection. If the number of objects is less than the maximum number are found,the feature value in that specific position is set to 0.

The pyramidal division of the images helps us consider smaller objects athigher levels (more cells) and bigger objects at lower levels (less cells). Thus,both small and big objects will be considered for the final prediction.

In order to define the feature ”Does the cell contain people?” we manuallyselected a set of person-related objects detected by the employed object detec-tion method. The concepts representing people that we selected are ”person”,”worker”, ”workman”, ”employee”, ”consumer”, ”groom” and ”bride”.

4 Results

This section describes the results obtained in a quantitative and qualitativeform. We compare the results obtained by variations of the proposed method ona self-made dataset of rich images.

Dataset: The dataset used for evaluating our model was acquired by the wear-able camera Narrative Clip 24, which takes a picture every thirty seconds auto-matically. The camera was worn during 15 days by 7 different people. Consideringthat on average the camera takes 1,500 images per day, our dataset consists of10,997 photographs.

The resulting data was labeled by neuropsychologist experts on MCI cogni-tion following the criteria that any rich image has to be 1) properly illuminated,2) not blurry and 3) contain one or more objects that are not occluded. Afterthis manual selection the acquired images where split in 6,399 rich images and4,598 non-rich images.

4 www.getnarrative.com

VII

In Fig. 5a we can see some examples of egocentric rich images and in Fig. 5bnon-rich images. We observe that rich images show people or recognizable places.However, non-rich images are meaningless or dark images (that can hardly beseen), including pictures of the sky, ceilings or floor.

(a) Rich images (b) Non-rich images

The resulting data was divided in training, validation, and test. Consideringthe pictures taken during the same day can be very similar, we proceeded torandomly separate the different days into the three different sets. First, thetraining set consists of 60% of the days, in this case 9. Second, 20% of the days,in this case 3, were defined as the validation set. Finally, the remaining 20% wasused for the test set.

Evaluation Metrics: In order to evaluate the different results and comparethem to get the best one, we make use of the F1-score (or F-measure) metric:

F1 = 2 ∗ 11

precision + 1recall

= 2 ∗ precision ∗ recallprecision + recall

where precision is the quotient between the number of True Positives objectsand the number of predicted positive elements; and recall is the quotient betweenthe number of True Positives objects and the number of real positive elements.

Quantitative Results: Currently, there are no previous works addressing thechallenge we introduce in this work. Thus, in order to compare the performanceof our proposed model, we have defined and compared several variations to ourmain pipeline (see results in Table 1).

As an alternative to our proposed approach (1), we tested an alternativefeature vector representation by means of using the (2) Word2Vec word embed-ding [12]. This word characterization is a 300-dimensional vector representationcreated by Google that represents words in space depending on their semanticmeaning (i.e. words with similar definitions will be represented close in space).The Word2Vec representation was used in two ways. On the one hand was used

VIII

for defining the set of concepts related to ”person” in the feature described as”Does the cell contain people?”. Thus, we computed the similarity between theword ”person” and any other concept detected in the image by the object detec-tion and the maximum similarity achieved was used as an alternative to a 0/1representation. On the other hand, the feature described as ”Object Class” wasreplaced by the 300-dimensions Word2Vec representation.

In the test setting (3) we additionally applied a PCA dimensionality reduc-tion to the Word2Vec representation. Finally, in (4) we used a Support VectorMachine (SVM) classifier instead of a Random Forest Classifier. We applied aGrid Search on the variables C and gamma for parameter selection over thevalidation set.

Precision Recall F1-score(1) RFC 0.79 0.79 0.79(2) RFC + Word2Vec 0.78 0.78 0.78(3) RFC + Word2Vec + PCA 0.74 0.75 0.75(4) SVM 0.68 0.67 0.68

Table 1: Comparison of the results

In conclusion we can see that using an RFC classifier (1) obtains better resultsthan SVM (4) and at the same time none of the Word2Vec representations (2)and (3) helped improving the base results.

Fig. 6: Example of rich (left) images selection, vs non-rich images rejection. Froman egocentric photostream composed by 972 images, 221 were considered rich.

Qualitative Results: Examples of the selected images by the proposed algo-rithm are shown in Fig. 6. On one hand, we can observe that rich images (left)are clearer, without shadows and with people or focused objects, which allowsthe user to infer what is happening in the scene. On the other hand, non-rich

IX

images (right) are discarded since they are not illustrative and make difficult thescene interpretation.

Images selected by the proposed model are rich in information and memorytrigger. We can foresee that the proposed model cannot only be used for seriousgames images selection, but also as a tool for images selection for autobiograph-ical memories creation.

5 Conclusions

In this work, we have introduced a novel type of wearable computing application,aiming to provide non pharmacological treatment for MCI patients and to im-prove their life quality. We discussed lifelogging pictures obtained from wearablecameras combined with serious games as a channel for personalized treatments.We also introduced and tested a novel computer vision technique to classify richand non rich images obtained from first-person point of view. We obtain 79%F1-score, promising results that will be further studied.

As future work, we will implement more serious games to be included in theapplication tool. Specialists will use it for MCI patients, aiming to prove thethe memory reinforcement hypothesis introduced in this work, as well as themotivation experienced by the subjects increase when using personalized richimages and serious games. Furthermore, in [25], positiveness from egocentricimages was addressed. Moreover, we will go deeper on the analysis of usersacceptance over the proposed technology, their willingness to use it, and thefactors that determine their acceptance toward it. Further improvements of themethodology will be developed in order to obtain more accurate results.

Acknowledgements

This work was partially founded by Ministerio de Ciencia e Innovacion of the Go-bierno de Espana, through the research project TIN2015-66951-C2. SGR 1219,CERCA, ICREA Academia 2014, Grant 20141510 (Marato TV3) and GrantFPU15/01347. The funders had no role in the study design, data collection,analysis, and preparation of the manuscript. The authors gratefully acknowl-edge the support of NVIDIA Corporation with the donation of the Titan XpGPU used for this research.

References

1. Alves, J., Alves-Costa, F., Magalhaes, R., Goncalves, O.F., Sampaio, A.: Cognitivestimulation for portuguese older adults with cognitive impairment: a randomizedcontrolled trial of efficacy, comparative duration, feasibility, and experiential rel-evance. American Journal of Alzheimer’s Disease & Other Dementias R© 29(6),503–512 (2014)

2. Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich,J., Kong, E., Larraburo, Y., Rolle, C., Johnston, E., et al.: Video game trainingenhances cognitive control in older adults. Nature 501(7465), 97–101 (2013)

X

3. Askoxylakis, I., Brown, I., Dickman, P., Friedewald, M., Irion, K., Kosta, E.,Langheinrich, M., McCarthy, P., Osimo, D., Papiotis, S., et al.: To log or notto log?-risks and benefits of emerging life-logging applications (2011)

4. Bolanos, M., Dimiccoli, M., Radeva, P.: Toward storytelling from visual lifelogging:An overview. IEEE Transactions on Human-Machine Systems 47(1), 77–90 (2017)

5. Carne, M., Giro-i Nieto, X., Radeva, P., Gurrin, C.: Egomemnet: Visual memora-bility adaptation to egocentric images

6. Criminisi, A., Shotton, J., Konukoglu, E., et al.: Decision forests: A unified frame-work for classification, regression, density estimation, manifold learning and semi-supervised learning. Foundations and Trends R© in Computer Graphics and Vision7(2–3), 81–227 (2012)

7. Doherty, A.R., Hodges, S.E., King, A.C., Smeaton, A.F., Berry, E., Moulin, C.J.,Lindley, S., Kelly, P., Foster, C.: Wearable cameras in health. American journal ofpreventive medicine 44(3), 320–323 (2013)

8. Doherty, A.R., Pauly-Takacs, K., Caprani, N., Gurrin, C., Moulin, C.J., O’Connor,N.E., Smeaton, A.F.: Experiences of aiding autobiographical memory using thesensecam. Human–Computer Interaction 27(1-2), 151–174 (2012)

9. Flak, M.M., Hernes, S.S., Skranes, J., Løhaugen, G.C.: The memory aid study:protocol for a randomized controlled clinical trial evaluating the effect of computer-based working memory training in elderly patients with mild cognitive impairment(mci). Trials 15(1), 156 (2014)

10. Fratiglioni, L., Grut, M., Forsell, Y., Viitanen, M., Grafstrom, M., Holmen, K.,Ericsson, K., Backman, L., Ahlbom, A., Winblad, B.: Prevalence of alzheimer’sdisease and other dementias in an elderly urban population relationship with age,sex, and education. Neurology 41(12), 1886–1886 (1991)

11. Gauthier, S., Reisberg, B., Zaudig, M., Petersen, R.C., Ritchie, K., Broich, K.,Belleville, S., Brodaty, H., Bennett, D., Chertkow, H., et al.: Mild cognitive im-pairment. The Lancet 367(9518), 1262–1270 (2006)

12. Goldberg, Y., Levy, O.: word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722 (2014)

13. Gurrin, C., Smeaton, A.F., Doherty, A.R., et al.: Lifelogging: Personal big data.Foundations and Trends R© in Information Retrieval 8(1), 1–125 (2014)

14. Khosla, A., Raju, A.S., Torralba, A., Oliva, A.: Understanding and predictingimage memorability at a large scale. In: Proceedings of the IEEE InternationalConference on Computer Vision. pp. 2390–2398 (2015)

15. Lee, M.L., Dey, A.K.: Lifelogging memory appliance for people with episodic mem-ory impairment. In: Proceedings of the 10th international conference on Ubiquitouscomputing. pp. 44–53. ACM (2008)

16. Li, H., Li, J., Li, N., Li, B., Wang, P., Zhou, T.: Cognitive intervention for personswith mild cognitive impairment: A meta-analysis. Ageing Research Reviews 10(2),285–296 (2011)

17. Manera, V., Petit, P.D., Derreumaux, A., Orvieto, I., Romagnoli, M., Lyttle, G.,David, R., Robert, P.H.: kitchen and cooking,a serious game for mild cognitiveimpairment and alzheimers disease: a pilot study. Frontiers in aging neuroscience7 (2015)

18. McCallum, S., Boletsis, C.: Dementia games: A literature review of dementia-related serious games. In: International Conference on Serious Games Developmentand Applications. pp. 15–27. Springer (2013)

19. Petersen, R.C., Smith, G.E., Waring, S.C., Ivnik, R.J., Tangalos, E.G., Kokmen,E.: Mild cognitive impairment: clinical characterization and outcome. Archives ofneurology 56(3), 303–308 (1999)

XI

20. Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. arXiv preprintarXiv:1612.08242 (2016)

21. Robert, P.H., Konig, A., Amieva, H., Andrieu, S., Bremond, F., Bullock, R., Cec-caldi, M., Dubois, B., Gauthier, S., Kenigsberg, P.A., et al.: Recommendations forthe use of serious games in people with alzheimer’s disease, related disorders andfrailty. Frontiers in aging neuroscience 6 (2014)

22. Salthouse, T.A.: Mental exercise and mental aging: Evaluating the validity of theuse it or lose it hypothesis. Perspectives on Psychological Science 1(1), 68–87 (2006)

23. Saunders, N.L., Summers, M.J.: Attention and working memory deficits in mildcognitive impairment. Journal of Clinical and Experimental Neuropsychology32(4), 350–357 (2010)

24. Sellen, A.J., Whittaker, S.: Beyond total capture: a constructive critique of lifel-ogging. Communications of the ACM 53(5), 70–77 (2010)

25. Talavera, E., Strisciuglio, N., Petkov, N., Radeva, P.: Sentiment Recognition inEgocentric Photostreams (2017)