Towards Virtual Rehabilitation: Hand Gesture Controlled Games to

Improve Cognitive and Motor Skills of Children with Special Needs

Thiwanka A. Makumburage2013/MCS/048

2016 Supervisor

Dr. G.D.S.P. Wimalaratne

Nearly 34,000 of children in Sri Lanka has difficulties with cognitive and motor

skills

Support tools in existence are

either traditional or out of date.

Healthcare professionals are very hard to find and their

services are very expensive to be

afforded

What are Serious Games?

Can we play games using

hand gestures?

Will games Improve human

skills?

Introduction

Aim

The aim of the research on a specialized approach to help the children with special needs on their rehabilitation exercise, using serious games controlled by hand gestures, to improve their cognitive functions and motor skills.

Objectives Track the hand gestures so precisely, that it can be used to

identify the improvements of the human body. Uniquely identifying fingers, joints of a hand leads to better information input to the serious game.

Identify the type of serious games that are suitable for the cognitive and motor skills development of a child.

Develop a process by utilizing real-time precision hand gesture tracking to bridge the hand gestures with the game actions.

Collection and analyze of the game play data, measure the improvements of children’s cognitive and motor skills.

Research Literature review

Identifying serious games

Creating taxonomy of serious games

Evaluation of the related work

Developing serious games for children with cerebral palsy: case study and pilot trial [1]

Discusses a series of approaches used to develop appropriate serious games for children with cerebral palsy. They have used two sample games shown above, including data and feedback from one child with cerebral palsy who evaluated the games for entertainment, engagement and replay value. This research only focuses on improvements of the cognitive functions of the cerebral palsy effected children.

Help Me! A Serious Game for Rehabilitation of Children Affected by Cerebral Visual Impairment [2]

The system integrates an eye tracker system to correctly measure the performances of the child. The developed game was called HelpMe! And has two goals: to train the ability to see and touch an object at the same time and to train the problem-solving capabilities of the child.

Azahar: Serious games, improving autonomy [3]

Azahar is a platform for free, customizable applications and Serious Games designed to improve the quality of life and autonomy of people with autism and/or intellectual disability

Gesture based serious games for physical therapy [4]

This is a game developed by advancedmedialab.com in Saudi Arabia as a replacement for physiotherapy exercises. It’s an online flying game that can be played by a patient using hand and body gestures. The gestures used to play the game are the ones required to perform physical therapy. Difficulty level of the game can be changed automatically by the system or remotely by the therapist according to the recovery behavior of each patient.

Visited Sri Lankan therapy units National Hospital of Sri Lanka

Lady Ridgeway Hospital for Children

Evaluation of hand gesture recognition and hand gesture controllers.

Contribution Taxonomy built by categorizing the serious games. Identify the type of serious games that are suitable for the

cognitive and motor skills development of a child.

Develop a process by utilizing real-time precision hand gesture tracking to bridge the hand gestures with the game actions.

‘Leap Play’ application, to improve the cognitive and motor skills of the children with special needs.

Children who participated for the evaluation of the research, got their cognitive and motor skills improved.

System Architecture

Process

Process Flow

Algorithms Scoring

[5]

T = Theoretical minimum time spent to finish the level (in seconds)

Tx = Time spent to finish the level (in seconds)

N = Number of times repeated the level

M = Maximum marks can be achieved from the level

Coordinate conversion

Technology Java 8

Leap Motion Java SDK (Version 2.2.3)

JavaFX 2

MySQL

Evaluation Six users

Multiple Scenarios

Usability feedback

Critical evaluation of the results

Results

Gesture recognition

Evaluation Form

Sample (User B) :

Conclusion Using a Natural User Interface, LeapPlay application was able

to improve the skills of disabled children using the serious games specially designed for therapy.

This project introduces a new way of utilizing hand gestures controls for the wellbeing of the society.

The literature review on serious games and hand gesture recognition lead to better understanding of the knowledge area.

Visits to the National Hospital of Sri Lanka and Lady Ridgeway Hospital for Children gave insight of the problem in Sri Lankan context.

Detailed comparison between hand gestures controllers lead to the identification of the Leap Motion device as a good gesture tracker .

LeapPlay system was implemented according to the design with the limitations mentioned in the design chapter.

The project was evaluated successfully and the results were positive towards achieving the goal.

Finally this project will provide a strong case to support the fact that serious games can contribute towards the virtual rehabilitation.

Lessons learned Identified multiple ways to precision hand gesture tracking.

Fatigue is a major concern for long duration game play.

Future work Integrate the system with haptic feedback gloves.

Proof of Concept

References[1] Henschke, M., Hobbs, D., & Wilkinson, B. (2012, November). Developing serious games for children with cerebral palsy: case study and pilot trial. In Proceedings of the 24th Australian Computer-Human Interaction Conference (pp. 212-221). ACM [2] Ciman, M., Gaggi, O., Nota, L., Pinello, L., Riparelli, N., & Sgaramella, T. M. (2013). Help Me! A Serious Game for Rehabilitation of Children Affected by CVI. In WEBIST (pp. 257-262).[3] Azahar: Serious Games Improving Autonomy , http://seriousgamesmarket.blogspot.com/2015/09/azahar-serious-games-improving-autonomy.html [4] Serious Games for Physical Threapy , http://advancedmedialab.com/e-health/products/gesture-based-serious-games-for-physical-therapy.html[5] R.A.G.K. Ranathunga, L.A.V.N. Rajakaruna, S.A.T.N. Karunarathne, L.N.Y. Abeywardena,D.P. Nawinna, T. Halloluwa (2014) 'A Gamified Learning Tool for Sri Lankan Primary Schools ', PNCTM, Vol 3.

Thank You