Augmented Reality-based Exergames for Rehabilitation

Kevin Desai, Kanchan Bahirat, SudhirRamalingam, Balakrishnan Prabhakaran

The University of Texas at Dallas{Kevin.Desai, Kanchan.Bahirat,

Sudhir.Ramalingam,bprabhakaran}@utdallas.edu

Thiru Annaswamy, Una E. MakrisDallas Veterans Affairs and University of Texas

Southwestern Medical CenterThiru.Annaswamy@va.gov,

Una.Makris@UTSouthwestern.edu

ABSTRACTRehabilitation for stroke afflicted patients, through exercisestailored for individual needs, aims at relearning basic motorskills, especially in the extremities. Rehabilitation throughAugmented Reality (AR) based games engage and motivatepatients to perform exercises which, otherwise, maybe bor-ing and monotonic. Also, mirror therapy allows users toobserve one’s own movements in the game providing themwith good visual feedback. This paper presents an aug-mented reality based system for rehabilitation by playingfour interactive, cognitive and fun Exergames (exercise andgaming).

The system uses low-cost RGB-D cameras such as Mi-crosoft Kinect V2 to capture and generate 3D model of theperson by extracting him/her from the entire captured dataand immersing it in different interactive virtual environ-ments. Animation based limb movement enhancement alongwith cognitive aspects incorporated in the game can helpin positive reinforcement, progressive challenges and motionimprovement. Recording module of the toolkit allows fu-ture reference and facilitates feedback from the physician.10 able-bodied users, 2 psychological experts and 2 PhysicalMedicine and Rehabilitation physicians evaluated the userexperience and usability aspects of the exergames. Resultsobtained shows the games to be fun and realistic, and at thesame time engaging and motivating for performing exercises.

CCS Concepts•Computing methodologies → Mixed / augmentedreality; •Applied computing → Interactive learningenvironments; •Software and its engineering→ Inter-active games; •Information systems→Multimedia infor-mation systems;

1. INTRODUCTIONAccording to the Center for Disease Control and Preven-

tion (CDC), approximately 800,000 people suffer from strokeevery year in the United States [1]. About 130,000 of them

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.

MMSys’16, May 10-13, 2016, Klagenfurt, Austriac© 2016 ACM. ISBN 978-1-4503-4297-1/16/05. . . $15.00

DOI: http://dx.doi.org/10.1145/2910017.2910612

die every year, making stroke the fifth leading cause of death[1] and the leading cause for serious long term disability [1].Interruption in the normal flow of blood or bleeding in thebrain, damage the brain cells that start dying within min-utes leading to a condition known as Stroke. Damage to thebrain cells triggers symptoms in parts of the body they con-trolled, attention deficiency, long-term disability, weaknessand partial paralysis on localized regions or even death[2].Usually patients suffer from post-stroke impairments, whichresult in loss of ability to carry out normal daily tasks. Ap-proximately 80% of the stroke survivors suffer from upperlimb impairments [2]. More than 40% of the patients receiv-ing lower limb rehabilitation therapy sessions are not ableto walk freely even after completing therapy [18].

Rehabilitation for patients affected by stroke involves mo-tion recovery of the affected body parts, mainly extremi-ties, through intense, skillful training exercises. Variousstrategies for rehabilitation [9] include physical tasks suchas reaching, flexing and grasping objects of daily use (e.g.,coffee cup, phone, TV remote, ball, etc.) for upper limbenhancement. Lower limb rehabilitation involves task spe-cific therapy such as treadmill training with partial supportof body weight, hip and knee extension exercises. Throughrepetitive exercising under therapist supervision, the areasof the brain affected by stroke get stimulated. Over a periodof time, this can result in relearning the basic movements [2].Action observation of performed tasks through videos, im-ages or even live stream improves the movement recoverysignificantly [30]. In recent years, Exergaming (exercise andgaming) [32], also known as Serious Gaming, has emergedas a cost effective rehabilitation tool [29], making repetitiveexercising a fun and motivating task.

One way of improving motion recovery using action ob-servation suggested in [30, 26, 13] is the use of AugmentedReality (AR). Most of the AR systems proposed in the lit-erature for rehabilitation exercises need external devices orsensors to be worn by a person. This is invasive in the sensethat it inhibits the normal user movements making themuncomfortable [3, 5, 6, 10, 34], and hence cannot be usedfor in-home rehab. Non-invasive systems proposed in [15,31] use webcams to capture the person which is in 2D andhence needs another device to track the hand movementsin 3D. Having various different devices not only makes thesystem limited to complex laboratory setup, but also bumpsup the overall cost. Virtual avatars have been used in someof the systems to replicate the human movements. With theadvent of low-cost 3D cameras, it has become possible tocapture “live” human models/avatars and immerse them in

AR games. This approach is non-invasive in the sense thatno additional sensors need to be worn by a patient. Also,using such live human models/avatars helps in understand-ing the emotional experience of the persons carrying out theexergames. The challenge here is that obtaining, generating,and rendering 3D mesh models of the human present in thescene needs to be done in real-time. Along with this task of3D model capturing and rendering, a nice augmented envi-ronmental setup is needed in order to make the exercisingtask fun, engaging and non-monotonous.

1.1 Proposed SystemThe proposed architecture uses Microsoft Kinect V2 [8], a

low-cost and off the shelf depth sensing, non-invasive, mark-erless camera, to capture the entire scene in real-time. Apartfrom RGB and depth streams, Kinect V2 also provides askeletal stream that provides the movements of the humanbody joints’ movements in 3D space. This skeletal datastream is used to perform physically based interactions. Thecaptured 3D human model is placed into an AR scene forexercising by playing games using Mirror therapy [33, 35].Animation based virtual enhancement is carried out on thelimbs to provide positive reinforcement and motivation toperform tasks in the AR space that are not possible in thereal world. Based on this approach, a toolkit consisting of4 immersive exergames is made available for post-stroke re-habilitation. The toolkit also contains additional featuresfor game selection, session recording, and playback (for fu-ture reference and expert feedback), and user mode selection(physician vs. patient). The four immersive games in thetoolkit - ShotPut, Bowling, Balance, and RehabQuiz - tar-get different rehab exercises, namely Elbow Flexion, ElbowExtension, Elbow Rotation and Hip Abduction. Along withthe physical task being performed, the RehabQuiz game in-corporates a cognitive aspect by including general knowledgequestions to be answered.

1.2 ContributionsThe proposed exergaming system was tested and evalu-

ated by 10 people with no known disabilities, 2 psychologyexperts and 2 Physical Medicine and Rehab physicians. Inthis paper, we make the following contributions:

• Cost effective, easy to use, non-invasive system for live3D capture of a person and its use in various aug-mented reality settings.

• Mirrored view of participants help understand theiremotional experience during the game-play, especiallyfor physicians reviewing the recorded game sessions.

• Use of animation based virtual enhancement performedon the skeletal joints as well as mirror therapy to assistpatient rehabilitation.

• The feedback from the 2 Physical Medicine and Rehabphysicians and the 2 psychology experts shows that theaugmented reality based exergaming system is enjoy-able, engaging, assistive and motivating, having goodpotential in rehabilitation for stroke affected patients.

2. RELATED WORKSerious games for rehabilitation are designed by keeping in

mind that they should: (a) be fun to play; (b) have a level of

uncertainty in output; (c) be accompanied by awareness ofa virtual reality [4]. Essential components of effective strokerehabilitation therapy, as mentioned in [20], include:

• Adaptability to motor skill level, meaningful tasks, ap-propriate feedback, theory appropriate range of mo-tion, focus diversion from exercise, intentional move-ment and quantitative measures.

• Include criteria such as personalization, ease of use,intuitive interface, default mode settings and efficientsetup.

• Provide appropriate challenge, motivational feedback,easy availability, and affordability.

• Immerse the person in the virtual scene at a sufficientlyfast rate so as to have a good, smooth and lag-freerendering.

Mirror therapy [33, 35] is the use of the mirrors to re-flect the non-paretic arm movements to make the affectedhand function as if it were normal. It has been proved thatmirror therapy improves the hand functioning more thangeneral rehabilitation as shown in [35]. Two groups tookpart in the experiments: one just performed the task andthe other group people, before performing the actual task,went through 30 minute session of observing their own non-paretic hand movements through a mirror giving them anillusion of a virtual hand performing the same movementtasks. The group that used mirror therapy performed bet-ter in the normal tasks and proved that such illusion orvirtual enhancement can be used to facilitate and speed therehabilitation. Augmented Reality Technology (ART) sys-tem proposed in [23] is able to manipulate the experience ofreality around the aspects of belief, interactivity, predictabil-ity and decoupling and demonstrates the potential of AR toimprove health and well-being.

In lower limb rehab, robot based technologies along withtreadmill training with body weight support are used forrepetitive locomotive gait rehabilitation with a special focuson children with cerebral palsy [7, 16, 19]. A VR-based soc-cer scenario was developed to provide interactive elements toengage patients during robot assisted treadmill training [3].The study used 18 children who were asked to walk on theLokomat, which is equipped with sensors at hip and kneejoints. The study showed that the use of VR scenario in-duced an immediate effect on motor output thereby helpingin gait rehabilitation.

Over the past few years, various attempts have been madeat developing serious games by using robotic/haptic controlsalong with commercial games as well as standalone con-sole games created for specific rehabilitation tasks. Becausethese games are considered fun, patients are more likely toengage in the activities and this may lead to improved sus-tainability of exercise routines. Robot based rehabilitationis used for reducing the expert therapist involvement andprovide for mass repetitive training. Some robotic devicesfor upper limb rehabilitation include MIT-MANUS [17], Bi-Manu Track [14] and T-WREX [25]. Such systems motivatethe patients to perform repetitive tasks by making it easyand forcefully assistive, but at the same time the invasivenature, cost and lab setup requirements make it unsuitablefor in-home rehab. Use of various commercial cameras suchas Sony EyeToy, Nintendo Wii, OptiTrack, Microsoft Kinect

Figure 1: System schematic showing the major modules: 3D model Generation, Exergame Toolkit and Virtual Enhancement.

and other webcams have increased to generate augmentedreality setups for rehab by capturing and imitating usermovements. Such off the shelf cameras make the game cre-ation and deployment easy and affordable, and hence haverecently gained much interest in physical therapies. Apartfrom Kinect, almost all other commercial camera setups re-quire additional invasive procedures for the 3D capture.

Sony PlayStation 2 EyeToy is used in [22] to generatea video-capture VR platform for creating games but needspeople to stand and play, making it inconvenient for thosehaving balance impairments to perform exercises. Ther-aMem system showed in [24], “fools the brain” by visuallyamplifying users’ hand movements - small actual hand move-ments lead to perceived larger movements. The results fromforty-five non-clinical participants validate the system butit is specifically for hand and finger movements. GestureTherapy system showed in [31] provides a very cheap al-ternative by using webcams for capture and interact withvirtual games by performing arm movement exercises. Theuse of webcam in rehabilitation restricts the movement in 2Dplane and hence needs an additional gripper having a col-ored ball to track the 3D arm position. Use of such systemneeds the stroke patient to have the ability to hold the ballwithout occlusion and the 3D movement is also restricted.Theragame[15] is a really cost efficient and easy to use VRsystem using a single webcam to perform in-home rehabil-itation. However, it is not able to capture the 3D motionmaking it ineffective for general rehab.

3D infrared passive motion capture system in [10] uses 10cameras for recording the user movements. Reflective mark-ers are placed on the body, such as hand, arm, shoulder andtrunk so as to get their positions in 3D space. Participantssat on a table and performed reaching movements towardsreal objects as well as virtual objects, with guidance from vi-sual and audio feedback in the large display in front. Majorcons for using such a setup for in-home therapy include highcost and size of the system as well as the need of wearing ex-

tra markers on the body. Games are developed using Kinectfor non-invasive tasks such as EEG-based functional brainmapping as shown in [27]. A Kinect based motor rehabilita-tion system is developed and evaluated in [12]. The systemis shows an exergame with support for multiple biomechan-ical movements along with analysis and postural compensa-tion. This prototype system performs well on the evaluatedhealthy users but does not provide any assistive feedback interms of virtual enhancement. A single Microsoft Kinect, isused by [11] in creating a duck punching game to performreaching tasks by punching the virtual duck targets placedin different virtual environments. The movement of the pa-tient is emulated using a virtual arm that can be enhancedin a controlled manner. The game tracked the 3D locationand showed the virtual arm but lacked the complete visualaspect of showing the real person in the game.

OptiTrack system consisting of 8 near infrared cameras isused in [5, 6, 34] to capture and track the reflective wrist-band worn by the person. Based on the tracked location andthe pressure feedback obtained from the chair and the table,correct movement of the person is obtained. Both audio andvisual feedbacks are provided by the system during rehabil-itation. The OptiTrack system is not as expensive as [10]but it needs expert knowledge for set up and use. Also, itis invasive, needing the person to wear reflective markers onthe affected limbs, and it is not easy to deploy for in-homerehab.

3. PROPOSED SYSTEM DESIGN & ARCHI-TECTURE

The proposed system consists of 3 major parts, as shownin Figure 1: (i) Human 3D model generation; (ii) Augmentedreality toolkit for exergames; and (iii) Virtual enhancement.An exergame toolkit consisting of 4 augmented reality basedgames is developed as explained in section 3.1. The partici-pant, a patient or a physician, needs to be placed in the ARscenario by capturing and generating a 3D mesh model as

explained in section 3.2. Animation based virtual limb en-hancement, section 3.3, is then incorporated along with themirror therapy to assist with patient’s movements in the ARspace.

Figure 3: Exergaming toolkit, showing game selection basedon body part, user profile creation/loading and enhancementratio selection.

3.1 Exergame ToolkitAn easy to use interface is provided for browsing and se-

lecting a game based on the name or the body joint that onewants to perform exercise on, e.g. elbow, hip, all, etc., asshown in Figure 3. User profiles can be created and saved soas to use them for future reference or for therapist/physicianreview. A drop down menu is provided for selecting the de-sired assistance ratio provided by the system in terms ofvirtual limb enhancement.

Different exercises that are showcased include Elbow Flex-ion, Elbow Extension, Elbow Rotation and Hip Abduction,as shown in Figure 2. The exercises mentioned are adaptedfrom publicly available websites supervised or created by cer-tified rehabilitation specialists - http://www.stroke-rehab.com/ & http://www.hep2go.com/. Based on the exercises, 4different games having one-to-one correspondence to the ex-ercises are developed - ShotPut (Figure 8) for Elbow Flexion(Figure 2a), Bowling (Figure 9) for Elbow Extension (Fig-ure 2b), RehabQuiz (Figure 10) for Elbow Rotation (Figure2c) and Balance (Figure 11) for Hip Abduction (Figure 2d).RehabQuiz not only makes the person perform the physicalexercise, but also presents a focus on cognitive aspect to an-swer some basic set of general knowledge questions. In thistoolkit, we facilitate exercises on lower part of the body i.e.hip abduction in Balance, which not many researchers havefocused on.

A recording facility is provided with the system that willallow both physician and the patient to view the action per-formed for future reference and modifications. The live feedof the entire scene is recorded while the game is being playedand a video is generated at the end of the game playing. Thisallows for session recording and a feedback for the physicianto understand the correctness of performed motions as wellas the patient’s emotional experience. Although the gamehas a scoring method, the patient might not adhere to thecorrect motions exactly and hence, the video generated canact as an added check for correctness of movements. Alongwith the entire scene recording, a facility is provided for the

physician to record the correct movements to be performedin each game that can act as reference for the patient. Figure4 shows the system in which the physician can interactivelyselect a game. After a countdown, the system allows thephysician to perform 15 seconds of recording for the partic-ular game selected. The recorded video of the physician canserve as a reference for the patient during in-home rehab.

All the games use the same underlying architecture: (a)person identification and tracking using Kinect; (b) physi-cally based interaction with virtual objects; and (c) virtuallimb movement enhancement. Virtual scoring is maintainedto motivate people to compete with themselves for improv-ing their skill in turn leading to improvement in their motorabilities. Different scene locations such as an Olympic sta-dium, an indoor bowling arena, beach and a closed room areput up so as to showcase the AR capabilities of the proposedtoolkit as well as to increase the interest and fun aspect ofthe game. To incorporate action observation, an animationbased model is used incorporating the virtual enhancementbased motions, as explained in section 3.3.

Figure 4: Recording system showing the person trying torecord a game by selecting it.

3.2 3D Model GenerationKinect V2 is used to capture the scene and generate the

3D model of the person present in the scene. Depth andcolor are the major streams provided by Kinect. Randomdecision forest algorithm is run on the depth image to ex-tract the features which are used in the estimation of skeletallimbs, as described in [28]. A human skeleton is fit on theextracted features to obtain a 25 joint skeleton model asshown in Figure 5a. Point cloud is obtained per frame usingthe depth image. Using the depth point cloud and the RGBimage obtained from Kinect V2, a mesh is generated. Densemeshing strategy is used [21], which is a 2D meshing strategythat connects the neighboring vertices if their depth valuesare not significantly apart. Figure 5b shows one such meshthat is generated corresponding to the skeleton extracted.

The enhancement procedure (Section 3.3) works on thegiven skeleton information and generates a modified skele-ton. Figure 5c shows one such case where enhancement isseen in the raised right leg as well as left arm. Rendering ofthe 3D model requires the mesh to be modified correspond-ing to the modified skeleton. An animation based enhance-ment strategy is incorporated as explained in [21]. Here,the enhanced skeleton, the original skeleton and the originalmesh are used to generate the new enhanced mesh of the per-son, as shown in Figure 5d. The entire point cloud obtainedis first segmented into different body part segments basedon the skeletal joint using a Voronoi decomposition basedapproach. All the points present in each segment are trans-formed to a spherical co-ordinate system based on a control

(a) (b) (c) (d)

Figure 2: Exercises used in games generated (a) Elbow Flexion (b) Elbow Extension (c) Elbow Rotation (d) Hip Abduction.

(a) (b) (c) (d)

Figure 5: Complete animation procedure with changesmarked: (a) Original Skeleton as captured by Kinect (b)Mesh generated on the original skeleton and captured pointcloud (c) Skeleton obtained after enhancement (d) Mesh an-imated using the original mesh, original skeleton and en-hanced skeleton.

point, i.e., the skeletal joint. For the enhanced skeleton, anew control point (skeletal joint) is used to place the spher-ical points in each segment back in the original Cartesiancoordinate system. Merging all body segments yields thefinal animated model using the virtually enhanced skeleton.

3.3 Virtual EnhancementFor patients suffering from motor impairments, one of the

major hindrances in improvement is the lack of motivation.Positive reinforcement accelerates the movement recoveryduring rehabilitation and can be provided by enhancing themotion of the paretic limb in the virtual world [11]. Us-ing this concept, we propose an animation based virtual en-hancement of the limb movements in which we animate theskeleton based on the previous skeleton by enhancing itsmovement.

In order to apply virtual enhancement on the skeleton,it is important to understand the direction and speed ofthe movement. Skeletal information is obtained in terms of25 joints that are updated at a refresh rate of more than30 frames per second. At any point in time, we maintaintwo skeleton structures: St−1 and St. Displacement of anyjoint p is obtained using its position in skeleton at two timeframes, pt and pt−1. Direction of displacement vector givesthe direction for virtual enhancement. The magnitude of dis-placement, along with an enhancement factor determined bythe therapist, decides the actual enhancement applied to thejoint. For realistic enhancement, modification in any jointmust also be applied to other joints connected to it, basedon the local hierarchy of parent and child bones. We alsoprovide two constraints for each bone describing a minimumangle and maximum angle of rotation between current boneand its parent bone. Though the proposed enhancement canbe applied to any joint movement, we focus only on arm andleg.

(a) (b) (c)

Figure 6: Illustration of enhancement for a simple wristmovement (a) pose of skeletal joints at time instance t − 1and t and enhancement estimation for wrist joint withoutnormalization shown with orange point and (b) with nor-malization shown with red point and (c) enhanced pose ob-tained with the proposed method

Figure 6 illustrates the application of proposed virtual en-hancement to wrist movement where w, e, s represent wrist,elbow and shoulder joints respectively. To enhance the wristmovement captured between the time instance t − 1 and t,we compute a directional vector between wt and wt−1 asillustrated in Figure 6, using the following equation:

| ~E| = |~d| ∗ Ef = |wt − wt−1| ∗ Ef (1)

where ~d indicates movement displacement vector. Direc-

tion of ~d provides direction of enhancement ~E. Magnitude of

enhancement | ~E| is computed based on magnitude of ~d andEf , the enhancement ratio determined by physician. We

apply ~E on wt to compute intermediate enhanced positionof wrist wi as indicated by orange dot in Figure 6. Directapplication of ~E on wt will result into unnatural elongationof bone. To resolve this issue, we normalize the current

bone length between joint et′

and wi by actual bone lengthby moving the estimated intermediate enhanced position of

wrist joint in the direction−−−→wiet

′as follows:

η = 1− |−−→wtet|

|−−−→wiet

′|

(2)

wt

′= wi − η

−−−→wiet

′(3)

where η represents the normalization factor and wt′

is finalestimated enhanced position of wrist joint.

Consider a hierarchy for wrist, elbow and shoulder jointdescribed by w ∈ e ∈ s where ∈ represent the relation “ischild of”. Movement in elbow will also move the wrist due

(a) (b) (c)

Figure 7: Illustration of enhancement of a complex move-ment involving multiple joints (a) Enhancement with nor-malization applied to elbow joint and its child join: wrist(b) Enhancement with normalization applied to wrist joint(c) Final enhanced pose

to the parent child relation. Figure 7 describes the com-plex motion involving movement of two joints: wrist andelbow. Ordering of the bones leads to enhancing the parentbone first and then proceeds down in the hierarchy. Hence,

enhancement is first applied to elbow bone to obtain et′.

The displacement computed while enhancing et is also ap-plied to wt to obtain wt

1. Enhancement in wt is computedbased on displacement vector between wt and wt−1. Esti-mated enhancement is applied to wt

1 to have accumulativeenhancement at wt.

4. EXERGAMESAugmented reality based exergames for rehabilitation are

designed and developed using Unity3D game engine. Shot-Put, Bowling, RehabQuiz and Balance are the 4 exergamesdeveloped, each focusing on a different exercise. A simplestrategy is employed: use close fist to grab an object andopen fist to indicate release of an object. The open andclose fist detection is performed using the Kinect V2 SDK.Each game consists of virtual instructions appearing on thescreen. A scoring module calculates the score for a partic-ular game and updates it on the screen to provide a senseof challenge. All games require the person to be standingor sitting in front of the camera, which is put right underthe TV. Virtual enhancement is incorporated into the limbmovements to assist and motivate the person and can be ac-tivated or deactivated at any moment during the game. Thefollowing subsections describe the unique characteristics ofeach game.

4.1 ShotPutElbow Flexion is the exercise performed to improve the

motion of bending and straightening the elbow, as shown in

Figure 8: ShotPut game, associated with Elbow Flexion,showing a person throwing the ’shot’ as far as possible basedon which score for the ’put’ is calculated.

Figure 2a. The game of Shot Put is selected and modifiedso that it would replicate the elbow flexion exercise. Thegame starts off by placing the person in the virtual sceneconsisting of a shot put arena made in an open stadium.The task is to throw the ball as far away as possible in thevirtual world by taking the hand holding the ball as far backas possible and then bringing it forward and releasing theball, making it travel in the direction of the motion withthe velocity calculated using the change in the fist positionwith respect to the time taken. The further one can extendthe arm, longer the distance the ball covers from the foulline and higher the score obtained. A virtual camera tracksand follows the ball once released. Figure 8 shows a personplaying the game and gives a camera angle, when the ballis thrown and it lands on ground, along with score updatebased on distance from the foul line.

4.2 Bowling

Figure 9: Elbow Extension exercise practiced by the per-son using Bowling game and getting the scores updated byhitting the pins.

Another important exercise for elbow rehabilitation is El-bow Extension. As shown in Figure 2b, the exercise involveskeeping the elbow straight by stretching it out as far back aspossible and bringing it in front. A virtual 10-pin Bowlingalley, similar to the real game, is generated and the person isplaced in it. The game starts with the person, either sittingor standing, and holding the ball normally on the side par-allel to the body. The idea of the Elbow Extension exercisematches the concept of bowling which is to try and stretchthe arm as far back as possible, bring it in front and releasethe ball by opening the fist to make it roll down the alleyand hit the pins. Figure 9 shows one such situation wherethe ball is hitting the pins down. Based on the pins hit,a scoring mechanism as used in real bowling game is usedto engage people and push them towards self-improvement.To allow for not getting demotivated by missing the targets,score for the current set starts only if there is at least one pinknocked down. For ease of use and motivation, the bowlingalley is in-built with side extensions to stop the ball rollinginto the gutter, pushing it back into the lane. Force anddirection applied on the ball is based on the angle in whichthe person takes the ball back and releases it forward.

4.3 RehabQuizIt maybe necessary to associate a cognitive aspect with the

exergame, along with the physical aspects involved. We pro-pose a trivia contest known as RehabQuiz to be performedalong with Elbow Rotation (Figure 2c). Here, the partici-pant places the elbow and the lower arm close to the bodyand rotates the elbow to make the hand go away from thebody. A virtual room is set up in which the participant is

Figure 10: Person exercising Elbow Rotation in the Re-habQuiz, a trivia game, showing him picking up a card formdeck and answering the question.

placed sitting on a chair. A virtual table is shown in thegame that has a deck of cards, each card having a questionfor the quiz, along with 3 bins where the cards are to bedropped. The person performs internal rotation of the el-bow to bring the hand close to the card deck. S/he closesthe fist to pick up a card, allowing a question to pop up,Figure 10. Depending on the answer to the question, s/hedoes external rotation of the elbow to go near the correctbin and drops the card by opening the fist. An appropriatemessage of wrong or correct is shown on table in front of thebins and the score is updated. The game engages the playerin the quiz, while performing the elbow rotation exercise.

4.4 Balance

Figure 11: Balance game with person practicing Hip Abduc-tion exercise by standing on 1 leg on water floating plank.

As mentioned earlier, lower limb rehab is important butmajority of the rehabilitation exergames created using vir-tual reality focus on upper limbs. We propose Balance, agame with a specific focus on the lower limb, hip abductionexercise. Figure 2d shows a person performing the hip ab-duction exercise by standing on one leg and stretching inand out the other leg so as to flex the hip joint. A nicebeach setup with a lighthouse and some trees is made so asto give a positive environmental feel. A wooden plank isplaced on the water making it float and rotate when thereis some unstable force applied to it. The game starts withthe person standing on one leg, by default left, on the float-ing wooden plank. The person moves along with the planksimulating the scenario of person actually being present onit. Being a game of balance, the person is placed a bit awayfrom the plank’s center of gravity, making it unstable byapplying some constant force on the plank in the directionof the person’s center of mass. This creates an illusion ofperson losing balance and gradually falling into water. Ifthe angle of the plank comes up to be too close to the wateri.e. greater than 80 or less than 280, the person is virtuallymade to appear drown. To avoid getting virtually drowned,

the person is motivated to move the right leg away fromthe body and perform hip abduction, Figure 11, shifting thecenter of mass of person to the left of the body and applyreverse force on plank, making the plank tilt in opposite di-rection. Time based scoring is performed by showing actualtime duration one can stay on the plank without falling inwater. Longer one stays on plank, higher is the score.

5. EXPERIMENTAL SETUPEntire setup requires only one computer for processing,

one display to see and enjoy the games and a single MicrosoftKinect V2 [8] for motion capture. The computer should beable to run graphics intensive application and thus shouldhave a good graphics card, sufficient RAM and relativelynew processor. The computer used in the experiment has aGTX 970 graphics card, Intel i7 2.4 GHz processor and 32GB RAM. A Samsung 3D TV is used as the primary display.A single Microsoft Kinect V2 connected to the computerusing a USB 3.0 is used to capture and extract the personpresent inside the scene. 3D model is generated on top of thecaptured information and transmitted at approximately 30frames per second to provide real-time rendering. The entiremotion capture and enhancement framework is written inC++ and the game development is done in Unity3D gameengine using OpenGL rendering and PhysX for physics.

Normal, enhanced and restrained modes are designed forall the games. Balance game is played in only the normalgame mode, where the person performs hip abduction ex-ercise by standing at a particular location marked on floor.RehabQuiz game was calibrated to be played by sitting on achair in front of camera (with change in calibration, it couldalso be played standing up). In the bowling game, peo-ple were allowed to stand or sit whichever was convenientwith them. In both these games, RehabQuiz and Bowling,people played the game in normal as well as virtually en-hanced mode. The game of ShotPut had one more specificrestrained setup on top of both normal and enhanced mode.The restrained setup required the person to be tied up withchair using a bungee cord so as to restrict their arm move-ments. It was designed mainly to simulate the real world pa-tient scenario and test the system on that. Figure 12 showsa person playing the ShotPut game with right arm tied tothe back of chair so as to restrict ball releasing motion.

6. USER STUDYA study was performed with a small group of able-bodied

people to test the playability and the usability of the toolkit

Figure 12: Experimental setup showing a person playingShotPut game with his right arm tied up with bungee cordto simulate patient condition.

Table 1: User study questionnaire & average scores for allgames by 10 able-bodied people, 2 psychology experts and2 physicians, using a 5 point Likert-type scale from -2 to 2.Question Avg ScoreOverall experience 1Visual quality - Game environment and person 1.46Ease of use 1.26Correctness - How correct does game replicate usermotion

1.14

Likely to play again 1.12Fun to play 1.22Motivating to exercise by engaging in the gamerather than the focus on exercise

1.38

Hard work - does it feel that you exercised enough 1.04Enhancement Mode -Visual quality - Game environment and person 0.1Improved performance 0.5Restrained Setup -Simulates handicap situation 1.25Difficult than normal 1.63Improved performance by enhancement 1

in rehabilitation scenario. A total of 10 healthy adults, 2psychology experts and 2 Physical Medicine and Rehab ex-perts volunteered for the study. 8 of the participants weremale and 6 were female. Age of the participating people wasfrom 23 to 50 with the median age of 28 years. Out of them,10 had experience in playing video games and 4 of them hadan idea about using VR for serious games.

The research team initially explained the task required tobe performed by giving out a demonstration for each game.Based on the handedness selection, right or left, situation ofthe game and their comfort level, people were asked to eitherstand or sit on a chair in front of a 60” 3D LED Smart TV.They were then asked to play the games one after other andfill out a questionnaire for each game. The questionnaireasked about different parameters used in the games, virtualenvironments used for immersion and exercise motivation forstroke patients. Most of the questions asked were objectiveso as to get a quantitative analysis. A number of subjectivecriticisms, both positive and negative, were also providedas qualitative analysis. Table 1 shows the set of questionsasked for the games played. Answers had to follow the 5point Likert-type scale, ranging from -2 being very bad to 2being very good.

Based on results shown in Table 1, overall the exergameswere considered favorably. From individual game results,almost everyone liked the Balance game better (perhaps,because no interaction was involved with virtual objectsin terms of picking or dropping). People remarked thatthough the interaction with the virtual objects, such as ballor card is needed and performed relatively well, there werestill scenarios where the Kinect could not detect the closedor open fist properly. Specific comments were put forwardon the game of bowling being really engaging and fun, butat the same time annoying since the ball did not reflect themovements properly. There were few comments regardingthe restrained setup that seemed to be really tough or in-convenient. However, when explanations were given aboutthe post-stroke rehabilitation usage, the participants agreedwith the method and even said it would be really helpful forsuch patients.

Two Psychology experts reviewed the system duringthe user study and provided few insightful comments. If aperson is really good in a game, say bowling, one would ex-pect to score and perform better. But, the camera trackingis not as fast as real world and so it was difficult to track fast

movements of a player, leading to bad performance. To addto that, if the enhancement mode is turned on, performancewould even deteriorate, since it is counter-intuitive for thelong trained muscles to slow down in compensation to theenhancement added. Enhancement made sense in the re-strictive mode, where the muscles are forced to slow down,as shown in Figure 12, leading to improved performance.The experts concluded that enhancement would be reallyhelpful for post-stroke patients since they have inherent re-strictive motion. The concept of using cognitive aspect withexercise was really appreciated as well. A user study com-paring the use of live 3D models vs 3D avatars was proposedby the psychology expert and is currently being set up.

The two Physical Medicine and Rehab expertsbased their analyses on the direct usage of the system withreal patients with stroke. Even though the exergames wereintuitive for stroke patients, they may have varying degreesof difficulty with the activity depending on the type of strokeand degree of physical impairment. Thus, the acceptabilitymight not only depend on the patient’s preferences, but alsoon the severity of stroke and the corresponding level of im-pairment. Given that stroke patients are a heterogeneousgroup, it is difficult to extrapolate information from thisstudy on able-bodied individuals and apply it to patientsafter stroke. They feel that this type of inter-disciplinaryinnovative system with customizable games is ripe for fur-ther evaluation with post-stroke patient populations.

Some of the discouraging factors included the showing upof artifacts, for instance, due to fast sudden arm movementsresulting in improper mesh prediction leading to degradationof visual quality in the virtual enhancement mode. Also,the open-close fist detection had false positives and nega-tives leading to confusion during the games. Other thanthese issues, almost everyone felt that the games created afun, engaging and motivating environment for performingin-home exercises.

A video showcasing the exergames is available at :https://www.youtube.com/watch?v=vPnOmuHkgVU

7. CONCLUSION & FUTURE WORKThe entire system is built with a focus on exergames for re-

habilitation. The proposed exergame toolkit creates an im-mersive and mirrored augmented environment for perform-ing exercise while playing a game. A positive reinforcementis provided with the use of mirror therapy and virtual en-hancement on the live 3D model. The games turn out to befun, engaging and at the same time motivating enough toplay again and hence perform repetitive exercise on affectedlimbs. The setup is affordable, easy to use and suitable forin-home rehabilitation due to the fact that it requires onlyone computer, a TV and a Kinect camera.

In terms of improving the proposed games, we plan to im-prove the close and open fist detection algorithm providedby Kinect. We also plan to improve the animation basedvirtual enhancement as well as the mesh prediction strategyto get rid of the artifacts, especially during fast movementsand self-occlusions. There are other body parts that can beexercised by playing different virtual games, thus increasingthe versatility of the toolkit even more. Based on the inputsfrom Psychology experts, a detailed study is being plannedto compare the use of virtual 3D avatar vs. live 3D humanmodels. Given our preliminary positive findings for the us-ability and satisfaction with our system, we are pursuing re-

search along with the rehab experts to evaluate the feasibil-ity and effectiveness in stroke patient populations. Accord-ing to rehab experts, this technology holds great promiseand has potential to improve in-home rehab in several vul-nerable populations, including patients affected with strokeas well as with chronic degenerative musculoskeletal painconditions.

8. ACKNOWLEDGMENTSThis material is based upon work supported by the Na-

tional Science Foundation under Grant No. 1012975. Anyopinions, findings, conclusions or recommendations expressedin this material are those of author(s) and do not necessarilyreflect the views of the National Science Foundation.

9. REFERENCES[1] Center for disease control and prevention: Stroke

facts, 2015.

[2] J. G. Broeks, G. J. Lankhorst, K. Rumping, andA. J. H. Prevo. The long-term outcome of armfunction after stroke: results of a follow-up study.Disability and Rehabilitation, 21(8):357–364, 1999.PMID: 10503976.

[3] K. Brutsch, T. Schuler, A. Koenig, L. Zimmerli,S. Merillat, L. Lunenburger, R. Riener, L. Jancke, andA. Meyer-Heim. Research influence of virtual realitysoccer game on walking performance in roboticassisted gait training for children. J. Neuroeng.Rehabil, 15(7):1–9, 2010.

[4] R. Caillois. Les jeux et les hommes: le masque et levertige. Folio. Essais. Gallimard, 1991.

[5] Y. Chen, M. Baran, H. Sundaram, and T. Rikakis. Alow cost, adaptive mixed reality system forhome-based stroke rehabilitation. In Engineering inMedicine and Biology Society, EMBC, 2011 AnnualInternational Conference of the IEEE, pages1827–1830, Aug 2011.

[6] Y. Chen, N. Lehrer, H. Sundaram, and T. Rikakis.Adaptive mixed reality stroke rehabilitation: Systemarchitecture and evaluation metrics. In Proceedings ofthe First Annual ACM SIGMM Conference onMultimedia Systems, MMSys ’10, pages 293–304, NewYork, NY, USA, 2010. ACM.

[7] R.-J. Cherng, C.-F. Liu, T.-W. Lau, and R.-B. Hong.Effect of treadmill training with body weight supporton gait and gross motor function in children withspastic cerebral palsy. American journal of physicalmedicine & rehabilitation, 86(7):548–555, 2007.

[8] L. Cruz, D. Lucio, and L. Velho. Kinect and rgbdimages: Challenges and applications. In Graphics,Patterns and Images Tutorials (SIBGRAPI-T), 201225th SIBGRAPI Conference on, pages 36–49, Aug2012.

[9] B. H. Dobkin. Strategies for stroke rehabilitation. TheLancet Neurology, 3(9):528 – 536, 2004.

[10] M. Duff, Y. Chen, S. Attygalle, J. Herman,H. Sundaram, G. Qian, J. He, and T. Rikakis. Anadaptive mixed reality training system for strokerehabilitation. Neural Systems and RehabilitationEngineering, IEEE Transactions on, 18(5):531–541,Oct 2010.

[11] P. Dukes, A. Hayes, L. Hodges, and M. Woodbury.Punching ducks for post-stroke neurorehabilitation:System design and initial exploratory feasibility study.In 3D User Interfaces (3DUI), 2013 IEEE Symposiumon, pages 47–54, March 2013.

[12] D. Q. Freitas, A. Da Gama, L. Figueiredo, T. M.Chaves, D. Marques-Oliveira, V. Teichrieb, andC. Araujo. Development and evaluation of a kinectbased motor rehabilitation game. Proceedings ofSBGames, pages 144–153, 2012.

[13] A. Henderson, N. Korner-Bitensky, and M. Levin.Virtual reality in stroke rehabilitation: a systematicreview of its effectiveness for upper limb motorrecovery. Topics in stroke rehabilitation, 14(2):52–61,2007.

[14] S. Hesse, C. Werner, M. Pohl, S. Rueckriem,J. Mehrholz, and M. Lingnau. Computerized armtraining improves the motor control of the severelyaffected arm after stroke a single-blinded randomizedtrial in two centers. Stroke, 36(9):1960–1966, 2005.

[15] R. Kizony, P. L. T. Weiss, M. Shahar, and D. Rand.Theragame: A home based virtual realityrehabilitation system. International Journal onDisability and Human Development, 5(3):265–270,2006.

[16] A. Koenig, M. Wellner, S. Koneke, A. Meyer-Heim,L. Lunenburger, and R. Riener. Virtual gait trainingfor children with cerebral palsy using the lokomat gaitorthosis. Studies in health technology and informatics,132:204–209, 2007.

[17] H. Krebs, B. Volpe, M. Aisen, and N. Hogan.Increasing productivity and quality of care:robot-aided neuro-rehabilitation. Journal ofrehabilitation research and development,37(6):639–652, 2000.

[18] S. E. Lord, K. McPherson, H. K. McNaughton,L. Rochester, and M. Weatherall. Communityambulation after stroke: how important andobtainable is it and what measures appear predictive?Archives of Physical Medicine and Rehabilitation,85(2):234 – 239, 2004.

[19] A. Meyer-Heim, C. Ammann-Reiffer, A. Schmartz,J. Schafer, F. H. Sennhauser, F. Heinen, B. Knecht,E. Dabrowski, and I. Borggraefe. Improvement ofwalking abilities after robotic-assisted locomotiontraining in children with cerebral palsy. Archives ofDisease in Childhood, 94(8):615–620, 2009.

[20] J. C. Perry, J. Andureu, F. I. Cavallaro, J. Veneman,S. Carmien, and T. Keller. Effective game use inneurorehabilitation: user-centered perspectives.

[21] S. Raghuraman, K. Venkatraman, Z. Wang,B. Prabhakaran, and X. Guo. A 3d tele-immersionstreaming approach using skeleton-based prediction.In Proceedings of the 21st ACM InternationalConference on Multimedia, MM ’13, pages 721–724,New York, NY, USA, 2013. ACM.

[22] D. Rand, R. Kizony, and P. T. L. Weiss. The sonyplaystation ii eyetoy: low-cost virtual reality for use inrehabilitation. Journal of Neurologic PhysicalTherapy, 32(4):155–163, 2008.

[23] H. Regenbrecht, S. Hoermann, C. Ott, L. Muller, andE. Franz. Manipulating the experience of reality for

rehabilitation applications. Proceedings of the IEEE,102(2):170–184, Feb 2014.

[24] H. Regenbrecht, G. McGregor, C. Ott, S. Hoermann,T. Schubert, L. Hale, J. Hoermann, B. Dixon, andE. Franz. Out of reach? : A novel ar interfaceapproach for motor rehabilitation. In Mixed andAugmented Reality (ISMAR), 2011 10th IEEEInternational Symposium on, pages 219–228, Oct 2011.

[25] D. J. Reinkensmeyer and S. J. Housman. “ if i can’t doit once, why do it a hundred times?”: Connectingvolition to movement success in a virtual environmentmotivates people to exercise the arm after stroke. InVirtual Rehabilitation, 2007, pages 44–48. IEEE, 2007.

[26] G. Saposnik, M. Levin, S. O. R. C. S. W. Group, et al.Virtual reality in stroke rehabilitation a meta-analysisand implications for clinicians. Stroke,42(5):1380–1386, 2011.

[27] R. Scherer, G. Moitzi, I. Daly, and G. R.

MAijller-Putz. On the use of games for noninvasiveeeg-based functional brain mapping. IEEETransactions on Computational Intelligence and AI inGames, 5(2):155–163, June 2013.

[28] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp,M. Finocchio, R. Moore, A. Kipman, and A. Blake.Real-time human pose recognition in parts from asingle depth image. In CVPR. IEEE, June 2011.

[29] J. Sinclair, P. Hingston, and M. Masek.Considerations for the design of exergames. InProceedings of the 5th International Conference onComputer Graphics and Interactive Techniques inAustralia and Southeast Asia, GRAPHITE ’07, pages289–295, New York, NY, USA, 2007. ACM.

[30] K. Stefan, J. Classen, P. Celnik, and L. G. Cohen.Concurrent action observation modulatespractice-induced motor memory formation. EuropeanJournal of Neuroscience, 27(3):730–738, 2008.

[31] L. Sucar, R. Luis, R. Leder, J. Hernandez, andI. Sanchez. Gesture therapy: A vision-based systemfor upper extremity stroke rehabilitation. InEngineering in Medicine and Biology Society(EMBC), 2010 Annual International Conference ofthe IEEE, pages 3690–3693, Aug 2010.

[32] K. Tanaka, J. Parker, G. Baradoy, D. Sheehan,J. Holash, and L. Katz. A comparison of exergaminginterfaces for use in rehabilitation programs andresearch. Loading..., 6(9), 2012.

[33] H. Thieme, J. Mehrholz, M. Pohl, J. Behrens, andC. Dohle. Mirror therapy for improving motorfunction after stroke. Stroke, 44(1):e1–e2, 2013.

[34] J. Vieira, M. Sousa, A. Arsenio, and J. Jorge.Augmented reality for rehabilitation using multimodalfeedback. In Proceedings of the 3rd 2015 Workshop onICTs for improving Patients Rehabilitation ResearchTechniques, pages 38–41. ACM, 2015.

[35] G. Yavuzer, R. Selles, N. Sezer, S. Sutbeyaz, J. B.Bussmann, F. Koseoglu, M. B. Atay, and H. J. Stam.Mirror therapy improves hand function in subacutestroke: a randomized controlled trial. Archives ofphysical medicine and rehabilitation, 89(3):393–398,2008.