This is a preprint of the paper presented at the Federated Conference on Computer Science and Information Systems 2016,September 04-07 2016 in Gdansk, Poland. DOI: 10.15439/2016F507

Applications for investigating therapy progress ofautistic children

Agata Kołakowska, Agnieszka Landowska, Michal R. WrobelFaculty of Electronics, Telecommunications and Informatics,

Gdansk University of Technology, PolandEmail: {agatakol,nailie,wrobel}@eti.pg.gda.pl

Dominika Zaremba, Dominika Czajak,Anna Anzulewicz

Harimata Sp. z o.o., PolandEmail: {dominika.zaremba,domnika,ania}@harimata.co

Abstract—The paper regards supporting behavioral therapyof autistic children with mobile applications, specifically appliedfor measuring the child’s progress. A family of five applicationsis presented, that was developed as an investigation tool withinthe project aimed at automation of therapy progress monitoring.The applications were already tested with children with autismspectrum disorder. Hereby we analyse children’ experience withthe games, as a positive attitude towards the application is the keyfactor enabling practical application of the solutions in therapy.Two evaluation methods were applied: a behavioral study ofvideo recordings of children interaction with the games and on-line behavioral tagging performed during measurement sessions.The paper also outlines the main challenges, encountered duringsessions with autistic children. The study might be interestingfor both researchers and practitioners applying e-technologies inautistics therapy.

I. INTRODUCTION

Autism is a developmental disorder, that influences theability to socialize, communicate as well as learning skills [1],[2]. Autistic children have diverse level of deficits in languageunderstanding, speaking and other areas, that make the therapyand education very difficult [3]. Autism is not only a personalor family matter, but a social problem, as the number ofchildren diagnosed with autism rises all over the world.Our motivations to develop e-technologies supporting autisticchildren lies in a belief, that an appropriate therapy adjustedto the deficits of an individual, may result in independent lifeas an adult, including not only daily routines, but also findinga job. Therefore a reliable diagnosis and effective therapy arecrucial for solving the social problem of autistics exclusionfrom society [4].

Educating children with autism spectrum disorder could bea challenge in the best of circumstances [5]. There are severalpremises for supporting autistics therapy with e-technologies[6], [7]. Most of the autistic children are eagerly usingcomputers and tablets once they are taught how to use them[5]. Autistics require repetitive environment for functioningand learning. Technologies are able to perform the sameactivities in exactly the same way and with indefinite patience.

This research is supported by the National Centre for Research andDevelopment, Poland under grant AUTMON no IS-2/6/NCBR/2015, as wellas DS Programs of the Faculty of Electronics, Telecommunications andInformatics, Gdansk University of Technology. This research is also inspiredby European Cooperation in Science and Technology (COST) Action TD1309Play for children with disabilities (LUDI).

Moreover, systems and devices might be customized in orderto adjust to a unique set of deficits of an individual [4].

This study is performed under the AUTMON project, thataims at development of methods and tools to allow for theautomatic evaluation of the therapy progress among childrenwith autism spectrum disorder (ASD) [8]. Therapy monitoringis based on automatic detection of behavioral patterns in tabletand application usage. During the project 5 applications weredeveloped or adjusted to fit the deficits of autistic children.The apps have the potential of being applicable in the therapyprogress monitoring. However, the crucial question is: whetherthe autistic children are eager to use those games. This studyaims at verification of this issue. In this paper, the applicationsand their adaptation to autistic users is described, and thedata from interaction sessions is analyzed. Moreover, wereport our observations on autistics interaction with tablets andapplications. The study might be interesting for both producersof solutions for autistic children as well as for researchersinvestigating effective methods for ASD support and therapywith e-technologies.

The paper is organized as follows: Section II reports workrelated to this study and especially previous works on mon-itoring autistic users performance. Section III presents thestudy context and study design, including the description ofdeveloped applications and their adaptation to acquisition oftherapy progress data. Section IV and V present the resultsof games evaluation, and those sections are followed bya summary of results and discussion (Section VI) and theconclusions describing future works (Section VII).

II. RELATED WORK

New technologies are used while working with people suf-fering from autism both at the stage of diagnosis and therapy.The use of technology in this field is usually based on softwareimplementations for common therapeutic tasks performed inmost cases without a computer. Supporting diagnosis is limitedto the use of computer versions of diagnostic questionnaires.Although there are some advanced technological solutionsthat can help to diagnose autism, they are still experimentalmethods, used in laboratories only or tested on small groupsof people. In many cases they require specialized hardware.Some of these solutions are based on:

• eye tracking, which reveals that people with autism rarelyfocus attention on the area of eyes if compared to otherparts of face. It can be used in the early diagnosis of ASD[9], [10], [11];

• automatic analysis of video recordings presenting chil-dren performing tasks that are commonly evaluated byspecialists [12];

• automatic analysis of individual hand movements whenperforming tasks, which require indicating objects on atouch screen [13];

• machine learning algorithms applied to predict the stateof a person on the basis of information taken fromcommonly used diagnostic questionnaires [14], [15].

Much more often than in the diagnosis, new technologiesare used in the therapy of autism. In most cases, thesesolutions are computer implementations of therapeutic tools,that usually are in a paper form. There are numerous pro-grams supporting daily activities, allowing for arranging andfollowing procedures that consist of sequences of activitiesrepresented as images or text [4]. Sometimes the images arealso accompanied by appropriate sound recordings, to helpautistic people to communicate and in particular express theirneeds and desire.

There are also programs that support learning of certainbehaviors, e.g. by providing stories illustrating course of typi-cal relationships. Thanks to the new technologies, such scenescan be more interactive and therefore more realistic. Typicalinterpersonal situations may be also trained using an onlinechat moderated by a therapist [16]. Other available populartools improve speech and verbal communication (VOCAs,ang. Voice Output Communication Aids) that read the namesof symbols or whole sentences and also support the trainingof word pronunciation, expressing requests and desires andanswering questions. Moreover they encourage spontaneousreactions during interaction with another person.

Very often children with autism spectrum disorders cannotrecognize faces among other objects and also are not able torecognize emotions from facial expression. Therefore, numer-ous tools are created to help to acquire these skills. Theseinclude programs that help in learning face recognition andemotions based on facial expressions [17], learning expressingemotions through facial expressions [18] and recognizing emo-tions of encountered people [19]. There are also solutions, thatincorporate special hardware measuring physiological signalsto interpret an emotional state of a person, which may beespecially valuable information in case of autistic children,who often are not able to show their emotions [20], [21].

Apart from solutions designed for people with autism, thereare also some applications supporting work of the therapists,e.g. ABPathfinder or ACE (Autism Curriculum Encyclopedia).They support diagnosis, preparation of individualized edu-cational programs, collecting information about the learningprocess and its analysis, detection of situations requiringintervention, recommendations for therapy depending on theprogress of the skills acquired.

III. STUDY CONTEXT AND STUDY DESIGN

A. Study context and purpose

In 2015 a project named AUTMON (Automated therapymonitoring for children with autism spectrum disorder) wasstarted by a consortium consisting of Gdansk Universityof Technology, Harimata LTD and Hippotherapy Foundation[8]. Its goal is to develop methods and tools for automatedmonitoring of progress in the therapy of children with autism.

The goal is addressed by (1) the development of technologyidentifying behavioral patterns associated with autism, whichcould be used to test and evaluate the progress of therapy andby (2) implementation of a system that automatically tracksthese patterns during therapy. A specially designed tabletapplications have been created to record the behavioral pat-terns during the childs interaction with tablet. These patterns,which may be analyzed for diagnostic or therapeutic purposes,contain information on: touch screen gestures, e.g. tap andswipe device movements identified by accelerometer; theapplication usage, i.e. navigationpatterns, objects that attractattention (goals/distractors), strategies of decision making etc.The research goal of the AUTMON project is to verify, that theproposed solution might support the therapists by providingthem with objective data on the progress of therapy of autisticchildren in various areas of their development.

The first step in the project research is verification of thechildren’ experience with the games, as attitude towards theapplication is the key factor enabling practical applicationof the solutions in therapy. Reporting the study of childrenexperience with the games is the main goal of this paper.

The key research question addressed by the presented studymight be formulated as follows: Which applications provideexperiences positive enough for the child to perform andcontinue interaction allowing (as a result) to monitor progress.

B. Applications under investigation

The applications developed during the project were devel-oped in the cooperative and iterative process engaging psychol-ogists, therapists and data scientists. They were designed for

Fig. 1. Boxes game

the tablet devices. The solution set includes five games, whichallows to gather information about children interactions withthe device, using touch screen sensors and gyroscope.

Each game consists of training phase and actual game.During the training phase child may get acquainted with thegame concept. At the beginning of training session interactivetutorial is presented. Later child may try to use application byherself/himself. During the whole training therapists shouldinstruct the child and may use either verbal and physicalguidelines. Depending on game, training session lasts from 2to 5 minutes. At any time training session may be terminatedby tapping three fingers at the top of the screen.

After finishing training session, actual game, which is usedto collect data, begins. In order to obtain the undisturbedresults, during this phase, the therapist should not interact withthe child. Like during the training phase the session can beterminated, when a child looses interest in the game.

1) Boxes: Boxes is a game designed for warm-up. The goalis to place the balls in boxes by matching corresponding colors,as shown in Fig. 1. The child should touch and drag each outof five balls into proper box. It is very simple, yet allows tomeasure color recognition, simple instruction understandingand matching ability. It also becomes visible if a child canproperly use the touch screen (swipe movements).

Fig. 2. Sharing game

2) Sharing: The goal of the second game is to sharefood among four animated children. Child has to tap on thedisplayed food article (watermelon, apple, cake) and swipeportions to four plates in front of the children. The gameconsists of learning phase and identical test phase. Thistask requires more complex instruction understanding, motorprecision, perseverance (the game is all about sharing for 5min) and to some extend attentional control, as screen isfilled with clickable distractors. Distractors are aimed to drawchild’s attention and prevent child from completing trials. Trialalso cannot be completed if child places the food unevenly(some children are left without their portion) and does notread the mimic cues given by characters(if child doesn’t gethis/her portion, his/her face turns sad, while fed children gethappy faces). Fig. 2 presents the game screenshot.

3) Cat & Dog: Based on the experimental paradigm ofGo/NoGo, the game is intended to be used by older children.Typically developing 3-4 years old’s are not ready to proceedwith this cognitively demanding task. Series of stimuli ispresented randomly at a fast pace and a child has to reactto target stimuli while refraining from reaction to distractingstimuli. For example, in Cat condition, as shown in Fig. 3, achild has to tap the screen as fast as possible when a cat or

Fig. 3. Cat & Dog game

Fig. 4. Pinwheel game

meowing sound is presented and inhibit reaction for dog orbarking. This task demands prolonged alertness, fast decisionmaking, good perceptual and attentional skills and executivecontrol of intentional actions. It can be used as a measureof impulsivity and control processes. As this game can befrustrating to some children, it is advised to omit the gameif the child displays loss of interest and negative emotions(usually auto-stimulatory behaviors).

4) Pinwheel: This game requires some flexibility of mentalrepresentation of tablets affordances that were learned inprevious games. Performance in Pinwheel game (Fig. 4) isdetermined by efficiency and speed of learning new ways touse objects, fine motor proficiency, instruction understanding,executive control and attention, inhibition, color recognitionand matching and many other basic cognitive abilities. Thepinwheel is slowly turning around while a color ball balancesat the base of its stem. A child has to flip the tablet (previouslyonly touch screen data were gathered, now accelerometer andgyroscope provide the data) precisely to move the color ballto the corresponding ”pinwheel petal”. Should ball fall off thestem, the child has to start over with the next trial, the samehappens if the ball reaches wrong petal (for example blueball hits yellow petal). Patient waiting, anticipatory movementplanning and ball control can be challenging for younger users,so the game is recommended for older children.

5) Creativity: Drawing and coloring pictures seem to bethe most rewarding, the game for most children. A child hasto exhibit a good understanding of the instruction, stability andprecision of movement, as well as ability to imitate (mimic)the lines displayed as a prompt. Each step of drawing a pictureis shortly displayed on the screen and child’s task is to leadthe finger on the dotted line. At the end child gets to colorthe finished picture – leaving behind even more movementpatterns for the analysis. Screenshots of the game are shownin Fig. 5.

Fig. 5. Creativity game

C. Study design

Two evaluation methods were applied: (1) a behavioralstudy of video recordings of children interacting with thegames and (2) on-line behavioral tagging during measurementsessions.

The first part of the study was performed before measure-ments sessions were started and aimed at evaluation of userexperience with the games. The second part of the study isperformed on-line during each measurement session and thispaper summarizes the results as a validation of the chosenapproach. The target group in those studies are childrenaged 3 up to 7 (kindergarten education level), diagnosed (orduring diagnosis) with autism spectrum disorder. As childrenmeasurements require consent from parents, randomization ofselection to sample was not possible. Children were recruitedamong pupils of ten therapeutic centers in Poland. Parentsagreement rate was very high (more than 90%) only fewrefused to agree that their child would take part in the study.

1) User experience study: This study was aimed at ana-lyzing the experience of the children play. The first step ofthis study was to find a definition and unambiguous indicatorsof good UX. Previous research on the subject mentioned:engagement, playfulness and fun as indicators of good UX

in children [22], [23], [24]. The proper measurement couldbe made only if a child pays attention and makes effort tointeract with tablet and the good UX was defined from thatperspective.

Previous studies [25] used some combinations of behav-ioral observation with survey and interview techniques. Thisapproach was hard to apply in this study, because children withautism are mostly nonverbal or of poor communication skills,so we couldn’t ask them straightforward for their opinions.Sometimes children with ASD present paradoxical mimicor vocal reactions, so to discover their inner feelings andattitudes was a challenge of this study. Therefore in this studybehavioral observation was chosen as an investigation tool.Moreover, the gameplay was recorded and analysis were madepost-hoc by independent observers. This approach allowed formeasuring consistency of manual tagging in order to achievehigher reliability. Steps of the study preparation:

1) An exploratory observation of gameplay was performedresulting in a list of observational indicators of interestand having fun with the games. A small subset of videosof children gameplay were chosen purposefully the onesrecorded with autistic children playing the games forthe first time. The videos presented two boys (3 and5 y.o.), while the younger expressed uneasy behaviorduring gameplay session. Three independent judges (twopsychologist with specialization in autism therapy anda naive observer not familiar with the symptoms ofautism) were watching videos and listing observationalindicators of good/bad experience of play.

2) Behaviors listed by observers in the previous step wereclustered into three categories of: engagement, under-standing and enjoyment.

3) The three factors were conceptualized and operational-ized with three behavioral indicators each forming 9-item behavioral observation scheme.

4) Each item was assigned a five-point discrete scale (1-5)with descriptive explanations assigned to first and lastitem. As a result an observation sheet was prepared, thatwas used in manual tagging of the video recordings.

5) Using the elaborated scale and observation sheet, manualtagging of videos was performed by 5 independentobservers. Total number of 21 recordings were analyzed.

6) The manual markers from observers were checkedagainst consistency criteria using commonly used Kappacoefficient [26]. The data was used to evaluate theapplications using the understanding, engagement andenjoyment criteria.

7) Final results were formulated regarding inclusion or ex-clusion of the games from the measurement procedure.Some additional recommendations were formulated forthe measurement procedure.

Understanding factor characterizes, that the device func-tions and the games are understandable for a child. First ofall, child realizes that tablet screen can be used to interact withthe tablet. There is a range of algorithms of interaction (based

on set of possible actions provided) that result in successfulinteraction with the game, meaning when child achieves thegoal of the game. Child wins the game if (s)he acquires a finalstage of task completion, e.g. feeds all four children, drawsa picture, paints a picture etc.) Children may get to win bythemselves or with a standarized verbal or motor prompt fromthe experimenter.

Engagement factor characterizes, that the function of thetablet and the game is engaging, meaning, that tablet andgame attracts childs attention. If in childs sight, in proximityof his/her hands there is a possibility of engaging. It can beobserved how often, regular and persistent is childs interactionwith the touch screen.

Enjoyment factor characterizes, whether interaction withtablet results in growth of positive affect symptoms, whichwere operationalized as laughing and vocal expressions, verbalexpressions, mimic expressions and motor and postural expres-sions (clapping hands, jumping, shrugging). Facial expressionswere not observable due to to the fact, that video materialswere filmed from the chin down (which was explicitly statedin the parent-experimenter agreement for the privacy reasons).

The results of the study are provided in Section IV.2) Validation during measurement sessions: The second

study is on-line observation made during the measurement ses-sions. A validation procedure was designed with the aim of thecomfort of the child. Children are seated by the experimenterat the table with tablet placed on it. The preparatory work(turning on the device, WiFi connection) is done before a childenters the room, so (s)he gets to play the game without waiting.The first part of the procedure is a tutorial session, in whichanimated prompts are displayed to give children clues on theintended use of the game. If the instruction is insufficient, theexperimenter or therapists are allowed to intervene and provideadditional motor or verbal instructions. The test phase of theprocedure lasts for 5 minutes and ends automatically. In thispart, experimenter and therapists are not allowed to interruptthe child or help them in any way. If the child loses interestin the game and walks away, adults can encourage the childto keep up with the game, but no specific instructions shouldbe given.

After the game is finished, the therapists ask a child togive him/her the tablet and they are waiting for the data tobe processed and send. In the meantime, the experimenter canoffer the child some toys or engage in a small talk. When thedata is loaded, next game can be launched. If all of the games(tutorial session + test phase) are completed, the study is over.

The children are supposed to play all five games duringthe sessions. The detailed criteria used in the usability studywere significantly simplified for the measurement stage. Fromthe 9-item scale only 2 indicators were used in the on-linetagging procedure. For each game the following two factors areestimated by an observer: the level of difficulty, which may be1-easy, 2-adequate, 3-difficult; the level of interest, which maybe 0-none, 1-low, 2-average, 3-high. If a child does not wantto play a game in spite of some encouragement, 0 is assignedto the level of interest and no value is given as the level of

TABLE ICHILDREN TAKING PART IN USABILITY TESTS

Child Age Sex Game

Box

es

Shar

ing

Pinw

heel

Cre

ativ

ity

Cat

&D

og

1 4 M X X2 4 M X X X X3 4 M X X X4 3 M X X X X5 6 M X X6 6 M X X X7 4 M X8 7 M X9 5 M X

difficulty. If, for some reasons, a therapist decides not to try toplay a game, neither difficulty nor interest are assigned a value.The reasons for childs intentional exclusion from play mightfall into following categories: a child performed an extremelynegative reaction to the game (eg. fear of sounds); a child is inbad disposition on this specific session day (regular therapistopinion); a child got upset during the study procedure beforeattempting to play the game.

The difficulty factor corresponds to understandability cri-teria from the UX study, the interest factor corresponds toengagement criteria from the UX study.

There were multiple reasons for that simplification. First ofall, the primal focus of the sessions was to measure the data,not to measure usability. Secondly on-line tagging is muchmore difficult than video tagging, as multiple activities mustbe performed simultaneously and there is no possibility to re-play the recording. Moreover, the two factors were the mostimportant from the perspective of measurements reliability.

IV. USABILITY EVALUATION OF APPLICATIONS FORPROGRESS MONITORING

This section provides results of usability evaluation of theapplications. Table I presents the distribution of the dataanalyzed. 21 recordings of nine children have been annotated.All children were boys, aged 3-7. Each of them played fromone to four of the created games.

To evaluate the consistency of the annotations done by fourobservers, kappa coefficient was calculated[26]. The valuesobtained for all nine behavior indicators defined in sectionIII-C and averaged over all games are shown in Table II. Thehighest agreement was obtained for engagement, moderate forunderstanding and the lowest for enjoyment.

To provide final evaluation scores for the three categories(engagement, understanding, enjoyment) the following proce-dure has been followed independently for each game:

1) in each category the scores for three behavior indicatorswere summed getting total category scores;

2) the total category scores were averaged over four judges;as a result three values for engagement, understandingand enjoyment were obtained for each recording;

TABLE IIKAPPA COEFFICIENTS OF MANUAL TAGGING

Behavioral indicator Kfree

Engagement 1 0,79Engagement 2 0,70Engagement 3 0,61Understanding 1 0,57Understanding 2 0,54Understanding 3 0,43Enjoyment 1 0,40Enjoyment 2 0,30Enjoyment 3 0,38

3) the obtained recording scores were averaged over allfilms presenting a given game.

The final results obtained in this way are presented inTable III. As there were only two recordings for Boxes,this game should not be considered in the analysis. Sharingreceived the highest scores for understanding and engagement.Surprisingly, Cat & Dog took the second place in these twocategories and the first one in enjoyment. That game seemedto be rather difficult, some of the children did not play it atall. In spite of these obstacles, it turned out to be entertaining.Creativity game received the lowest scores for understandingand enjoyment, Pinwheel for engagement. Although Creativityseemed to be understandable for the children, as most of themintuitively drew lines with their fingers, precise drawing turnedout to be quite difficult bringing about the low scores. It can beobserved that more understandable games get higher scores inengagement and enjoyment, however the relationship betweenthe variables would require further analysis.

V. ON-LINE TAGGING - EXECUTION AND RESULTS

Eight therapy centers take part in this study. The centers arevisited once a month. Each of them has been already visitedat least three times. To summarize the results the first threesessions have been taken into account. These are the first threesessions for each child, which is not necessarily equivalentto the first three visits in a center. For example, if a childis absent during session 2, then his results from session 3are assigned to session 2. Due to some absence rate not allparticipants have already gathered three data records. From thetotal number of 42 children, 29 of them were present during allthree sessions, 6 - during two sessions and 7 had one sessiononly. The absence rate is typical for the kindergarten childrenage range.

The results obtained for each game are averaged over allchildren. The fields with no values are not taken into accountin the calculations. The final scores for the levels of difficultyand interest are shown in Table IV.

It can be seen that three of the games (Sharing, Boxes,Creativity) turned out to be much more interesting than othertwo (Pinwheel, Cat and Dog). Some relation between interestand difficulty may be observed. The more difficult game,the less interesting and vice versa. Boxes and Sharing werethe easiest ones. Pinwheel, which requires motor skills whileflipping the tablet, and Cat & Dog requiring being focused,

TABLE IIIUSER EXPERIENCE RESULTS OBTAINED FOR THE APPLICATIONS

Game name Understanding Engagement EnjoymentMin Max Avg SD Min Max Avg SD Min Max Avg SD

Boxes 10 13 nd nd 15 15 nd nd 8 11 nd ndSharing 8 15 12,8 3,3 14 15 14,7 0,6 7 12 8,9 2,3Pinwheel 4 15 9,5 5,2 8 15 12,9 3,1 4 12 8,8 3,3Creativity 6 13 8,9 3 11 14 13,4 1,5 2 10 7,8 1,8Cat & Dog 6 15 11,3 4,3 12 15 14,2 1,5 5 13 9,3 3,3

TABLE IVINTEREST AND DIFFICULTY LEVELS FOR THREE MEASUREMENT SESSIONS

Game name Interest Difficulty

Total* 1stsession**

2ndsession**

3rdsession** Total* 1st

session**2nd

session**3rd

session**Boxes

n*=100 n**=282,5

(0,8)2,4

(0,8)2,4

(0,7)2,5

(0,8)1,9

(0,5)1,8

(0,6)1,9

(0,6)1,9

(0,5)Sharing

n*=104 n**=282,6

(0,6)2,7

(0,5)2,7

(0,6)2,6

(0,5)1,8

(0,5)1,9

(0,5)1,9

(0,5)1,7

(0,5)Pinwheel

n*=85 n**=221.9

(0,9)2,1

(0,9)1,8

(1,0)2,1

(0,8)2,6

(0,6)2,6

(0,5)2,6

(0,5)2,4

(0,7)Creativity

n*=102 n**=282,4

(0,8)2,6

(0,7)2,5

(0,7)2,4

(0,7)2,3

(0,5)2,2

(0,5)2,4

(0,5)2,2

(0,5)Cat & Dog

n*=49 n**=131,8

(1,0)1,9

(0,9)1,9

(1,0)1,9

(1,0)2,7

(0,6)2,6

(0,8)2,8

(0,4)2,7

(0,5)n* - total no of gameplays evaluated (all children, up to 3 sessions)n** - no of gameplays evaluated (only children with 3 sessions)

were the most difficult, just as it was expected. Some of thekids were not able to play Cat & Dog at all.

Another interesting observation is the fact that the levels ofinterest and difficulty do not change much over the time. Onemight presume that difficulty would decrease as the kids playmore. However, it is not the case. Neither the level of difficultynor the level of interest becomes lower. This is a relevantfeature from the point of view of the AUTMON project. Itmeant that data collected while playing are not affected byconfounding factor such as history effect.

VI. SUMMARY OF RESULTS AND DISCUSSION

The aim of this paper was to evaluate, whether the appli-cations developed for monitoring progress of children withautism have a chance to be used in practice. The question wasraised, whether the applications are engaging, understandableand enjoying enough to trigger and maintain childs focusand interaction. Two studies were performed: one detailedusability study based on observation and manual tagging ofvideo recordings and an on-line tagging during first threemeasurement sessions.

The main results might be formulated as follows:• understanding, engagement and enjoyment are partially

dependent, eg. some children are fascinated with pin-wheel although they do not understand the purpose of thegame, for the others, if they do not understand a game,they are not interested in it.

• both studies confirmed, that three of the games: Sharing,Creativity and Boxes are mostly understandable for thechildren; out of the three, Sharing seems to be the favoritegame;

• game Cats & Dogs and Pinwheel seems less intuitive andhalf of the children refuse to use them, however, once achild understands the game, it seems to be engaging,

• although it seemed, that some of the children do re-member the games after a month (average time distancebetween recording sessions), the data do not confirm thehistory effect, which is convenient for measurement.

All recording and measurement sessions revealed also somequalitative observations:

• Most of the children with autism were eager to usetablets, typical behavior was to grab a tablet in theproximity of hand and to follow it, when it was takenaway,

• Most of the children know very well, how to turn thetablet on and how to switch to another game, we haveused the parent mode in order to prevent children fromswitching the games off;

• Some lower functioning children got frustrated, whenthey did not understand the game;

• Some higher functioning children got bored easily andeven though they started eagerly, it was hard to keep themplay for longer than a minute;

Authors are aware of the fact, that this study is not free ofsome limitations, such as small number of videos analyzed andlow consistency of manual tagging for some of the variables.Videos for usability evaluation were taken before the finalapplications were launched (and the application were correctedafterwards). Children are challenging subjects to follow someof them moved around the room, took the tablet and turned, gotoff the chair to sit on the floor and so on. From the recordedvideos we have chosen for the analysis those, that were the

most complete and had clearly visible actions on the tabletscreen.

In this study the main reason for the relatively low consis-tency is that we have assumed using the common scale forthe games. In the future research we are planning to defineusability criteria (engagement, understanding and enjoyment)with more detailed behavioural scales separately for eachgame, which is expected to increase consistency of manualtagging.

Despite some limitations of the study, the performed anal-ysis allowed us to determine, which applications provideexperiences positive enough for the child to perform andcontinue interaction allowing (as a result) to monitor progress.The goal of the study was achieved.

VII. CONCLUSIONS

Our study shows that potential for use of tablet-based tech-nology in children therapy should be considered as more validand reliable tools emerge. Children are generally enthusiasticabout tablets and other mobile devices and this rapture can beutilized in the area of research, diagnosis and therapy. Futureworks in this discipline should focus on to developing toolsfor monitoring progress and better means for their evaluation.As our study concludes, its not easy to evaluate in terms ofchild experience, especially in autistic population, where ratersagreement is hard to reach.

Objective measures of therapy progress can bring qualitydata to the therapy providers. As they have better knowledgeof childs state and developmental pace and direction, thy canprovide adjusted intervention to best fit the individual needsof the child. Autistic children often develop disharmoniously,so they are progressing faster or slower in certain areas ofdevelopment, say fine or gross motor skills, communicationor self-reliance. When these (sometimes minor) differencesare recognized and taken into account while planning theintervention a child gets highly personalized and evidence-based therapy. Better therapy means better life for childrenwith ASD, as enormous amount of evidence shows thatpersonalized and early onset intervention improve their futurequality of life (in terms of occupational, educational andprivate situation and coping).

REFERENCES

[1] W. H. Organization et al., The ICD-10 classification of mental andbehavioural disorders: clinical descriptions and diagnostic guidelines.Geneva: World Health Organization, 1992.

[2] A. . A. P. Association et al., “Diagnostic and statistical manual of mentaldisorders,” 1994.

[3] F. R. Volkmar, R. Paul, A. Klin, and D. J. Cohen, Handbook of Autismand Pervasive Developmental Disorders, Diagnosis, Development, Neu-robiology, and Behavior. John Wiley & Sons, 2005, vol. 1.

[4] A. Landowska and M. Smiatacz, “Mobile activity plan applications forbehavioral therapy of autistic children,” in Man–Machine Interactions4. Springer, 2016, pp. 115–125.

[5] L. Winerman, “Effective education for autism,” Monitor on Psychology,vol. 35, no. 11, pp. 46–49, 2004.

[6] A. Landowska, A. Kołakowska, A. Anzulewicz, P. Jarmołkowicz, andJ. Rewera, “E-technologie w diagnozie i pomiarach postepow terapiidzieci z autyzmem w Polsce,” e-mentor, no. 4 (56), pp. 26–30, 2014.doi: 10.15219/em56.1120

[7] A. Landowska, A. Kolakowska, A. Anzulewicz, P. Jarmołkowicz, andJ. Rewera, “E-technologie w edukacji i terapii dzieci z autyzmem wPolsce,” EduAkcja. Magazyn edukacji elektronicznej, vol. 8, no. 2, pp.42–48, 2014.

[8] “Automated therapy monitoring for children with developmental disor-ders of autism spectrum, project website,” http://autmon.eti.pg.gda.pl/,accessed: 2016-05-08.

[9] A. Klin, W. Jones, R. Schultz, F. Volkmar, and D. Cohen, “Visual fixationpatterns during viewing of naturalistic social situations as predictorsof social competence in individuals with autism,” Archives of generalpsychiatry, vol. 59, no. 9, pp. 809–816, 2002.

[10] A. Saitovitch, A. Bargiacchi, N. Chabane, A. Phillipe, F. Brunelle,N. Boddaert, Y. Samson, and M. Zilbovicius, “Studying gaze abnor-malities in autism: Which type of stimulus to use?” 2013.

[11] W. Jones and A. Klin, “Attention to eyes is present but in decline in2-6-month-old infants later diagnosed with autism,” Nature, vol. 504,no. 7480, pp. 427–431, 2013. doi: 10.1038/nature12715

[12] J. Hashemi, T. V. Spina, M. Tepper, A. Esler, V. Morellas, N. Pa-panikolopoulos, and G. Sapiro, “Computer vision tools for the non-invasive assessment of autism-related behavioral markers,” IEEE In-ternational Conference on Development and Learning and EpigeneticRobotics, 2012. doi: 10.1109/devlrn.2012.6400865

[13] E. B. Torres, M. Brincker, R. W. Isenhower, P. Yanovich, K. A. Stigler,J. I. Nurnberger, D. N. Metaxas, and J. V. Jose, “Autism: the micro-movement perspective.” Frontiers in Integrative Neuroscience, vol. 7,2013. doi: 10.3389/fnint.2013.00032

[14] D. P. Wall, R. Dally, R. Luyster, J.-Y. Jung, and T. F. DeLuca, “Useof artificial intelligence to shorten the behavioral diagnosis of autism,”PloS one, vol. 7, no. 8, p. e43855, 2012.

[15] D. Wall, J. Kosmicki, T. Deluca, E. Harstad, and V. Fusaro, “Use ofmachine learning to shorten observation-based screening and diagnosisof autism.”

[16] L. Cheng, G. Kimberly, and F. Orlich, “Kidtalk: online therapy foraspergers syndrome,” Microsoft Research, 2002.

[17] J. W. Tanaka, J. M. Wolf, C. Klaiman, K. Koenig, J. Cockburn,L. Herlihy, C. Brown, S. Stahl, M. D. Kaiser, and R. T. Schultz, “Usingcomputerized games to teach face recognition skills to children withautism spectrum disorder: The lets face it! program,” Journal of ChildPsychology and Psychiatry, vol. 51, no. 8, pp. 944–952, 2010.

[18] D. Deriso, J. Susskind, L. Krieger, and M. Bartlett, “Emotion mirror:a novel intervention for autism based on real-time expression recogni-tion,” in Computer Vision–ECCV 2012. Workshops and Demonstrations.Springer, 2012, pp. 671–674. doi: 10.1007/978-3-642-33885-4 79

[19] R. El Kaliouby and P. Robinson, “The emotional hearing aid: anassistive tool for children with asperger syndrome,” Universal Accessin the Information Society, vol. 4, no. 2, pp. 121–134, 2005. doi:10.1007/s10209-005-0119-0

[20] A. Landowska, K. Karpienko, M. Wrobel, and M. Jedrzejewska-Szczerska, “Selection of physiological parameters for optoelectronicsystem supporting behavioral therapy of autistic children,” in Societyof Photo-Optical Instrumentation Engineers (SPIE) Conference Series,vol. 9290, 2014. doi: 10.1117/12.2075020

[21] M. Jedrzejewska-Szczerska, K. Karpienko, and A. Landowska, “Systemsupporting behavioral therapy for children with autism,” Journal ofInnovative Optical Health Sciences, vol. 8, no. 03, p. 1541008, 2015.doi: 10.1142/s1793545815410084

[22] L. Hanna, K. Risden, and K. Alexander, “Guidelines for usability testingwith children,” Interactions, vol. 4, no. 5, pp. 9–14, 1997.

[23] L. Hanna, K. Risden, M. Czerwinski, and K. J. Alexander, “The role ofusability research in designing childrens computer products,” The designof childrens technology, pp. 3–26, 1999.

[24] J. C. Read, S. MacFarlane, and C. Casey, “Endurability, engagementand expectations: Measuring children’s fun,” in Interaction design andchildren, vol. 2. Shaker Publishing Eindhoven, 2002, pp. 1–23.

[25] G. Sim, S. MacFarlane, and J. Read, “All work and no play:Measuring fun, usability, and learning in software for children,”Computers & Education, vol. 46, no. 3, pp. 235–248, 2006. doi:10.1016/j.compedu.2005.11.021

[26] J. J. Randolph, “Free-marginal multirater kappa (multirater k [free]):An alternative to fleiss’ fixed-marginal multirater kappa.” in JoensuuLearning and Instruction Symposium. ERIC, 2005.