Seeds of empowerment mobile learning for blinds and visually impaired children

All Rights Reserved by Seeds of Empowerment

Spatial Intelligence and ExecutiveSpatial Intelligence and Executive Function of Blinds

in Mobile Learning Environment

Paul Kim

Donggil SongDonggil Song

Arafeh Karimi

1


• The first study was conducted on NovemberThe first study was conducted on November 11, 2010 in SMPK Setapak (Asarama) school located at Jalan Genting Kelang 53300 Kualalocated at Jalan Genting Kelang,53300 Kuala Lumpur, Malaysia

• Participants were 25 total blind children age• Participants were 25 total blind children age 12‐17 years old.

2

Entrance‐ November


11 2010‐ 3 PM We met the principle and teachers. They kindly gathered 30 total blind children in the library for our study.

Our total blind participants welcomed us warmly and thenwelcomed us warmly and then the teacher asked them to go to the classroom beside the library. Non of them used stick to find the way. Some of them were holding y geach others hand for assistance.

After greetings, we explained the purpose of the study.

We told them we heard you areWe told them: we heard you are very smart and doing well in school. We are working on designing mobile games for you and this is our deviceand this is our device.

3

First Touch•Then we handed in the


•Then we handed in the devices to them. As we didn’t have enough TMs

They shared the devices.

•They started to explore TMs right away and were discussing about it with each other.each other.

• 2 participants were holding TMs with 90 degree rotation saying h f lthey feel more comfortable in that position.

•We asked them simple pquestions like how many buttons does the device have ? They started to talk to each other and thento each other and then 90% answered correctly .

•They were excited! 4

Exploration


Exploration

•4 boys figured out the up, down, rightthe up, down, right left concept of the buttons very quickly. I asked them if they h d ihad any experience working with computers and the answer was positive. pThey said “they like computers and they want to play computer games”!computer games ! Later I checked with their teacher and she said these guys are competent in using computers.

5

Exploration


Different ways of holding and using TM is demonstrated in pictures. We suggest further study on usabilityand interfaceand interface design.. .

6

Exploration‐Hand/Mouth/Nose


Exploration Hand/Mouth/Nose

We observed 4 cases using their mouth/nose for exploration. They

lli thwere smelling the device.

Any idea?

7

Connection


Connection

After the introductory session, we have dividedsession, we have divided them to groups of 6 to be able to conduct the study with our 6 TM and be able t itto manage it.

We gave them the device and asked them: Can you help us turning on thehelp us turning on the devices?

They first were playing individually then started to discuss and collaboratewith each other to solve the problem.

8

Connection‐ Found it!


Aina, 13yrs old was the first one figuring out how to turn on the device after 3 minutes. Sh d ’t hShe doesn’t have any experience of playing computer games .

9

Connection


Connection

The audio test was set as the default program in TM so it started playing the audible test after Aina turned it on and everyone were excited ,working harder on their device to fi t h t t itfigure out how to turn it on.

Next one was the boy with gray T‐shirt (The Wong) with 5min record.

After they solved the problem they were guiding each other and sharing the experience.

We talked about the process. They said “I knew the power is in corner but I didn’t know I should hold it”

10

Interaction


Interaction

After everybody learnt how to turn on TMs it was interaction time.

First, all of them listened to the first question and then played with navigation buttons. They had fun when they were pushing the buttons and hearing North, South, West, East in their g , , ,own language (Malay). Later they found out they have to push blue button to confirm their answer and get feedback. Exciting!

The program didn’t include aural instructionThe program didn t include aural instruction on how to work with the device, therefore it took them a while to figure out how to do it. However we gave them time to explore by themselves. For Next groups we guided themthemselves. For Next groups we guided them on how to use the device because of the time issue.

They were fast in finding what each button does unfortunately current version of audiodoes, unfortunately current version of audio test doesn’t have any reaction for two upper buttons (green, yellow). They were expecting to hear something after touching each button. 11

Interaction‐Level1


Interaction Level1

Level 1 questions was described as “easy” by q y ythe participants.

Almost 85% of them answered to the level 1 questions correctly. They were able to follow q y ythe instruction and answer the questions.

No Interaction and discussion happened among the participants while were working g p p gon level 1 questions.

They seemed like they are enjoying it.

All of them knew the cardinal directionsAll of them knew the cardinal directions.

12

Interaction‐Level 2


Interaction Level 2• The moment they heard 180 degree rotation in questions, we observed

expressions like “wow phww mmm oooh ” Mostly from boysexpressions like “wow, phww, mmm,oooh…” Mostly from boys. • 3 participants (2boy, 1girl) asked us to repeat level 2‐q2,3 • They started talking to each other and discussing the answers. More

collaboration.• At Level 2 question, 2 girls started demonstrating the scenario. One of

them stood up and literally followed all the instructions such as 180 rotation. Facing North, turning to right, left. She answered all the questions correctlyquestions correctly.

• The other girl was following the instruction by demonstrating lines on palm of her hand with her index finger .

• One boy (13yrs) started to show signs of frustrations, holding his head i h hi h d i hi l h l d i lwith his hands, moving his leg, he only answered one question correctly.

• Whenever they received “correct” feedback from the device they were excited sharing their success stories with their friends.

• Longer response time was reported compare to level 1Longer response time was reported compare to level 1

13

Interaction‐Level 3


Interaction Level 3

More requests for repetition of the questions.the questions.

Demonstrating girls answered all the questions correctly.

More difficulty following the y gscenario specially when question had two levels of rotation.

M i i dMore interaction and discussion on answers. Sharing success stories.

More incorrect answersMore incorrect answers

More frustration signs but none of them stopped playing.

Longer response timeg p

14

Reflection


Reflection

• We asked them who answered to all the questions correctly? 5 of them stood up, we announced their name and everybody else applaud them and congrats them

• We interviewed them after finishing the data collection. What do you think about the game, what was the problem,…

“I lik d it” “I lik t l i I d ’t lik h ” “I t t• “I liked it”, “I like to play science game, I don’t like geography”, “I want to answer English questions” “Level 3 was hard” “ I want repetition feature” “do you have more games?” “ what does this button (yellow, green) do”

• Because of lack of time we weren’t able to conduct TOL study with all of them soBecause of lack of time we weren t able to conduct TOL study with all of them so we decided to play the instruction on computer and receive feedback.

• They listened to the scenario carefully then gave us following comments:– “too hard to follow”

– “why do you want to put fruits on sticks”

– "hard to imagine" , "why are we moving the fruits", "why we can only move one fruit at a time". One girl (13yrs old asked me if I have the sticks and fruits somewhere she can touch it" “do you have any other game?”

15


• Blind children collaborate with each other to solve more complicated problems

• Blinds need an authentic scenario and enjoy problem solving questions.

• Blind children can perform audible tests using TM however the Instruction and scenario design, narration language, length, and authenticity should be further t di dstudied.

• Usability and interface design study would add more richness to the experience.

• More study on Blinds’ cognition and comprehension process is required (spatial intelligence why did they started demonstrating the scenario physically how dointelligence , why did they started demonstrating the scenario physically, how do they imagine the aural instruction? )

• The more cognition required the more response time needed.

• Participants requested for more aural feedback when they were interacting withParticipants requested for more aural feedback when they were interacting with the device.

16


2nd Study: Tower Of London Game

17


• Study was conducted on 16 November 2010 inStudy was conducted on 16 November 2010 in SMPK Setapak (Asarama) school located at Jalan Genting Kelang 53300 Kuala LumpurJalan Genting Kelang,53300 Kuala Lumpur, Malaysia

• Participants were 12 total blind and 1 visually• Participants were 12 total blind and 1 visually impaired children age 14‐17 years old.

18

Plan


Plan

• Due to School holiday number of participantsDue to School holiday, number of participants was decrease to 12. However quantitative and qualitative data was collectedqualitative data was collected.

• Quantitative data was collected through TeacherMate while participants were playingTeacherMate while participants were playing with Tower of London game.

Q li i d ll d h h i• Qualitative data was collected through semi‐structured interview and focus group sessions.

19

Entrance


Entrance

• Upon our arrival we were welcomed byUpon our arrival we were welcomed by participants from previous study.

we said “Hello guys” they replied “Hi where iswe said Hello guys , they replied Hi, where is your TM?”

Th h k d i f h ill h llThen they asked us to wait for them till they call up their friends.

20

Introduction


Introduction

We didn’t have much to do anymore. Our little friends managed everything. We just sat there and observed.

Old group gathered their friends and passionately explained who we are and what we do. Then they asked us to give them some TeacherMates so they can

l i lexplain more accurately.

Next question “Do you have a new game for us today?”

They invited us to their class and showed us around.

21

Touch listen touch


Touch listen touch

22

New ideas


New ideas

• While we were waiting for teachers to prepareWhile we were waiting for teachers to prepare the new study setting, we started to discuss with our friends about their last experiencewith our friends about their last experience with TM and asked them how do they want to use it? What kind of game do they suggest?use it? What kind of game do they suggest? How are they planning to use TM for their study?study?

23

New ideas


New ideas

• They started to• They started to discuss with each other and sharingother and sharing ideas.

24

New ideas


New ideas

After a funAfter a fun discussion with lots ofwith lots of jokes and laugh theylaugh they concluded:

25


“I like it when I listen to TM and I can control the game I likeI like it when I listen to TM and I can control the game. I like playing with navigation buttons.”

26

“I like hearing different


I like hearing different voices, I feel I’m getting to know new people.”p p

We had two English narration with Korean and Malaysian accent and one sentence in Spanish, they were excited about new accents and were imitating th t d l hithe accents and laughing among themselves .

27

“I want to have all my


school books in TM so I don’t have to sit and read all the braills. I can play with TeacherMate and listen to my lessons.”

Two other participants instantly suggested they will record the lessons for him!

28

“We like audio games ” They


games. They patiently explained to us how they interact with the environment and emphasized on pauditory channel as one of the most important methodsimportant methods of communication.

“When we listen to any instruction we imagine it in our mind You knowWhen we listen to any instruction we imagine it in our mind. You know blind people have a better memory than sighted kids!!!!”

29

Other suggestions about using TM in class continued as:


continued as:“We can use TM to do our assignments.” Azli, 14years old asked us how we created the14years old asked us how we created the games with audible test and if it is easy for their teacher to use the audio maker. Thentheir teacher to use the audio maker. Then problem solved.

“we will ask our teacher to record thewe will ask our teacher to record the assignment in audio test maker and we will submit our audio files to her”.

Currently they are using braille typewriters to do their assignment.

30

Tie was excited “ wow, we don’t have to carry


typing machine everywhere anymore!”

TeacherMate vs braille typewriterTeacherMate vs. braille typewriter

31

Next idea:


“Can we take all of our exam with TM?

It would be more fun”It would be more fun

They discussed about pros and cons of taking exam with TM but at the end agreed that it’s aexam with TM but at the end agreed that it s a better option for them if audio maker supports repeating the question and they aresupports repeating the question and they are able to revise the answer.

32

New Ideas


New Ideas

Next lesson for us was learning how they replied to g y pthe “cardinal direction” game.

“I imagined myself walking while following the instruction”instruction”

“I imagined I’m driving a car and am following the direction to get to the destination.”direction to get to the destination.

“I imagined leading my friend, holding his hand and guiding him”

We continued our discussion after playing TOL game.

33

Tower of London


Tower of London

• To design the audible version of TOL game we followed participatory design approach and changed the original game based on feedback received from our blind designers.

• V1‐Original TOL using color to differentiate the donuts‐ Our little fiends commented that color is not authentic to them and it’s rather a label and namecommented that color is not authentic to them and it s rather a label and name than a comprehensive concept.

• V2‐Using fruit and skewer concept. You have to move the fruits and rearrange the order‐We had to justify to them “why do you want to move them” why did you j y y y y yput the fruit on sticks”

• V3‐Using a barbecue party and role playing concept, problem solving‐ Players are expert chefs who are taking different orders to prepare various arrangements of hi k fi h d l bchicken, fish and lamb.

• V4‐ Local version of barbecue party with fish, and shrimp ball and chicken suggested by total blind participants. (Malay translation)

34

Sample Scenario


Sample Scenario

• Order 1. Start position: The left skewer holds up O de . Sta t pos t o : e e t s e e o ds uptwo foods, chicken and shrimp. The chicken is at the bottom, and the shrimp is on the chicken. The

ddl k h ld f h b ll h bmiddle skewer holds up a fish ball at the bottom. The right skewer holds up nothing.G l iti Th l ft k h ld thi• Goal position: The left skewer holds up nothing. The middle skewer holds up two foods, fish ball and chicken. The fish ball is at the bottom, andand chicken. The fish ball is at the bottom, and the chicken is on top of the fish ball. The right skewer holds up a shrimp.

35

Game Design


Game Design

Original TOL has 5 levels with 52 questionsOriginal TOL has 5 levels with 52 questions.

For this design we planned to test only 3 levels with 21 questionswith 21 questions.

36

TOL Study


TOL Study

12 Students12 Students gathered to participate inparticipate in our study

37

TOL Study


TOL Study

We divided the participants to two groups to be able to use our devices and facilitate observation process.

38

TOL Study


TOL Study

Participants had a t c pa ts adenough time to explore the TM.

l d hWe explained the concept and what they have beenthey have been asked to do and they eagerly y g ystarted to play the game.

39

TOL‐Level 1


TOL Level 1

Beginning level of the TOL game. Participants were focusing on g g g p gthe instruction to figure out how this game works.

40

TOL‐Level 1


TOL Level 1The level of participation and collaboration at level 1,beginning of the game was low and they separately were concentratingof the game, was low and they separately were concentrating

on their tasks.

41

TOL‐Level 2


TOL Level 2After they passed level 1 and understood the game concept the

participation and collaboration started and they looked more p p yrelaxed and seemed they are having fun with the game

42

TOL‐Level 3


TOL Level 3

• Level 3 questions are a bit harder and moreLevel 3 questions are a bit harder and more complicated compare to previous questions but participants had the highest level of collaboration and they were talking together about their performance.

• The TOL game provides feedback to players and informs them if they replied correctly or

P i i h d hi h l l fnot. Participants showed high level of concern about correct/incorrect feedback.

43

TOL‐Level 3


TOL Level 3

44

TOL‐Level 3


TOL Level 3

45

Su continued playing till level 5 although b d l h l f


everybody else has left ….

46

More ideas


More ideas

• After data collection we had a chance of discussing more gabout their feedback and learning from their suggestion.

• They labeled our game as Geography game (Cardinal di ti ) d Mi d (TOL )direction) and Mind game (TOL game).

• 10/12 participants found the mind game more challenging and interesting rather than geography gameg g g p y g

• All participants asked us about our next game

• 2 participants called up their visually impaired friends to play with TM.

• 3participants asked for science game (2boy 1 girl) and 8 asked for more mind game and 1 asked for literacy gamefor more mind game and 1 asked for literacy game.

47

More ideas


More ideas

• Natasha: “ I like mind game to improve my memory so I can g p y yremember more, it makes my brain smarter”

• All participants suggested to add repeat feature and more t l th flcontrol over the game flow.

• They asked for more group games.

They informed us about their everyday activities such asactivities such as playing “goal ball”, Chess, riding a bicycle, li t i t di dlistening to radio and TV.

48

African stories


African storiesWe shared Seeds’ f hAfrican stories with

them (Jomopira and Kakama) playing onKakama), playing on iPad and Android phone They answeredphone. They answered to all the story questions qcorrectly.

49

Zambia and India plan



Next we shared Seeds’ plan to go to India andNext we shared Seeds plan to go to India and Zambia and design games for blind children. We explained children’s educational situation to themexplained children s educational situation to them and our young friends started to propose solutions.

50




They asked many questions such as how old are they, if they don’t y y q y, yhave typewriter how do they take exam?

What language they speak?

How could they learn more about them?

Why are they facing difficulties in their life?

What courses they like?What courses they like?

Do you have more stories from them?

51

Zambia and India suggestions


Zambia and India suggestions

“Their teacher can record all of their courses on TM and they can listen to them all the time, It’s OK if they don’t know braille““We can record all of our lessons for them and send it to them”“They can use TM to practice their courses do assignments takeThey can use TM to practice their courses, do assignments, take exam”“We can design audio games for them to play after school. Do you h i d ?”have more mind games?”“We can record our stories for them and send on TM”They offered us their help and support and proposed revising our ey o e ed us t e e p a d suppo t a d p oposed e s g oufuture games to make it more suitable for blinds!We were asked if we can give TMs to them to play in school, unfortunately we didn’t have enough devices to shareunfortunately we didn t have enough devices to share.

52

Next?


Next?

• Revising TOL game and create multi‐lingual versions

• More study on Instructional design and narration instruction design for blinds

• Pilot testing the audio game with children from varied socioeconomic g gbackground

• Increase the sample size

• Design game in different context: science math literacyDesign game in different context: science, math, literacy,…

• Study on collaborative and participatory learning among blinds

• User interface and tactile experience studies

S d bli d ’ i i i lli bili d d l• Study on blinds’ cognitive intelligence, ability and development

53

Acknowledgement


Acknowledgement

Special thanks to Nurzyanti Mohammad Khan forSpecial thanks to Nurzyanti Mohammad Khan for her generous help in translating and narrating all the questions and great help in conducting the q g p gstudies. Thanks to Maryam Sarkhail for her assistance in conducting the Cardinal direction study. We couldn’t do any of this without you.

We acknowledge our appreciation toward principle g pp p pand teachers of SMPK SETAPAK school specially MS Rosi.

54

Education

Seeds of empowerment mobile learning for blinds and visually impaired children