Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Development and Implementation of Behaviours for a Socially Assistive Robot for the Elderly
By
Zhonghe (Jacob) Li
A thesis submitted in conformity with the requirements
for the degree of Masters of Applied Science
Mechanical and Industrial Engineering University of Toronto
© Copyright by Zhonghe Li 2015
Development and Implementation of Behaviours for a Socially
Assistive Robot for the Elderly
Zhonghe (Jacob) Li
Masters of Science
Mechanical and Industrial Engineering University of Toronto
2015
Abstract
Bingo is a cognitively stimulating recreational activity popular in long-term care facilities and
retirement homes. This thesis focuses on the development of the human-like socially assistive
robot Tangy for the autonomous facilitation of Bingo. A set of assistive social behaviours are
developed for multi-user and single user interactions the robot may encounter while facilitating
the Bingo game. In addition, development of actuation modules for establishing eye contact,
controlling the arms, and interacting with players using audio communication is discussed. A set
of experiments verified the system performance of these behaviours and actuation capabilities
and the efficacy of the social robot behaviours with elderly residents in long-term care. The
results demonstrate that the robot determines and executes the appropriate behaviours during the
facilitation of a Bingo game, and that elderly residents enjoyed interacting with Tangy and
wished to participate in Bingo games facilitated by the robot in the future.
ii
Acknowledgements
I would like to thank my supervisor, Professor Goldie Nejat, for her continued support, input, and
guidance throughout my research. I would also like to thank my M.A.Sc. thesis committee for their
time and feedback. I would also like to thank Geoffrey Louie, Chris Mohamed, Frank Despond,
Tiago Vaquero and Vincent Lee for their work on the collaborative Tangy project. Thank you to the
Dr. Robot Inc. developers for their generous technical support with the robot platform. I would like
to thank my friends and family, and my lab-mates for their advice and comradeship.
iii
Table of Contents
Acknowledgements ........................................................................................................................ iii
Table of Contents ........................................................................................................................... iv
List of Tables ............................................................................................................................... viii
List of Figures ................................................................................................................................ ix
Chapter 1 Introduction ..............................................................................................................1
1.1 Motivation ............................................................................................................................1
1.2 Tangy the Socially Assistive Robot .....................................................................................2
1.3 Problem Definition...............................................................................................................2
1.4 Proposed Methodology ........................................................................................................3
1.4.1 Literature Review.....................................................................................................3
1.4.2 Gathering Impressions/Design Considerations ........................................................3
1.4.3 Robot Behaviours.....................................................................................................3
1.4.4 Actuation Modules ...................................................................................................4
1.4.5 System Performance Review ...................................................................................4
1.4.6 HRI User Studies .....................................................................................................4
1.4.7 Conclusions ..............................................................................................................4
Chapter 2 Literature Review.....................................................................................................5
2.1 Cognitive Training Activities ..............................................................................................5
2.1.1 Specific Cognitive Training Programs .....................................................................5
2.1.2 General Cognitive Training Programs ....................................................................6
2.1.3 Bingo as a Cognitive Training Intervention .............................................................7
2.2 Socially Assistive Robots for Older Adults .........................................................................7
2.2.1 Single User Scenarios ..............................................................................................7
iv
2.2.2 Multi-User Scenarios ..............................................................................................8
2.3 Social Human-Robot Interaction .........................................................................................9
2.3.1 Eye Contact ............................................................................................................10
2.3.2 Physical Gestures ..................................................................................................11
2.4 Chapter Summary ..............................................................................................................13
Chapter 3 Gathering Impressions and Design Considerations ...............................................15
3.1 Focus Group Study ............................................................................................................15
3.1.1 Single User Scenarios ............................................................................................15
3.1.2 Participant Demographics .....................................................................................17
3.1.3 Thematic Analysis .................................................................................................18
3.2 Thematic Sets .....................................................................................................................18
3.2.1 Bingo .....................................................................................................................18
3.2.2 Telepresence ..........................................................................................................19
3.2.3 Tangy’s Appearance .............................................................................................19
3.2.4 Acceptance of the Robot .......................................................................................20
3.2.5 Ease of Use ............................................................................................................21
3.3 Suggested Features/Activities ............................................................................................22
3.3.1 Reminders/Prompting ............................................................................................22
3.3.2 Simple Interactions with Cognitively Impaired Residents ...................................22
3.3.3 Other Recreational Activities .................................................................................23
3.3.4 Physical Tasks and Aiding Residents’ Autonomy ................................................23
3.3.5 Music Therapy .......................................................................................................23
3.3.6 Multilingual Support .............................................................................................23
3.4 Chapter Summary ..............................................................................................................24
v
Chapter 4 Robot Behaviours...................................................................................................25
4.1 The Bingo Scenario............................................................................................................25
4.2 Behaviour Determination ...................................................................................................26
4.2.1 Sensory Capabilities ..............................................................................................27
4.3 Robot States During the Bingo Game ................................................................................26
4.3.1 Pre-Game State .....................................................................................................28
4.3.2 Multi-User State .....................................................................................................29
4.3.3 Transition State .....................................................................................................30
4.3.4 Single User State ...................................................................................................31
4.3.5 Post-Game State ....................................................................................................33
4.4 Behaviour Determination ...................................................................................................33
4.5 Chapter Summary ..............................................................................................................38
Chapter 5 Robot Actuation Modules ......................................................................................39
5.1 Arm Control Module..........................................................................................................39
5.1.1 Motion Planning for the Arms ...............................................................................39
5.1.2 Optimization Parameters ........................................................................................41
5.2 Eye Contact Module ..........................................................................................................43
5.3 Audio Interactions ..............................................................................................................43
5.4 Chapter Summary ..............................................................................................................45
Chapter 6 System Performance Review .................................................................................47
6.1 Finite State Machine Performance Review........................................................................47
6.1.1 Methods .................................................................................................................47
6.1.2 Results ....................................................................................................................48
6.2 Arm Control Accuracy Experiment ...................................................................................48
6.2.1 Methods .................................................................................................................48
vi
6.2.2 Results and Discussion ..........................................................................................52
6.3 Arm Control Accuracy Experiment ...................................................................................55
6.3.1 Methods .................................................................................................................55
6.3.2 Results and Discussion ..........................................................................................56
6.4 Chapter Summary ..............................................................................................................59
Chapter 7 Human Robot Interaction User Studies .................................................................60
7.1 Participants .........................................................................................................................60
7.2 Methods..............................................................................................................................61
7.3 System Performance Results ..............................................................................................62
7.4 Human-Robot Interaction Results......................................................................................63
7.4.1 Participant Questionnaire Results .........................................................................63
7.5 Discussion ..........................................................................................................................65
7.6 Chapter Summary ..............................................................................................................66
Chapter 8 Conclusion .............................................................................................................67
8.1 Summary of Contributions .................................................................................................67
8.1.1 Gathering End-User Feedback ..............................................................................67
8.1.2 Assistive Robotic Behaviours and Actuation Capabilities ...................................67
8.1.3 Experimental Results ............................................................................................68
8.2 Discussion of Future Work ................................................................................................68
vii
List of Tables
Table 1: Focus Group Questions....................................................................................................16
Table 2: Participant Demographics ................................................................................................17
Table 3: Computer and Robot Experience of Focus Group Participants .......................................17
Table 4: Pre-Game Behaviours ......................................................................................................28
Table 5: Multi-User State Behaviours ...........................................................................................30
Table 6: Transition Behaviours ......................................................................................................31
Table 7: Single user Behaviours ....................................................................................................32
Table 8: Post-Game Behaviour ......................................................................................................33
Table 9: Behaviour-Determining States.........................................................................................34
Table 10: Manufacturer Servo Specifications for Tangy's Arms .................................................43
Table 11: Finite State Machine Performance .................................................................................48
Table 12: Robot Right Arm Poses .................................................................................................51
Table 13: Accuracy Performance Review for Tangy's Right Arm ................................................53
Table 14: Accuracy Performance Review for Tangy's Left Arm ..................................................54
Table 15: Mean Errors in Joint Accuracy Performance over Three Trials ....................................55
Table 16: Duration and Energy Consumption Differences between Sets of Twenty-Five
Executions of Optimized and Un-Optimized Gestures ..................................................................58
Table 17: Number of Participants with Experience with Computers or Robots............................60
Table 18: System Performance Results .........................................................................................63
Table 19: Post-Bingo Session Questionnaire Results ....................................................................64
viii
List of Figures
Figure 1: Tangy the Socially Assistive Robot ...................................................................... 2
Figure 2: Tangy's System Architecture ..........................................................................................26
Figure 3: Off-board and On-board Sensors ...................................................................................27
Figure 4: Tangy’s FSM ..................................................................................................................36
Figure 5: Tangy Behaviours during Bingo Game ..........................................................................37
Figure 6: CAD Model of Tangy and Kinematic Model of Tangy’s Arm ......................................40
Figure 7: Face Detection Rotation Invariance ...............................................................................44
Figure 8: Face Tracking .................................................................................................................45
Figure 9: Servo Number Scheme (Right Arm) ..............................................................................49
Figure 10: Texas Instruments CC2650 SensorTag ........................................................................50
ix
1
Chapter 1 Introduction
1.1 Motivation:
As people age, they experience greater risks for cognitive impairments such as declines in
memory, abstract reasoning, attention skills and verbal skills [1]. These age-related cognitive
impairments among the elderly diminish their capability to independently perform activities of
daily living, such as eating, dressing and toileting [2]. Older adults with cognitive impairments
who are not able to live independently frequently turn to long-term care facilities where they can
receive support with these activities of daily living, medical services, as well as scheduled leisure
activities [3]. Long-term care facilities also provide the elderly with opportunities for social
engagement, which have been shown to improve the quality of life of long-term care residents
[4].
As the population is rapidly aging, the demand for long-term care facilities is growing larger and
larger [5]. Currently, the population of older adults above the age of 60 is 901 million, and is
projected to reach 3.2 billion by 2050 [6]. Conversely, the long-term care workforce, which
already traditionally sees higher rates of turnover, is facing lower numbers of healthcare staff
each year [5]. The diminishing long-term care workforce has led to gaps in service for elderly
residents, such as in the provision of recreational activities in these facilities [5]. The increasing
pressure on the long-term care health system has led researchers to explore assistive technologies
in order to provide aids for healthcare staff to lessen their workloads.
Cognitive training interventions are a form of therapy-based activities which are designed to
maintain or reduce the rate of age-related cognitive decline [7]. Various types of cognitive
training exist, with both specific and non-specific cognitive training activities [7]. Specific
cognitive training interventions target particular functions in the brain such as memory,
reasoning, or speed of processing [7]. Non-specific cognitive interventions do not target any
particular functions, but are designed to give general cognitive stimulation through recreational
activities [7].
1
2
1.2 Tangy the Socially Assistive Robot
One of the main objectives of the Autonomous Systems and Biomechatronics Laboratory
(ASBLab) is to develop and integrate robotic technologies in order to aid in cognitive
interventions for older adults. A key research area in the ASBLab focuses on developing the
socially assistive robot, Tangy, which can aid in facilitating general cognitive interventions in the
form of recreational activities. Tangy is a human-like socially assistive robot which is being
developed in order to autonomously facilitate the cognitively stimulating multi-user recreational
activity Bingo for retirement home and long-term care settings (Figure 1).
Figure 1: Tangy the Socially Assistive Robot
Currently, Tangy is being designed to directly incorporate common forms of communication
used in natural social interactions between people. The advantage of a socially assistive robot is
to be able to provide an intuitive and effective set of assistive behaviours upon employment in a
long-term care/retirement home setting due to its ability to communicate within existing social
structures. Moreover, the robot is capable of lessening the workloads of the healthcare staff,
while providing beneficial social and cognitive stimulation to elderly residents.
1.3 Problem Definition
The research behind this thesis focuses on the implementation of a socially assistive robot for
facilitating a multi-user cognitive intervention for older adults. In particular, Tangy will facilitate
a Bingo game by performing a set of structured behaviours for a group of elderly users. These
behaviours involve sensing the environment, determining a set of assistive actions based on the
2
3
state of the environment, robot and game, and performing these actions using a set of physical
and auditory actuation techniques. The objective of this thesis is to develop the decision-making
capabilities needed in order to generate the interactive robot behaviours for Tangy, as well as the
various software modules to determine and then physically display the social behaviours on the
robot. These modules discussed in detail in this thesis will include the robot deliberation, arm
motion planning, eye contact, and audio interaction systems.
1.4 Proposed Methodology
The overall methodology for the design of Tangy’s behaviours and actuation interfaces
comprises of the following components with corresponding reference to the thesis chapters:
1.4.1 Literature Review
In Chapter 2, literature review for the following areas, which are critical to the development of
socially assistive robots in long-term care, is presented: (i) cognitive interventions for older
adults; (ii) socially assistive robots used in healthcare settings for older adults; and (iii) social
behaviours implemented on robotic systems.
1.4.2 Gathering Impressions/Design Considerations
In Chapter 3, the results from focus group studies which gathered the impressions and
suggestions that residents, their family members and healthcare staff at a long-term care facility
and a retirement home had about socially assistive robots, including Tangy, are presented to
inform the design of the robot’s functionality and behaviours. The chapter discusses the
methodology of the focus groups, and then the various thematic sets which were drawn from the
discussions of the participants.
1.4.3 Robot Behaviours
In Chapter 4, a brief description of the Bingo multi-user scenario is first presented. Then, each
behaviour that the robot performs in the scenario is detailed. Lastly, the chapter presents the
development of the behaviour determination module for the robot, which focuses on choosing
the robot’s appropriate behaviours based on the current robot, game, and world states.
3
4
1.4.4 Actuation Modules
In Chapter 5, the design of the software interfaces used in Tangy’s physical implementation of its
behaviours is presented. These modular interfaces are described within the context of Tangy’s
overall layered system architecture. The functionalities of each layer as well as the individual
modular structure of each layer are explained. The inputs and outputs of each module are
outlined in order to present its function and role in the architecture. In particular, the section
describes the development of the arm control, eye contact, and audio interaction modules. With
respect to the arm control, both the motion planning and optimization techniques for choosing
the most efficient motion plan are discussed.
1.4.5 System Performance Review
In Chapter 6, experiments conducted in our lab to verify the success rates of the robot’s
behaviours during Bingo games and the optimization technique for choosing motion plans for the
robot’s gestures are presented. The optimization technique has been tested to ensure that the
optimized motion plans have lower time and energy costs when compared to a control set of
motion plans.
1.4.6 User Studies
In Chapter 7, detailed experiments and user studies at a long-term care facility and a retirement
home are presented to evaluate the implementation of the robot behaviours and the control
architecture for the intended population. User studies examined the use of the robot behaviours
in social interactions with elderly residents. Discussions to illustrate the effectiveness of the
proposed designs are also presented.
1.4.7 Conclusion
Lastly, Chapter 7 presents concluding remarks on the development of Tangy’s behaviours and
architecture, highlighting the main contributions of the thesis and future work.
4
5
Chapter 2 Literature Review
This section provides an overview of work related to socially assistive robots for older adults. In
Section 2.1, various types of cognitive training programs for the elderly are discussed,
concluding with a discussion about the recreational activity Bingo. In Section 2.2, literature on
socially assistive robots for single user and multi-user scenarios with the elderly is discussed.
Lastly, in Section 2.3, literature on social human-robot interaction is discussed.
2.1 Cognitive Training Activities
Cognitive training activities are therapy-based activities which aim to stimulate cognitive
functions in order to delay or halt age-related cognitive decline [1]. Cognitive training includes
both specific cognitive interventions and general cognitive training activities [7],[8],. Specific
cognitive training programs aim to stimulate particular functions such as memory, reasoning, or
speed of processing, while general cognitive stimulation activities involve general activities
which stimulate individuals through active cognitive and social engagement [7],[9].
2.1.1 Specific Cognitive Training Programs
Several specific cognitive training programs have been shown to reduce rates of cognitive
decline and improve functioning among those already suffering from cognitive impairments [10].
For example, the IMPACT (Improvement in Memory with Plasticity-based Adaptive Cognitive
Training) program was aimed to improve memory and attention in participants using exercises
self-administered on a computer [11]. The IMPACT training program was administered to older
adults above the age of 65, and included exercises to test their processing speed of speech-related
audio information. Results from the program indicated that there was a significant improvement
in auditory-based cognition when comparing participants who were administered the training
program and participants who participated in a control program.
Another specific cognitive training program was the ACTIVE (Advanced Cognitive Training for
Independent and Vital Elderly) program, which employed certified personnel to teach strategies
and administer exercises to improve performance in memory, reasoning and speed of processing
[8]. More specifically, the exercises included training verbal episodic memory, the ability to
5
6
solve problems which follow a serial pattern, and the ability to visually search and identify
information in a divided-information format. For training verbal episodic memory, participants
were taught mnemonic strategies and then given practice exercises related to everyday tasks. For
reasoning training, participants were given number or letter series and asked to identify the
patterns in the series. Finally, for speed-of-processing training, participants were asked to
perform speed tasks on a computer with continually increasing difficulties for the tasks. The
ACTIVE program was shown to improve the targeted cognitive abilities by magnitudes
equivalent to the declines in those abilities expected in an elderly person without dementia over 7
and 14 year periods (depending on the ability).
The success of these specific cognitive training programs often depends on adherence to self-
administered training and professional supervision. This reliance limits their availability for
institutionalized older adults [12].
2.1.2 General Cognitive Training Programs
General cognitive training programs which have been demonstrated to improve cognitive
functioning in older adults include Reality Orientation Training (ROT) [13], rehabilitative training
based on procedural memory [14], and/or recreational activities and art therapies [7],[16]. ROT
involves repetitively presenting the participant with orientation information (such as location or
time) throughout the day or in group meetings [13]. ROT has been demonstrated to significantly
improve Mini-Mental State (MMS) scores and other measures of cognition such as word list
memory for older adults [13],[14].
Rehabilitative training based on practicing activities of daily living has been used to stimulate
procedural memory [15]. This training has also been shown to provide improvement in the
performance of those activities by older adults and promote cognitive functioning [15]. After an
occupational therapist conducted procedural memory training with a group of participants for a
three week period by prompting, instructing and modelling thirteen activities of daily living, the
participants significantly increased their speed for performing those activities.
Participation in recreational activities has also shown benefits in cognitive functioning in the
elderly [7],[17]. Moreover, recreational activities provide secondary benefits such as greater
happiness, better physical functional abilities, and a reduction in mortality for older adults [18].
6
7
When elderly participants with mild Alzheimer’s disease participated in conversation, singing,
party games and dancing over six weeks as part of a cognitive training program, they experienced
improvements in behavioral and everyday functioning [7]. Another example of recreational
activities being used for cognitive training was presented in [17], where the AKTIVA (Aktive
Kognition Stimulation – Vorbeugung im Alter) program was used to educate and promote the
benefits of participating in cognitively stimulating recreational activities for older adults in the
community. Participants in the program were encouraged to read, play games, and play musical
instruments [17]. Results from the study showed the program enhanced speed of processing,
improved self-reported subjective memory decline, and increased voluntary participation in
cognitively stimulating activities.
2.1.3 Bingo as a Cognitive Training Intervention
A recreational cognitively stimulating activity that is popular among older adults is Bingo.
Bingo’s simplicity allows it to be accessible to a large range of player skills, and can be
increasingly challenging through the addition of cards a user simultaneously plays [19]. Bingo
has been shown to have therapeutic benefits for cognitive functions such as memory, recall, and
recognition [4]. Moreover, the activity provides a structured social setting which encourages
engagement among players. The social interactions players experience in a Bingo game are
important, as improving an elderly individual’s social network has a variety of benefits including
lowered mortality rates, reducing isolation and depression, and delaying age-related cognitive
decline [20],[21].
2.2 Socially Assistive Robots for Older Adults
Socially assistive robots have been designed as potential aids for facilitating cognitive training
programs for the elderly [22]-[27]. A variety of socially assistive robots have been designed to
engage in recreational activities with the elderly in both single user and multi-user scenarios. The
robots range from semi-autonomous facilitation of an activity with some assistance from a
human operator to fully autonomous.
2.2.1 Single User Scenarios
7
8
Socially assistive robots have been designed to engage in cognitively stimulating recreational
activities with the elderly such as memory card games [22], music games [23], and read-alongs
and arithmetic quizzes [24]. In [22], the human-like robot Brian 2.1 autonomously engaged
elderly individuals in a memory card game in which users memorize the locations of pairs of
unique cards which are briefly shown before being flipped over [22]. The study found that the
elderly users enjoyed interacting with Brian 2.1, thought that it was sociable, and were both
engaged in the interaction and compliant during the robot’s prompts. In [23], the small child-like
robot Bandit played a music-guessing game, in which excerpts of songs were played and
participants pressed the button corresponding to the song name, with cognitively impaired
elderly residents of a long-term care facility [23]. Through the activity, the study found that
Bandit was able to maintain and improve the impaired residents’ cognitive attention. In [24], the
small animated YORISOI Ifbot engaged older adults in conversational read-alongs of Japanese
folk tales and simple arithmetic calculations by speaking to the adults, asking questions and
instructing users to input their answers. This study found that all participants actively engaged in
the activities, and felt increasing affection for the robots.
2.2.2 Multi-User Scenarios
While socially assistive robots have been effective in engaging and instructing elderly residents
during single user activities as previously mentioned, those scenarios limit many of the social
benefits that can be achieved through multiplayer recreational activities. Multiplayer activities
instead can provide social engagement among players, which can reduce symptoms of
depression [20], decrease physical impairment [21], lessen the negative impact of widowhood on
health [21], and increase functional independence [21]. As such, socially assistive robots have
also been developed to facilitate or participate in recreational activities with multiple users.
Robots that are designed for multi-user scenarios with the elderly include: Ifbot, which facilitated
educational games [25]; Aibo, which played a physical catch game [26]; and Matilda, which
facilitated Bingo and Hoy games in [27].
In [25], a group of elderly long-term care residents in Japan participated in a study where
participants engaged in school-style recreational and academic activities with the Ifbot robot.
The activities were structured similar to primary school classes, with language studies, sing-
alongs, tongue twisters and arithmetic exercises. The robot would facilitate these various
8
9
activities by presenting exercises to the elderly users, receiving their solutions to the exercises,
and responding with an assistive behaviour based on their inputs. The interactions between the
robot and the participants were mediated by a human operator, who had to repeat the robot’s
prompts and manually input their responses into the robot. The results of the study indicated that
the robot and the activities were received positively, and the elderly participants actively engaged
in the exercises.
A different type of activity was investigated in [26], where an AIBO animal-like robot played a
physical recreational game with elderly nursing home residents in Japan with moderate
dementia. The game was a three-fold ball-catching activity in which: i) participants would throw
the ball and the robot would fetch it; ii) participants would throw the ball into a basket
shouldered by the robot and compete for the most number of points; and iii) participants would
cooperate and pass the ball among each other in order to keep it away from the robot. The
recreational activity was facilitated by a human coordinator who explained the game to the
residents. The results from the study showed that interacting with the robot and with the other
participants in this recreational activity improved a player’s emotion control and social skills.
In [27], the affective communication robot Matilda was used to facilitate Bingo and Bingo-like
games for nursing home residents in Australia. The robot worked cooperatively with caretakers
in a nursing home to facilitate Bingo and Hoy (a Bingo-clone which replaces the Bingo numbers
with playing cards). Matilda’s task was to call out and display the Bingo numbers and Hoy cards,
while caretakers were responsible for providing one-on-one help to players if they were not able
to hear the numbers, or if they believed they had a winning card. Reactions from the nursing
home residents about Matilda were very positive—residents enjoyed the robot’s body language
and playing the game with the robot.
In multi-user scenarios, the socially assistive robots received positive feedback when they
engaged elderly participants in recreational activities. However, the robots required some human
assistance to engage in multiple interactions with the users throughout the activities.
2.3 Social Human-Robot Interaction
In order for users to adopt a social robot as an assistive aid in the facilitation of recreational
activities, users should display a desire to use the robot and trust in the robot’s advice when it
9
10
performs an assistive behaviour. Intent to use a robotic system is directly correlated with
perceived enjoyment of using the system by the Almere model [28]. As the sociability of a
robotic system has a positive relationship with perceived enjoyment, intent to use is then
positively influenced by how sociable the robot is [28]. Trust in a robot has also been shown to
be influenced by the amount of social intelligence and social abilities in a robotic system [29].
The development of a socially assistive robot should, therefore, take into account types of social
expressions to implement in order to maximize users’ intent to use the robot. Two typical types
of social actions include maintaining eye contact and using arms to gesticulate. The following
sections discuss the use of the non-verbal forms of communication of eye contact and physical
gestures in human-robot interactions.
2.3.1 Eye Contact
Eye contact plays an important role in human communication in directing the flow of
communication and providing cues to help people guess the internal states of the other person
[30],[31]. There have been several research projects which have investigated the role of eye
contact in communication between humans and robots [32]-[34]. One example of a gaze-
communicative robot was presented in [32], which investigated a stuffed toy-like robot that
monitored a user’s gaze direction and performed a gesture or uttered an expression of
acknowledgement whenever eye contact was made. Twenty-two people aged twenty-one to forty
years old were shown the robot placed in between two monitors which showed two animations
randomly as stimuli. They were instructed to look at the robot as it turned its head to look at the
participants and the screens, and then asked what the robot had in mind. One of the primary
conclusions from the study found that the eye contact between the user and the robot provided a
direct evocation of a favorable feeling, and the robot’s gaze was able to draw the attention of a
user to an external stimulus.
In [33], the interactive humanoid robot Robovie was able to display human-like behaviours
through actuators, and vision and audio sensors. In order to test Robovie’s communication with
humans, a version of the robot with eye contact and a version without were given to two groups
of researchers and engineers within the laboratory of the investigators leading the study and were
tasked to obtain the attention of the humans and direct it towards a poster on a wall. The study
found that the participants with the version of the robot with eye contact all indicated that they
10
11
saw a poster, while the majority of the other participants indicated that they saw Robovie’s hand.
The study demonstrated that eye contact was an important factor in obtaining the attention of the
users, as it indicated the robot’s intention to communicate with them.
Another example of eye contact to direct attention during communication between a human and
a robot occurs in [34], when a robot shared joint attention with a human through the use of gaze
tracking. In [34], the expressive humanoid robot Leonardo detected and examined peoples’ faces
in order to determine their expressions as part of a set of imitative behaviours to mimic during
natural social interactions between humans. As part of the face detection, Leonardo also tracked
the user’s eye gaze in order to learn and follow the source of his/her attention. This way,
Leonardo was capable of following the flow of the communication, as well as discovering new
salient aspects of the environment.
2.3.2 Physical Gestures
Robots can use gesturing in order to maintain the attention of a person, explicitly direct
someone’s attention during the flow of communication or express emotion [35]-[39]. The use of
engagement gestures during a human-robot communication interaction can be important to
engage the human and maintain his/her attention [35]. A socially communicative penguin robot
Mel was used to study the efficacy of engagement interactions with human participants in [35].
The robot had two 2 degrees-of-freedom (DOF) shoulders to actuate its wings, a 2 DOF neck to
orient its head, and a 1 DOF beak to open its mouth when speaking. A version of the robot
performed gestures while speaking such as moving its beaks, wings and head, while a second
version of the robot merely used speech and beak movements. 37 participants from the summer
staff at the laboratory ranging from 20 to 50 years of age were split into two groups to interact
with either the fully moving version or the non-moving version of Mel. The participants were
shown a demo of Mel, and then given a questionnaire to investigate their feelings on the robot.
The results found that the participants were more engaged when Mel performed more
movements, and turned their attention to other objects in the environment fewer times during the
demo than the group with the version of the non-gesturing robot. Moreover, participants in the
group where the robot gestured tended to respond to the robot more frequently than the
participants in the other group.
11
12
Arm and head gestures in robots can also play a significant part in displaying the intent of the
robot during a communication interaction, such as when coordinating joint activity between a
robot and a human [36]. A study in [36] investigated the impact of nonverbal communication on
the efficiency and robustness of human-robot teamwork using the humanoid robot Leonardo.
Leonardo was a 65 DOF expressive robot which was capable of orienting its head, moving its
arms to gesture or perform tasks, and shift its body position. Participants from the local
university campus ranging from 20 to 40 years of age were asked to teach Leonardo a task, check
to make sure it knew the procedure to complete the task, and acknowledge when the robot was
able to complete it. During the study, participants relied purely on the non-verbal gestures like
shrugging, nodding, and confusion gestures to diagnose when the robot was not able to
understand instructions. The results of the study demonstrated that having a greater number of
intuitive social gestures significantly increased the speed at which the humans were able to teach
the robot the task. The intuitive social gestures used by Leonardo were able to display the robot’s
intent to participants such that they could immediately adjust their actions in order to better work
with the robot.
Physical pointing in particular is a useful gesture for communicating the intent of a robot towards
a specific focus of attention. In [37], the anthropomorphic upper-torso robot called Nico
performed a series of tasks with participants in order to gauge aspects of social human-robot
interaction such as greetings, cooperation between the user and the robot, and trust in the robot’s
advice. 65 undergraduate students, graduate students and administrative staff at Yale University
participated as subjects in the study. During part of the study, Nico would first wave to a
participant in the room to greet him/her. It would later present a task to the participant by
physically pointing to a stack of books in the room and then to a close-by bookcase to indicate
that the participant needed to move the books from the stack to the bookcase. Nico would only
use this pointing gesture to explain the task. The participants in the room with Nico all correctly
interpreted the implied task from the pointing gesture and moved the books. When Nico then
instructed participants to do the unusual task of throwing out the books by pointing to the stack
of books and to the garbage can, many participants showed some confusion at the task, but still
complied with the request. The study demonstrated that participants could identify the robot’s
intent with only a non-verbal pointing gesture, and generally complied with a request from the
robot even if it was unusual.
12
13
Gestures could also be used to help robots display emotion. The human-like WE-4RII robot in
[38] displayed emotions using facial expressions and upper body movements. The robot
demonstrated emotions including disgust, fear, sadness, happiness, neutrality, surprise and anger.
Each emotion involved a facial expression, head orientation, and arm gestures. When each
emotion was shown to 18 participants with an average age of 21, all of the participants were able
to correctly identify the emotions of surprise, sadness, anger and disgust. The only emotion
which participants had difficulty with was fear, which was many times misidentified as disgust.
The study compared its results with a previous experiment by the same group of researchers in
which a version of the robot without hands demonstrated the same types of emotions to other
participants. The comparison showed that the version of the robot with the complete arms was
able to demonstrate its emotions more effectively to humans, with higher recognition rates across
almost every emotion.
Another example of a robot which incorporated arm gestures when displaying emotions is the
humanoid Nao robot in [39]. The robot generated the emotions of anger, fear, sadness and joy
using the methods of body movements, sounds or eye colors. Nao utilized its entire body
including its arms to display its emotions, and were modeled after analogous human body
movements. Each type of the robot’s methods to display its emotions was demonstrated to 42
participants ranging in age from 19 to 29. Subjects were then asked to identify the emotions
behind the robot’s expressional cues. The results of the study showed that participants
understood the body movements the best out of all the types of emotional cues. The recognition
rates for the emotions behind the body movements indicated that human emotional expression
with body movement was also applicable for expressions with humanoid robots.
2.4 Chapter Summary
This chapter discussed various types of cognitive training programs to help with cognitive and
functional impairments. There are two general types of cognitive training programs, including
specific programs, which target certain cognitive functions, and general programs, which provide
non-specific cognitive stimulation. One type of general training program is recreational
activities, which offer many benefits including cognitive stimulation, social engagement and
health benefits. Bingo is one such recreational activity which can be therapeutically beneficial
for older adults. In order to facilitate Bingo and other recreational activities, social robots have
13
14
been designed as potential activity facilitators. Single user and multi-user social robots have been
designed to facilitate or participate in a wide range of recreational activities. However, many of
these robots require some human intervention during their facilitation of an activity. This thesis
aims to design an autonomous robot which can facilitate a recreational cognitive training activity
without the need for a human operator.
In order to design a socially assistive autonomous robot which can facilitate the recreational
activity Bingo, the social capabilities of the platform must be considered. Two types of non-
verbal social behaviours which can be implemented on a robot are eye contact and physical
gestures. Eye contact has been demonstrated to have many important uses during communication
between humans and robots, including directing the flow of communication and providing cues
to help people guess the robot’s internal states. Gesturing is also important for maintaining the
attention of a person, explicitly directing someone’s attention during the flow of communication
or expressing emotion.
14
15
Chapter 3 Gathering Impressions and Design Considerations
Gather the opinions of end users through a focus group study can aid in the development of a
new technology. This chapter discusses the results of a focus group study done in a long-term
care facility and a retirement home regarding the adoption of Tangy as a recreational activity
facilitator for older adults. The study was done in order to inform the design of various existing
features on Tangy, as well as gain suggestions on features which may be implemented in the near
or long-term future.
3.1 Focus Group Study
Focus group studies were conducted at both a long-term care facility and a retirement home by
our team in order to collect the opinions and concerns of elderly residents, their family members
and healthcare staff about Tangy and other healthcare robots. Written informed consent was
obtained prior to commencement of the study following our Research Ethics Protocol. The focus
groups offered their perspectives on having a socially assistive robot in a long-term
care/retirement home setting, impressions of Tangy from a quick descriptive video of its
capabilities, and suggestions for potential features for Tangy’s design. In this thesis, the
responses were then transcribed and thematically analyzed in order to group common themes or
suggestions across all focus groups.
3.1.1 Description
The long-term care home had recreational programmers who organized regular social activities,
staff to perform housekeeping, laundry and meal preparation duties, and physiotherapists who
guided physical rehabilitation for residents. The long-term care healthcare staff also aided
residents with their activities of daily living, including toileting, eating, dressing, mobility, and
bathing. The retirement home provided residents with housekeeping, laundry and meal
preparation services, as well as regular recreational activities.
There were a total of 16 sessions, each with an average of five participants and lasting
approximately 45 minutes. Each session included exclusively residents, residents’ family
15
16
members, or healthcare workers. Trained moderators from our research group led discussions
according to the set of structured questions listed in Table 1.
Table 1: Focus Group Questions [40]
Focus Group Session Questions 1a) How do you feel about having Tangy assisting residents living at the LTC home? 1b) How do you feel about having Tangy assisting residents living at the retirement home? 2a) What activities do you think Tangy can assist with around the LTC home? 2b) What activities do you think Tangy can assist with around the retirement home? 3a) What specific daily tasks can Tangy personally help you with? (residents) 3b) What specific daily tasks can Tangy help the residents with? (staff/family) 4) How do you feel about having Tangy organize and run Bingo games? 5a) How do you feel about using Tangy to schedule and make video calls to your family and friends? With your doctor? (residents) 5b) How do you feel about using Tangy to schedule and make video calls between residents and their family and friends? With a residents’ doctor? (staff/family) 6a) What features do you think are important in the design of Tangy to make the robot acceptable at the LTC home? 6b) What features do you think are important in the design of Tangy to make the robot acceptable at the retirement home? 7a) What do you see as advantages and disadvantages of having a health-care robot at the LTC home? 7b) What do you see as advantages and disadvantages of having a health-care robot at the retirement home? 8) Do you have any final comments about the robot?
After question 3b) as listed in Table 1 was asked, the moderators showed the participants a
descriptive video of Tangy which depicted it facilitating a Bingo game, and then hosting a
telepresence activity. In the first part of the video, Tangy greeted Bingo players, called out
numbers, and navigated to assist a player with correctly marking his card when he raised his
hand. A segment with Tangy detecting a winning card and celebrating was shown as well. In the
second part of the video, Tangy navigates to an older adult and displays an incoming video call
on its tablet. The user answers the call, and begins having a conversation with the incoming
caller.
16
17
3.1.2 Participant Demographics
Participant information was self-reported using a questionnaire, developed by members of our
research team [40], which asked for gender, age, experience with computers and experience with
robots. Forty residents from the long-term care facility and the retirement facility participated in
the focus group sessions. Residents were only included in the study if they were above the age of
60, spoke fluent English, had good auditory acuity, and had a Cognitive Performance Scale
Score [40] under 3 (indicating mild to no cognitive impairments). Ten family members of
residents participated in the study. Thirty-one healthcare staff from the two facilities (holding
positions including personal service workers, nurses, staff educators, program managers, and
private caregivers) with an average age of 38 participated in the focus groups. Varying levels of
experience with computers were seen across all groups; however, no participants indicated any
experience with robots other than having seen them in museums, science centers, or television
shows. The demographics and experience levels of all participants are displayed in Table 2 and
Table 3, respectively.
Table 2: Participant Demographics
Demographic Average Age (Std Dev.) Female Male
Residents 82.4 (10.3) 33 7 Family Members 66.1 (9.9) 7 3 Healthcare Professionals 38.2 (12.8) 28 3
Table 3: Computer and Robot Experience of Focus Group Participants
Demographic Computer Experience* Robot Experience**
None Beginner Intermediate Advanced None Beginner Intermediate Advanced
Residents 20 13 4 3 33 7 0 0 Family Members 3 1 3 3 8 2 0 0
Healthcare Professionals 2 7 8 14 16 15 0 0
*Beginner (email, use simple programs) Intermediate (internet, chat) Advanced (editing documents, use complex programs) ** Beginner (seen robots at museums/science centers or stores, or have watched shows with robots) Intermediate (have worked with/used commercial robots) Advanced (have worked on robot developmental aspects including hardware/ software design)
17
18
3.1.3 Thematic Analysis
The focus group sessions were audiotaped and later transcribed. The transcriptions were
analyzed with the goal of drawing out thematic sets involving common attitudes, suggestions or
perspectives about the robot and the robot’s role in the long-term care setting. The analysis used
common techniques for decoding focus group results in qualitative research. Participants’
comments were categorized into general impressions, concerns and suggestions about healthcare
robots, or specific impressions, concerns and suggestions for Tangy. Each of these six categories
were then compiled together and qualitatively inspected for similar themes and suggestions. The
thematic sets were then used to guide Tangy’s design.
3.2 Thematic Sets
3.2.1 Bingo
Residents from both facilities indicated that although Bingo was offered regularly at both
facilities, none of them present in the focus groups actually played Bingo. They indicated that the
activity was not particularly interesting to them; however, the residents almost unanimously
stated that they would give Bingo a try if Tangy was there to facilitate the game. Residents also
believed that their friends or neighbours in the facilities would also enjoy participating in the
game with the robot. One elderly participant explained that she would play at least once “from a
curiosity point of view.”
Healthcare staff members thought the robot brought novelty to the Bingo activity, which would
help attract residents to play the game. Some brought up the potential benefits by having a robot
facilitate recreational activities like this, such as lessening the program staff’s workload so that
they have time to create personalized interventions for residents. After showing the descriptive
video of Tangy facilitating a Bingo game for our research team, staff members suggested the use
of alternate methods of interaction between residents and the robot in order to encourage
accessibility, such as the use of leg movements or voice commands. They indicated that the
method of requesting assistance from Tangy demonstrated in the video where participants raised
their hands to get Tangy to come over and help them would present an accessibility barrier to
residents who had impairments with their arms.
18
19
3.2.2 Telepresence
The video demonstration of the telepresence activity was extremely well-received by participants
across all demographics and sessions. Residents were very supportive of the activity as it would
help facilitate communication between them and their family members. They indicated that the
activity gave them freedom so that they wouldn’t have to wait for infrequent visits from their
children and grandchildren. Elderly residents from both facilities gave positive feedback on this
feature due to its ability to facilitate more social interactions between them and their family.
Healthcare staff members indicated that the activity would be very useful and attractive to
residents who had family members who didn’t live close by to the facilities. Staff members also
suggested that the button on the robot’s screen to receive a call be made larger to accommodate
residents with poor eyesight or fine-motor control.
Family members were also largely supportive of the feature and gave similar motives as the
residents. They cited the difficulties in arranging visit times, large distances between them and
the facilities, and their busy schedules as barriers to communication, and noted how the
telepresence activity could overcome these barriers. One family member stated: “I definitely
would use it a lot. I would be using it almost every day.”
3.2.3 Tangy’s Appearance
Opinions about the robot’s appearance varied highly among participant demographics. Elderly
residents in both the long-term care facility and the retirement home were accepting of the
robot’s appearance. Several residents supported the idea that any appearance for Tangy was fine
as long as its physical design was functionality-driven rather than aesthetically driven; one
resident in particular described: “If it looks like a machine, that's fine. I think the most important
thing is what it does.” Several other residents mentioned that Tangy’s machine-like look would
even be more preferable to having the robot look more humanoid. One resident gave her opinion:
“…it could look like a machine, you know…But the idea of having it look like a human being,
which is purposefully done for sure, is wrong, I think.”
Staff participants in both facilities believed that the robot would be more accepted among the
residents if its physical appearance didn’t have as many overtly mechanical features. One staff
member stated that Tangy’s exterior had “too much metal”, and suggested that the robot should
19
20
be made to look “…a little friendlier. If it was furry, or if it looked like an animal or something.
Less machine-like somehow.” One common suggestion among healthcare staff across both the
long-term care facility and the retirement home was to have the robot dressed in everyday
clothing in order to cover up some of its gearing and wiring.
Family members brought up Tangy’s appearance, but were divided on their views about it. Many
participants agreed that the robot’s appearance was agreeable, with some describing: “…he’s got
a very nice expression…”, and they especially liked that Tangy had “…wide, open curious eyes,
and his mouth doesn’t turn down…” However, other family member participants believed the
robot needed to be more human-like in order for residents to be more accepting of it.
3.2.4 Acceptance of the Robot
Some variation of the concern that older adults would not be able to accept the robot because
they had very little experience with similar technologies was brought up in most of the focus
group sessions. Before viewing the descriptive video of Tangy facilitating the Bingo and
telepresence activities, several resident participants indicated that the idea of having a robot in
their facilities scared them; one described: “When you're not used to something, it can be scary.
Certainly at our age, I think.” Nonetheless, several residents gave clear sign that they would be
open to the idea of learning how to use and adopting Tangy in their facilities if its benefits were
clear. One participant stated: “I would say, it'd be non-productive on anybody's part to say they
weren't prepared to look at what the next stage might be…” However, another participant stated
that any desire to adopt a robot may still be tempered by their ability to learn how to use the
technology: “… a learning curve at our age is, generally speaking, not as good as a younger age
like yourselves who have been brought up with all these technologies. That would be another
reaction that I would have, [why] I might be a little bit more resistant to it…”
After being introduced to the robot’s capabilities through the demonstration video and the
moderator’s explanations, the residents were more amenable to having Tangy in the facilities. A
participant who had expressed unease with the robot explained her shift in opinion: “Anything
new, at our age, I tend to look at very warily, but now I feel a great deal more warmth towards a
robot than I ever did before, simply because I've seen it, I understand it, you've explained what
20
21
it's all about.” Similarly, several participants believed that clear explanation of the robot’s
functionality and goal would help alleviate fears about the technology among other residents.
Healthcare staff members brought up one concern about how residents who had cognitive
impairments would react to the robot. They expressed concerns about how moderate to severely
impaired residents may react very aggressively out of fear to something they do not understand.
One participant stated: “[Because] the ones who are stuck in the fifties in their mind, robots were
things way into the future…”, they would not be able to understand the robot or its role in the
facilities.
Similar to the residents and the staff members, family members were unsure whether residents
would be able to understand the purpose or the functionality of the robot.
3.2.5 Ease of Use
An important aspect of the robot’s design among participants was the ease of use of the robot.
Residents required that the technology be easy for them to adopt; one participant expressed: “[the
robot]’s good, as long as it’s easy [to use]…” Moreover, participants pointed out that Tangy’s
benefits would only be relevant if it worked reliably—otherwise, “If it doesn't work, it's a pain.”
Healthcare staff members explained that the usability of the robot would be critical in preventing
feelings of frustration in residents when using it. A staff member pointed out that not only
Tangy’s user interface for residents needed to be clear and intuitive, but also the robot’s expected
behaviours as well: “it will be extremely frustrating to the residents [because] if, here’s
something, and it looks like it’s a machine to help make something easier, and they expect it to
be competent, but it’s not, [then] it’s worse than human beings failing you.” Besides the ease of
use for residents, staff members also wondered about the ease of use for themselves as well:
“What would be the ease of use? ...Would it be really difficult to turn him on and set him up?”
Family members brought up the residents’ lack of knowledge about modern technology again to
highlight the need for the robot to be easy to use.
21
22
3.3 Suggested Features/Activities
3.3.1 Reminders/Prompting
Residents and staff members suggested Tangy be able to give reminders to them and prompt
them about their schedules. Resident participants acknowledged that their memories were being
affected by their age, and gave examples of how they were beginning to forget items on their
schedules. One resident gave an anecdote: “Reminders would be very helpful…I forgot I was
going to get a cup of coffee with a former student yesterday afternoon …I can't imagine doing
that, it's crazy that I should have forgotten that.” Other items suggested for Tangy to remind
residents about included taking medication, responding to missed calls, and going to scheduled
activities.
Healthcare staff members gave similar recommendations for the reminders, and included
prompting during meals and when transporting large groups of residents as a possible feature for
Tangy. Staff participants explained that residents sometimes had difficulties staying focused
during mealtimes or when moving in large groups, and sometimes wandered off. In those
situations, staff members may be focused on another resident or leading a group, so having a
robot to help keeping residents focused would be very helpful.
3.3.2 Simple Interactions with Cognitively Impaired Residents
Healthcare staff and family members believed that having Tangy engage with cognitively
impaired residents would greatly help with their quality of life. Staff participants explained that
they attempted to interact with all residents regularly but sometimes spend less time engaging
with them if they have other critical duties. For residents with cognitive impairments, the social
engagement is simple; any conversation is enough. Participants suggested that Tangy hold
simple conversations with these residents if they were not able to interact with them that day.
Family members similarly pushed for this feature for residents as they believed the social
interaction would stimulate them and help stave off potential loneliness. One participant
described: “They do need that. It stimulates them, they don’t feel so alone, and just sitting there,
no communication, and [the robot] can go around and just talk to them.” Family members
suggested that the conversation need not be complex, as the residents only desired something to
listen to them.
22
23
3.3.3 Other Recreational Activities
As the resident participants in the focus group study were not regular Bingo players, they
suggested a variety of other group recreation activities which were popular at the facilities.
Participants strongly enjoyed trivia and trivia-like games with focuses on history, geography and
grammar. One participant mentioned the benefits of mentally-intensive games like trivia: “you're
asking people to try and recall information, which is another good mental activity. [Which is]
what seniors need. A lot of mental activity.”
3.3.4 Physical Tasks and Aiding Residents’ Autonomy
Having Tangy perform physical tasks in order to help residents and staff members around the
facility was a popular topic of discussion among residents. Resident participants described a
variety of small errands which they or other residents they knew had troubles with: carrying
objects, pushing their wheelchairs around, or helping them out of bed. The only option currently
for residents who needed physical assistance is to call and wait for a staff member or a private
social worker at the facilities. Participants indicated they would rather have a robot help them
rather than wait for a human. One resident also gave another reason that he wanted Tangy to
provide physical assistance: “I always found embarrassing, since I wasn’t in rehabilitation or
anything…There was a time when I couldn't get out of bed by myself. And all I would call the
nurse for was to come and help me out of my bed so I could go to the washroom. Well, that [is
why] I could call a robot, and…that would be acceptable to me.”
3.3.5 Music Therapy
Throughout the study, participants recommended that Tangy incorporate music as a feature.
Residents brought up karaoke and sing-alongs as popular pastimes at the facilities, and suggested
that Tangy be involved in these activities. Staff members mentioned the therapeutic impact of
music, and believed that residents would strongly enjoy having Tangy play music to them.
3.3.6 Multilingual Support
As the long-term care facility and the retirement home had diverse ethnic resident populations,
staff members and family members brought up multilingual support as a feature on Tangy. They
suggested that the robot be able to communicate in other languages both vocally and through text
on its screen in order to connect with residents whose may not be fluent in English. Staff also
23
24
desired Tangy to be able to act as an interpreter between them and residents if there were any
communication gaps.
3.4 Chapter Summary
This chapter investigated the results from a focus group study conducted with three
demographics at a long-term care facility and a retirement home. Residents, family members and
healthcare staff were asked about their general and specific opinions about social robots for the
elderly and Tangy. In general, the three groups of participants believed that Tangy could be
useful in the facilities if it was able to assist with the tasks that were demonstrated in the video
during the study. Participants strongly enjoyed the aspects of the demonstrated activities which
would help facilitate human-human interaction. With respect to Bingo, participants believed that
Tangy would bring an aspect of novelty to the game which would attract more residents to the
weekly games. Healthcare staff in particular identified that the demonstrated method of
requesting assistance from the robot during the game was unfeasible for residents with physical
impairments to their upper body. Participants also brought up the appearance and ease of use of
the robot as important factors to consider when designing the robot to be accepted by the
residents. In particular, all three groups indicated that the robot should have an intuitive interface
and be easy to use, both for residents and the healthcare staff. Finally, participants presented a
variety of suggestions for future features for Tangy, including reminders, prompting, simple
conversations with residents, other recreational activities, physical aid, music therapy and
multilingual support.
24
25
Chapter 4 Robot Behaviours
Tangy’s behaviours are based on the objective of facilitating a Bingo game without the need for
any intervention by a human operator. The robot’s behaviours were designed in order to
encapsulate all of the duties of the human Bingo facilitators at the long-term care and retirement
home facilities in which user studies were planned. From repeated observations of Bingo games
at both facilities, the Bingo facilitator was responsible not only for calling out Bingo numbers,
but also engaging residents with jokes and conversation, assisting players with mistakes, and
celebrating with players when they had a winning card. Tangy’s behaviours imitate the actions of
the human Bingo facilitator by also engaging in social interactions with players by giving jokes
during the game, assisting players one-on-one when they request aid, and performing celebration
dances with players who have winning cards. The robot was developed with five major types of
behaviours corresponding to the various stages of the Bingo game: i) initiating the Bingo
activity; ii) facilitating the game for all players; iii) transitioning from multi-user to single user
behaviours; iv) assisting a single player with his/her card; and v) ending the Bingo activity. In
order to determine which behaviour to implement, Tangy uses a finite state machine. The finite
state machine outputs the appropriate behaviour for the robot given particular trigger events
which change the state of the world, robot, or the players’ cards.
4.1 The Bingo Scenario
A Bingo session begins with Tangy located at the front of the room facing four players seated at
a long table. An hour-long session comprises two or three Bingo games, depending on the
lengths of the games. In front of each player are a Bingo card, markers, and an assistance request
device. Tangy initiates the Bingo session by greeting the residents with a wave and introducing
itself. The robot then explains the rules of the game before beginning to call out the Bingo
numbers. As the robot calls out numbers, players place markers on their card corresponding to
these numbers. Players can request Tangy to come to view their card at any time using their
assistance request device. Tangy is able to offer assistance with incorrectly marked or missing
markers on players’ cards, or award players with victory if it identifies a winning Bingo card.
Tangy concludes calling out numbers when a Bingo game is finished, which occurs either when
one or more players have winning cards, or when all numbers have been called. Depending on
25
26
the game mode, a winning configuration has to have a horizontal, vertical or diagonal line of five
markers on the Bingo board, a fully marked card, or two diagonal lines of five markers.
4.2 System Architecture
Tangy was designed using a multi-layered system architecture, (Figure 2) in order to
autonomously facilitate Bingo games for groups of elderly residents. The architecture consists of
four major layers: the sensory layer, which takes in inputs from the environment; the input
interpretation layer, which transforms the sensory inputs into world, game and robot state
variables; the behaviour determination layer, which uses the state variables in order to determine
the robot’s next actions; and the actuation layer, which transforms the determined behaviours
into outputs that are visible in the environment. Each layer is separated into a modular structure
which helps to organize the different sensory input types.
Figure 2: Tangy's System Architecture
This thesis focuses on the development of the Behaviour Determination layer and the eye
contact, arm control and verbal interactions modules in the Actuation layer. The Behaviour
Determination layer is discussed in the following subsection 4.4 of this chapter. The
development of the actuation layer is also covered in this thesis in the next chapter. In particular,
Chapter 5 discusses the actuation modules of speech, eye contact and arm control.
Behaviour Determination
Actuation State Interpreter Sensing
Face Tracking
Assistance Identification
Bingo Card Detection
Assistive Behaviour Deliberation
Navigation System
Localization
Off-Board IR Sensor
Voice, Screen and
Motors
Laser Range Finder
2D Cameras
Hardware Controllers
26
27
4.2.1 Sensory Capabilities
The sensory layer in the robot architecture employs an array of onboard sensors on Tangy as well
as an off-board sensor in the Bingo activity environment, displayed in Figure 3. An ASUS Xtion
Pro IR sensor is mounted on a stand behind Tangy during a Bingo game in order to capture any
assistance requests from residents. Two sensors help Tangy localize and navigate itself within
the environment: a URG-04LX-UG01 laser range finder captures live data about obstacles in
close proximity to the robot, while optical encoders on Tangy’s wheels continuously record the
distance travelled by each wheel. Tangy’s right eye is a 2D Axis M1031-W camera, and provides
video of users when the robot is assisting them in order to help Tangy track and follow their
faces. Finally, Tangy uses a 2D Logitech Pro C920 webcam mounted on its forehead to provide
high resolution imaging such that it can detect the Bingo card and the card state.
Figure 3: Off-board and On-board Sensors
4.3 Robot States During the Bingo Game
There are five overarching robot states during the Bingo game. The pre-game state occurs before
the game begins, when Tangy gives an introductory speech to the players. The multi-user state
then occurs when Tangy is facilitating Bingo for multiple users at the front of the room. The
transition state defines when Tangy begins to transition in response to an assistance request. The
2D Logitech Pro C920 Camera
2D Axis M1031-W Camera
URG-04LX-UG01 Laser Range Finder
ASUS Xtion Pro IR Sensor
Assistance Request Device
Bingo Markers Bingo Card
27
28
single user state occurs when Tangy is in front of a single user and in the process of assisting
him/her. The final post-game state determines whether the robot concludes the game with a
speech, or continues with a new game. Each state contains a set of pre-determined behaviours,
described in the following sub-sections.
4.3.1 Pre-Game State
The pre-game state is defined before a Bingo game begins, when Tangy gives appropriate
information to players depending on whether this is the first game of the session or a subsequent
game. During the pre-game state, Tangy does not take in any sensory inputs. The behaviours
which Tangy may perform are listed in Table 4.
Table 4: Pre-Game Behaviours
Behaviour Description
Robot Physical Action
Verbal Communication Text on Screen
Introduction Wave “Hi!” “Hi!”
N/A “My name is Tangy! I am so excited to play bingo with you today! It is one of my favorite games.”
“My name is Tangy!”
Instructions Point to screen “I will call a series of numbers, and also display them on my screen. If your card has the same number I called, please mark the number with a red marker.”
“How to play Bingo”
N/A “If you need help with your Bingo card. Or if you think you have Bingo, please press the button on your table.”
“Press button for help”
Starting another game after first
N/A “Let’s play another game of Bingo!”
“Let’s play another game of Bingo!”
N/A “Please clear your cards now.” “Please clear your cards now.”
28
29
4.3.2 Multi-User State
Tangy facilitates the Bingo game for multiple players by calling out numbers and telling jokes
when no assistance requests have been currently detected. The robot selects random numbers
from a vector of strings containing all of the possible Bingo numbers. After a number has been
selected, it is written to a text file for tracking progress while the corresponding string in the
vector is removed. The progress file is used in situations where the robot is unable to continue
playing a game because of a program or server crash. In these situations, a human operator can
restart the Bingo game from the point of failure without having to clear the players’ cards or
restart the introduction speech.
During calling out numbers, Tangy swivels its head to randomly look at pre-set positions where
players would likely be during the game. The head movement allows Tangy to simulate paying
attention to the players from the front of the room by looking at them. The head movement also
provides a more dynamic feel to the robot during the multi-user scenario, when it is mostly
stationary at the front of the room.
Tangy occasionally tells jokes in between calling out numbers. The two-line jokes are stored in a
text file, which the robot reads before beginning the game. After every seventh number is called
out, the robot chooses a random joke from the vector of jokes populated from the original joke
database. After a joke has been told, it is removed from the vector of remaining jokes so as to not
be chosen again. After telling a joke, the robot laughs and brings its hand up to its mouth in a
laugh gesture.
During the multi-user scenario, Tangy constantly scans the environment in order to detect any
assistance requests. The behaviours which Tangy may perform during this state are listed in
Table 5.
29
30
Table 5: Multi-User State Behaviours
Behaviour Description
Robot Physical Action
Verbal Communication Text on Screen
Calling out numbers
Point to screen (every other number)
“The next number is:” “The next number is:”
Head swivel
Bingo Number [Tangy calls out a Bingo number]
Bingo Number [The Bingo number is displayed on the screen]
N/A “One more time” Bingo Number
Head swivel Bingo Number Bingo Number
Telling a joke
N/A Joke set-up (e.g. “What's the best way to carve wood?”)
Joke set-up
N/A Joke punchline (e.g. “Whittle by whittle.”)
Joke punchline
Laugh gesture “Hee hee hee.” Joke punchline
4.3.3 Transition State
Upon detecting an assistance request, Tangy will begin to transition from facilitating the Bingo
game for multiple users to assisting a single user. When an assistance request is detected, the
robot will immediately turn its head towards the location of the request and nod towards the user
in order to acknowledge the request. Tangy will then finish calling out the current number or
state the joke. Subsequently, it will inform the other users that it will help someone and pause the
calling of numbers in order to navigate to the player in need.
30
31
Table 6: Transition Behaviours
Behaviour Description
Robot Physical Action
Verbal Communication
Text on Screen
Acknowledgement of help request
Nod towards location of assistance request.
N/A N/A
Interrupting the game N/A “Please give me a second to help someone.”
"Coming to Help..."
Navigating to the location of the assistance request
Move from front of the room to the assistance request location
N/A "Coming to Help..."
4.3.4 Single User State
Tangy will assist a player, one-on-one if he/she request assistance. If multiple users request
assistance at the same time, or one or more users request assistance when Tangy is currently
helping a user, Tangy will help them in the order that it detects their help requests.
During an assistance sequence with one player, Tangy may move its head in order to look at the
person’s face to establish eye contact. Tangy will attempt to maintain eye contact anytime while
it is interacting with the player. The robot will discontinue eye contact when it is attempting to
view the player’s card.
Tangy identifies any mistakes or winning combinations on a player’s card by viewing it with a
webcam located on the robot’s head. The robot will attempt to look at the card by bowing its
head so its face is angled downwards towards the table in front of it. If it is not able to identify a
card in its field of view, Tangy changes the angle at which its face is bowed two times in order to
capture a field of view closer and further away. The adjustment of its head position is done in
order to ensure that Tangy captures a full range of the view in front of it, so as to not miss seeing
the card.
The list of behaviours Tangy may perform in its single user state is listed in Table 7. How the
robot determines which assistive behaviour to execute is discussed in Section 4.4.
31
32
Table 7: Single user Behaviours
Behaviour Description
Robot Physical Action
Verbal Communication Text on Screen
Greet player Establish eye contact
“Let's check your card to see how you are doing.” OR “Let's take a look at your card.”
“Checking Bingo Card”
Detect card state Bow robot head to look at card (adjust if unsuccessful)
N/A “Checking Bingo Card”
Prompt player to move card into robot’s field of view
Establish eye contact
“I'm sorry. I cannot read your card. Please move it slightly closer and make sure nothing is blocking it.”
“Please Move Card”
Prompt player to fix incorrectly placed markers
Establish eye contact AND Point to screen
“Oops! You have some misplaced markers. Please remove these markers from the following numbers on your card.” OR “Oh jeepers, I think you have some misplaced pieces. Please see if you can remove these markers.” AND List of missing Bingo numbers
“Please place markers on:” AND List of missing Bingo numbers
Prompt player to mark missing numbers
Establish eye contact AND Point to screen
“Oh no! You have some missing markers. Please place markers on the following numbers on your card.” OR “I think you have some missing numbers. Can you place markers on the following spots?” AND List of missing Bingo numbers
“Please place markers on:" AND List of missing Bingo numbers
32
33
Praise player and then encourage him/her
Establish eye contact
“You're doing amazing!” OR “You must have played this before!” OR “Your card looks great!” OR “Wow, you are close to getting Bingo!”
“GREAT JOB!”
N/A “Keep up the good work!” OR “You're almost there!” OR “Just a few more numbers before you're there!”
“GREAT JOB!”
Congratulate player on a winning card
Establish eye contact AND Do celebration dance
“Congratulations! You have won Bingo!”
"BINGO!!!!!!!!"
4.3.5 Post-Game State
The post-game state is similar to the pre-game state, where Tangy takes in no sensory inputs and
states a few farewell statements. Tangy’s possible behaviours during the post-game state are
given in Table 8.
Table 8: Post-Game Behaviour
Behaviour Description
Robot Physical Action
Verbal Communication Text on Screen
Give farewell
N/A “Well that's it! Wasn't that fun? Thank you very much for playing Bingo with me. I had so much fun!”
“Thank You!”
Wave “Goodbye!” “Goodbye!”
4.4 Behaviour Determination
Robot behaviours are determined using the robot, card, and world states. The robot states govern
the flow of the Bingo game and are discussed in Section 4.2. The card states are only used when
33
34
Tangy is assisting a single user with his/her card, and determine the specific assistive behaviour
discussed in Section 4.3.4. The world state determines when Tangy enters the transition state
described in Section 4.3.3, as well as how the robot navigates through the environment. The
various states and their possible values are listed in Table 9.
Table 9: Behaviour-Determining States
Type of State Value
Robot State
Pre-Game State
Multi-User State
Transition State
Single user State
Post-Game State
Card State
Occluded
Correctly marked
Missing number(s)
Incorrectly marked number(s)
Winning combination
World State
Location of assistance requests in environment
Obstacles in the environment
Location of robot in the environment
Tangy, as the game facilitator, utilizes a finite state machine (FSM) in order to choose which
behaviours to implement. The FSM was developed so that it was structured around the flow of a
typical Bingo game. Namely, the robot chooses and executes its behaviours based on the
progression of the game. The progression of a Bingo game requires the following primary events
to occur: i) the game begins; and ii) Bingo numbers are consecutively presented to players.
Secondary, player-dependent events may also occur if: i) a player indicates he/she requires
assistance during the game; or ii) a player claims to have a winning card and informs the game
facilitator. With respect to the primary Bingo events, the Bingo facilitator is responsible for
initiating the game and calling out numbers. With the secondary events, the facilitator is
responsible for aiding a player with questions about the game or about his/her card, and either
34
35
accepting or declining a claim for a winning card. Any other responsibilities, such as social
interactions between the facilitator and players, or post-game actions are supplementary to the
facilitator’s core behaviours.
The FSM implements Tangy’s behaviours required for the primary events before all others.
Namely, the robot executes the pre-game and multi-user state behaviours described in sections
4.2.1 and 4.2.2. The FSM only implements the secondary behaviours if there is a world state
change where an assistance request has been located in the environment. These behaviours
include the transition and single user state behaviours described in sections 4.2.3 and 4.2.4. The
post-game behaviour in section 4.2.5 is supplementary, and occurs after the primary and
secondary behaviours have been executed.
Tangy’s FSM is presented in Figure 4, in which the circles represent the individual behaviours
and the arrows represent the behaviour transitions which would trigger the consequent behaviour
to execute. Each behaviour is color-coded in order to categorize them based on the robot state
that the behaviour belongs in.
35
36
Figure 4: Tangy’s FSM – Pre- and post-game states are in light blue; multi-user state behaviours are in green; transition actions are in grey; and the individualized help behaviours are in pink.
Every Bingo session begins with Tangy in the pre-game state. Upon completion of its
introduction and instruction behaviours, Tangy will then begin to facilitate the game in its multi-
user state.
When facilitating the game for multiple users, Tangy will continuously execute its calling out
Bingo number and telling joke behaviours. The joke behaviour is executed after every seven
Bingo numbers have been called. The robot will only progress from these behaviours upon a
detection of an assistance request in the environment.
36
37
After detecting an assisting request, Tangy will perform an acknowledgement behaviour
immediately, and navigate to the location of the request only after it finishes calling out the
current number.
Upon reaching the first player who requested assistance, the robot will then immediately perform
its single user greeting behaviour, and then progress to attempt card detection. If the occluded
card state occurs, then the robot will prompt the player to move the card into its field of view.
Otherwise, if the incorrectly marked card state occurs, the robot will prompt the player to fix the
incorrectly placed markers. Then, if the missing markers card state occurs, the robot will prompt
the player to place markers on the missing spots. The robot will execute these three behaviours
repeatedly until the card state becomes correctly marked. If at this point the card does not contain
a winning combination, Tangy executes its praise and encouragement behaviour. If the card does
contain a winning combination, Tangy executes its celebration behaviour.
After assisting one person, Tangy will re-enter the transition state until all assistance requests
have been fulfilled. Should a winning card not been identified after every player has been helped,
Tangy will re-enter the multi-user state and begin calling out numbers again. Otherwise, Tangy
will enter its post-game state and begin giving its farewell behaviour.
Figure 5: Tangy Behaviours during Bingo Game: (a) Greeting and Introduction at the beginning of a game; (b) Calling out and pointing to Bingo numbers displayed on its screen; (c) Navigating to player; (d) Giving corrective assistance aurally and visually through its screen; (e) Celebrating a winning card by dancing with its arms swaying from side to side in the air; (f) Saying and waving goodbye at the end of a session.
37
38
4.5 Chapter Summary
This chapter presented the development of the robot behaviours and behaviour determination
technique for Tangy to autonomously facilitate the Bingo scenario. The robot behaviours were
designed to imitate the behaviours of a human Bingo facilitator, and encapsulate both the multi-
user and single user interactions seen in a typical Bingo game in a long-term care facility and
retirement home. Tangy’s behaviours fell within five major categories: i) pre-game greeting and
instructions; ii) multi-user interactions; iii) transition actions; iv) single user assistive behaviours;
and v) post-game farewell. For each behaviour, the robot performs some combination of a
physical action, a verbal auditory statement, and a visual display of a text statement on its tablet
screen. Tangy determines its behaviours using an FSM. The FSM outputs a behaviour based on
triggering events which occur when one of the world, robot or card states change.
38
39
Chapter 5 Robot Actuation Modules
The following sections discuss the development of the arm control, eye contact and audio
interactions modules.
5.1 Arm Control Module
Tangy displays body language primarily through arm gestures and head movements. The arm
gestures currently implemented include waving to players, pointing to the robot’s onboard
screen, covering the robot’s face when laughing, and swaying its arms above its head when
celebrating a winning card. Tangy’s gestures are used to draw players’ attention to important
information on its screen, provide natural non-verbal communication common in social
interactions, and amuse players.
5.1.1 Motion Planning for the Arms
Tangy implements ROS MoveIt! [42] in order to create motion trajectories for the arm gestures.
Motion plans consist of a set of joint angles for a set kinematic chain over a trajectory from an
initial position to a final position. MoveIt! generates point-to-point motion plans for kinematic
chains dynamically using motion planners in the Open Motion Planning Library [43]. The real-
time generation of the plans provides unique trajectories for the arms, in order to have Tangy’s
arm gestures seem less mechanical and machine-like. Moreover, dynamically planning the
gestures allows for the option of generating obstacle-avoiding trajectories.
In order to generate plans for Tangy’s arm gestures, a 3D assembly of the robot was first created
in the solid modelling computer aided design (CAD) software SolidWorks in order to establish
the kinematic chains for the planner. The models of the arm linkages were created according to
the measured dimensions of each link in Tangy’s arms. The robot’s upper torso, and the tablet on
the front of the robot, was also modeled using the dimensions of the actual robot torso and tablet,
as it was important to properly define the workspace of the robot’s arms. The joints of the
modelled robot were constrained to rotate about the same axes as the analogous joints on the
physical robot, including the pitch and roll capabilities of the 2 DOF neck, the pitch, yaw and roll
DOF shoulder, the pitch DOF elbow, and the pitch and roll capabilities of the 2 DOF wrist. A
picture of the model is displayed in Figure 6.
39
40
Figure 6: CAD Model of Tangy and Kinematic Model of Tangy’s Arm The CAD model of the robot was imported into the MoveIt!-specific Unified Robot Description
Format (URDF) using the SolidWorks to URDF exporter plugin for SolidWorks, which
automatically generated a series of physical properties for all of the parts of the modelled robot.
The URDF file defines the robot model in a large XML format file, with linkage dimensions,
physical and collision properties, and joint movement limitations defined through pre-set fields.
The URDF file is also used to set up key configuration files for Tangy in MoveIt!, such as the list
of the kinematic chains on the robot available to the planners, the initial poses for the kinematic
chains, and the collision matrices of each of the robot’s parts which is calculated from their
dimensions and physical properties.
The arm motion plans are generated using a serial kinematic chain, a starting pose, and an end
pose as initial inputs. A set of ten plans is always generated in tandem for each gesture, from
which an optimal plan with the smallest time duration and the smallest battery discharge amount
is chosen. The optimal criteria were chosen to provide the most responsive gesture while
preserving the robot’s battery levels. The optimizer calculates time duration based on timestamps
native to each point in motion plans, and energy consumption based on the Reverse Newton-
Euler Algorithm [44] used in inverse dynamics for kinematic chains.
40
41
5.1.2 Optimization Parameters
The two parameters of the motion plans chosen to be optimized were time and energy
consumption of the implemented plans. Motion plans with shorter time durations have fewer
points within the plans, which generally result in fewer inefficient movements through the
gesture. The time parameter for a motion plan for one of Tangy’s arm movements is defined to
be the amount of time the robot takes to move from an initial arm pose to a final arm pose. The
duration of a plan is calculated using the duration field specified in each point in an arm motion
plan class object. The total duration of the plan is equal to the sum of the durations of each point
in the motion plan.
𝐷𝐷𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = � 𝐷𝐷𝑝𝑝𝑝𝑝
# 𝑃𝑃𝑝𝑝𝑃𝑃.𝑖𝑖𝑝𝑝 𝑃𝑃𝑝𝑝𝑝𝑝𝑝𝑝
𝑖𝑖=0
(1)
The estimated energy consumption was chosen to be optimized in order to minimize battery
usage and increase the amount of Bingo games the robot can facilitate one session. As the
robot’s primary power drains include arm movements, head movements, navigation with the
mobile base, and idling all of the servos and sensors, reducing the amount of energy required for
performing the gestures during a Bingo game could result in measurable differences in runtime
on a single charge of the battery.
The optimization approach determines the energy scores of a motion plan by first solving the
inverse dynamics problem for the kinematic chain for each point to point movement in the plan.
The Recursive Newton-Euler Algorithm is used to solve for the torques at the joints. Using
forward recursion from the first link (shown in Figure 6 in the first coordinate frame of the
kinematic model) in the robot’s arm to the last, the angular velocity ωi, the angular αi and linear
accelerations ai of each link i can be obtained, assuming that the base link does not move and
initial linear and angular velocities and accelerations are 0:
41
42
𝜔𝜔𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝜔𝜔𝑖𝑖−1 + 𝑧𝑧𝑖𝑖 ∙ 𝜃𝜃�̇�𝚤
(2)
𝛼𝛼𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝛼𝛼𝑖𝑖−1 + 𝑧𝑧𝑖𝑖 ∙ �̈�𝜃𝑖𝑖 + 𝜔𝜔 × 𝑧𝑧𝑖𝑖 ∙ 𝜃𝜃�̇�𝚤 (3)
𝑎𝑎𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝑎𝑎𝑖𝑖−1 + �̇�𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1 + 𝜔𝜔𝑖𝑖 × (𝜔𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1) (4)
𝑎𝑎𝑐𝑐𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝑎𝑎𝑖𝑖−1 + �̇�𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1,𝑐𝑐𝑖𝑖 + 𝜔𝜔𝑖𝑖 × (𝜔𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1,𝑐𝑐𝑖𝑖) (5)
Backwards recursion can then be applied in order to find the forces and torques on the joints of
the robot. Using the results from equations (2) to (5), the forces fi and torques τi can be calculated
working from the last link to the first:
𝑓𝑓𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖+1𝑖𝑖 ∙ 𝑓𝑓𝑖𝑖+1 + 𝑚𝑚𝑖𝑖 ∙ (𝑎𝑎𝑐𝑐𝑖𝑖 − 𝑹𝑹𝑇𝑇 ∙ 𝑔𝑔)𝒊𝒊
0 (6)
𝜏𝜏𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖+1𝑖𝑖 ∙ 𝜏𝜏𝑖𝑖+1 − 𝑓𝑓𝑖𝑖 × 𝑟𝑟𝑖𝑖−1,𝑐𝑐𝑖𝑖 + 𝑹𝑹𝑇𝑇𝑖𝑖+1
𝑖𝑖 ∙ 𝑓𝑓𝑖𝑖+1 × 𝑟𝑟𝑖𝑖,𝑐𝑐𝑖𝑖 + 𝜔𝜔𝑖𝑖 × (𝐼𝐼𝑖𝑖 ∙ 𝜔𝜔𝑖𝑖) + 𝐼𝐼𝑖𝑖 ∙ 𝑎𝑎𝑖𝑖 (7)
Subsequently, the optimization technique would then take the vector of torques τ obtained for
each series of movements and find the energy required for each set of movements from one point
to another through a motion plan. The power P for each movement increment is found by
multiplying the torques τ by the angular velocity �̇�𝜃. The energy for each movement E is then the
power multiplied by the duration of the increment dt between the points. The energy for an entire
plan for one joint Ejoint is the sum of all of the energies for each point in the plan for that joint:
𝐸𝐸𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝 = � 𝑃𝑃(𝑡𝑡)𝑑𝑑𝑡𝑡𝑝𝑝
0= � (𝑃𝑃𝑖𝑖(𝑖𝑖) ∗ 𝑑𝑑𝑡𝑡)
𝑝𝑝
𝑖𝑖=1
𝐸𝐸𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝 = � �𝜏𝜏𝑖𝑖 ∗ 𝜃𝜃�̇�𝚤(𝑖𝑖) ∗ 𝑑𝑑𝑡𝑡�𝑝𝑝
𝑖𝑖=1
(8)
However, the energy required to move the arm in the motion plan must be adjusted for the type
of servos in each joint in order to accurately estimate the energy required to execute the plan.
Tangy’s manufacturer, Dr. Robot Inc., supplied the servo model numbers for each of the joints.
The servo models and their operating energy consumption values are listed in Table 10. The
operating power consumption of each joint was used to adjust the sum in equation (8) by
weighting the energies according to the type of servo at the joint. The weighting is done through
multiplication of a weighting factor Wi.
42
43
𝐸𝐸𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = � 𝐸𝐸𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝 𝑖𝑖 ∗ 𝑊𝑊𝑖𝑖
# 𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝𝑃𝑃
𝑖𝑖=1
(9)
The weighting factor is calculated by normalizing the power consumptions for the joints so that
the servo with the highest power consumption has a weight of 1.
Table 10: Manufacturer Servo Specifications for Tangy's Arms [45]
Joint: Servo Model: Operating Power Consumption: Shoulder Roll Joint HS-805BB 830 mA @ 6.0V = 4.98 W Shoulder Pitch Joint HS-805BB 830 mA @ 6.0V = 4.98 W Shoulder Yaw Joint HS-785HB 285 mA @ 6.0V = 1.71 W
Elbow HS-785HB 285 mA @ 6.0V = 1.71 W Wrist Pitch HS-645MG 350 mA @ 4.8V = 1.68 W Wrist Roll HS-645MG 350 mA @ 4.8V = 1.68 W
The techniques to calculate duration and energy costs used in the simulation described in the
previous section were then implemented on Tangy using the following technique. The
optimization approach on the robot takes in ten plans from the motion planner, and calculates the
duration costs for each. If the differences between the lowest duration plan and subsequent
plan(s) are within 0.25 seconds of each other (which is visually very similar to one another), then
the optimization approach takes all of these similar plans and calculates the energy costs for
each. The optimization approach searches for the lowest duration plan first as gestures which
take longer to execute than needed may look unnatural either due to its slow speed or extraneous
movements. After the energy costs have been calculated for each plan, the plan with the lowest
energy cost is finally chosen to be executed.
5.2 Eye Contact Module
In order to provide eye contact as a social behaviour for players when assisting them in single
user scenarios, Tangy tracks the location of a user’s face and attempts to follow that location
with its head in order to make and maintain eye contact with him. This feature was implemented
in order to mimic the eye contact that residents would receive when requesting one-on-one
assistance from a human facilitator.
43
44
The eye contact module uses a video stream from the 2D Axis M1031-W camera in Tangy’s
right eye to scan for and track faces in the robot’s field of view. The OKAOTM Vision software
library [44] searches for facial features within 2D images from the video stream in order to
identify players’ faces (as displayed in Figure 8). The module is rotation invariant within the
following ranges: 30°, 20° and 360° in the yaw, pitch or roll directions (Figure 7). The location
of every face within the field of view of the 2D video stream is obtained as a set of pixel
coordinates with respect to the horizontal and vertical axes. The coordinates are defined by an
origin at the top left hand corner of the video stream, and increasing X and Y coordinates going
downwards and to the right respectively. The center of the 2D image is considered to be the
direct line-of-sight; as the resolution of the video feed is 640 x 480 pixels, the direct line-of-sight
is considered to be located at the coordinates (320, 240).
Figure 7: Face Detection Rotation Invariance Tangy identifies the person closest to its direct line-of-sight as the “person of interest”, and
assumes that this is the player who requested assistance. If the location of the person of interest’s
face is offset from center of the robot’s direct line-of-sight, Tangy will actuate its neck servos in
the pitch and roll axes in order to reorient its line-of-sight. The specific servo commands sent to
the neck are converted from the offset distances in pixels through an empirically tested
conversion technique. The conversion technique sends a maximum servo increment if the offset
distance value is over 150 pixels, and a parameterized servo increment (c * offset_value) if the
offset is under 150 pixels. The parameter c was chosen to be 0.4, after several tests with the eye
contact module. The parameter was chosen in order to limit the movement of Tangy’s head when
44
45
attempting to make eye contact with a player to speeds which are similar or slightly slower to a
human making eye contact with another human. Tangy continuously sends incremental servo
commands until the offset distance value approaches 0, allowing the robot to maintain unbroken
eye contact.
Figure 8: Face Tracking
5.3 Audio Interactions
Tangy employs verbal communication and music to interact aurally with residents. During the
multi-user aspect of the Bingo activity, the robot plays ragtime music in the background in order
to keep residents engaged during moments of silence between numbers. At the same time, Tangy
also continuously calls out Bingo numbers and tells jokes. During the single user aspect of the
Bingo activity, Tangy pauses the music in order to provide clear verbal instructions to players
when assisting them. Both the verbal communication and music are handled by the audio
interactions module.
In order to provide the verbal communication, Tangy uses a pitch-adjusted female voice
synthesized by GoogleTM powered text-to-speech. The background songs are stored and played
at random continuously during the game when Tangy is calling out Bingo numbers. All audio
files are played using an audio player developed using the Simple DirectMedia Layer library
[47].
5.4 Chapter Summary
This chapter discussed the eye contact module, the arm control scheme and optimization
technique, and the audio interactions module. Tangy controls its arms using a set of motion
(246,138)
45
46
planners in the Open Motion Planning Library, which generate point-to-point plans that take a
kinematic chain from an initial to a final pose. An optimization technique was developed in order
to minimize both the duration of arm movements during the execution of a gesture, and their
energy consumption as well. The optimization was done in order to avoid unnaturally slow arm
movement speed or extraneous movements, and to increase the runtime of the robot on a single
charge of battery.
Tangy initiates eye contact with a Bingo player by first tracking his/her face in a 2D video
stream obtained by a webcam in its eye. The robot then maintains eye contact by sending servo
commands to its neck to reorient its own face to decrease the offset distance between the player’s
face and its direct line-of-sight.
Finally, Tangy uses a speech synthesizer to provide verbal interactions with players. The robot
also plays background music to engage residents during silences in between calling out numbers.
46
47
Chapter 6 System Performance Review
To evaluate the robotic system, the finite state machine and the arm control were tested internally
in the research lab. The performance review for the finite state machine included playing the
equivalent of 50 Bingo games with five players and recording the success rates of behaviours.
The performance review for the arm control included two separate experiments; the first tested
the accuracy of the arms, and the second tested the optimization technique to minimize duration
of execution and energy consumption when performing a gesture.
6.1 Finite State Machine Performance Review
The finite state machine was tested in tandem with the sensory capabilities of the robot. The
performance review was done in order to verify that the robot was able to choose and execute its
behaviours appropriately when the world, robot, or card state changes.
6.1.1 Methods
In order to test the performance of the robotic system, the robot played the equivalent of 50
Bingo games by performing its multi-user, transition and single-user interactions repeatedly.
Each repetition of the game involved the robot calling out numbers and telling jokes,
transitioning to assist with an incorrectly played Bingo card, providing appropriate assistive
instructions until the card has been fixed, and continuing the game until a winning card has been
identified. During Tangy’s implementation of its multi-user interaction, the instances of the
Bingo number-calling behaviour and the joke-telling behaviour were counted to ensure that they
were occurring with the right frequency. Each time an assistance request device was activated,
the robot was monitored to identify the success rate of properly acknowledging the request and
then properly transitioning to the location of the assistance request. During the single-user
interaction, Tangy was tested to investigate whether it was able to identify cards with both
incorrectly marked and missing markers, and whether it was able to provide the appropriate
instructions to correct the card. The robot was also tested to see whether it would properly
identify correctly marked cards or cards with winning combinations, and whether it would
consequently provide the praise and encouragement or celebration behaviours respectively.
47
48
6.1.2 Results
The finite state machine performance results are compiled in Table 11. The results demonstrate
that Tangy was able to successfully choose the correct behaviour provided that a state change
was properly identified and sent to the finite state machine as an input.
Table 11: Finite State Machine Performance
State Change Input Expected Robot Behaviour Success Rate (%)
Playing Bingo (no state changes) Robot calls out random Bingo numbers from 1 to 75 100
7 Bingo Numbers Called Robot tells a joke or a trivia fact about Bingo 100 Help Request Detected in the
World Robot acknowledges assistance request by nodding its
head towards the user 100
Help Request Location Detected; Robot Location
Changes Robot navigates to a user in order to provide help 100
Bingo Card State Detected Robot: 1) instructs user to unmark incorrect
numbers/mark missing numbers; 2) provides praise; or 3) congratulates user and does a celebration dance.
100
6.2 Arm Control Accuracy Experiment
The arm control system was tested by moving individual joints at a time to investigate the
accuracy of the movements being executed by the arm servos. The experiment sought to test the
hardware accuracy of Tangy’s arms by taking measurements of the pose of the arm with an
external sensor after given a pre-determined goal pose.
6.2.1 Methods
The accuracy of the arm servos was tested for each joint in both of Tangy’s arms (Figure 9), with
the exception of the finger joints. The experiment procedure consisted of sending a goal rotation
of a certain angle for each joint and then measuring the actual movement of the arm to compare
the actual rotation of the joint with the goal. In order to simplify the analysis, each joint was
tested independently so as to more easily obtain the angle difference from the initial to the final
position. The finger joints were not considered as they are not used in any of the gestures
implemented on Tangy.
48
49
Figure 9: Servo Number Scheme (Right Arm) [48]
The external sensor used to measure the angle changes from the rotation of Tangy’s joints was a
three-axis accelerometer. The Texas InstrumentsTM CC2650 SensorTag, which contains a suite
of various sensors including a three-axis accelerometer, was specifically chosen for this
experiment for its cheap cost, small profile and its pre-established data collection interface. The
SensorTag measures 5.1cm x 3.7cm x 0.9cm, and connects to a smartphone app through a
BluetoothTM connection. The smartphone app can automatically upload data onto a cloud-based
storage system. The SensorTag’s accelerometer is the MPU-9250 model from sensor
manufacturer InvenSenseTM. The MPU-9250 has a ±60 mg tolerance in the X and Y axes at rest,
and a ±80 mg tolerance in the Z axis.
49
50
Figure 10: Texas Instruments CC2650 SensorTag
In order to convert accelerometer readings to angle measurements, simple vector geometry is
used. If the accelerometer readings for gravity in frames of the starting arm pose and final arm
pose are 𝒂𝒂𝟏𝟏 = �𝑎𝑎𝑥𝑥1,𝑎𝑎𝑦𝑦1,𝑎𝑎𝑧𝑧1� and 𝒂𝒂𝟐𝟐 = �𝑎𝑎𝑥𝑥2,𝑎𝑎𝑦𝑦2,𝑎𝑎𝑧𝑧2� respectively, then the difference in angle
between the gravity vectors in the two frames can be found from the dot product of the two
vectors:
𝒂𝒂𝟏𝟏 ∙ 𝒂𝒂𝟐𝟐 = |𝒂𝒂𝟏𝟏||𝒂𝒂𝟐𝟐| ∗ cos (∆𝜃𝜃) (10)
∆𝜃𝜃 = cos−1
⎝
⎛ 𝑎𝑎𝑥𝑥1 ∗ 𝑎𝑎𝑥𝑥2 + 𝑎𝑎𝑦𝑦1 ∗ 𝑎𝑎𝑦𝑦2 + 𝑎𝑎𝑧𝑧1 ∗ 𝑎𝑎𝑧𝑧2
�𝑎𝑎𝑥𝑥12 + 𝑎𝑎𝑦𝑦12 + 𝑎𝑎𝑧𝑧12 ∗ �𝑎𝑎𝑥𝑥22 + 𝑎𝑎𝑦𝑦22 + 𝑎𝑎𝑧𝑧22 ⎠
⎞ (11)
In this experiment, the joints are only given rotational goals about the major X, Y, and Z axes of
the world frame. The rotational goals which Joints 4-8 are given are all 90°. The rotation goal
given for Joint 3 is 45°, due to the fact that the joint is limited to a hardware range of ±45°. The
resulting poses from the rotation of the joints is illustrated for the right arm in Table 12, and is
mirrored for the left arm. The initial starting position for the test of Joints 3, 5, 7, and 8 will be
the neutral position. The starting position for the test of Joints 4 and 6 will be Pose 2 and 5
respectively, as the accelerometer is not capable of detecting roll about the vertical Z axis in the
world frame.
The performance review investigated the error between a measured rotation and the goal rotation
for each joint.
50
51
Table 12: Robot Right Arm Poses
Pose Number Pose Description Pose Picture
1 Neutral Position – All joints are at 0°
2 Joint 8 (shoulder roll joint) is rotated 90°
3 Joint 7 (shoulder pitch joint) is rotated 90°
4 Joint 6 (shoulder yaw joint) and Joint 8 are rotated 90°
5 Joint 5 (elbow joint) is rotated 90°
51
52
6 Joint 4 (wrist roll joint) and Joint 5 are rotated 90°
7 Joint 3 (wrist pitch joint) is rotated 45°
6.2.2 Results and Discussion
The results from the accuracy test for the right and left arms are listed in Table 13 and Table 14.
The mean errors for each joint in both arms are listed in Table 15. In general, the accuracy of the
movements of all of the joints in the right arm was within 5% of the goal pose, while the
accuracy of the left arm was within 8%. The errors of the joint movements for both arms could
be attributed to noise when attempting to measure the pose of the arm with the external sensor;
Tangy’s arms vibrate slightly, which resulted in a large range of accelerometer readings when
testing the right arm. The accelerometer values in Table 13 are the mean of the readings taken
from 10 consecutive accelerometer readings.
The performance review demonstrated that Tangy’s right arm was more accurate when executing
a goal pose than the left arm. These results suggest that any gestures which require any level of
precision, such as a pointing gesture, should be implemented on Tangy’s right arm rather than its
left arm.
52
53
Table 13: Accuracy Performance Review for Tangy's Right Arm
Trial 1
Testing Joint
Initial Pose
Final Pose
Accelerometer X Axis (g’s)
Accelerometer Y Axis (g’s)
Accelerometer Z Axis (g’s)
Δθ (Degrees) % Error
- 1 1 0.01 0.97 0.02 0 -
8 1 2 0.96 0.03 -0.03 87.6 3%
7 1 3 0.04 -0.01 0.96 89.4 1%
6 2 4 0.02 -0.04 0.99 91.1 1%
5 1 5 -0.02 -0.01 0.97 89.4 1%
4 5 6 0.98 0.01 0.04 88.8 1%
3 1 7 -0.01 0.683 0.685 43.9 2%
Trial 2
Testing Joint
Initial Pose
Final Pose
Accelerometer X Axis (g’s)
Accelerometer Y Axis (g’s)
Accelerometer Z Axis (g’s)
Δθ (Degrees) % Error
- 1 1 0.01 0.97 0.02 0 -
8 1 2 0.96 0.03 -0.03 87.6 3%
7 1 3 0.04 -0.01 0.96 89.4 1%
6 2 4 0.02 -0.04 0.99 91.1 1%
5 1 5 -0.02 -0.01 0.97 89.4 1%
4 5 6 0.98 0.01 0.04 88.8 1%
3 1 7 -0.01 0.683 0.685 43.9 2%
Trial 3
Testing Joint
Initial Pose
Final Pose
Accelerometer X Axis (g’s)
Accelerometer Y Axis (g’s)
Accelerometer Z Axis (g’s)
Δθ (Degrees) % Error
- 1 1 0.01 0.97 0.02 0 -
8 1 2 0.96 0.03 -0.03 87.6 3%
7 1 3 0.04 -0.01 0.96 89.4 1%
6 2 4 0.02 -0.04 0.99 91.1 1%
5 1 5 -0.02 -0.01 0.97 89.4 1%
4 5 6 0.98 0.01 0.04 88.8 1%
3 1 7 -0.01 0.683 0.685 43.9 2%
53
54
Table 14: Accuracy Performance Review for Tangy's Left Arm
Trial 1
Testing Joint
Initial Pose
Final Pose
Accelerometer X Axis (g’s)
Accelerometer Y Axis (g’s)
Accelerometer Z Axis (g’s)
Δθ (Degrees) % Error
- 1 1 -0.1 1.04 -0.01 0 0
8 1 2 -0.92 0.03 0.01 82.6 8%
7 1 3 0.1 0.06 -0.92 86.3 4%
6 2 4 -0.07 -0.02 -0.99 90.2 0%
5 1 5 0.05 0.07 -0.93 85.4 5%
4 5 6 -0.92 -0.09 0.1 90.1 0%
3 1 7 0.04 0.707 0.707 46.1 3%
Trial 2
Testing Joint
Initial Pose
Final Pose
Accelerometer X Axis (g’s)
Accelerometer Y Axis (g’s)
Accelerometer Z Axis (g’s)
Δθ (Degrees) % Error
- 1 1 0.04 0.99 0.03 0 0
8 1 2 -0.92 -0.07 -0.13 96.8 8%
7 1 3 -0.01 0.03 -0.93 89.9 0%
6 2 4 0.09 0.09 -0.95 86.1 4%
5 1 5 -0.04 0.03 -0.95 90.0 0%
4 5 6 -1.01 -0.04 -0.09 94.7 5%
3 1 7 0.05 0.693 0.696 43.3 4%
Trial 3
Testing Joint
Initial Pose
Final Pose
Accelerometer X Axis (g’s)
Accelerometer Y Axis (g’s)
Accelerometer Z Axis (g’s)
Δθ (Degrees) % Error
- 1 1 0.1 0.96 -0.09 0 0
8 1 2 -0.91 -0.04 0.01 98.4 9%
7 1 3 -0.09 -0.01 -1 85.7 5%
6 2 4 0 -0.08 -0.97 89.3 1%
5 1 5 0.05 0.02 -0.9 83.0 8%
4 5 6 -1.02 -0.02 0.09 97.4 8%
3 1 7 0.1 0.75 0.66 46.4 3%
54
55
Table 15: Mean Errors in Joint Accuracy Performance over Three Trials
Joint Right Arm Mean Errors
Left Arm Mean Errors
8 3% 8% 7 1% 3% 6 3% 2% 5 2% 4% 4 2% 5% 3 4% 3%
6.3 Arm Optimization Approach Experiment
The optimization technique in the arm control was tested by comparing the measured duration
and energy consumption of the optimized and un-optimized execution of various gestures. The
experiment tested the wave, point to screen, laugh, and celebration gestures to ensure that the
optimization approach was effective in optimizing different lengths of plans and ranges of
motion.
6.3.1 Methods
Tangy performed the wave, point to screen, laugh and celebration gestures each 25 times in a
trial which implemented the optimization technique. The robot planned a new trajectory after
each execution of a gesture, with a total of 100 unique plans created and executed during one
optimized trial. Then, Tangy performed the same procedure with the gestures in a trial which did
not optimize for time or energy consumption. The durations and energy consumption readings
for the optimized and un-optimized cases were taken for each group of repetitions of gestures. A
baseline energy consumption reading when the robot remained idle was taken for each gesture to
establish the amount of energy consumed by other power drains such as the robot base, neck
servos and the laptop. In order to measure the energy consumption during the trials, voltage
readings from Tangy’s battery were taken before and after each set of repetitions of gestures. The
energy consumption of the gestures will be defined as by the depth of discharge percentage
value. The depth of discharge is calculated by determining the initial and final states of charge of
the battery when compared to a theoretical full and empty charge (36 V and 25.9 V respectively,
as according to the manufacturer’s standards [48]). The equation to calculate the depth of
55
56
discharge D.O.D. from the initial battery reading Vinit, final battery reading Vfin, theoretical full
charge Vfull and theoretical empty charge Vempty is as follows:
𝐷𝐷.𝑂𝑂.𝐷𝐷. =𝑉𝑉𝑖𝑖𝑝𝑝𝑖𝑖𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦𝑉𝑉𝑓𝑓𝑓𝑓𝑝𝑝𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦
−𝑉𝑉𝑓𝑓𝑖𝑖𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦𝑉𝑉𝑓𝑓𝑓𝑓𝑝𝑝𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦
(12)
6.3.2 Results and Discussions
The results from the two trials were compiled in Table 16. The first column of the tables
specifies the particular trial for each gesture, including the optimized or un-optimized trials, or
the baseline case. Next, the total durations in seconds for executing each set of gestures for each
trial are given. The third and fourth column state the initial and final battery levels for each trial,
which are measured by polling the robot’s power state before and after the execution of the
gestures. The fifth column displays the voltage drops from the execution of the gestures, which
are calculated by taking the difference between the initial and final battery levels. The sixth
column includes the depths of discharge calculated from the voltage drops as according to
Equation (12).
The results show that the duration of the optimized plans for the gestures was lower in all cases,
and that the optimizer was successful in choosing motion plans which require less time to
execute. The speed difference when performing the optimized gestures when compared to the un-
optimized gestures ranges from 1% for the point to screen gesture to 20% for the celebration
gesture. The small difference in speed when performing the point to screen gesture is most likely
due to the small range of motion required, making it unlikely that the motion planners would
plan highly variant trajectories for the gesture. On the other hand, the celebration gesture not
only requires large movements from Tangy’s arms, it requires both the right and left arm to
execute the behaviour. This introduces much more possibility for variation in the plans produced
by the motion planners, and a greater difference in execution speeds for an optimized compared
to an un-optimized celebration gesture.
The energy consumption also appeared to be lower among the optimized gestures than the un-
optimized. However, the depth of discharge results seem to indicate that for certain gestures, the
difference in energy consumption between optimized and un-optimized gestures may be small,
such as in the case of the point to screen gesture. For the point to screen gesture, the energy
56
57
consumption from executing the optimal motion plans was indistinguishable from the energy
consumption when executing the un-optimized motion plans. On the other hand, large gestures
such as the celebration gesture can significantly drain the robot’s battery. The difference in depth
of discharge for the optimized compared to the un-optimized celebration gestures was 5% of the
robot’s battery over 25 repetitions of the gesture. Although for some gestures, the optimization
approach offered minimal benefits for providing the most energy efficient plans, the results of
the experiment demonstrate that the optimization technique did provide energy savings across
many repetitions of the same gesture.
57
58
Table 16: Duration and Energy Consumption Differences between Sets of Twenty-Five Executions of Optimized and Un-Optimized Gestures
Point to Screen Gesture
Trial: Total
Duration (s)
Initial Battery Reading (V)
Final Battery Reading (V)
Total Voltage
Drop (V)
Depth of Discharge (%)
Baseline 537 38.5 37.8 0.7 7%
Optimized 537 39.1 38.3 0.8 8%
Un-optimized 542 38.7 37.9 0.8 8%
Celebration Gesture
Trial: Total
Duration (s)
Initial Battery Reading (V)
Final Battery Reading (V)
Total Voltage
Drop (V)
Depth of Discharge (%)
Baseline 890 38.3 37.1 1.2 12%
Optimized 890 38.1 36.2 1.9 19%
Un-optimized 1068 38.9 36.5 2.4 24%
Laugh Gesture
Trial: Total
Duration (s)
Initial Battery Reading (V)
Final Battery Reading (V)
Total Voltage
Drop (V)
Depth of Discharge (%)
Baseline 718 38.8 37.7 1.1 11%
Optimized 718 39.1 37.9 1.2 12%
Un-optimized 797 38.4 37.1 1.3 13%
Wave Gesture
Trial: Total
Duration (s)
Initial Battery Reading (V)
Final Battery Reading (V)
Total Voltage
Drop (V)
Depth of Discharge (%)
Baseline 765 37.9 37 0.9 9%
Optimized 765 39.1 37.9 1.2 12%
Un-optimized 796 38.7 37.3 1.4 14%
The results from the performance review could be used to improve the robot’s behaviour
deliberation system by providing information on the energy consumption of some of its
behaviours. Namely, one potential application of the results from this experiment may be to
allow the robot to factor in the average energy consumption of a single gesture into its
determination of a behaviour. For example, Tangy may decide to perform certain gestures with
relatively higher energy consumption rates fewer times such as the laugh gesture when the
robot’s battery is at a low state of charge.
58
59
6.4 Chapter Summary
Several experiments were conducted to investigate the performance of the robot in a controlled
setting in the research lab. Tangy’s finite state machine performed with 100% success rates in
performing its expected behaviour over all 50 games, provided that the state change was detected
by the robot.
The arm control accuracy was studied by sending Tangy goal arm poses and measuring its
executed pose with an external sensor. The accuracy of the right arm was found to be very high –
Tangy was able to reach within 5% of the targeted goal pose. The left arm appeared to be less
accurate, with one joint reaching within 8% of its targeted orientation. The accuracy of the arms
was most likely impacted by instability in the servos, including vibrations or jerking motions
when performing a movement or attempting to maintain a pose.
The optimization technique in the arm control was tested to investigate its ability to choose the
lowest duration and lowest energy consumption motion plans. Tangy performed several gestures
using the un-optimized motion planner and the optimized motion planner in independent trials.
The results from the trials indicate that the optimization approach was able to choose shorter
duration and lower or equivalent energy consumption plans for the robot.
59
60
Chapter 7 Human Robot Interaction User Studies
A human-robot interaction pilot study was conducted by our research team to investigate the
efficacy and acceptance of Tangy as an autonomous Bingo facilitator for elderly residents in a
long-term care facility. Several Bingo game sessions with the robot took place with several
residents over the course of two weeks. Development of the study’s methodology and analysis of
the results were frequently performed collaboratively with other members of the research team.
This chapter investigates specifically the system performance of the robot, and the reactions from
participants to the Bingo scenario with the robot in this study.
7.1 Participants
Seven residents participated in a total of six Bingo sessions facilitated by Tangy. The residents
were aged 66-79 years old, with a mean of 79.3 years old and standard deviation of 11.7 years.
Participants were chosen to meet the following criteria: they were 1) cognitively intact or with
mild cognitive impairment (Cognitive Performance Scale level of 2 or less [40]), 2) over the age
of 60, 3) fluent in English, and 4) could hear normal levels of speech. Several residents played
multiple sessions with Tangy, with each resident participating in at least two Bingo games.
Participants’ experience levels with computers and robotics were gathered at the end of each
game, with the results listed in Table 17. Most participants had experience with computers, but
only one participant had any experience with robots. Written informed consent was obtained
prior to commencement of the study.
Table 17: Number of Participants with Experience with Computers or Robots
Experience with Computers* Experience with Robots** No experience 1 6
Beginner 2 1 Intermediate 1 0
Advanced 3 0 *Beginner (email, use simple programs) Intermediate (internet, chat) Advanced (editing documents, use complex programs)
** Beginner (seen robots at museums/science centers or stores, or have watched shows with robots) Intermediate (have worked with/used commercial robots) Advanced (have worked on robot developmental aspects including hardware/ software design)
60
61
7.2 Methods
Prior to playing in the Bingo games with Tangy, participants were given a demonstration of the
robot’s capabilities and actions during the game. The demonstration included a game facilitated
by the robot with members of the research team. The purpose of the demonstration was to
introduce the robot and the scenario to the participants so that they would have the opportunity to
present any questions or concerns before playing the game themselves. The demonstration
displayed every potential behaviour that the robot could perform during the game.
The study was conducted in a first floor activity room in the long-term care facility. Each session
lasted approximately 45 minutes to an hour, and consisted of either two or three Bingo games.
The sessions were video recorded for analysis purposes. Tangy’s system performance was
measured using the video recordings of the sessions by other members of the research team. The
system performance review was done in order to assess the execution success rates of the robot
behaviours developed in this thesis in a real-world environment during the study. Two members
of the research team measured the execution success rates by counting the number of times the
robot executed a behaviour which it was supposed to, and when it didn’t execute, or performed
the behaviour incorrectly. This thesis provides a reflection on the execution success rates of the
robot behaviours later on in this chapter.
After each Bingo session, participants were also given a post-interaction questionnaire if it was
their last session. The questionnaire contained closed questions investigating the participants’
acceptance and attitudes towards Tangy. The closed questions were developed by myself and
another member of our group, and categorized into constructs developed from the Almere model
[28] by this member of the research team. These questions were answered using the five-point
Likert scale (where 1= strong disagree, 2= somewhat disagree, 3= neutral, 4= somewhat agree,
5= strongly agree) and are listed in Table 19. The questionnaire also contained open-ended
questions developed in this thesis, which investigated the features on Tangy that residents liked
or found helpful, and other features or activities which they wanted to see on Tangy. Thematic
analysis was performed on the results on the open-ended questions, and the thematic sets are
provided later on in this chapter as well.
61
62
7.3 System Performance Results
The system performance results are documented in Table 18. In general, the success rates for the
robot’s behaviours were 100%, with the exception of two behaviours: the acknowledgement of a
player’s assistance request, and initiating eye contact. Tangy did not acknowledge every request
by nodding at players when two players pressed their buttons at the same time. Upon
investigation, this failure was attributed to the fact that the robot finite state machine did not
include a provision for processing two requests to nod at the same time. If Tangy received one
set of servo commands to nod toward the location of the first assistance request and the second
immediately afterwards, the second set of servo commands would pre-empt the first set.
Tangy failed to perform the eye contact behaviour properly, with instances occurring during the
single user assistance scenarios where the robot would attempt to make eye contact with players
sitting beside the player who requested assistance. The failure of the eye contact was attributed to
the programming logic behind the eye contact module. Tangy would attempt to make eye contact
with the person whose face appears closest to the center of the field of view of the robot.
However, this would only be the correct person to maintain eye contact with if the robot had
navigated perfectly in front of him/her. Should the robot be skewed in orientation or shifted in
location to either the right or the left of the correct player upon initiation of the eye contact
module, it might capture the face of his/her neighbour instead. These failures occurred more
often because participants would frequently move closer to each other to view the robot when it
came to assist a player.
62
63
Table 18: System Performance Results1
Robot State Robot Location Card State
Expected Robot Behaviour
Success Rate
Pre-Game At front of room - Greet players 100% Multi-User Interactions At front of room - Call out Bingo
number 100%
Multi-User Interactions At front of room - Provide Jokes and
Facts 100%
Multi-User Interactions At front of room Acknowledge
player(s) request(s) 92.68%
Transition At front of
room/At another player’s location
- Navigate towards location of player 100%
Single User Interactions
At player’s location -
Localize player’s face and initiate eye
contact 63.64%
Single User Interactions
At player’s location
Occluded card
Prompt player to move card 100%
Single User Interactions At player’s
location Incorrectly
marked
Request to remove marker(s) from
incorrectly marked numbers
100%
Single User Interactions At player’s
location Missing markers
Request to mark missing number(s) on
the Bingo card 100%
Single User Interactions At player’s
location Correctly marked
Provide encouragement 100%
Single User Interactions
At player’s location
Winning card
Provide congratulations 100%
Post-Game - - Say farewell to player 100% 1Measured by two other members of the research team
7.4 Human-Robot Interaction Results
7.4.1 Participant Questionnaire Results
The statements measured with the Likert scale for the questionnaire are presented in Table 19
with the mean scores for each question.
63
64
Table 19: Post-Bingo Session Questionnaire Results2
Statement Mean I enjoy playing Bingo with Tangy. 5.00
I think Tangy could help me during the game. 4.00 Tangy is able to help me. 4.57
I will play Bingo with Tangy again. 5.00 I will ask Tangy for help again. 4.71
I will ask Tangy for help in the future. 5.00 I think Tangy should host Bingo games again. 4.86
I think Tangy is helpful to other players. 4.86 I think Tangy makes the Bingo game interesting 4.86
I like Tangy’s appearance 4.71 It feels like Tangy is looking at me when I am
playing the game with it 4.00
I am comfortable interacting with Tangy 5.00 I find Tangy intimidating* 1.00
I trust Tangy’s help 4.86 I will follow what Tangy asks me to do in the
Bingo game 5.00 2Questionnaire developed collaboratively with another member of the research team; statistical analysis done by same member
The responses to the open-ended questions from the questionnaire revealed that participants
wished to see Tangy facilitate recreational games which were more popular among residents,
such as card games, board games, or physical activities. Tangy’s voice was praised by three
participants, who indicated that it was “clear” and “human-like”. Tangy’s arm gestures were
singled out by two residents as very enjoyable; one participant stated that the robot’s body
language was very expressive and the robot’s celebration dance demonstrated clearly that
“Tangy was excited”. Several participants noticed Tangy looking at them when they were
playing the Bingo game; one participant described “[I] really enjoyed when Tangy looked at
[me] during the game”. An aspect of the game considered to have a negative impact on four
participants’ enjoyment of the session was the slow pacing of the game by the robot facilitator. A
participant gave his opinion: “Personally, for me, I would like the game to be faster. But I think
it was a good speed for the others.” Finally, the majority of participants indicated that they had
no preference for either the verbal communication of the robot or the visual display of
information on its tablet, but that both were important sources of information during the game.
64
65
7.5 Discussion
The robot was demonstrated to be consistent in properly executing all of its behaviours at the
appropriate times during the Bingo game except the acknowledgement behaviour and the eye
contact behaviour. The failure of the acknowledgement behaviour can be avoided in the future
by adding a provision for multiple consecutive nods in the code when sending servo commands
to the robot’s neck. The provision should preserve the intent of the acknowledgement behaviour,
which is to signal to every player who requested assistance that the robot has detected his/her
request; as such, the provision should simply queue up nodding gestures and complete them one
after the other. The failure of the eye contact behaviour requires a more complex solution. As
non-zero tolerances must be assumed with the robot’s navigation system, Tangy cannot be
assumed to always position itself with the face of the player who requested assistance directly in
the center of its field of view. One possible method to prevent this failure may be to store
player’s faces within the robot’s database and link them uniquely to the assistance request
devices. Thus, the robot would be able to immediately know who requested assistance even
before navigating to the player. It would then be able to choose the correct player within its field
of view to focus on and initiate eye contact with.
The results of the post-game questionnaire showed that the participants enjoyed the Bingo
session with Tangy as the facilitator and would be willing to play Bingo with the robot again in
the future. The comfort participants felt when interacting the robot may have influenced their
enjoyment of these sessions and their desire to interact with the robot in the future [48]. As
comfort has been linked with the sociability of a robotic system, the sociability of Tangy may
have allowed participants to feel more comfortable interacting with the robot [50]. In [50], the
expressive robot iCat had two types of conversations with older adults at a long-term care
facility: one type which involved social behaviours such as gazing at users and using facial
expressions, and the other which didn’t involve the social behaviours. The study demonstrated
that older adults felt more comfortable with the more sociable robot. Similarly, in our study,
participants indicated that they were comfortable and not intimidated by interacting with Tangy.
This may have been motivated by the various types of social behaviours performed by the robot
including speech, eye gaze and physical gestures, which participants alluded to consistently in
their responses to the open ended questions in the post-game questionnaire.
65
66
Participants’ feedback on the Bingo session also provided insights into potential improvements
on Tangy’s features. Several comments about the robot’s pacing of the game and of the slow
speed of its interactions with single users came up during the post-game questionnaire. A
possibility for increasing the robot’s pacing when facilitating the Bingo game can be to increase
the speed at which Tangy calls out Bingo numbers. The single user interactions can also be sped
up by increasing the navigation speed of the robot through the environment. All participants
indicated that they enjoyed the supplementary behaviours such as the robot’s jokes and
background music. Their enjoyment of these features can inform Tangy’s design by increasing
the amount of these engaging interactions in the design of the robot’s behaviours.
7.6 Chapter Summary
This chapter presented the human-robot interaction pilot study done at a long-term care facility
to investigate the efficacy and acceptance of Tangy as a robot facilitator of the recreational
activity Bingo among long-term care residents. Tangy facilitated six Bingo sessions with 7
residents. The sessions were analyzed for the robot’s system performance, participant
compliance with Tangy’s requests, and participant engagement with the activity. The results
demonstrated that Tangy was able to perform most of its behaviours appropriately during the
activity and participants indicated that they enjoyed playing the game with Tangy. Several
participants explained that they especially enjoyed some of Tangy’s social features. They
brought up the clearness of the robot’s voice, the usefulness and expressiveness of the robot’s
gestures, and their enjoyment of the eye contact during the game. Participants mentioned that the
speed of the Bingo game and of the robot’s interactions with players could be increased. Overall,
participants indicated that they would interact with Tangy again in the future, which may be due
to their high level of comfort with the robot.
66
67
Chapter 8 Conclusion
8.1 Summary of Contributions
This chapter provides a summary of the developmental work on Tangy’s behaviours and
actuation capabilities and the experiments designed and implemented to investigate their
performance and efficacy for older adults.
8.1.1 Gathering End-User Feedback
Focus group studies were performed at a long-term care facility and a retirement home in order
to obtain the opinions and design considerations of elderly residents, family members and
healthcare staff about socially assistive robots and Tangy. Participants were shown a demo of an
early implementation of the robot, and asked to give feedback on the robot’s activities, its
features and the physical robot itself. The suggestions participants gave about the various
activities the robot could do, including recreational activities other than Bingo, prompting and
reminding, simple interactions with residents with cognitive impairments, and physical aid
around the facilities, provides a large amount of potential future work for implementing socially
assistive robots in long-term care. Moreover, important feedback was gained about the
interaction methods between the robot and residents. The results of the study indicated that
participants strongly believed the robot’s interface needed to be highly intuitive and easy to use.
Participants also brought up the issue of accessibility for physically impaired residents, which
needed to be considered when designing the interaction methods between residents and the robot.
8.1.2 Assistive Robotic Behaviours and Actuation Capabilities
The behaviours and the behaviour determination approach for the facilitation of the recreational
activity Bingo game were developed and implemented on Tangy. The behaviours of the robot
were chosen to imitate human Bingo facilitators at a long-term care facility and a retirement
home. The assistive social robot behaviours uniquely covered both the crucial multi-user and
single user interactions that a facilitator may have with players during the facilitation of a Bingo
game in order to allow the robot to facilitate the game completely autonomously. The robot
performs five major types of behaviours depending on the current junction of the game: i) pre-
67
68
game introduction; ii) multi-user behaviours; iii) transition actions; iv) single user assistive
behaviours; and v) post-game farewell. Which behaviours were chosen to be executed was
determined by the robot’s finite state machine. Tangy’s FSM determined appropriate behaviours
based on the state changes in the world, robot or detected card of a player who requested
assistance.
The other contribution made in this thesis relates to the actuation modules which allow Tangy to
perform the assistive behaviours of verbally communicating with players, making eye contact
with players when assisting them, and physically gesturing. The robot uses a synthesized voice in
order to call out Bingo numbers, instruct players on fixing their misplayed Bingo cards, praise
and encourage correctly played cards, and congratulate players with winning cards. Tangy
maintains eye contact by tracking a player’s face in the 2D video stream from the webcam in its
eye and then reorienting its head in order to keep that player’s face as close to its direct line of
sight as possible. Finally Tangy gestures using its arms by executing trajectory plans created by
motion planners. Tangy chooses the optimal motion plan with the lowest duration or the lowest
energy consumption requirements.
8.1.3 Experimental Results
Tangy’s system performance with respect to its finite state machine was evaluated by having the
robot run the equivalent of 50 Bingo games and performing every behaviour possible in the
experiment. Tangy’s arm control was assessed in both the accuracy of each individual joint and
the effectiveness of the optimization technique used to select the lowest duration and energy cost
motion plans. The experiments demonstrated that the arm servos were fairly accurate, with each
joint the right arm reaching within 5% of a pre-determined goal movement and each joint in the
left arm reaching within 8% of the same goal movement. Moreover, the experiments
demonstrated that the optimization approach to the arm control succeeded in choosing the plans
with the shortest duration and the lowest battery voltage consumption.
8.2 Discussion of Future Work
Future work for the implementation of Tangy should include investigating the possibility of the
robot facilitating other recreational activities other than Bingo. This thesis develops several
Bingo-specific behaviours, but also social and assistive behaviours which could be implemented
68
69
in other activities, like the popular trivia and trivia-like activities in the long-term care and
retirement home facilities.
Future research should also investigate the efficacy of the social robot behaviours for elderly
residents during the Bingo game in order to inform the design of social behaviours for
autonomous socially assistive robots for the elderly.
Finally, future work can be done on creating an integrated system which can use the energy
consumption values found in the experiment described in section 6.3. The integrated system can
not only choose the optimal motion plan for the execution of an arm gesture, but also optimize
the usage of gestures in order to increase the runtime of the robot on a single charge of its
batteries.
69
70
References
[1] I. Deary et al, “Age-associated cognitive decline”, British Medical Bulletin, Vol. 92, No.1,
pp. 135-152, 2009.
[2] G. Lafortune and G. Balestat, “Trends in severe disability among elderly people: assessing
the evidence in 12 OECD countries and the future implications,” OECD Publishing, 2007.
[3] S. Sharkey, “People Caring for People: Impacting the Quality of Life and Care of
Residents of Long-Term Care Homes,” Ontario Ministry of Health and Long-term Care,
Toronto, ON, 2008.
[4] B. Sobel, “Bingo vs. physical intervention in stimulating short-term cognition in
Alzheimer’s disease patients”, American Journal of Alzheimer’s Disease and Other
Dementias, Vol. 16, No. 2, pp. 115-120, 2001.
[5] F. Colombo et al., “Help wanted? Providing and paying for long-term care,” OECD
Publishing, Paris, 2011.
[6] United Nations, “World Population Prospects The 2015 Revision,” Key Findings and
Advance Tables, New York, 2015.
[7] E. Farina, et al, “Evaluating two group programmes of cognitive training in mild-to-
moderate AD: Is there any difference between a ‘global’ stimulation and a ‘cognitive-
specific’ one?”, Aging and Mental Health, Vol. 10, No. 3, pp. 211-218, 2006.
[8] K. Ball, D. Berch, and K. Helmers, "Effects of Cognitive Training Interventions with Older
Adults", The Journal of the American Medical Association, Vol. 288, No. 18, pp. 2271-
2281, 2002.
[9] R. Anand et al., “Brain health fitness: beyond retirement”, Educational Gerontology, Vol.
37, No. 6, pp. 450-465, 2011.
[10] N. Gates and M. Valenzuela, “Cognitive exercise and its role in cognitive function in older
adults”, Current Psychiatry Reports, Vol. 12, No. 1, pp. 20-27, 2010.
[11] G. Smith et al., “A Cognitive Training Program Based on Principles of Brain Plasticity:
Results from the Improvement in Memory with Plasticity‐based Adaptive Cognitive
Training (IMPACT) Study”, Journal of the American Geriatrics Society, Vol. 57, No. 4,
pp. 594-603, 2009.
70
71
[12] A. Kueider et al., “Computerized cognitive training with older adults: a systematic
review”, PLoS One, Vol. 7, No. 7, pp 1-13, 2012.
[13] A. Spector, et al., “Efficacy of an evidence based cognitive stimulation therapy programme
for people with dementia”, British Journal of Psychiatry, Vol. 183, pp. 248–254, 2003.
[14] V. Breuil, et al., “Cognitive stimulation of patients with dementia: Preliminary results.
International Journal of Geriatric Psychiatry”, Vol. 9, pp. 211–217, 1994.
[15] O. Zanetti, et al., “Effectiveness of procedural memory stimulation in mild Alzheimer’s
disease patients: A controlled study”, Neuropsychological Rehabilitation, Vol. 11, pp. 263-
272, 2001.
[16] S. Zarit, J. Zarit, & K. Reever, “Memory training for severe memory loss: Effects on senile
dementia patients and their families” Gerontologist, Vol. 22, pp. 373–377, 1982.
[17] V. Tesky, C. Thiel, W. Banzer, and J. Pantel, “Effects of a group program to increase
cognitive performance through cognitively stimulating leisure activities in healthy older
subjects,” GeroPsych: The Journal of Gerontopsychology and Geriatric Psychiatry, Vol.
24, No. 2, pp. 83-92, 2011.
[18] V. Menec, “The relation between everyday activities and successful aging: A 6-year
longitudinal study”, The Journals of Gerontology Series B: Psychological Sciences and
Social Sciences, Vol. 58, No. 2, pp. S74-S82, 2003.
[19] T. Laudate. Neargarder et al., “Bingo! Externally supported performance intervention for
deficient visual search in normal aging, Parkinson's disease, and Alzheimer's disease,”
Aging, Neuropsychology, and Cognition, Vol. 19, No. 1-2, 102-121, 2012.
[20] T. Glass et al., “Social engagement and depressive symptoms in late life longitudinal
findings”, Journal of Aging and Health, Vol. 18, No. 4, pp. 604-628, 2006.
[21] J. Unger, C. Johnson, and G. Marks, “Functional decline in the elderly: evidence for direct
and stress-buffering protective effects of social interactions and physical activity”, Annals
of Behavioural Medicine, Vol. 19, No. 2, pp. 152-160, 1997.
[22] D. McColl, W. G. Louie, and G. Nejat, “Brian 2.1: A Socially Assistive Robot for the
Elderly and Cognitively Impaired,” IEEE Robotics & Automation Mag., Vol. 20, No. 1, pp.
74-83, 2013.
71
72
[23] A. Tapus, C. Tapus, and M. Mataric, “The use of socially assistive robots in the design of
intelligent cognitive therapies for people with dementia,” IEEE Conference on
Rehabilitation Robotics, pp. 924-929, 2009.
[24] Y. Oida, et al., “Development of a Robot-Assisted Activity Program for Elderly People
Incorporating Reading Aloud and Arithmetic Calculation,” Asian Perspectives and
Evidence on Health Promotion and Education, pp. 67-77, 2011
[25] M. Kanoh et al., “Examination of practicability of communication robot-assisted activity
program for elderly people”, Journal of Robotics and Mechatronics, Vol. 23, No. 1, pp. 3-
12, 2011.
[26] T. Hamada, et al., “Robot therapy as for recreation for elderly people with dementia-Game
recreation using a pet-type robot”, in IEEE International Symposium on Robot and Human
Interactive Communication, pp. 174-179, 2008.
[27] R. Khosla et al., “Embodying care in Matilda: an affective communication robot for the
elderly in Australia,” in ACM SIGHIT Proceedings of International Health Informatics
Symposium, pp. 295-304, 2012.
[28] M. Heerink, et al., "Assessing acceptance of assistive social agent technology by older
adults: the almere model", International Journal of Social Robotics, Vol. 2, No. 4, pp. 361-
375, 2010.
[29] B. De Ruyter, et al. "Assessing the effects of building social intelligence in a robotic
interface for the home." Interacting with computers, Vol. 17, No. 5, pp. 522-541, 2005.
[30] D. Miyauchi, et al. "Human-robot eye contact through observations and actions." Pattern
Recognition, ICPR 2004. Proceedings of the 17th International Conference on. Vol. 4,
2004.
[31] A. Kendon, “Some functions of gaze-direction in social interaction”, Acta Psychologica,
26:22–63, 1967.
[32] T. Yonezawa et al., “Gaze-communicative Behaviour of Stuffed-toy Robot with Joint
Attention and Eye Contact based on Ambient Gaze-tracking”, Proceedings of the 9th
international conference on Multimodal interfaces, pp. 140-145
[33] H. Ishiguro et al., “Development of an Interactive Humanoid Robot “Robovie” – An
interdisciplinary approach”, Robotics Research, Springer Berlin Heidelberg, pp.179-191,
2003.
72
73
[34] C. Breazeal, et al. “Learning from and about others: Towards using imitation to bootstrap
the social understanding of others by robots”, Autonomous Robots, Vol. 11, No. 1-2, 2005
[35] C. Sidner, et al. “Explorations in engagement for humans and robots.” Artificial
Intelligence, Vol. 166, No. 1, pp. 140-164, 2005.
[36] C. Breazeal, et al. "Effects of nonverbal communication on efficiency and robustness in
human-robot teamwork." Intelligent Robots and Systems, IEEE, 2005.
[37] W. Bainbridge, et al. "The effect of presence on human-robot interaction." Robot and
Human Interactive Communication, 2008. RO-MAN 2008. The 17th IEEE International
Symposium on. IEEE, 2008.
[38] H. Miwa, et al., “Effective Emotional Expressions with Emotion Expression Humanoid
Robot WE-4RII,” in Intelligent Robots and Systems, 2004. (IROS 2004). Proceedings.
2004 IEEE/RSJ International Conference on, Vol.3, pp. 2203-2208, 2004.
[39] M. Haring, N. Bee and E. Andre, "Creation and evaluation of emotion expression with
body movement, sound, and eye color for humanoid robots," in IEEE International
Symposium on Robot and Human Interactive Communication, pp. 204-209, 2011.
[40] G. Louie et al., “A Focus Group Study on the Design Considerations and Impressions of a
Socially Assistive Robot for Long-Term Care”, Robot and Human Interactive
Communication, 2014 RO-MAN: The 23rd IEEE International Symposium, 2014.
[41] J.N. Morris et al., “MDS cognitive performance scale,” Journal of Gerontology, Vol. 49,
No. 4, pp. M174-M182, 1994.
[42] I. Sucan, and S. Chitta, “MoveIt!”, [Online] Available: http://moveit.ros.org [Accessed
March 2015], 2014.
[43] I. Sucan, M. Moll, and L.E. Kavraki, “The Open Motion Planning Library”, IEEE Robotics
& Automation. Mag, Vol. 19, No. 4, pp. 72–82, 2012.
[44] R. Murray, et al. A mathematical introduction to robotic manipulation. CRC press, 1994.
[45] ServoCity, Available: https://www.servocity.com/html. [Accessed Aug. 2015].
[46] Omron, “OKAO Vision”, Available: http://www.omron.com/r_d/coretech/vision/okao.html
[Accessed May 2013], 2007.
[47] Simple DirectMedia Layer, [Online] Available: https://www.libsdl.org [Accessed March
2015]
73
74
[48] Dr. Robot Inc., [Online] Available: http://www.drrobot.com/products_H20.asp [Accessed Sept. 2015]
[49] E. Broadbent, E., R. Stafford, and B. MacDonald, “Acceptance of healthcare robots for the
older population: Review and future directions,” International Journal of Social Robotics,
Vol. 1, No. 4, pp. 319-330, 2009.
[50] M. Heerink et al., “The influence of a robot's social abilities on acceptance by elderly
users,” IEEE International Symposium on Robot and Human Interactive Communication,
pp. 521-526, 2006.
74