Autism and Nao: “It’s a Very

Compelling Device”

Re-published with permission from

Family Center on Technology and Disability

Technology Voices – October 2010

Autism and Nao: “It’s a Very Compelling Device”

An Interview with Notre Dame Nao senior team members

Dr. Joshua Diehl, Assistant Professor;

Dr. Charles Crowell, Associate Professor and

Dr. Michael Villano, Research Assistant Professor

When encountering the Nao robot, Dr. Michael Villano remarks, “people behave as if

an infant has just been brought into the room. Women are drawn to it as are small

children. I’ve never seen anyone of any age step away from it. It’s such a compelling

device, which is what’s going to make this study so easy to run.”

The Notre Dame team’s focus in working with the robot is

on developing therapeutic protocols and software to

support those protocols. They are not interested, says team

member Dr. Joshua Diehl, in the commercial potential of

such software. “We are dedicated only to the science of

this research,” stresses Dr. Diehl. “As a team, we are

interested in critically examining the role of the robot in therapy and then publishing

our findings.”

Enhancement of Intervention

Dr. Diehl’s team is in the early stages of piloting. “We’ve gone through our first round

of piloting in which our aim, in part, was to see how the robot would fit into the

therapy session. We wanted to discover what went well, what aspects needed

improvement and how the interface was going to function.

“Now we’ve moved on to a development phase during which we’re improving the

interface’s ease of use. We’re developing comparison conditions that examine the

benefit of an actual robot in the room as opposed to other versions that are less

expensive than Nao.

“We’re committed to discovering Nao’s best application and, most importantly, to

learning which children will benefit most from this robot. Even if it’s a given that Nao

will be effective for children with autism, it is not yet certain that it will be effective for

all children with autism. We want to make certain that we know which kids derive the

most benefit from this kind of intervention enhancement, because that is what Nao

really is, an enhancement of intervention.”

Essentially, he explains, “ours is an applied behavior analysis therapeutic program

into which the robot is integrated in order to provide an enhancement to the existing

program.”

A Fully Functional Humanoid Robot

Dr. Charles Crowell, one of the study team’s senior researchers, describes the 22-inch

humanoid robot that has prompted an unusually positive emotional response from

children with autism and their families.

Nao’s sensory capabilities, Dr. Crowell explains, include

cameras that mimic sight; speakers that allow the user to

project a synthesized voice; and a microphone that will

pick up a voice from interactions. “The robot has a touch-

sensitive area on its head and sonar to detect obstacles to

its front and slightly to its side.”

Nao’s motor capabilities, he continues, include motors in its

joints that allow it to move legs, knees, arms, shoulders,

elbows and to some extent fingers. “Nao has three fingers

wedded together on each hand and an internal computer

that runs a Unix operating system that can serve as a

platform for developing and controlling the robot.” Nao

can communicate via a tethered Ethernet communication line or with Wi-Fi.

Nao Simplifies Human Behavior

According to Dr. Diehl, the Applied Behavioral Analysis (ABA) sessions they conduct

tend to be programmatic and repetitive for good reason: the children need to

practice and drill to learn skills. “What the robot adds to those sessions is an element

that’s intrinsically interesting and engaging for the children. It adds spice to the

therapy sessions, but Nao’s value goes way beyond that.”

There are many potential benefits to robot use over low-tech, Dr. Diehl insists. “Nao

allows us to better simulate and simplify human behavior to make that behavior

more understandable. It provides a safer environment for the child to practice social

skills, like carrying on a conversation.

“Trying to practice conversation skills with an eight-year-old child with autism and an

eight-year-old peer is much more difficult and the setting is less safe, with more

potential variables to control, with the therapist not knowing precisely how the peer

will react to some situations. Robot use gives us a safe environment so that we can

structure a conversation that enables the child to practice these skills multiple times.”

Behaviors Can Be Developed: Creating a Complex Interface

Nao, Dr. Crowell adds, “is a complicated device for which behaviors can be

developed using techniques to control the motors and to feed the robot information.

Performance is induced by a command-driven interaction style or it can be set up to

run in an autonomous mode in which its task capabilities are dependent on user

inputs.”

Given that functionality, he says, “we believed it was possible to integrate the Nao

into a therapeutic setting along with the therapist as a third person in the room who

would provide an adjunct to the therapy.”

“We took a simplified approach to the interface creation; baby steps, in fact,” Dr.

Villano comments. “We’ve built a Wizard of Oz interface. The program looks much

like a universal remote control on a Windows PC. Users can initiate specific Nao

behaviors by pushing certain buttons. For instance, a positive feedback behavior will

initiate a football referee’s touchdown signal by Nao and a voice response that tells

a child, ‘way to go,’ or Nao can assume a generic instructor’s stance with hand on

hip and a voiceover that tells the child, ‘I’m not sure I understood you.’ There are

many other similar behaviors that we can initiate from the universal remote control.”

Nao Text-to-Speech Capabilities

Text-to-speech is another major functionality the team uses to improve interactivity,”

Dr. Villano says. “The vehicle can be as simple as a text box. Anything typed into the

text box will be immediately transmitted into the Nao. The Nao will then speak the

words that have been typed into the text box by employing its text-to-speech

conversation in a humanlike voice.” The goal, he adds, “is to carry on a conversation

but also record pre-set dialogues.” Recently, he says, “we implemented a way for

the Nao to play Simon Says with a child in therapy. Nao can either lead Simon Says

or enable the child to lead Simon Says and can participate in the Simon Says sessions

by making pre-set movements initiated from this interface.”

“It’s amazing how lifelike and yet how unhumanlike these movements can be,”

Dr. Diehl observes. For example, he adds, “we pre-program behaviors through

controls so precise that we can subtly manipulate gestures so that each gesture

conveys a different meaning, the same way subtle manipulations in our own gestures

can change the meaning of what we are trying to communicate.”

Customizable Capabilities

Thanks to the interface, Dr. Villano points out, Nao’s capabilities are fully

customizable. “The behaviors associated with the button labels are driven by a

computer configuration. The result is that customized behaviors can be developed in

a relatively short time frame and targeted toward a child with whom we’re

working.”

For example, he continues, “if a child is a fan of the movie Toy Story or enjoys

swimming or any other specific activity we can develop some custom behaviors,

with companion voiceover, to match those preferences. We have one that says,

‘Hey, why don’t you watch me do my backstroke?’ Then the robot performs

backstroke movements. The kids become very excited when they see a robot

perform an activity with which they’re familiar. The configuration files allow us to

customize the activities for individual children.”

The Project’s Genesis: a Shared Interest in Human/Technology Interaction

While Dr. Diehl has a layman’s interest in technology fueled initially by

his younger brother’s use of AAC devices, for Dr. Crowell and Dr.

Villano technology is a vocation. Their shared interest: how individuals

react to technology. The result: the genesis of Notre Dame’s Nao

project.

Says Dr. Crowell, “Examination of various forms of human-technology interaction is a

major focus of our lab, whether it’s humans interacting with computers, humans

interacting with virtual environments like video or avatars on a screen, or humans

interacting with robots.”

Several years ago, he recalls, he became interested in robot-human interaction. “Dr.

Villano and I acquired a conventional robot for our lab. We were attending some

conferences where we were trying to present the early studies we conducted with

this ‘people-bot’ robot that doesn’t look very humanlike. We had long been

interested in acquiring another robot with characteristics that were somewhat more

humanlike, because we are very interested in examining the social dynamics

between humans and robots, in particular how humans might be influenced by a

piece of technology that they perceive to possess humanlike characteristics,

whether the technology is an artificially intelligent computer program or a robot with

the appearance and actions of a conventional robotic device.

“We were searching for ways to acquire a more human-looking robot. Then, last

year, at a robotics conference, we saw a French company, Aldebaran Robotics

(http://www.aldebaran-robotics.com/en) demonstrate robots named ‘Nao.’ We

were intrigued by the robots’ capabilities. Dr. Villano and I strategized on the way

home from the conference as to how we might be able to assemble the funding in

order to acquire one or two of these devices.”

They eventually obtained funding for two robots. “Even before we got them to our

lab we understood that children with ASD might possess a special affinity for a Nao

device in part because these children tend to be more object-oriented than

social. We felt that having these robots in our lab provided us with an opportunity to

convince some of our colleagues to collaborate with us to see how the use of these

robots in therapy for children with autism might play itself out. That caused us to go

next door and find Dr. Diehl and tell him about the Nao. He was very excited about

the concept and we began to assemble the current team beyond the three of us.”

Parents Ask, “What Can We Do with This Research?”

“Most of my work has been centered on understanding language

and communication difficulties in children with autism,” explains

Dr. Villano. “I heard over and over from parents, ‘There’s so much

work underway aimed at trying to understand autism, trying to

pinpoint its causes, but what can we do with this research?’”

He was already leaning toward therapy intervention work “because there was such

a need in the community for more knowledge about what works and what doesn’t.

Nao fit into what I wanted to do in terms of expanding my research in order to

provide useful information to the community while also learning more about what we

could do to enhance therapies.”

Before returning to Notre Dame, where he spent his undergraduate years, Dr. Villano

spent 15 years in research and development in industry where he sought ways to

adopt technologies and technological solutions to business problems, focusing on

human-computer interaction and employee training issues. “I was actually looking to

return to an academic environment and helped create a position for myself at Notre

Dame, which led me to resume a collaboration with Dr. Crowell that began when I

was an undergrad.” Since then Dr. Villano has taught courses on interactions

between humans and computers.

“I’ve seen recent research that intrigued me, such as Rosalind Picard’s work at MIT

(http://ist.mit.edu/sites/default/files/migration/isnews/archive/ist.v23n04.pdf),

where technologies were being developed to help children with autism identify other

people’s expressions or their own expressions. I was impressed with the ways

technologies were applied in this area.

“We had a graduate student in our department then who was developing software

that could be utilized with children with autism. The software the student was

developing was open source and was intended to be freely available to children

and their families. I was impressed with that effort. He began at a low level by

engaging other undergrad students to build small games that would be applicable.

This spurred discussions between Dr. Crowell and I about other ways in which we

could become involved in utilizing technology to impact autism.”

Will a Robot Ever Be Affordable for Families?

Will families be able to afford Nao or a robot that performs similar

tasks and offers similar therapeutic opportunities?

Says Dr. Diehl, “If we get to a point where we determine that

there are clear scientifically proven benefits in research, the next

question is, ‘What is the best way to integrate this concept in a cost-effective

manner?’

“We’re already beginning to address that question in the design of our research

studies to see if there are less expensive versions of Nao that provide equal benefit.

Nevertheless, in an economic crisis, affording Nao is difficult. If Nao works, how can

we make it available?” Perhaps, he suggests, through a recorded video screen

program that a therapist could use. “Maybe we don’t even need a robot. Maybe

there’s an animation program that is just as effective. The trouble is, we don’t have

that information yet.”

“There’s something else of interest to Dr. Crowell and me,” Dr. Villano declares. “We

are already beginning to look at alternatives to Nao. Nao is a great research and

development platform that was specifically designed for an academic

setting. There are many low-cost alternative robots that we might be able to

manage. First, we need to identify the Nao features that are most engaging and that

exert the most impact on the therapy.”

Perhaps, for example, “there’s a device that possesses Nao’s physical movements

but those movements aren’t as graceful as Nao’s. There might be less expensive but

effective robots that cost under $1,000 as opposed to Nao’s nearly $20,000 price tag.

This would be far more affordable for therapists to use for multiple children and might

be more affordable for families as well. As Dr. Diehl says, super low-cost alternatives

would provide therapists and families with a piece of software with animation, like a

video game.”

“Maybe there’s an aspect of such an advanced robot like Nao that may prove

critical. Maybe that high level of technology is what’s needed. At that point the

question is, “How do we make this technology more available?”

With tongue only half embedded in check, Dr. Crowell replies, “We may take Dr.

Diehl to France to persuade the Nao manufacturer to produce a lower cost robot

specifically for this purpose.”

Dr. Villano responds, “If the company can be convinced that it’ll get 100,000 orders

instead of 1,000 orders I’m sure it could lower the cost significantly.”

Robotic Applications for Other Children with Disabilities

Would robotic applications prove appropriate for other children with disabilities

beyond those with ASD? According to Dr. Diehl, the simple answer is yes.

“If we see the science playing towards beneficial response from kids with autism, we

can say, ‘It was easy for us to make the connection between children with autism

and robots or technology in general. It’s not that big a step to reasonably hope that

other children with developmental disabilities, especially those with social

interaction, are going to respond it similar ways.’ That would be a next application.

The programs may differ from those used for children with ASD, but that’s something

for research to point the way to.”

The Team Approach: How It Works

For Dr. Diehl, a team approach to research is the only approach he’s known during

his professional career. “The usual barriers in any team effort include the inability to

gather all team members for meetings, organizational issues and the struggle to

constantly make certain that all team members are on the same page. Fortunately,

in this project we’ve encountered very few of the usual potholes, probably because

most of us are quite experienced in team situations.

“This is a project I could never have come close to handling alone,” he admits. “This

team has so many individuals with specific areas of expertise. Each of us has his or

her own piece. It’s not like this project is one individual’s private preserve and the

other team members are only assisting. Each of us has our own areas of expertise,

from the primary researchers to the therapists to the expertise contributed by the

many students who are helping out. There is no downside to this effort.”

“Coming from industry, says Dr. Villano, “I’ve been on highly functional teams and

teams that are highly dysfunctional. This team works extraordinarily well. We have so

many skill sets that complement each other. There’s a tremendous amount of mutual

respect and lack of ego, which makes it a pleasure to get together.”

Nao Training for Children and Therapists: Control by

iPad?

Nao training for children is very basic, Dr. Diehl says. “All

that’s necessary is for children to know the ground rules

before their ABA session.” Hopefully, he adds, “those rules are no different than for

any ABA session other than there is a robot involved. For example, one of the ground

rules would be that the child cannot pick up and throw the robot. Our goal is to

integrate the robot as seamlessly as possible into the existing and widely used

behavior therapies for children.”

For the therapist, he notes, “robot control in the same room as the session is our

aspiration, replacing the current wizard located in a room adjoining the therapy

session. Eventually we’d like to take that interface and combine it with a mechanism

that can be controlled by the therapist during sessions by hand via an iPad-like

device. This would enable the therapist to have full control for the integration of the

device into the therapy session. We want to make that interface sufficiently simple

and adaptable so that only minimal training would be required for a therapist with a

basic knowledge of computers to control this robot. The ultimate goal is to make the

device as simple and straightforward as possible.”

Cool Steps: An Enhancement Wish List

Beyond therapist control, Dr. Crowell says, “there are several cool steps we can

imagine, one of which might be that the Nao doesn’t necessarily have to be

physically present during therapy sessions.” To replace Nao’s physical presence, he

points out, “all that would be needed is a video of the Nao that the therapist can

arrange to behave like the real Nao by invoking pre-recorded sequences. We might

be able to create an iPad-like tool enabling the therapist to call up the Nao in a

virtual way and get the Nao to behave like the current Nao because we would have

learned that Nao’s behaviors were the most effective methods of interaction with

the child. That’s just one of the many possible next steps.”

Dr. Villano cites Dr. Crowell’s observation that Nao possesses many sensing

capabilities. “One of the problems I see is that many people try to make the robots

autonomous or semi-autonomous. There’s an awful lot of effort expended in time

and research trying to determine how to get the robot to understand the voice of

the child, what the child is saying, or react to movements the child might make by

sensing its movement or doing face recognition.

“There is a gamut of artificial intelligence or computer science-based applications

aimed at producing a robot that is not human-controlled but instead robot-

controlled in that the robot interacts directly with the child. For me that is a very

difficult research path.”

Many promises have been made over the past 20-30 years since he was in graduate

school “that have not been satisfactorily fulfilled,” Dr. Villano declares. “In our

research we didn’t experiment with an autonomous or semi-autonomous Nao, but in

terms of possible future research directions it would be beneficial if aspects that were

successful in robot autonomy research could be implemented.”

For example, he says, “it would be very helpful if a robot could track a child’s

face. The Nao does not currently perform that function and performing it would

make the robot experience far more compelling for a child with autism. There are

similar approaches we might adopt in order to create a more integrated experience

thereby turning over a little control to the robot itself to see if that would make the

robot more engaging and even more useful in a therapy session.”

Personally, he adds, “I suspect that it is the physical embodiment of the robot in the

room that creates the most excitement for children. It’s one thing to see an image of

a robot on a computer screen – children are accustomed to that – but quite another

for kids who have never seen a robot live. The children have an incredible reaction

to the robot in the room.”

Dr. Villano and Dr. Crowell continue to utilize an industrial Peoplebot

(http://www.mobilerobots.com/researchrobots/researchrobots/peoplebot.asp)

robot in some of their research. “It resembles the robots that are visible traveling

through hospital corridors. It’s a tall open metal cylinder. We ran studies using that

robot. Some people jumped away from it. They were frightened when it came into

the room. Women responded to it very differently than did the men.”

Translating Research into Fun for Kids and Families

“I said this to some students recently when I was speaking to them

about careers in psychology: Of all the projects I’ve been on this is the

most fun and rewarding. I’ve worked on seven-year development

efforts with a core team of people in my corporate life but there is

something about doing something for other people and watching

technology benefit the greater good that produces a very different

level of satisfaction, which is a major aspect of Notre Dame’s academic mission and

one that all three of us buy into and to which we are deeply committed.”

Dr. Crowell attributes much of the team’s success so far “to my young colleague, Dr.

Diehl. Of all the people who could make or break the project he is in a position to do

it. He has responded not only with a tremendous amount of enthusiasm and energy

but also humility. He doesn’t care who gets the credit. He’s focused on getting the

project done and is committed to the research. When you have an individual like

that who is the lead person on a team with recognition and expertise in this area we

can’t help but be a success. So he’s going to make the rest of us look good.”

Concludes Dr. Diehl, “We’re having a lot of fun and really enjoying this experience.

Now the goal is to translate our enjoyment into fun for the kids and their families.”