Research Article Human-Robot Interaction Learning Using ...downloads.hindawi.com/journals/ijdsn/2013/782043.pdfe repulsion of robots can be reduced by di erent motor primitives. Di

Hindawi Publishing CorporationInternational Journal of Distributed Sensor NetworksVolume 2013 Article ID 782043 8 pageshttpdxdoiorg1011552013782043

Research ArticleHuman-Robot Interaction Learning UsingDemonstration-Based Learning and Q-Learning ina Pervasive Sensing Environment

Yunsick Sung1 Seoungjae Cho2 Kyhyun Um3 Young-Sik Jeong3

Simon Fong4 and Kyungeun Cho3

1 The Department of Game Mobile Contents Keimyung University Daegu 704-701 Republic of Korea2Department of Multimedia Engineering Graduate School of Dongguk University Seoul 100-715 Republic of Korea3 Department of Multimedia Engineering Dongguk University Seoul 100-715 Republic of Korea4Department of Computer and Information Science University of Macau Macau 3000 China

Correspondence should be addressed to Kyungeun Cho ckedonggukedu

Received 28 August 2013 Accepted 16 October 2013

Academic Editor James J Park

Copyright copy 2013 Yunsick Sung et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Given that robots provide services in any locations after they move toward humans the pervasive sensing environment can providediverse kinds of services through the robots not depending on the locations of humans For various services robots need tolearn accurate motor primitives such as walking and grabbing objects However learning motor primitives in a pervasive sensingenvironment are very time consuming Several previous studies have considered robots learning motor primitives and interactingwith humans in virtual environments Given that a robot learnsmotor primitives based on observations a disadvantage is that thereis no way of defining motor primitives that cannot be observed by a robot In this paper we develop a novel interaction learningapproach based on a virtual environmentThemotor primitives are defined by manipulating a robot directly using demonstration-based learning In addition a robot can apply119876-learning to learn interactions with humans In an experiment using the proposedmethod the motor primitives were generated intuitively and the amount of movement required by a virtual human in one of theexperiments was reduced by about 25 after applying the generated motor primitives

1 Introduction

In pervasive sensing environments robots can provide vari-ous services in an active manner Irrespective of the locationsof humans robots can provide services after they movetoward humans based on the information of the humansrsquodaily life [1]However the following problemsmay occur aftera robot learns interactionswith a human First a robot cannotlearn interactions with humans rapidly which leads to learn-ing time problems during interaction learning Therefore itis necessary to learn interactions with humans without theparticipation of humans Second an interaction between arobot and a human could injure the latter because of theincomplete perception of robotsTherefore protective equip-ment is required by humans

Previous studies have considered interaction learningwith a human in virtual environments which can solve theproblems described above [2ndash4] A virtual robot can generateits motor primitives by observing a virtual human in virtualenvironments and utilizing demonstration-based learningHowever unobservable motor primitives cannot be gener-ated In addition because of the differences in the appearanceof a robot and a human the motor primitives of robots maydiffer from the movements of humans and it might not bepossible to perform the motor primitives generated for arobot Thus the methods used to generate different motorprimitives need to be improved Further research is requiredto determine how to teach motor primitives to a robot whilelearning interactions with humans in a virtual environment

2 International Journal of Distributed Sensor Networks

In this paper we propose a virtual pervasive sensingenvironment-based interaction learning method that utilizesdemonstration-based learning to learn motor primitives and119876-learning to executemotor primitivesThemotor primitivesare defined during manipulations based on demonstrationlearning so the motor primitives can be generated intuitivelyby users who are not programmers The application of 119876-learning allows the newly generated motor primitives to beperformed without modifying any of the algorithms aftertheir production

The remainder of this paper is organized as follows Sec-tion 2 introduces demonstration-based learning approachesand virtual environment-based learning Section 3 proposesan interaction learningmethod for a virtual pervasive sensingenvironment Section 4 presents the results of interactionlearning experiments in virtual pervasive sensing environ-ments Finally we provide our conclusions in Section 5

2 Related Work

Various types of learning algorithms are required to allowrobots to interact with humans In this section we summarizerelated research into the learning of motor primitives and thelearning of interactions with humans in virtual environment

The motor primitives learned by robots are very impor-tant for achieving their goals The repulsion of robots canbe reduced by different motor primitives Different types ofresearch are ongoing to produce motor primitives for robotsthat appear more natural like those of humans For examplea related study defined natural motor primitives for followingthe shortest path [5 6] A genetic algorithm was used togenerate these movements Following mutation the motorprimitives that failed to follow the shortest path were elim-inated and new motor primitives were generated Anotherapproach is to use demonstration-based learning [7ndash9]Demonstration-based learning algorithms learn each motorprimitive separately based on repetition before analyzing thesame learned motor primitives [7 10] Another approachinvolves learning motor primitives by dividing a series ofmovements [8] where each motor primitive is defined as apart of the series of movements Furthermore an approachwas proposed that generates motor primitives as a hierarchi-cal tree [9 11] Within the same hierarchical tree a robot exe-cutes the samemotor primitive initially but executes differentmotor primitives in different states The motor primitivesare usually generated by planning algorithms [12] Howeversome problemsmay occur if planning algorithms are appliedFor example planning algorithms are defined based on thegenerated motor primitives If the motor primitives changethe planning algorithms must also be changed to executethe motor primitives An advantage of demonstration-basedlearning is that humans can define motor primitives withoutany requirement for programming However this advantagedoes not apply to planning algorithmsTherefore algorithmsare required that are not affected by changes to the motorprimitives

There is amethod that learns the interactionwith humansby utilizing motor primitives after generating the motor

Virtual robot

Virtual environment

Real-life environment

Real human

Virtual human

Real robot

(4) Deployment

(3) Collaboration learning

(1) Human modeling

(2) Motor primitive learning

(5) Collaboration

Figure 1 Process used for learning interactions

primitives using demonstration-based learning [13] A previ-ous study defined a virtual human and a virtual robot wherethe former is a virtual agent that behaves in virtual environ-ments in the same way as a human in a virtual environmentwhile the latter behaves like a real robot Therefore a virtualrobot interacts with a virtual human to learn an interactionwith a real human If a virtual human executes a motorprimitive the virtual robot also executes the motor primitiveat the same time However virtual-based interaction learninghas problems For example the motor primitives used by avirtual robot cannot be generated if a virtual human does notexecute the motor primitives because they are generated byobserving the virtual human Therefore another approach isrequired for generating motor primitives

Thus we propose a new approach for defining the motorprimitives for a virtual robot We also apply 119876-learning tosolve the problem of executing motor primitives which doesnot require any changes after the modification of motorprimitives

3 Virtual Learning Framework forHuman-Robot Interaction

31 Concept In a pervasive sensing environment it takes along time to learn interactions with humans and the numberof interactions with robots is limited Therefore the numberof interactions should be reduced to increase the amount ofthe learning to facilitate the high quality execution of motorprimitives In our approach the interactions are learned via avirtual pervasive sensing environment so no interactions arerequired in real pervasive sensing environments as shown inFigure 1

We define two types of virtual agents for learning in avirtual pervasive sensing environment a virtual human anda virtual robot The virtual human acts like a human whilethe virtual robot executes motor primitives to collaboratewith the virtual human The virtual robot learns interactionswith real humans by interacting with virtual humans Thelearning result is then embedded in the real robot The realrobot executesmotor primitives based on the results of virtuallearning to interact with a real human

International Journal of Distributed Sensor Networks 3

Table 1 Approaches used in different stages of interaction learning by robots

Stage Type of agent Learning approachMotor primitive learning Virtual robot Direct manipulation of a robotCollaboration learning Virtual robot Interaction with a virtual human by 119876-learning [13]

Motorprimitivegenerator

Policygenerator

Motormeasurer

Motorprimitiveexecutor

Robot serverReal robot


Virtual robot

Operator

Resident

Manipulation

Live

OperatorVirtual human

Manipulation

Live

Deployer

Virtual pervasive sensing environmentReal pervasive sensing environment

(1) Humanmodeling

(2) Motorprimitivelearning

(4) Deployment

(5) Collaboration

(3) Collaborationlearning

Figure 2 Framework for interaction learning

There is no requirement for interactions with realhumans The learning time problem is always invoked if ahuman is involved during learning processes which makes itvery hard to reduce the learning time However the learningtime can be reduced more by increasing the speed of inter-actions between a virtual human and a virtual robot This isbecause a virtual human and a virtual robot do not need toexecute motor primitives at the same speed as a real humanand a real robot

In our approach interaction learning includes humanmodeling motor primitive learning collaboration learningdeployment and collaboration stages In this paper weonly propose the processes used during the motor primitivelearning stage and the collaboration learning stage as showninTable 1During the humanmodeling stage humans controla virtual human to make them act like humans by execut-ing predefined motor primitives The virtual humans learnhow to execute motor primitives by analyzing the humancontrol process During the motor primitive learning stagehumans control the virtual robots directly to teach themhow to move and the virtual robots then generate their ownmotor primitives Next the virtual robot interacts with avirtual human by executing the learnt motor primitivesDuring this interaction the virtual robot learns how toprovide services to humansThe results obtained frommotorprimitive generation and from interactions are then appliedin a real robot which can interact with real humans

32 Human-Robot Interaction Framework and Processes Theroles of real humans are divided into two groups duringwhole learning processes one for residents and the otherfor operators Operators teach real robots while residents

live in pervasive sensing environments All of the virtualhumans in the virtual pervasive sensing environment arevirtual residents We also define a robot server as a serverthat generates motor primitives and policies which transfersdata between a real robot and a virtual robot Our proposedframework is shown in Figure 2

First an operator controls a virtual human via a userinterfaceDuring themotor primitive learning stage there aretwo modules in a real robot a motor measurer and a motorprimitive generatorThemotor measurer is deployed in a realrobot When the operator manipulates a real robot directlythe motor measurer determines the degrees of the joints inthe real robotThemotor primitive generator is embedded inthe robot server rather than the real robot which separatesthe dependency of the motor primitive generator from therobot platformThe generated motor primitives are deployedin the virtual robot and the real robot

During the collaboration learning stage a policy gener-ator and a motor primitive executor are utilized to learn theinteractions between a resident and a real robot based on theinteractions that occur between a virtual human and a virtualrobot The motor primitive executor executes the generatedmotor primitives and the policy generator then generates theresults of the interaction The interaction results are thendeployed in the real robot Finally the real robot can providevarious services by executing the motor primitives based onthe interaction learning results

In our approach a robot executes multiple motor primi-tives119872

119894is the 119894th motor primitive A motor primitive is de

fined as a part of a series of movements which is described bymultiple joints of the robotTherefore119872

119894comprisesmultiple

joints The 119896th joint of the 119894th motor primitive is defined


M1

M

Time t11 t12 t13 t14 t1M11 M111 M112 M113 M114 M11

middot middot middot

M1120585 M11205851 M11205852 M11205853 M11205854 M1120585

M2

Time

t21 t22 t23 t24 t2M21 M211 M212 M213 M214 M21


M2120585 M21205851 M21205852 M21205853 M21205854

Mi

Time

ti1 ti2 ti3 ti4 tiMi1 Mi11 Mi12 Mi13 Mi14 Mi1


Mi120585 Mi1205851 Mi1205852 Mi1205853 Mi1205854 Mi120585

M2120585


Figure 3 Configuration of the motor primitive set

by 119872119894119896 119872119894119896ℎ

is the ℎth measured 119872119894119896 If 120585 is the number

of joints 119872119894is ⟨119872

1198941 119872

119894119896 119872

119894120585⟩ Each joint moves

irregularly 119905119894ℎ

denotes the time when 119872119894119896ℎ

is executedFinally the setM is a motor primitive set Figure 3 shows theexample of the configuration of the motor primitive set

To eliminate any differences between motor primitives ofa virtual robot and a real robot themotor primitive generatorgenerates the same motor primitives for both To reducethe number of movements measured any movements areeliminated that do not change as much as the difference cal-culated using (1) After similarmovements are eliminated themotor primitives are generated using the remaining mea-sured movements Consider

(1198721198941198961minus119872119894119896minus11)2+ (1198721198941198962minus119872119894119896minus12)2+ sdot sdot sdot lt 120575

2 (1)

Given that pervasive sensing environment is usually complexthe policy generator used by our approach utilizes119876-learning[14] to execute the generated motor primitives because 119876-learning has the advantage that a model of the environmentdoes not need to be defined In addition 119876-learning algo-rithmdoes not need to bemodified after themotor primitivesare generated The policy generator selects motor primitivesdepending on the current state 119904 and sends the selectedmotorprimitive to themotor primitive executor for execution Afterexecuting each motor primitive the corresponding reward ofthe executed motor primitives is calculated and transferredback to the policy generatorThe policy generator updates theQ-values with the reward using

119876 (119904119872) larr997888119876 (119904119872) + 120572

times 119903 + 120574 timesmax119876(11990410158401198721015840) minus 119876 (119904119872) (2)

where 119872 is an executed motor primitive 119904 is a state 119903 is areward after executing 119872 1199041015840 and 1198721015840 are the next state andthe next motor primitive respectively 120572 denotes the learningrate and 120574 is a discount factor

The motor primitive executor receives motor primitivesfrom the motor primitive generator and executes the motorprimitives according to the decisions made by the policygenerator After executing the motor primitives the corre-sponding reward of the executed motor primitives is trans-ferred to the policy generator

4 Experiment

41 Configurations of the Real and Virtual Pervasive SensingEnvironments In our experiment we used a Nao as a realrobot We also built a model house which was a suitable sizefor theNao as shown in Figure 4Themodel house containeda kitchen living room and bedroomTheNao learned duringinteractions with a real human

The objective of the Nao was to transfer the objectsrequired by a real human After recognizing the object theNao moved toward the object initially Next it grabbed theobject moved toward the real human and gave the object tothe real human In the experiments we used the objectsshown in Table 2 There were two types of objects staticobjects that could not be moved and movable objects whicha Nao and a human could grab carry and put down

The state space must be defined in advance to use 119876-learning In this experiment we denoted the positions of thehuman and the robot based on their grid coordinates aftertaking a picture using an omnicamera placed on the ceilingand dividing the picture into the grid shown in Figure 5 Thesize of each cell was set to the width of the NaoThus 50 cellswere defined We defined each state based on the coordinatesof the human the robot and the object located nearest to thehuman

To learn interactions between a real human and a realrobot the virtual pervasive sensing environment used in thisexperiment was modeled in exactly the same way as thereal pervasive sensing environment as shown in Figure 6Therefore the structure and size of the virtual pervasivesensing environment were the same as the real pervasive


Kitchen

(a)

Living room

(b)

Bedroom

(c)

Figure 4 Model house as a pervasive sensing environment

Table 2 Objects used in the experiments

Location Object Object type

Kitchen

Cup Movable objectKettle Movable objectChair Movable object

Kitchen table Static objectStove Static object

Living room

TV table Static objectTV (assumed) Static object

Couch Static objectRemote controller Movable object

Newspaper Movable objectRoom Bed Static object

Figure 5 Grid environment of the real pervasive sensing environ-ment used for interaction learning

sensing environment Objects were also deployed in the sameway as the real pervasive sensing environment We utilizedtwo virtual agents as a virtual human and a virtual robot

42 Configuration of the Motor Primitives A real operatorcontrolled a virtual robot while a virtual human and a robotserver were also used depending on the stage The robotfollowed a different process during each stage and the real

Figure 6 Virtual pervasive sensing environment used for interac-tion learning

Table 3 Predefined motor primitives for a virtual robot and a realrobot

Notation Name Description

1198720

Standing beforegrabbing

If a real robot has notgrabbed an object it standsand waits to execute thenext motor primitive

1198721

Standing aftergrabbing

If a real robot has grabbedan object it stands andwaits to execute the next

motor primitive

11987210 Walking

A real robot follows a ballwhile remaining at a fixeddistance from the ball

operator also controlled the state of the real robot by touchinga touch sensor on the head of the real robot

Themotor primitives of the robot were defined as followsThe real operator manipulated the robot directly to make therobot learn the motor primitives There were two types ofmotor primitives First a type of motor primitive was pre-defined by programming as shown in Table 3 For examplegiven that an initial motor primitive was required and thatit was very hard to define a walking motor primitive by


Table 4 Virtual human animations

Name DescriptionStanding Standing with arms down

One-hand grabbing Stretching arms grabbing objects andcarrying objects while standing

One-hand placing Stretching arms and placing one of thegrabbed objects while standing

Touching Turning the switch of a light or stoveon or off

Receiving Receiving an object with the right handGiving Giving an object with the right handWalking Walking toward a specific objectSitting Sitting on a chair or couchLaying Laying down on a bed

manipulation the real robot executed two preprogrammedstanding motor primitives and one walking motor primitiveThe other type of motor primitive was defined by themanipulations performed by the operator

For the walking motor primitive the algorithm deter-mined a path from the current coordinates to specific coor-dinates We used the 119860lowast search algorithm because the gridsof the virtual and real pervasive sensing environments werenot complex and they only comprised 50 cells For exampleif a real operator was in the specific position where a virtualhuman needed to move the virtual human moved to theposition while avoiding objects and walls

While the real robot was learning the motor primitivesthe real robot measured its joints every 500ms and trans-ferred the values of the joints to the robot server If the intervalis set under 500ms the joints are not measured accuratelywhich delays the performance of the real robot

We predefined the animation of the virtual human asshown in Table 4 The objective of the Nao was to transferobjects for a virtual human so the animation of the virtualhuman also focused on transferring objects

43 Motor Primitive Generation Experiment The first exper-iment aimed to generate motor primitives for the Nao Anoperator defined the motor primitives from 119872

2to 1198727by

manipulating the arms and touching the touch sensors on thearms as shown in Table 5 In this experiment the operatoronly controlled the arms because the legs only moved whenthe robot walked

The real robot executed a series of motor primitives Theend of a motor primitive was connected to the end of thenext motor primitive in a natural mannerThus the standingmotor primitives were executed after each motor primitiveand the next motor primitive started after the end of thestandingmotor primitiveTherefore we defined the sequenceof motor primitives as shown in Figure 7

Some of motor primitives could not be connected withthe standing motor primitive because of the grabbed objectsTherefore standing after grabbing was added Standing after

Table 5 Motor primitives learned during the manipulations


1198722

One-hand grabbingStretching arms

grabbing objects andcarrying objects while

standing

1198723

One-hand placingStretching arms andplacing one of the

grabbed objects whilestanding

1198724

TouchingTurning the switch ofa light or stove on or

off1198725

(Reserved)

1198726

Receiving Receiving an objectwith the right hand

1198727

Giving Giving an object withthe right hand

Walking

Touching

Standing not after grabbing

One-handplacinggiving

Receivingone-handgrabbing

Standingafter grabbing

Walking

Figure 7 Sequential relationships among the motor primitives

grabbing was performed after executing receiving or one-hand grabbing followed by one-hand placing or giving

Each motor primitive was generated based on separatemanipulation performed by a real human Figure 8 showsfour of the generated motor primitives Only five joints weremeasured which were all related to the right hand Thegenerated motor primitive was then performed by the virtualrobot

44 Interaction Learning Experiment We specified a scenariofor learning the interactions First we applied our approachto the scenario where a human stood up sat on a couch andthen read a newspaper after picking it up as shown in thefollowing list (a)

Interaction Learning Results

(a) Scenario where a virtual human lives alone is as fol-lows

(i) a virtual human sleeps(ii) the human wakes up on a bed(iii) the human walks to a couch(iv) the human sits on the couch for a while(v) the human stands up on the couch


minus05

0

05

1

15

2

RShoulderPitchRShoulderRollRElbowYaw

RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand

(b) One-hand grabbing

minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand

(c) One-hand placing

minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving

Figure 8 Four motor primitives produced for a robot

(vi) the human walks to a newspaper(vii) the human picks up the newspaper and(viii) the human reads the newspaper

(b) Scenario where a virtual robot provides services is asfollows

(i) a virtual human sleeps(ii) the human wakes up on a bed(iii) the human walks to a couch(iv) while the human sits on the couch

(1) a virtual robot walks to a newspaper and(2) picks up the newspaper

(v) when the human stands up on the couch(1) the robot walks to a virtual human and(2) gives the newspaper

(vi) the human receives the newspaper and(vii) the human reads the newspaper

Figure 9 shows the accumulated rewards according to theincrease in the amount of interaction learning After 14000

0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning

Figure 9 A virtual robot delivers a newspaper to a virtual human

the robot started to learn the interaction The previous list(b) shows the changed scenario by the virtual robot based onthe result of the interaction after the interaction learning Ifa virtual human lived alone the virtual human walked to thenewspaper and picked it up for itself However if a virtualrobot was present the virtual robot walked to the newspaperand picked it up then walked to the virtual human and gaveit the newspaper


5 Conclusion

In this paper we developed an approach to virtual pervasivesensing environment-based interaction learning where theoperators taught motor primitives to a real robot by manipu-lating its arms directlyThe learnedmotor primitiveswere uti-lized by a virtual robot and executed to learn interactionswitha human The operators defined the motor primitives usingmanipulations so various different types of motor primitivescould be defined intuitively which overcame the problems ofprevious approaches

The virtual human and the virtual robot used in our pro-posed method and 119876-learning are suitable for single agent-based learning algorithms so it is necessary to improve ourproposedmethod by applyingmulti-agent-based119876-learningA method is also required to allow a virtual robot to provideservices to multiple virtual humans Finally an approachwill be developed to facilitate the application of the learnedinteraction results to a real robot

Acknowledgments

This work was supported by the Basic Science ResearchProgram through the National Research Foundation ofKorea (NRF) funded by the Ministry of Education Scienceand Technology (2011-0011266) And this work was alsosupported by the Basic Science Research Program throughthe National Research Foundation of Korea (NRF) fundedby the Ministry of Education Science and Technology(2012R1A1A2009148)

References

[1] T Teraoka ldquoOrganization and exploration of heterogeneouspersonal data collected in daily liferdquoHuman-Centric Computingand Information Sciences vol 2 no 1 pp 1ndash15 2012

[2] M Lim and Y Lee ldquoA simulation model of object movementfor evaluating the communication load in networked virtualenvironmentsrdquo Journal of Information Processing Systems vol9 no 3 pp 489ndash498 2013

[3] H T Panduranga S K Naveen Kumar and H S SharathKumar ldquoHardware software co-simulation of the multipleimage encryption technique using the xilinx system generatorrdquoJournal of Information Processing Systems vol 9 no 3 p 4992013

[4] Y Sung and K Cho ldquoCollaborative programming by demon-stration in a virtual environmentrdquo IEEE Intelligent Systems vol27 no 2 pp 14ndash17 2012

[5] S Ra G Park C H Kim and B-J You ldquoPCA-based geneticoperator for evolving movements of humanoid robotrdquo inProceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo08) pp 1219ndash1225 Hong Kong China June 2008

[6] G Park S Ra C Kim and J-B Song ldquoImitation learning ofrobot movement using evolutionary algorithmrdquo in Proceedingsof the 17th World Congress International Federation of Auto-matic Control (IFAC rsquo08) pp 730ndash735 Seoul Republic of KoreaJuly 2008

[7] S Calinon F Guenter and A Billard ldquoOn learning repre-senting and generalizing a task in a humanoid robotrdquo IEEE

Transactions on Systems Man and Cybernetics B vol 37 no 2pp 286ndash298 2007

[8] N Koenig and M J Mataric ldquoBehavior-based segmentation ofdemonstrated taskrdquo in Proceedings of International Conferenceon Development and Learning (ICDL rsquo06) 2006

[9] M N Nicolescu and M J Mataric ldquoNatural methods for robottask learning instructive demonstrations generalization andpracticerdquo inProceedings of the 2nd International Joint Conferenceon Autonomous Agents and Multiagent Systems (AAMAS rsquo03)pp 241ndash248 Melbourne Australia July 2003

[10] S Calinon and A Billard ldquoA probabilistic programming bydemonstration framework handling constraints in joint spaceand task spacerdquo in Proceedings of the IEEERSJ InternationalConference on Intelligent Robots and Systems (IROS rsquo08) pp367ndash372 Nice France September 2008

[11] M N Nicolescu and M J Mataric ldquoExtending behavior-basedsystems capabilities using an abstract behavior representationrdquoin Proceedings of the AAAI Fall Symposium on Parallel Congni-tion pp 27ndash34 2000

[12] M J Mataric ldquoSensory-motor primitives as a basis for imi-tation linking perception to action and biology to roboticsrdquoin Imitation in Animals and Artifacts pp 391ndash422 MIT Press2000

[13] Y Sung and K Cho ldquoA method for learning macro-actionsfor virtual characters using programming by demonstrationand reinforcement learningrdquo Journal of Information ProcessingSystems vol 8 no 3 pp 409ndash420 2012

[14] C J C H Watkins and P Dayan ldquoQ-learningrdquo MachineLearning vol 8 no 3-4 pp 279ndash292 1992

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014


Active and Passive Electronic Components

Control Scienceand Engineering

Journal of



RotatingMachinery


Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design



Shock and Vibration


Civil EngineeringAdvances in

Acoustics and VibrationAdvances in



Electrical and Computer Engineering

Journal of

Advances inOptoElectronics


Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of


Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014


Chemical EngineeringInternational Journal of Antennas and

Propagation




Navigation and Observation



DistributedSensor Networks



In this paper we propose a virtual pervasive sensingenvironment-based interaction learning method that utilizesdemonstration-based learning to learn motor primitives and119876-learning to executemotor primitivesThemotor primitivesare defined during manipulations based on demonstrationlearning so the motor primitives can be generated intuitivelyby users who are not programmers The application of 119876-learning allows the newly generated motor primitives to beperformed without modifying any of the algorithms aftertheir production

The remainder of this paper is organized as follows Sec-tion 2 introduces demonstration-based learning approachesand virtual environment-based learning Section 3 proposesan interaction learningmethod for a virtual pervasive sensingenvironment Section 4 presents the results of interactionlearning experiments in virtual pervasive sensing environ-ments Finally we provide our conclusions in Section 5

2 Related Work

Various types of learning algorithms are required to allowrobots to interact with humans In this section we summarizerelated research into the learning of motor primitives and thelearning of interactions with humans in virtual environment

The motor primitives learned by robots are very impor-tant for achieving their goals The repulsion of robots canbe reduced by different motor primitives Different types ofresearch are ongoing to produce motor primitives for robotsthat appear more natural like those of humans For examplea related study defined natural motor primitives for followingthe shortest path [5 6] A genetic algorithm was used togenerate these movements Following mutation the motorprimitives that failed to follow the shortest path were elim-inated and new motor primitives were generated Anotherapproach is to use demonstration-based learning [7ndash9]Demonstration-based learning algorithms learn each motorprimitive separately based on repetition before analyzing thesame learned motor primitives [7 10] Another approachinvolves learning motor primitives by dividing a series ofmovements [8] where each motor primitive is defined as apart of the series of movements Furthermore an approachwas proposed that generates motor primitives as a hierarchi-cal tree [9 11] Within the same hierarchical tree a robot exe-cutes the samemotor primitive initially but executes differentmotor primitives in different states The motor primitivesare usually generated by planning algorithms [12] Howeversome problemsmay occur if planning algorithms are appliedFor example planning algorithms are defined based on thegenerated motor primitives If the motor primitives changethe planning algorithms must also be changed to executethe motor primitives An advantage of demonstration-basedlearning is that humans can define motor primitives withoutany requirement for programming However this advantagedoes not apply to planning algorithmsTherefore algorithmsare required that are not affected by changes to the motorprimitives

There is amethod that learns the interactionwith humansby utilizing motor primitives after generating the motor

Virtual robot

Virtual environment

Real-life environment

Real human

Virtual human

Real robot

(4) Deployment

(3) Collaboration learning

(1) Human modeling

(2) Motor primitive learning

(5) Collaboration

Figure 1 Process used for learning interactions

primitives using demonstration-based learning [13] A previ-ous study defined a virtual human and a virtual robot wherethe former is a virtual agent that behaves in virtual environ-ments in the same way as a human in a virtual environmentwhile the latter behaves like a real robot Therefore a virtualrobot interacts with a virtual human to learn an interactionwith a real human If a virtual human executes a motorprimitive the virtual robot also executes the motor primitiveat the same time However virtual-based interaction learninghas problems For example the motor primitives used by avirtual robot cannot be generated if a virtual human does notexecute the motor primitives because they are generated byobserving the virtual human Therefore another approach isrequired for generating motor primitives

Thus we propose a new approach for defining the motorprimitives for a virtual robot We also apply 119876-learning tosolve the problem of executing motor primitives which doesnot require any changes after the modification of motorprimitives

3 Virtual Learning Framework forHuman-Robot Interaction

31 Concept In a pervasive sensing environment it takes along time to learn interactions with humans and the numberof interactions with robots is limited Therefore the numberof interactions should be reduced to increase the amount ofthe learning to facilitate the high quality execution of motorprimitives In our approach the interactions are learned via avirtual pervasive sensing environment so no interactions arerequired in real pervasive sensing environments as shown inFigure 1

We define two types of virtual agents for learning in avirtual pervasive sensing environment a virtual human anda virtual robot The virtual human acts like a human whilethe virtual robot executes motor primitives to collaboratewith the virtual human The virtual robot learns interactionswith real humans by interacting with virtual humans Thelearning result is then embedded in the real robot The realrobot executesmotor primitives based on the results of virtuallearning to interact with a real human





Policygenerator

Motormeasurer




Virtual robot

Operator

Resident

Manipulation

Live


Manipulation

Live

Deployer


(1) Humanmodeling


(4) Deployment

(5) Collaboration















M1

M



M1120585 M11205851 M11205852 M11205853 M11205854 M1120585

M2

Time

t21 t22 t23 t24 t2M21 M211 M212 M213 M214 M21


M2120585 M21205851 M21205852 M21205853 M21205854

Mi

Time



Mi120585 Mi1205851 Mi1205852 Mi1205853 Mi1205854 Mi120585

M2120585



by 119872119894119896 119872119894119896ℎ


of joints 119872119894is ⟨119872

1198941 119872

119894119896 119872







2 (1)


119876 (119904119872) larr997888119876 (119904119872) + 120572




4 Experiment






Kitchen

(a)

Living room

(b)

Bedroom

(c)




Kitchen



Living room










1198720



1198721



motor primitive

11987210 Walking
















2to 1198727by






1198722



standing

1198723



1198724


off1198725

(Reserved)

1198726


1198727


Walking

Touching





Walking









minus05

0

05

1

15

2


RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand


minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand


minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving









0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning




5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation













Policygenerator

Motormeasurer




Virtual robot

Operator

Resident

Manipulation

Live


Manipulation

Live

Deployer


(1) Humanmodeling


(4) Deployment

(5) Collaboration















M1

M



M1120585 M11205851 M11205852 M11205853 M11205854 M1120585

M2

Time

t21 t22 t23 t24 t2M21 M211 M212 M213 M214 M21


M2120585 M21205851 M21205852 M21205853 M21205854

Mi

Time



Mi120585 Mi1205851 Mi1205852 Mi1205853 Mi1205854 Mi120585

M2120585



by 119872119894119896 119872119894119896ℎ


of joints 119872119894is ⟨119872

1198941 119872

119894119896 119872







2 (1)


119876 (119904119872) larr997888119876 (119904119872) + 120572




4 Experiment






Kitchen

(a)

Living room

(b)

Bedroom

(c)




Kitchen



Living room










1198720



1198721



motor primitive

11987210 Walking
















2to 1198727by






1198722



standing

1198723



1198724


off1198725

(Reserved)

1198726


1198727


Walking

Touching





Walking









minus05

0

05

1

15

2


RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand


minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand


minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving









0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning




5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










M1

M



M1120585 M11205851 M11205852 M11205853 M11205854 M1120585

M2

Time

t21 t22 t23 t24 t2M21 M211 M212 M213 M214 M21


M2120585 M21205851 M21205852 M21205853 M21205854

Mi

Time



Mi120585 Mi1205851 Mi1205852 Mi1205853 Mi1205854 Mi120585

M2120585



by 119872119894119896 119872119894119896ℎ


of joints 119872119894is ⟨119872

1198941 119872

119894119896 119872







2 (1)


119876 (119904119872) larr997888119876 (119904119872) + 120572




4 Experiment






Kitchen

(a)

Living room

(b)

Bedroom

(c)




Kitchen



Living room










1198720



1198721



motor primitive

11987210 Walking
















2to 1198727by






1198722



standing

1198723



1198724


off1198725

(Reserved)

1198726


1198727


Walking

Touching





Walking









minus05

0

05

1

15

2


RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand


minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand


minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving









0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning




5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










Kitchen

(a)

Living room

(b)

Bedroom

(c)




Kitchen



Living room










1198720



1198721



motor primitive

11987210 Walking
















2to 1198727by






1198722



standing

1198723



1198724


off1198725

(Reserved)

1198726


1198727


Walking

Touching





Walking









minus05

0

05

1

15

2


RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand


minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand


minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving









0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning




5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation





















2to 1198727by






1198722



standing

1198723



1198724


off1198725

(Reserved)

1198726


1198727


Walking

Touching





Walking









minus05

0

05

1

15

2


RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand


minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand


minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving









0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning




5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










minus05

0

05

1

15

2


RElbowRollRHand

(a) Giving

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31minus05

0

05

1

15

2


RElbowRollRHand


minus05

0

05

1

15

2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29


RElbowRollRHand


minus05

0

05

1

15

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21


RElbowRollRHand

(d) Receiving









0

20000

40000

60000

80000

100000

112

5225

0337

5450

0562

5675

0787

5810

009

1126

012

511

1376

215

013

1626

417

515

1876

620

017

2126

822

519

2377

025

021

2627

227

523

2877

430

025

3127

6

Accu

mul

ated

rew

ards

Number of learning




5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation










5 Conclusion



Acknowledgments


References


















RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation











RoboticsJournal of





Journal of



RotatingMachinery





VLSI Design



Shock and Vibration







Journal of



Volume 2014


SensorsJournal of





Propagation









Documents

Research Article Human-Robot Interaction Learning Using ...downloads.hindawi.com/journals/ijdsn/2013/782043.pdfe repulsion of robots can be reduced by di erent motor primitives. Di