Phase Estimation for Fast ActionRecognition and Trajectory Generationin Human-Robot Collaboration

Journal TitleXX(X):1–16c©The Author(s) 2015

Reprints and permission:sagepub.co.uk/journalsPermissions.navDOI: 10.1177/ToBeAssignedwww.sagepub.com/

Guilherme Maeda1, Marco Ewerton1, Gerhard Neumann2, Rudolf Lioutikov1 and JanPeters1,3

AbstractThis paper proposes a method to achieve fast and fluid human-robot interaction by estimating the progress ofthe movement of the human. The method allows the progress, also referred to as the phase of the movement,to be estimated even when observations of the human are partial and occluded; a problem typically found whenusing motion capture systems in cluttered environments. By leveraging on the framework of Interaction ProbabilisticMovement Primitives (ProMPs), phase estimation makes it possible to classify the human action, and to generate acorresponding robot trajectory before the human finishes his/her movement. The method is therefore suited for semi-autonomous robots acting as assistants and coworkers. Since observations may be sparse, our method is basedon computing the probability of different phase candidates to find the phase that best aligns the Interaction ProMPwith the current observations. The method is fundamentally different from approaches based on Dynamic TimeWarping (DTW) that must rely on a consistent stream of measurements at runtime. The phase estimation algorithmcan be seamlessly integrated into Interaction ProMPs such that robot trajectory coordination, phase estimation, andaction recognition can all be achieved in a single probabilistic framework. We evaluated the method using a 7-DoFlightweight robot arm equipped with a 5-finger hand in single and multi-task collaborative experiments. We comparethe accuracy achieved by phase estimation with our previous method based on DTW.

KeywordsClass file, LATEX 2ε, SAGE Publications

1 Introduction

Assistive and collaborative robots must have the abilityto physically interact with the human, safely andsynergistically. However, pre-programming a robot for alarge number of tasks is not only tedious, but unrealistic,especially if tasks are added or changed constantly.Moreover, conventional programming methods do notaddress semi-autonomous robots—robots whose actionsdepend on the actions of a human partner. Once deployed,for example in a domestic or small industrial environment,a semi-autonomous robot must be easy to program, withoutrequiring the need of a dedicated expert. For this reason,this paper proposes the use of interaction learning, a data-driven approach based on the use of imitation learning(Schaal 1999) for learning tasks that involve human-robot interaction. In this paper, we exploit the benefits ofaugmenting the interaction learning method with a temporalmodel of the distribution of phases of a human movement.

An important aspect of collaborative robots is the abilityto recognize the action of the human, and to quickly

generate a corresponding robot trajectory that matches thepredicted human trajectory. As illustrated in Figure 1, byobserving the movement of the human, a robot partner mustnot only decide if it should hand over a screw, a plate, orhold a screwdriver; but once the action is recognized, therobot must spatially coordinate its trajectory w.r.t the humanmovement. Probabilistic models have been used to addresseither action recognition or trajectory generation, but fortheir realization, many methods require the time alignmentof training data such that the spatial correlation can beproperly captured (Calinon et al. 2007; Ye and Alterovitz

1 Intelligent Autonomous Systems Group, TU Darmstadt, Germany2 Computational Learning for Autonomous Systems Group, TUDarmstadt, Germany3 Max Planck Institute for Intelligent Systems, Tuebingen, Germany

Corresponding author:Guilherme Maeda, Intelligent Autonomous Systems Group, TUDarmstadt, GermanyEmail: maeda@ias.tu-darmstadt.de

Prepared using sagej.cls [Version: 2015/06/09 v1.01]

2 Journal Title XX(X)

Act

ion

reco

gniti

on

Screw handover Hold toolPlate handover

Tra

ject

ory

coor

dina

tion

Figure 1. A robot coworker must recognize the intention of the human before deciding which action to take. In this case, to handover a screw, to hand over a plate, or to hold the tool for the human partner. Once the action decision is made, the robot mustthen coordinate its trajectory with the trajectory of the human, such that a handover of a plate is successful, for example. Thegoal of this paper is to allow the robot to quickly take such actions at the early stages of the human movement, possibly underpartial occlusion of the human movement.

2011; Dong and Williams 2012; Ben Amor et al. 2014;Perez-D’Arpino and Shah 2015). Under severe changes inthe duration of human trajectories, such models requirereliable measurements of positions for online alignmentof the observations before the interaction model can bequeried. This requirement poses a problem of practicalimportance since occlusions and interrupted streams ofposition are prone to occur in many of the collaborativeenvironments of interest such as hospitals, homes andfactories. Other methods will be discussed in the relatedwork section.

Instead of systematically computing distances betweentrajectories for an exact time-alignment as in dynamictime warping (DTW), the principle of our proposedmethod is to test phase candidates by computing thelikelihood of models with different durations. Thisapproach allows for querying the model with a minimumnumber of observations. The general idea is illustratedin Figure 2 where training data collected with differentdurations are first normalized to a single nominalduration. A probabilistic representation of movementprimitives (Paraschos et al. 2013) is used to learn theparameters of the distribution under the normalized time.During execution, the method then computes temporalvariations of the model, which are governed by the phase.From Figure 2, it is clear that given the same observations,the model with duration T1 is more likely to represent thecurrent human motion than the model with duration T2.

To achieve coordination with a robot partner, the methodof Interaction ProMPs are used (Maeda et al. 2016), whichallows the robot trajectory to be generated in conjunctionwith the prediction of the human motion. Basically, anInteraction ProMP provides a model that correlates the

T2

T1

Training data Probabilistic model

T

Execution

Figure 2. A cartoon of the phase estimation problem: undersparse observations there may be many ways in which theobservation can be explained by variants of a model that differby a temporal scaling. Each variation leads to different humanprediction and collaborative robot trajectories. The mostprobable time scaling is assumed to generate the correcthuman-robot coordination.

parameters that describe elementary trajectories of a humanand a robot when executing a task in collaboration. Usinga probabilistic treatment, the trajectory of the robot isgenerated by inference, where the Interaction ProMP isconditioned on the observations of the human. The robotstates are subsequently sampled from the resulting posteriordistribution.

Prepared using sagej.cls

Maeda et al. 3

The contribution of this paper is a single probabilisticframework that allows a robot to estimate the phase of thehuman movement online, and to associate the outcome ofthe estimation to address different tasks and their respectiverobot motions. Phase estimation not only allows the robotto react faster—as the trajectory inference can be done withpartial and even occluded observations—but also eliminatesthe need for time-alignment. An initial version of this paperwas presented in Maeda et al. (2015) where the algorithmfor phase estimation was introduced. Here, we provide amore elaborated an extended version with new experimentalresults and a significant number of additional evaluations.The full probabilistic framework including its relation withInteraction ProMPs is also explained in more detail.

This paper is organized as follows. Section 2 addressesthe relevant literature related to the framework with anemphasis on phase and time representations. Section 3describes the proposed method with a brief backgroundon ProMPs, followed by Interaction ProMPs, phaseestimation, and action recognition. Finally, Section 4provides experiments and discussions on the application ofthe method in a single-task handover experiment and in amulti-task assembly scenario.

2 Related WorkThis section initially discusses approaches that addressthe problem of robot movement generation or humanaction recognition under temporal variations. The usualapproach is based on time alignment of trajectories. Otherworks, however, address time as transition probabilities,or consider phase as a dynamical system. An importantdifference with our work, is that the great majorityeither address the problem of movement generation oraction recognition, thus lacking a principled, unified wayto connect these two problems. Due to the practicalapplication, this section also briefly discusses the handoverliterature.

Dynamic Time Warping (DTW) (Sakoe and Chiba1978) has been used in many robotics applications fortemporally aligning trajectories. For example, as part ofan algorithm that estimates the optimal, hidden trajectoryprovided by multiple expert demonstrations (Coates et al.2008) or as a pre-processing step for the generation ofprobabilistic representations, notably Gaussian MixtureModels (GMM) (Calinon et al. 2007; Ye and Alterovitz2011) but also other forms of representations (Dong andWilliams 2012; Perez-D’Arpino and Shah 2015). AlthoughDTW is suitable for off-line processing of data, its onlineapplication is challenging as its principle mechanismrelies on systematic distance computations between fulltrajectories. Dong and Williams (2012) and Ben Amoret al. (2014) presented DTW formulations for onlineapplications useful when the stream of data is consistent.

Under partial/occluded observations, as illustrated in Figure2, the use of DTW becomes impractical.

Calinon et al. (2007, 2012) propose explicitly encodingtime dependency in a mixture model. Thus, by directlyconditioning the model on time, smooth temporal solutionsusing Gaussian Mixture Regression can be achieved. In ourwork, however, observations are provided by the human—as opposed to the movements of a controlled robot. Dueto the many ways the human can change the speed, thetime index does not reflect the phase of the movement,hindering the possibility to condition the model directly. Infact, an important difference of our work is the coordinationbetween human and robot, rather than the single agentrobot scenario. Hidden Markov Models (HMMs) have alsobeen used in (Calinon et al. 2006) and in (Lee andOtt 2011) particularly to avoid the pre-processing of dataduring online execution. While the temporal informationis addressed by the transition probabilities of the HMM,HMMs alone does not suffice to completely representcontinuous distribution of trajectories, often requiringadditional mechanisms to overcome its discrete nature.

The temporal alignment or the phase estimation problemcan be alleviated when velocities or the goal of the humantrajectory are known. The measurement or estimation ofvelocity, for example, by differentiation of a consistentstream of positions, eliminates the ambiguity illustratedin Figure 2 and allows for the realization of onlinealgorithms that cope with very fast dynamics (Kimet al. 2010, 2014). Such methods, however, rely ona planned environment free from occlusions and fasttracking capabilities; requirements difficult to achieve inenvironments where semi-autonomous robots are expectedto make their biggest impact, such as in small factories,hospitals and home care facilities. Englert and Toussaint(2014) presented a method to reactively adapt trajectoriesof a motion planner due to changes in the environment.This was achieved by measuring the progress of a task witha dynamic phase variable. While this method is suited forcases where the goal is known—as the phase is estimatedfrom the distance to the goal—a semi-autonomous robotis not provided with such information: the goal must beinferred from the observation of the human movement,which in turn requires an estimate of the phase.

Dynamical Movement Primitives (DMPs) have theability to modulate temporal variations with a phasevariable (Ijspeert et al. 2013). The phase variable is usedto govern the spread of a fixed number of basis functionsthat encode parameters of a forcing function. Recently, amodified form of DMPs where the rate of phase changeis related to the speed of movement has been presentedby Vuga et al. (2014). The method uses ReinforcementLearning and Iterative Learning Control to speed up the

Prepared using sagej.cls

4 Journal Title XX(X)

execution of a robot’s movement without violating pre-defined constraints such as carrying a glass full of liquidwithout spilling it. A similar form of iterative learningwas used to learn the time mapping between demonstratedtrajectories and a reference trajectory (Van Den Berg et al.2010). With their approach, a robot was able to perform asurgical task of knot-tie faster than the human demonstrator.Although such methods exploit phase or durations to adapta velocity profile, they do not address the inverse problemof estimating the phase itself. ProMPs use the concept ofphases in the same manner as DMPs, with the differencethat the basis functions are used to encode positions. Thisdifference is fundamental for the tractability of InteractionPrimitives since estimating the forcing function of thehuman (a DMP requirement) is nontrivial in practice.

Several works have addressed the action recognitionproblem. Graphical models, in particular, have been widelyused. In human-robot interaction, HMMs have been usedhierarchically to represent states and to trigger low-levelprimitives (Lee et al. 2010). HMMs were also applied topredict the positions of a coworker in an assembly linefor tool delivery (Tanaka et al. 2012) while in Koppulaand Saxena (2013), Conditional Random Fields wereused to predict the possible actions of a human. Theprediction of actions and movements of human coworkershave been addressed by many authors using probabilisticapproaches to model trajectories (Dong and Williams 2012;Mainprice and Berenson 2013; Perez-D’Arpino and Shah2015). In common, the cited methods do not explicitlyaddress robot trajectory generation as part of the model,usually treating the design of the robot motion as anindependent step that must be executed once the action isrecognized. Robot trajectories were pre-programmed forthe recognized actions Koppula and Saxena (2013), orgenerated with motion planners Mainprice and Berenson(2013); Hayne et al. (2016). We exploit the use of correlatedmovement primitives to generate the robot trajectory suchthat action recognition and movement generation are bothgiven by the same probabilistic model and solved by similarcomputations.

Although some of the scenarios here presented ultimatelylead to the handover of objects, handovers are not theonly application of Interaction ProMPs and our methodis not intend to be a self-contained solution to the wholehandover problem. A handover is comprised of a complexseries of combined physical and social interaction steps.As previously investigated by Strabala et al. (2013), thesesteps range from (1) the social-cognitive cues that establishthe connection between the giver and the taker, (2) thecoordination of the location and the resulting trajectory as afunction of preferences and socially acceptable movements(Sisbot and Alami 2012), and (3) the final physicaltransfer that comprises interaction forces and compliances

(Kupcsik et al. 2015). While Interaction ProMPs does notencode or output trajectories based on such vast amountof information, as an imitation learning method, theyimplicitly encode user preferences from demonstrationswhich appears to be more suited for human interaction thanpure motion planning approaches (Cakmak et al. 2011).Human-robot handover offers an interesting application formethods that emphasize the fast response of the robot, andtherefore, it suits our investigations in phase estimation.In the handover context and under the assumption of asingle task, this paper shares similar challenges faced inYamane et al. (2013) where the robot trajectory had tobe coordinated according to the observation of the humanpartner during the passing of an object. Yamane et al. (2013)encoded the demonstrations in a tree-structured database asa hierarchy of clusters, which then poses the problem ofsearching matching trajectories given partial observationsusing a sliding window. In contrast, our method uses aflat representation which requires less computation whileallowing the recognition of different tasks.

This paper uses Gaussian distributions to encode thejoint distribution of human and robot movement primitivesover entire trajectories. This makes action recognitionstraightforward since each Gaussian represents one task.Also, this representation allows to quickly infer the mostappropriate robot movement primitive by conditioning thejoint distribution on the human task. Note that this isfundamentally different from the GMMs approach thatencode local variations across demonstrated trajectories(e.g. Calinon et al. (2012)). In the body of work of Calinonet al., the flexibility provided by multiple componentscan be exploited, for example, by a stiffness controllerwhich adapts the robot behavior as a function of thelocal uncertainty over the distribution of demonstratedtrajectories.

This paper consolidates our recent efforts in differentaspects of semi-autonomous robots. It leverages on ourdevelopments in human-robot interaction (Ben Amor et al.2014) and the ability to address multiple tasks (Ewertonet al. 2015b; Maeda et al. 2016). While our previousinteraction models were explicitly time-dependent andreliant on DTW, here, we introduce a phase-dependentmethod that is free from time alignment processing andallows for fast robot action recognition and trajectorygeneration.

3 Probabilistic Movement Primitives forHuman-Robot Interaction

This section introduces the basic concepts of ProMPs on aone dimensional case, followed by the multi-dimensionalProMP case. As it will become clear, there is a natural

Prepared using sagej.cls

Maeda et al. 5

transition from a multi-dimensional ProMP to the human-robot case with Interaction ProMPs. This section alsoaddresses how to compute the probabilities of a task whenmany Interaction ProMPs are used to represent differenttasks. This section finishes by introducing phase estimation,which also provides means to recognize human actions inmultiple-task scenarios under partial human observation.

3.1 Probabilistic Movement Primitive for aSingle Dimension

For each time step t a position is represented by ytand a trajectory of T time steps as a sequence y1:T . Aparameterization of yt in a lower dimensional weight spaceis proposed as

yt = ψTt w + εy, (1)

p(y1:T |w) =

T∏1

N (yt|ψTt w, σy), (2)

where εy∼N (0, σy) is zero-mean i.i.d. Gaussian noise,w ∈ RN is a weight vector that parameterizes thetrajectory, and ψt = [(ψt)1, ...(ψt)N ]T ∈ RN×1 has thevalues of each of the basis function at time t. The weightsare computed with linear regression with a time-dependentdesign matrix

w = (ΨT1:TΨ1:T )−1ΨT

1:Ty1:T (3)

where

Ψ1:T =

(ψ1)1 . . . (ψ1)N

.... . .

...(ψT )1 . . . (ψT )N

, (4)

The number of Gaussian bases N is often much lowerthan the number of trajectory time steps∗. The number ofbases is a design parameter that must be matched withthe desired amount of detail to be preserved during theencoding of the trajectory.

Assume M trajectories are obtained via demonstrations;their parameterization leading to a set of weight vectorsW = {w1, ...wi, ...wM} (the subscript i as inwi will beused to indicate a particular demonstration when relevant,and will be omitted otherwise). Define θ as a parameterto govern the distribution of the weight vectors in the setW such that w∼p(w;θ). The model p(w;θ) is assumedas a Gaussian with mean µw ∈ RN and covariance Σw ∈RN×N , that is θ = {µw,Σw}. This prior model allows usto sample trajectories with

p(yt;θ) =

∫p(yt|w)p(w;θ)dw

= N (yt|ψTt µw,ψTt Σwψt + σy).

(5)

The Gaussian assumption is restrictive in the sense thatthe training data must be time-aligned, for example byDTW, such that the spatial correlation can be capturedappropriately. In Section 3.4 we will introduce a methodto avoid such alignment.

3.2 Correlating Human and RobotMovements with Interaction ProMPs

Interaction ProMPs model the correlation of multipledimensions, here each dimension given by each of thedegrees-of-freedom (DoFs) of multiple agents. Let usdefine the state vector as a concatenation of the P numberof observed DoFs of the human, followed by the Q numberof DoFs of the robot

yt = [ yH1,t, ... yHP,t, y

R1,t, ... y

RQ,t ]T , (6)

where the upper scripts (·)H and (·)R refer to thehuman and robot DoFs, respectively. Similar to theone dimensional case, trajectories of each DoF areparameterized as weights such that

p(yt|w) = N (yt|HTt w,Σy), (7)

where HTt = diag((ψTt )1, ..., (ψ

Tt )P , (ψ

Tt )1, ..., (ψ

Tt )Q)

has P+Q diagonal entries. Each collaborativedemonstration now provides P+Q training trajectories.The weight vector wi of the i-th demonstration is now aconcatenation of all weight vectors involved in the i-thdemonstration. Thus, the many DoFs involved in theinteraction will be correlated

wi = [ (wH1 )T , ..., (wH

P )T , (wR1 )T , ..., (wR

Q)T ]T . (8)

A normal distribution from a set of M demonstrationsW = {w1, ...wM} with µw ∈ R(P+Q)N and Σw ∈R(P+Q)N×(P+Q)N can be computed.

The fundamental operation to infer the robot trajectoryis to compute a posterior probability distribution ofthe weights w∼N (µneww ,Σnew

w ) conditioned on theobservations of the human. Since the robot is not observedwe denote the observation vector as

yot = [ yH1,t, ... yHP,t, 0R1,t, ... 0RQ,t ]T. (9)

To contrast with a complete sequence [t : t′] we usethe notation [t− t′] ∈ RS×P to indicate that the sequenceof S observations in the interval is incomplete, thatis, some measurements in between are missing. The

∗In the particular case of the experiments here reported, trajectories havean average time of 3 seconds, sampled at 50 Hz. We used 20 basis functionsand thus, for a 1 DoF case the dimensionality is decreased from 3 × 50 =

150 samples to a weight vector w ∈ R20.

Prepared using sagej.cls

6 Journal Title XX(X)

Interaction ProMP is updated with yot−t′ using closed-formconditioning

µneww = µw +K(yot−t′ −Ht−t′µw),

Σneww = Σw −K(Ht−t′Σw),

(10)

where K = ΣwHTt−t′(Σ

oy + Ht−t′ΣwHT

t−t′)−1 and Σo

y

is the measurement noise. The upper-script (·)new is usedfor values after the update. The observation matrix Ht−t′ isobtained by concatenating the bases at the correspondingobservation steps, where the Q unobserved states of therobot are represented by zero entries in the diagonal. Fora single observation at time t,

Ht =

(ψTt )1 . . . 0 0 . . . 0

0. . . 0 0

. . . 0

0 . . . (ψTt )P 0 . . . 0

0 . . . 0 0P . . . 0

0. . . 0 0

. . . 00 . . . 0 0 . . . 0Q

(11)

with Ht ∈ R(P+Q)×(P+Q)N .Trajectory distributions that predict human and robot

movements are obtained by integrating out the weights ofthe posterior distribution

p(y1:T ;θnew) =

∫p(y1:T |w)p(w;θnew)dw. (12)

3.3 Multiple Interaction PatternsTo address multiple tasks we repeat the procedure describedin Section 3.2 for each task. The set of demonstratedtrajectories for each type of collaboration is labeled foraction recognition purposes and are trained independently.Given that aK number of tasks are presented, and given theobservation vector as defined in (9) the probability of eachtask k can be computed with

p(k|yot−t′) ∝ p(yot−t′ |k)p(k)

∝ p(yot−t′ |θk)p(k),(13)

where p(k) is a prior probability of the task. The actionis recognized by selecting the posterior with the highestprobability

k∗ = arg maxk

p(k|yot−t′), (14)

whose corresponding model is θnewk∗ .To compute the likelihood in (13) note that each

component k is governed by θk, therefore

p(yot−t′ ;θk) =

=

∫p(yot−t′ |HT

t−t′w,Σoy)p(w;θk)dw

= N (yot−t′ |HTt−t′µw,H

Tt−t′ΣwHt−t′ + Σo

y).

(15)

The most likely robot trajectory given the human actionis inferred by conditioning the most probable interactionmodel θnewk∗ with (10) on yot−t′ and by integrating out theweights

p(y1:T ;θnewk∗ ) =

∫p(y1:T |w)p(w;θnewk∗ )dw. (16)

3.4 Estimating Phases on Multiple Tasks

Previous works (Ewerton et al. 2015b; Maeda et al.2016) have only addressed spatial variability, but nottemporal variability of demonstrated movements. However,when demonstrating the same task multiple times, ahuman demonstrator will inevitably execute movementsat different speeds, thus changing the phase at whichevents occur. Previously, this problem was mitigated duringthe training phase by time-aligning the demonstratedtrajectories using a DTW method. Time alignment ensuresthat the weights of each demonstration can be regressedusing the same feature ψ1:T for all demonstrations. As aconsequence, during execution, the conditioning (10) canonly be used when the phase of the human demonstratorcoincides with the phase encoded by the time-alignedmodel, which is unrealistic in practice.

Under temporal variability and unobserved velocities,the problem is to retrieve the corresponding basis ψtfor an observation yot such that the conditioning can becomputed correctly. While time alignment allows for theencoding of spatial variability during training, it poses adifficult problem during execution since the observationsmust be aligned while the human is moving. Dong andWilliams (2012) and Ben Amor et al. (2014) have proposedonline variants of DTW given stream of positions. Suchapproaches, however, are not suitable when only a fewsparse points are measured as the estimation of distancesbetween trajectory segments is compromised. A practicalsolution to the alignment problem is to condition only at thefinal position of the human movement, since only for thisparticular case, the corresponding basis function is knownto be the last one ψT . This, however, causes a significantlag on the robot response as the robot has to wait for thehuman to finish his movement first.

To solve this problem, we propose incorporating thetemporal variance as part of the model by learning adistribution over phases from the same demonstrationspreviously used to create the Interaction ProMP. Thisenriched model not only eliminates the need for time-alignment, but also allows for faster robot reactions as theconditioning (10) can be applied before the end of thehuman movement. Initially, we replace the original timeindexes of the basis functions with a phase variable z(t).Thus, a trajectory of duration T is now computed relative

Prepared using sagej.cls

Maeda et al. 7

to the phase

p(y1:T |w) =

T∏1

N (y(zt)|ψT(zt)w,Σy). (17)

Define a nominal sequence {1 : Tnom}, for example,by taking the average final time of the demonstrations.Assuming that each of the i-th demonstrations has aconstant temporal change in relation to the nominalduration, we define a scaling factor

αi = Ti/Tnom, (18)

such that all demonstrations can be indexed by the samenominal time index. We then define fixed Gaussian basesspread over the nominal duration ψ1:Tnom

. The weightsof all demonstrations can then be regressed from thesame bases to obtain the parameters of the distributionθ = {µw,Σw} using a single design matrix with nominalduration Ψ1:Tnom

.Given a time step t from the nominal sequence, a

trajectory with different duration is found by using zt =αt in (17). This temporal model implicitly assumes thatthe human movement is governed by a single phase andtherefore, we will refer to α as a phase ratio. Note from (18)that each i-th demonstration in the training set provides avalue of αi. We will assume that the phase ratios of differentdemonstrations vary according to a normal distribution, thatis, α∼N (µα, σ

2α), where

µα = mean({α1, ... αi, ... αM}),σ2α = var({α1, ... αi, ... αM}),

and M is the number of demonstrations. Despite thesimplicity of this model, experiments have shown thatthese assumptions (single phase, normal distribution) holdin practice for simple, short stroke movements typical ofhandovers†.

Given a sparse partial sequence of observations yot−t′ , aposterior probability distribution over phases is given as

p(α|yot−t′ ,θ) ∝ p(yot−t′ |α,θ)p(α), (19)

where p(α) is the prior probability of the phase.For a specific α value the likelihood is

p(yot−t′ |α,θ) =

∫p(yot−t′ |w, α)p(w)dw

= N (yot−t′ |A(zt−t′)Tµw,

A(zt−t′)TΣwA(zt−t′) + Σo

y),

(20)

where

A(zt−t′) =

ψ(zt−t′)

T1 . . . 0

0. . . 0

0 . . . ψ(zt−t′)TP

, (21)

is the partition of the full matrix H in (11) that contains thebasis functions of the observed positions of the human—however, now indexed by the phase zt = αt.

Given the observations yot−t′ , the likelihood of eachsampled α candidate is computed with (20), and the mostprobable value

α∗ = arg maxα

p(α|yot−t′ ,θ) (22)

is selected. Intuitively, the effect of different phases isto stretch or compress the temporal axis of the prior(unconditioned) distribution proportionally to α. Themethod then compares which scaling value generates themodel with the highest probability given the observationyot−t′ .Once the most probable scaling value α∗ is found,its associated observation matrix H(z) can be used in (10)to condition and predict the trajectories of both human androbot. To efficiently estimate the phase during execution,one approach is to sample a number of values of α from theprior p(α) and precompute and store, for each of them, theassociated matrices of basis functions A(z1:T ) and H(z1:T )beforehand.

Figure 3 summarizes the workflow of the InteractionProMP with phase estimation. During the training phase,trajectories are scaled to a nominal time Tnom and thedistribution of the scaling values is encoded as a normaldistribution. The set of normalized trajectories are usedto learn the parameter θ = {µw,Σw} that correlatesthe trajectories of human and robot. In the figure, thedistribution modelled by θ is abstracted as a bivariateGaussian where each of the two dimensions are given bythe distribution over the weights of the human and robottrajectories. During execution, the assistive trajectory ofthe robot is predicted by integrating out the weights ofthe posterior distribution p(w;θnew). The operation ofconditioning is illustrated by the slicing of the prior, at thecurrent observation of the position of the human yot . Theconditioning requires finding the correct temporal scalingof the prior model that best fits the observations in time.Thus, the probability of many phase ratio candidates aretested on the sparse human observations and the mostprobable value is assumed to provide the optimal timeindexing of the observations.

In a multi-task scenario, we can now address therecognition of the task given the positions yot−t′ and

†For problems where this assumption does not hold, e.g. when themovement accelerates and decelerates along and between demonstrations,the estimation of multiple phases must be addressed. Initial investigationsreported in (Ewerton et al. 2015a) show that movements with multiplephases on synthetic data can be treated under the assumption that differentphases must be correlated along the motion. This assumption, however, istoo stringent for realistic purposes. We found that real human experimentswith short strokes are better addressed by the single phase methodpresented here.

Prepared using sagej.cls

8 Journal Title XX(X)

Human trajectories

Human observations

Human Robot

Trai

ning

In

fere

nce

Conditioning

Multiple demonstrations

Robot trajectories

Phase normalization Correlated model

…

Searching Trajectory tracking

Figure 3. The workflow of Interaction ProMP (after Maeda et al. (2016)) augmented with a step to learn the distribution overphases during training, and a step to infer the phase during execution. The figure illustrates a single task case where thedistribution of human-robot parameterized trajectories is abstracted as a bivariate Gaussian. The conditioning step is shown asthe slicing of the distribution at the observation of the human position. In the real case, this distribution has more dimensions.

unknown phase ratio. Let the parameter θk represent acollaborative task independently trained as an InteractionProMP. For each task k, the most probable α∗k isfirst searched with (22) rendering the set {α∗k,θk}. Thetask recognition is given by the procedure described inSection 3.3 with the difference that the likelihood (13) isnow

p(k|yot−t′) ∝ p(yot−t′ |α∗k,θk)p(k). (23)

Since {α∗k,θk} is the solution of the phase search foreach task, task recognition demands only the computationof (23). The two optimizations—to search for the correctphase, and to recognize the task—lead to an algorithm thatscales linearly in the number of sampled α’s and in thenumber of tasks.

4 Experiments with a Semi-AutonomousRobot

Experiments were conducted using a 7-DoF lightweightKUKA arm equipped with a 5-finger hand. To train theprimitives the joint encoder trajectories of the robot wererecorded by kinesthetic teaching. For each demonstration,the set of human-robot measurements were stored with asampling rate of 50 Hz.

The first set of experiments evaluates the responsivenessof the method in a single task scenario where the robottries to predict where the human will handover a cup.The Cartesian coordinates of the cup were tracked witha motion capture system by placing a marker directly onit. In the second set of experiments, a multi-task scenario

of collaborative box assembly was used to evaluate theproblem of task recognition under partial observations. Inthis case motion capture was used to track the Cartesiancoordinates of the wrist of the human.

4.1 Predictive HandoverAn important characteristic for fluid human-robot interac-tion is the capability of the robot to preemptively react to thehuman partner. In this experiment we evaluate this capabil-ity in an illustrative scenario of a cup handover. As shown inFigure 4, initially, a set of paired trajectories was collectedby simultaneously moving the robot in kinesthetic teachingmode while the human was bringing the cup towards therobot. A total of 20 pairs of Cartesian trajectories of the cupand robot joint encoders were recorded. The demonstratorenforced temporal variability by changing the speed of thedemonstration, with fast trajectories taking approximately2 seconds and slower trajectories taking approximately 3seconds.

We evaluated the effect of the number of candidatesof phase ratios α w.r.t the approximation error of thetrue duration of the movement. Also, we evaluated thevariation of the same parameter in relation to the accuracyof the prediction of the robot trajectory. To create asparse limited observation, only the first half of each testtrajectory was used, from which five states were randomlydrawn along the time axis as observations. One instance isshown in Figure 5 (a) where ten phase ratios candidateswere evaluated. The plot at the left shows the Cartesiancoordinate of the marker on the cup and randomly drawn

Prepared using sagej.cls

Maeda et al. 9

Figure 4. Collection of training data to create a singleInteraction ProMP for the handover of a cup. The marker isplaced on the top of the cup. Both cup and robot arm aremoved towards the same position, leading to pairs ofcollaborative trajectories. This figure shows two of suchdemonstrations at different positions.

observations represented as the circles. The blue patchrepresents ± 2 standard deviations around the mean andthe traced curve represents the ground truth. The plot atthe right shows the predicted trajectory distribution of oneof the joints of the robot arm. The many gray curves atthe background show samples that were drawn from thetrained model to illustrate the variability in both time andposition. In (a), the fact that the predicted distribution hasroughly the same duration as the test trajectory indicatesthat a reasonably good approximation to the true phaseratio could be found. The bottom row shows the samecase, however when only two phase ratio candidates wereevaluated leading to a coarse approximation of the truetrajectory duration.

This procedure was repeated systematically with leave-one-out cross validation (LOOCV) on 20 test trajectories.The number of phase ratio candidates was varied from 1 to15 for each test trajectory, resulting in a total of 15×20 runs.To systematically evaluate the different number of phasesunder repetitive conditions, we selected the candidates onan equally spaced grid within the interval covering 95percent (± 2 σ) of the normal distribution of phases. Figure6 (a) shows the error in predicting the correct phase ratioas the number of sampled candidates increases. The resultsshow that, on average, no improvement is achieved beyond6 sampled candidates.

Due to the nature of the handover task, it is impracticalto evaluate the accuracy of the robot in online experimentsas the human will invariably adapt to the robot’s response.Thus, we used the same LOOCV procedure to evaluatethe accuracy of the robot end-effector with respect to thecorresponding recorded human trajectory. This result isshown in Figure 6 (b), which shows the error of the finalpredicted positioning of the robot hand as the number ofsampled candidates increases. The final position error of therobot hand was computed via forward kinematics of the armgiven the mean of the predicted joint angle distributions.There is a lower limit of 3 cm in the accuracy of the robotprediction. This value suffices for tasks such as handovers

(a)

Time (s)

q2 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.540

50

60

70

80

90

100

Time (s)

y (c

m)

0 0.5 1 1.5 2 2.5 3 3.510

20

30

40

50

60

70

(b)

Time (s)

q2 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.545

50

55

60

65

70

75

80

85

Time (s)

y (c

m)

0 0.5 1 1.5 2 2.5 3 3.510

20

30

40

50

60

70Observations

Ground truth

Predicted distribution

Model samples

Human

Human

Robot

Robot

Figure 5. Examples of two different phase estimations underthe same test data and five observations up to 1 second of thehuman movement. (a) Ten phase candidates evaluated. Theposition error of the robot hand is 4.31 cm. (b) Two phasecandidates evaluated leading to a erroneous prediction of thetrajectory duration. The position error of the robot hand was of5.20 cm.

.

of objects and may be affected by the quality of the setup,such as the positioning of the markers and the non-idealdemonstrations that invalidate the Gaussian assumption. Asit will be discussed in Section 5, direct feedback trackingof the marker may be used to increase the accuracy of therobot.

It is also important to evaluate the effect of the durationof the observations of the human before the robot attemptsto infer its own trajectory. The sooner the robot can predictthe correct human trajectory, and as a consequence thefinal handover position, the quicker it can react to provideassistance. On the other hand, trying to predict the humanmotion too early may lead to poor coordination between thehuman and robot final positions. Figure 7 shows snapshotswhere the robot observed only 0.2 seconds of the humanmovement (approx. 8 % of the whole trajectory length)before generating its own trajectory after evaluating tenphase candidates. The final position of the robot hand doesnot match the final position of the human hand. Empirically,for this particular task we noticed that observing the humanfor at least 0.5 seconds (approx. 20 % of the full trajectorylength) while evaluating ten phase candidates lead to quitesatisfactory coordination between the final positions of thehuman and robot hands.

For a quantitative analysis, we used the same 20 testtrajectories of the previous analysis as ground truths andfixed the number of phase ratio candidates to ten. Each

Prepared using sagej.cls

10 Journal Title XX(X)

(b)

(a)Phase ratio evaluations

Phas

e ra

te e

stim

atio

n er

ror

0 2 4 6 8 10-0.2

0

0.2

0.4

0.6

Phase ratio evaluations

End-

effec

tor p

ositi

onin

g er

ror (

m)

0 2 4 6 8 100.01

0.02

0.03

0.04

0.05

0.06

Figure 6. Effects of the number of phase ratio evaluations on(a) the approximation to the true phase ratio value, and (b) theaccuracy of the prediction of the final position of the robothand.

1 2

3 43Figure 7. Snapshots of a handover trajectory where the robotreacts too early to predict the final human position accurately.In this example the robot observed only the first 0.2 seconds ofthe human movement before attempting to generate its owncorresponding trajectory.

of the test trajectories was used to generate ten randomlydrawn human observations. The span in which theseobservations were drawn varied between the first 10 %to 99 % of the total test trajectory length. Figure 8 (a)shows two examples where the trajectory of the humanis predicted from different observation durations. On theleft, only 5 % of the trajectory was observed; on the

Observation ratio (relative to the total duration of the test trajectory)E

nd-e

ffect

or p

ositi

oni

ng e

rror

(m

)

0 0.2 0.4 0.6 0.8 10.01

0.02

0.03

0.04

0.05

0.06

0.07

(a)

(b)

y (cm)

4050

6070

8090

10030

4050

5060

7080

90100

30

x (cm)x (cm)

Predicted mean

Ground truth

Observation

Model sample

Figure 8. Effect of the duration of the observation of thehuman position on the cup handover task. (a) Prediction of thehuman trajectory in the Cartesian space given ten randomlysampled observations. Left: 5 % of the trajectory observed.Right: 50 % of the trajectory observed. (b) Accuracy of thefinal robot hand position as a function of increasing durationsof the human observation.

right, 50 % of the trajectory was observed. As expected,the predicted human trajectory approximates the groundtruth as the duration of observation increases. As a fullstate estimator, not only the trajectory of the human iscomputed from the observations, but also the correspondingrobot trajectories. Figure 8 (b) shows the effect of differentobservation durations—indicated in the horizontal axis asthe ratio between the observed length and the total length ofthe trajectory—on the prediction of the location of the robothand. Similar to the previous analysis, the final position ofthe robot hand was computed by using forward kinematicsof the arm given the predicted joint angles. As expected,the longer the human is observed before the prediction isattempted, the better the final accuracy of the robot, at theexpense of lag in responsiveness.

A side-by-side comparison between short and longobservations of the human movement and its effect onthe robot responsiveness and accuracy can be watchedin the accompanying video and also by following thelink https://youtu.be/bUVn0AwAb1U. To control the robotmotion, the mean of the posterior distribution overtrajectories for each joint of the robot was used, and tracked

Prepared using sagej.cls

Maeda et al. 11

by the standard, compliant joint controller provided by therobot manufacturer‡.

4.2 A Multi-Task Semi-Autonomous RobotCoworker

As it was motivated at the introduction of this paper,we applied our method on a multi-task scenario wherethe robot plays the role of a coworker that helps ahuman assembling a toolbox. This scenario was previouslyproposed in Ewerton et al. (2015b). The assembly consistedof three different collaborative interactions. In one of them,the human extends his hand to receive a plate. The robotfetches a plate from a stand and gives it to the humanby predicting the location of the marker on his wrist. Ina second interaction, the human fetches the screwdriverand the robot grasps and gives a screw to the human asa pre-emptive collaborator would do. The third type ofinteraction consists of the robot receiving a screwdriversuch that the human coworker can have both hands free(the same primitive representing this interaction is alsoused to give the screwdriver back to the human). Eachinteraction of “plate handover”, “screw handover” andholding the screwdriver was demonstrated 15, 20, and 13times, respectively.

The upper row of Figure 9 shows snapshots of thetraining phase for each of the interactions and theirrespective multiple demonstrated trajectories in Cartesianspace. The markers were placed on the wrist of thehuman. The bottom row shows the execution of the learnedInteraction ProMPs from the demonstrations. On purpose,all human demonstrations started roughly at the sameposition in order to make the problem of action recognitionapparent.

We used the original training data previously presented inEwerton et al. (2015b). The original data set did not presentsufficient variability of phases and the correct phase couldbe reasonably well estimated with only two to three samplecandidates. Thus, the durations of each demonstration wasscaled by sampling different final times out of a normaldistribution centered at the mean final time of the trainingset and with a standard deviation of 0.5 seconds. Thisperturbation acts as a surrogate of a demonstrator movingat different speeds at each demonstration.

We evaluate the effect of observing different durationsof the human trajectory before attempting to recognize theaction. Similar to the experiments in Section 4.1, eachof the test trajectories was used to generate five, sparserandomly drawn, human observations. A total of ten phaseratio candidates were fixed. The duration from which theseobservations were drawn varied between the first 5 % to75 % of the total duration of the test trajectory. Figure 10shows, as circles, the observations of the “plate handover”trajectory drawn from the first 35 % of the trajectory length.

Note that while these observations clearly do not fit the firsttask of holding the tool, they can incorrectly explain thetask of “screw handover”. Selecting the tasks by comparingtheir probabilities provides a principled mechanism to makesuch a decision.

Figure 11 (a) shows the action recognition accuracyaccording to the ground truth trajectory labeled with thecorresponding action. The square marks represent the casewhen the trajectory of the “hold tool” task was used asa test data. The circle marks represent the case whenthe trajectory of “plate handover” task was the correctaction, and cross marks represent the case when the “screwhandover” was the correct action. The data measured fromthe motion capture system was used as a noiseless case.From the plot it is observed that even when only 5 %of the task was observed, the “plate handover” could becorrectly recognized in all tests. The subplot (b) repeatsthe same procedure but Gaussian noise with zero meanand standard deviation of 3.5 cm on the observed pointswere added. As expected the recognition deteriorates as thefew noisy observations could overlap significantly onto theincorrect distribution. Figure 11 (c) shows the case wherethe Gaussian noise has a standard deviation of 7.0 cm, inwhich case, practically the full human trajectory has tobe observed such that a proper action recognition can bemade. All cases in Figure 11 were evaluated using ten phasecandidates.

As mentioned in Section 3.4, in previous works (Ewertonet al. 2015b), the training data had to be time-alignedwith a method based on DTW. During execution, the robotwas then conditioned at the final position of the humanas a way to overcome the problem of aligning partial andsparse observations online. With phase estimation not onlyconsiderable pre-processing effort is avoided—the use ofDTW on both training and observations is not needed—but, the robot can now predict the collaborative trajectorybefore the human finishes moving, leading to a faster robotresponse. Figure 12 shows the difference in uncertainty andaccuracies between the time-aligned method using DTW(a) with two cases of phase estimation that differ in regardsto the durations of observations (b-c) for the task of “platehandover”. As shown in (a), since the Interaction ProMPis conditioned at the final position of the human hand—theprincipal state of interest for predicting the final positionof the robot—inference of the robot trajectory in this caseprovides a baseline to compare with the phase estimationmethod.

Using LOOCV over the training set, we quantifiedthe accuracy of the prediction of the phase estimation

‡Although not used in this paper, the ProMP framework also providesmeans to compute the feedback controller and the interested reader isreferred to Paraschos et al. (2013).

Prepared using sagej.cls

12 Journal Title XX(X)

(a) Demonstration of collaborative trajectories

(b) Execution

x (cm)y (cm)

z (c

m)

2040

6080

100

-50

0

50

-40

-20

0

20

x (cm)y (cm)

z (c

m)

2040

6080

100

-50

0

50

-40

-20

0

20

x (cm)y (cm)

z (c

m)

2040

6080

100

-50

0

50

-40

-20

0

20

hold toolplate handover screw handover

human

robot

human

robot

human

robot

hold tool plate handover screw handover

Figure 9. (a) Demonstrations of the three different interactions and their respective trajectories. (b) Human-robot collaborationusing the trained Interaction ProMP for each task. The robot actions are conditioned on the partial observations of the humanmovement. The scenario and training data appeared previously in Ewerton et al. (2015b) and Maeda et al. (2016) where trainingdata had to be time-aligned by DTW.

hold tool plate handover screw handover

Figure 10. Each of the three tasks conditioned on the same observations sampled from the “plate handover” task. The sameobservations can fit the task of “screw handover” with a lower probability. The thick solid line shows the predicted humantrajectory. The dashed lines is the “plate handover” ground truth, and the circles are observations.

method and the time-aligned model conditioned at theend of the observation (as illustrated in Figure 12). Theseresults are summarized in Figure 13 where the proposedmethod is represented as the circles and the previoustime-aligned method is shown by the square marker. Theresults show that for the “plate handover” and “hold tool”tasks the accuracy improves as longer observations are

made, as expected. At about 75 % of the observation,the accuracy achieved by the phase estimation method iscomparable to the time-aligned trajectories using DTW.This experiment show that the single phase assumptionis reasonably met, otherwise worst accuracy would beexpected in relation to the time-aligned trajectories. For thetask of “screw handover” no improvement is observed as

Prepared using sagej.cls

Maeda et al. 13

Observation ratio (relative to the total duration of the test trajectory)

Task

rec

ognt

ion

accu

racy

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

0.2

0.4

0.6

0.8

1

hold_tool

plate_handover

screw_handover

Observation ratio (relative to the total duration of the test trajectory)

Task

rec

ognt

ion

accu

racy

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

0.2

0.4

0.6

0.8

1

hold_tool

plate_handover

screw_handover

Observation ratio (relative to the total duration of the test trajectory)

Task

rec

ognt

ion

accu

racy

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

0.2

0.4

0.6

0.8

1

hold_tool

plate_handover

screw_handover

(a)

(b)

(c )

Figure 11. Improvement in action recognition as the durationof the observation increases under different amounts of noise.(a) Clean observations. (b) Observations added withzero-mean Gaussian noise and standard deviation 3.5 cm. (c)Observations added with zero-mean Gaussian noise andstandard deviation 7.0 cm.

the trajectories of the human and robot are not correlated bythe demonstration. In this case, the robot brings the screwto the mean of the unconditioned distribution (i.e. the priormean) regardless of the position where the human graspsthe screwdriver on the table.

5 Discussion of the Experiment andLimitations

Since the experiments aimed at exposing the predictivetrajectory capability of the Interaction ProMPs no directfeedback tracking of the marker on the human wristwas made. However, even humans are prone to inferthe wrong location when attempting to grasp an objectfrom a partner too quickly. We naturally use visualfeedback for corrections. In the same way, the InteractionProMP framework may potentially benefit when used in

combination with a feedback controller that tracks themarkers directly. In this case, the predictive trajectorygeneration that results from the method provides a richnonlinear behavior as a movement primitive learned fromdemonstrations. Direct tracking of the marker could thenact in a corrective/reactive manner to account for themistakes in prediction. Note, however, that it is notpossible to completely replace an Interaction ProMP bya tracking controller. In a multi-task scenario InteractionProMPs are essential for action recognition. Moreover, afeedback controller does not easily provide the flexibilityand richness of the trajectories that can be encoded in aprimitive learned from human demonstrations.

The presented method scales linearly in the number oftasks and in the number of phase ratio candidates to beevaluated. In practice, this scalability imposes an upperlimit on the number of tasks and samples α’s that canbe supported, which can be empirically evaluated. Theconstraint is that the total time required to compute theprobability of all sampled α’s for each possible tasksmust be less than the duration of the human movement.Otherwise, one can simply use the final state of the humantrajectory, as it was done in previous works with timealigned trajectories, to recognize the action and compute thecorresponding collaborative robot trajectory. Therefore, theduration of the human movement and the implementationefficiency dictate to which granularity the phases can beestimated, and how many tasks can be recognized. In ourpreliminary evaluations on the assembly scenario, a total of25 candidates of phase ratios α for each of the three taskswere used. This setting required 75 (25 samples × 3 tasks)calls to the computation of the probabilities (19) while thehuman was moving his arm. The whole process, from theprediction of the full trajectory with (12), observed duringthe first second of the human movement took in average0.20 seconds using Matlab code on a conventional laptop(Core i7, 1.7 GHz). In contrast, the duration of a handoverstroke can vary from 1 to 2.5 seconds.

In our previous work (Maeda et al. 2016), for the samecollaborative task, we evaluated the deterioration of actionrecognition when the duration of the observation differedfrom that of the time-aligned model. The frequency of thecorrect recognition of a “plate handover” decreased from100 % to 50 % when the observed trajectory had a duration1.25 longer than the duration of the time-aligned model.This sensitivity to temporal misalignments is an evidencethat, unless the human partner is very consistent in termsof the speed of his/her movement, phase estimation isessential for action recognition. In the same work, we alsocompared the accuracy of a single-task Interaction ProMPwith a baseline nearest-neighbor (NN) method on the taskof plate handover. The Interaction ProMP presented twice

Prepared using sagej.cls

14 Journal Title XX(X)

(a) Time aligned

(b) Observation ratio 0.1

Human

Robot

Human

Robot

Time (s)

q1 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.5-40

-20

0

20

40

60

80

Time (s)

q2 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.50

20

40

60

80

Time (s)q3

(de

g)

0 0.5 1 1.5 2 2.5 3 3.520

40

60

80

100

120

Time (s)

x (c

m)

0 0.5 1 1.5 2 2.5 3 3.550

60

70

80

90

100

Time (s)

y (c

m)

0 0.5 1 1.5 2 2.5 3 3.5-40

-20

0

20

40

60

Time (s)

z (c

m)

0 0.5 1 1.5 2 2.5 3 3.5-40

-30

-20

-10

0

(c) Observation ratio 0.5

Time (s)

q1 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.5-40

-20

0

20

40

60

80

Time (s)

q2 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.520

30

40

50

60

70

80

Time (s)

q3 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.520

40

60

80

100

120

Time (s)

x (c

m)

0 0.5 1 1.5 2 2.5 3 3.550

60

70

80

90

100

Time (s)

y (c

m)

0 0.5 1 1.5 2 2.5 3 3.5-40

-20

0

20

40

60

Time (s)

z (c

m)

0 0.5 1 1.5 2 2.5 3 3.5-40

-30

-20

-10

0

Human

Robot

Time (s)

q1 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.5-40

-20

0

20

40

60

80

Time (s)

q2 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.520

30

40

50

60

70

80

Time (s)

q3 (

deg

)

0 0.5 1 1.5 2 2.5 3 3.520

40

60

80

100

120Time (s)

x (c

m)

0 0.5 1 1.5 2 2.5 3 3.550

60

70

80

90

100

Time (s)

y (c

m)

0 0.5 1 1.5 2 2.5 3 3.5-40

-20

0

20

40

60

Time (s)

z (c

m)

0 0.5 1 1.5 2 2.5 3 3.5-40

-30

-20

-10

0

Figure 12. The patch represents the posterior distributions (± 2 standard deviations) for different methods to condition on thehuman observations. (a) Time-aligned trajectories (using DTW) conditioned at the final position. In (b) and (c), phase estimationwith an observation ratio of 10 % and 50 % relative to the total duration of the test data, respectively.

the accuracy of the nearest-neighbor given the same trainingset.

Measurement setup systems that allow for reliablevelocity measurements, or its estimation via positiondifferentiation, can greatly simplify the problem of phaseestimation. On the other hand, to become widely accepted,the deployment of semi-autonomous robots in the field mustcope with occluded and cluttered environments. Sparse

position measurements must be taken into account inrealistic scenarios, where noisy measurements are oftenprovided by low-cost sensors such as Kinect cameras.The results in Figure 11 show promise in regards to therobustness of the method to large amounts of noise.

When compared to representations based on multiplereference frames such as Dynamical Systems (Calinon et al.2012), and forcing functions as in DMPs, ProMPs have

Prepared using sagej.cls

Maeda et al. 15E

nd-e

ffect

orlp

ositi

onin

gler

rorl

(m)

0.05

0.1

0.15

End

-effe

ctor

lpos

ition

ingl

erro

rl(m

)

0.05

0.1

0.15

End

-effe

ctor

lpos

ition

ingl

erro

rl(m

)

0.05

0.1

0.15

Observationlratio0 0.2 0.4 0.6 0.8 Timelaligned

Holdltool

Platelhandover

Screwlhandoverl

Figure 13. Effect of the duration of the observation on thepredicted accuracy of the final robot hand position for each ofthe tasks. The time-aligned case represents the time-alignedmodel conditioned at the final position of the humanmeasurement. In the “screw handover” case the position of thehuman hand is not correlated with the location in which therobot gives the screw.

the limitation to only operate within the demonstrated set.Extensions to generalize ProMPs to other robot kinematicsand environments are currently being investigated.

6 ConclusionThis paper presented a method suited for collaborative andassistive robots whose movements must be coordinatedwith the trajectories of a human partner moving at differentspeeds. This goal was achieved by augmenting the previousframework of Interaction ProMPs with a prior modelof the phases of the human movement, obtained fromdemonstrated trajectories. The encoding of phases enrichesthe model by allowing the alignment of the observationsof the human in relation to the interaction model, underan intermittent stream of positions. We experimentallyevaluated our method in an application where the robot actsas a coworker in a factory. Phase estimation allowed ourrobot to predict the trajectories of both interacting agentsbefore the human finishes the movement, resulting in afaster interaction. The duration of a handover task could bedecreased by more than 50 % in the single task case, andour current evaluation on the multi-task scenario decreasestime by 25 %.

A future application of the method is to use the estimatedphase of the human to adapt the velocity of the robot. Aslowly moving human suggests that the robot should alsomove slowly, as an indication that a delicate task is beingexecuted. Conversely, if the human is moving fast, the robotshould also move fast as its partner may want to finishthe task quickly. This paper initially appeared in Maedaet al. (2015), and here an extended version with additionalevaluations was presented.

Acknowledgements

The authors would like to thank the reviewers of the first version ofthis paper, initially presented at ISRR 2015. Their comments wereinvaluable to improve the quality of this extended article, part ofthe special issue on the same conference.

The research leading to these results has received funding fromthe European Community’s Seventh Framework Programmes(FP7-ICT-2013-10) under grant agreement 610878 (3rdHand) andfrom the project BIMROB of the Forum fur interdisziplinareForschung (FiF) of the TU Darmstadt.

References

Ben Amor H, Neumann G, Kamthe S, Kroemer O and PetersJ (2014) Interaction primitives for human-robot cooperationtasks. In: Proceedings of the IEEE International Conferenceon Robotics and Automation (ICRA).

Cakmak M, Srinivasa SS, Lee MK, Forlizzi J and KieslerS (2011) Human preferences for robot-human hand-overconfigurations. In: Intelligent Robots and Systems (IROS),2011 IEEE/RSJ International Conference on. IEEE, pp.1986–1993.

Calinon S, Guenter F and Billard A (2006) On learning thestatistical representation of a task and generalizing it tovarious contexts. In: Proceedings of the IEEE InternationalConference on Robotics and Automation (ICRA). IEEE, pp.2978–2983.

Calinon S, Guenter F and Billard A (2007) On learning,representing, and generalizing a task in a humanoid robot.Systems, Man, and Cybernetics, Part B: Cybernetics, IEEETransactions on 37(2): 286–298.

Calinon S, Li Z, Alizadeh T, Tsagarakis NG and Caldwell DG(2012) Statistical dynamical systems for skills acquisition inhumanoids. In: Proceedings of the IEEE/RAS InternationalConference on Humanoids Robots (HUMANOIDS). pp. 323–329.

Coates A, Abbeel P and Ng AY (2008) Learning for controlfrom multiple demonstrations. In: Proceedings of the 25thinternational conference on Machine learning. ACM, pp.144–151.

Dong S and Williams B (2012) Learning and recognition ofhybrid manipulation motions in variable environments using

Prepared using sagej.cls

16 Journal Title XX(X)

probabilistic flow tubes. International Journal of SocialRobotics 4(4): 357–368.

Englert P and Toussaint M (2014) Reactive phase and task spaceadaptation for robust motion execution. In: Proceedings ofthe IEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS). pp. 109–116.

Ewerton M, Maeda G, Peters J and Neumann G (2015a) Learningmotor skills from partially observed movements executed atdifferent speeds. In: IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS). pp. 456–463.

Ewerton M, Neumann G, Lioutikov R, Ben Amor H, Peters Jand Maeda G (2015b) Learning multiple collaborative taskswith a mixture of interaction primitives. In: Proceedingsof the International Conference on Robotics and Automation(ICRA). pp. 1535–1542.

Hayne R, Luo R and Berenson D (2016) Considering avoidanceand consistency in motion planning for human-robotmanipulation in a shared workspace. In: Proceedings of theIEEE International Conference on Robotics and Automation(ICRA).

Ijspeert AJ, Nakanishi J, Hoffmann H, Pastor P and Schaal S(2013) Dynamical movement primitives: learning attractormodels for motor behaviors. Neural computation 25(2): 328–373.

Kim S, Gribovskaya E and Billard A (2010) Learning motiondynamics to catch a moving object. In: Proceedings of theIEEE/RAS International Conference on Humanoids Robots(HUMANOIDS). pp. 106–111.

Kim S, Shukla A and Billard A (2014) Catching objects inflight. IEEE Transactions on Robotics 30(EPFL-ARTICLE-198748).

Koppula HS and Saxena A (2013) Anticipating human activitiesusing object affordances for reactive robotic response. In:Robotics: Science and Systems.

Kupcsik A, Hsu D and Lee S (2015) Learning dynamic robot-to-human object handover from human feedback. In:International Symposium on Robotics Research (ISRR).IFRR.

Lee D and Ott C (2011) Incremental kinesthetic teachingof motion primitives using the motion refinement tube.Autonomous Robots 31(2-3): 115–131.

Lee D, Ott C and Nakamura Y (2010) Mimetic communicationmodel with compliant physical contact in humanhumanoidinteraction. The International Journal of Robotics Research29(13): 1684–1704.

Maeda G, Neumann G, Ewerton M, Lioutikov R, Kroemer Oand Peters J (2016) Probabilistic movement primitives forthe coordination of multiple human-robot collaborative tasks.In press: Autonomous Robots. Special Issue on Assistive andRehabilitation Robotics : 1–20.

Maeda G, Neumann G, Ewerton M, Lioutikov R and PetersJ (2015) A probabilistic framework for semi-autonomous

robots based on interaction primitives with phase estimation.In: Proceedings of the International Symposium of RoboticsResearch (ISRR).

Mainprice J and Berenson D (2013) Human-robot collaborativemanipulation planning using early prediction of humanmotion. In: Intelligent Robots and Systems (IROS), 2013IEEE/RSJ International Conference on. IEEE, pp. 299–306.

Paraschos A, Daniel C, Peters J and Neumann G (2013)Probabilistic movement primitives. In: Advances in NeuralInformation Processing Systems (NIPS). pp. 2616–2624.

Perez-D’Arpino C and Shah JA (2015) Fast target predictionof human reaching motion for cooperative human-robotmanipulation tasks using time series classification. In:Robotics and Automation (ICRA), 2015 IEEE InternationalConference on. IEEE, pp. 6175–6182.

Sakoe H and Chiba S (1978) Dynamic programming algorithmoptimization for spoken word recognition. Acoustics, Speechand Signal Processing, IEEE Transactions on 26(1): 43–49.

Schaal S (1999) Is imitation learning the route to humanoidrobots? Trends in cognitive sciences 3(6): 233–242.

Sisbot EA and Alami R (2012) A human-aware manipulationplanner. Robotics, IEEE Transactions on 28(5): 1045–1057.

Strabala KW, Lee MK, Dragan AD, Forlizzi JL, Srinivasa S,Cakmak M and Micelli V (2013) Towards seamless human-robot handovers. Journal of Human-Robot Interaction 2(1):112–132.

Tanaka Y, Kinugawa J, Sugahara Y and Kosuge K (2012) Motionplanning with worker’s trajectory prediction for assembly taskpartner robot. In: Proceedings of the IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS). IEEE,pp. 1525–1532.

Van Den Berg J, Miller S, Duckworth D, Hu H, Wan A, Fu X,Goldberg K and Abbeel P (2010) Superhuman performanceof surgical tasks by robots using iterative learning fromhuman-guided demonstrations. In: Proceedings of theIEEE International Conference on Robotics and Automation(ICRA). pp. 2074–2081.

Vuga R, Nemec B and Ude A (2014) Speed profile optimizationthrough directed explorative learning. In: Proceedings of theIEEE/RAS International Conference on Humanoids Robots(HUMANOIDS). IEEE, pp. 547–553.

Yamane K, Revfi M and Asfour T (2013) Synthesizing objectreceiving motions of humanoid robots with human motiondatabase. In: Robotics and Automation (ICRA), 2013 IEEEInternational Conference on. IEEE, pp. 1629–1636.

Ye G and Alterovitz R (2011) Demonstration-guided motionplanning. In: International symposium on robotics research(ISRR), volume 5.

Prepared using sagej.cls