Model-Based Probabilistic Pursuit via Inverse Reinforcement Learning

Florian Shkurti, Nikhil Kakodkar and Gregory Dudek

Abstract— We address the integrated prediction, planning,and control problem that enables a single follower robot (thephotographer) to quickly re-establish visual contact with a mov-ing target (the subject) that has escaped the follower’s field ofview. We deal with this scenario, which reactive controllers aretypically ill-equipped to handle, by making plausible predictionsabout the long- and short- term behavior of the target, andplanning pursuit paths that will maximize the chance of seeingthe target again. At the core of our pursuit method is theuse of predictive models of target behavior, which help narrowdown the set of possible future locations of the target to afew discrete hypotheses, as well as the use of combinatorialsearch in physical space to check those hypotheses efficiently.We model target behavior in terms of a learned navigationreward function, using Inverse Reinforcement Learning, basedon semantic terrain features of satellite maps. Our pursuitalgorithm continuously predicts the latent destination of thetarget and its position in the future, and relies on efficientgraph representation and search methods in order to navigateto locations at which the target is most likely to be seenat an anticipated time. We perform extensive evaluation ofour predictive pursuit algorithm over multiple satellite maps,thousands of simulation scenarios, against state-of-the art MDPand POMDP solvers. We show that our method significantlyoutperforms them by exploiting domain-specific knowledge,while being able to run in real-time.

I. INTRODUCTION

This paper proposes a model-based predictive pursuitalgorithm that enables a robot photographer to maintainvisual contact with a subject whose motion is independentof the photographer. We explicitly address the case wherevisual contact is lost and propose an algorithm that aimsto minimize the expected time to recover it. This is novelwith respect to previous work, particularly pursuit-evasiongames, which do not consider the scenario of intermittentloss of visibility, which is inevitable in practice1.

We make use of navigation demonstrations by the target,which we exploit via Maximum Entropy Inverse Reinforce-ment Learning (IRL) [1] in order to estimate a rewardfunction that expresses their preferences over the terrainfeatures of the map. For example, the target might preferto take paved roads, as opposed to grass or sand, in order tonavigate to its destination. We can integrate that informationin a behavior model for more accurate predictions, thus

*This work was supported by the Natural Sciences and EngineeringResearch Council (NSERC) of Canada, and the NSERC Canadian FieldRobotics Network

The authors are affiliated with the Mobile Robotics Lab, the Centerfor Intelligent Machines (CIM), and the School of Computer Scienceat McGill University in Montreal, Canada. {florian, nikhil,dudek}@cim.mcgill.ca

1In many variants of pursuit-evasion games, once line-of-sight is broken,the game ends.

Fig. 1. Red denotes destinations. (A) The follower sees the target. (B)Visual contact is lost. Destinations are predicted and particles (green) startto diffuse. (C) Particles split on two different roads. The follower choosesone group of particles as more promising. (D) The follower re-establishesvisual contact. Destinations on the top side of the map become unlikely.

increasing the chance of re-establishing visual contact. Mostimportantly, feature preferences can be learned in a data-efficient way from as few as ten demonstrations, on a singlemap, and are typically transferable to other maps that sharethe same appearance and semantic classes. Using the rewardlearned from IRL and treating the target as an efficientnavigator with respect to that reward, we identify a class ofstochastic policies that can very efficiently simulate target be-havior without having to perform planning at runtime for thepurpose of prediction. This learned predictive and generativemodel for plausible paths marks an improvement over ourprior work on this topic [2], which did not make use of priorknowledge of the target’s navigation behavior. We assumethat the robot photographer receives noisy observations ofthe target within a limited-range and field of view sensor,but has no other means of communication. We also assumethat the target navigates purposefully while heading for adestination, and its motion is independent of the actions ofthe photographer, as opposed to being evasive or cooperative.

Our method operates on both positive and negative obser-vations and maintains a belief distribution on the target’spossible locations. While the target remains unseen, thebelief expands spatially and uncertainty grows. Our planningalgorithm generates actions that shrink uncertainty or evencompletely clear it. The key enabling insight that allowsus to plan over the vast array of potential target paths isthe use of the learned navigation reward, which we use toreduce the set of possible hypotheses into a few possibilitiesthat are compatible with the target’s demonstrated navigationbehavior and the set of available destinations.

II. RELATED WORK

Our problem is related to the large body of existing workon the theoretical bounds related to maintaining persistenttracking of a moving target. This problem has receivedextensive consideration in the literature of pursuit-evasiongames. For most work in this category, issues related tothe optimization of visibility based on the structure of theenvironment are critical, so there are natural connections withthe classic “art gallery” problem and its variants [3].

Lavalle et al. [4] consider the problem of planning thetrajectory of a single observer trying to maximize visibilityof a fully-predictable or partially-predictable target. Gener-ally, continuous visibility maintenance has been consideredmostly for the case of adversarial targets. Such is the casefor example in [5], [6], [7], [8]. The question of identify-ing whether a target is being cooperative or evasive andmodifying the pursuer’s plans in response to this degreeof information is something that currently lacks a precisecharacterization in the literature. Nevertheless, it is worthmentioning that if the target is adversarial there exist settingsunder which the evader can escape the follower’s field ofview, no matter what the follower does ([5], [6]).

The persistent tracking problem has also been addressedby approximate sampling-based POMDP solvers such asSARSOP [9], which extends a Monte Carlo Search Treeand maintains both the upper and lower bounds of thestate-action value function. This enables pruning of provablysuboptimal actions, and has been an idea that has beenadopted in multiple subsequent solvers. The main limitation,however, when applied to our problem is that it works onlyon small-sized grids and runs in the order of minutes. RecentPOMDP solvers such as DESPOT [10] alleviate some ofthese problems, by sampling a finite set of plausible futurescenarios.

III. BACKGROUND

A. Maximum Entropy Inverse Reinforcement Learning

Making long-term and short-term predictions about thefuture behavior of a purposefully moving target requiresthat we know the instantaneous reward function that thetarget is trying to approximately optimize. Given a set ofdemonstration paths that trace the target’s motion on a map,we can infer which parametric transformation of features ofthe map explains them, using the framework of MaximumEntropy Inverse Reinforcement Learning (IRL) [1]. In thisframework the target is treated as planning in a MarkovDecision Process (MDP) whose parametric reward functionrθ(s, a) is unknown, and to be estimated from the demon-strated paths.

We assume the existence of feature vectors f(s) =[f1(s), f2(s), ..., fN (s)]> where s is a location on the map.In our case, these N feature maps are boolean images thatindicate semantic labels over satellite maps, such as roads,vegetation, trees, automobiles, buildings, as well as dilationsof these boolean maps. We also assume that the instan-taneous reward is a linear transformation of the features,

rθ(s, a) = θ>f(s), parameterized by the weight vector θ.Maximum Entropy IRL factors the probability distributionover trajectories τ = (s0, a0, s1, ..., aT−1, sT ) as an instanceof the the Boltzmann distribution:

p(τ |θ) =1

Z(θ)exp

(T∑t=0

θ>f(st)

)(1)

where Z(θ) is the partition function that acts as a normaliza-tion term. In this formulation, paths that have accumulatedequal reward will be assigned the same probability mass,and differences in cumulative reward will be amplifiedexponentially. This distribution arises from the constrainedoptimization problem of maximizing entropy subject to first-moment matching constraints, which specify that the ex-pected feature count at each state should be the same as theempirical mean feature count in the demonstration dataset.It is worth mentioning that Eqn. 1 assumes deterministicdynamics for the system being modeled, as well as no noisein the observation of the demonstrated trajectories.

Learning in the Maximum Entropy IRL model can be doneby maximizing the log-likelihood function of the trajectoriesin the demonstration dataset S, as follows:

θ∗ = argmaxθ

|S|∑i=1

log p(τi|θ) (2)

= argmaxθ

|S|∑i=1

Ti∑t=0

(θ>f(s

(i)t )− log Z(θ)

)(3)

This optimization problem is convex for deterministic dy-namics, so gradient-based optimization methods are suf-ficient to solve it. In our case we use the exponen-tiated gradient ascent method from [11] to obtain anestimate of the optimal reward weight parameters. Thegradient of the scaled log likelihood function L(θ) =1|S|∑|S|i=1

∑Tit=0

(θT f(s

(i)t )− log Z(θ)

)is

∇θL(θ) =1

|S|

|S|∑i=1

Ti∑t=0

(f(s

(i)t )−∇θlog Z(θ)

)(4)

=1

|S|

|S|∑i=1

Ti∑t=0

(f(s

(i)t )− dZ(θ)/dθ

Z(θ)

)(5)

=1

|S|

|S|∑i=1

Ti∑t=0

f(s(i)t )−

∑τ

p(τ |θ)fτ (6)

= f −∑s

p(s|θ, π)f(s) (7)

where fτ =∑s∈τ f(s) is the vector of features accumulated

by the trajectory τ , p(s|θ, π) is the probability of visitingstate s from a policy that solves the MDP with the inducedreward function rθ, and f is the empirical average of featuresin the dataset. The Maximum Entropy IRL algorithm forestimating the optimal reward weights is shown in Alg. 1.

Algorithm 1 MaxEnt IRL1: S : set of demonstrated trajectories2: θ ← θ0

3: while not converged do4: Use value iteration to solve MDP(θ)

for the optimal policy π(a|s, θ)5: Compute state visitation frequencies p(s|θ, π)

using dynamic programming as in [1]6: Compute gradient as in Eqn. 77: θ ← θ exp(α∇θL(θ)) [11]

IV. TERRAIN-BASED PREDICTION MODEL FORNAVIGATION

Once we learn the parameters of the target’s rewardfunction from a set of example paths we can make predic-tions about the target’s actions. We treat the target as anefficient navigator through the environment, approximatelyoptimizing its trajectory while trying to reach its destination.We want to account for the fact that the target’s optimizationprocess is approximate. In existing literature [12], [13], oneway to perform approximate planning on MDPs, and alsoin particular in step 4 of Alg. 1, is done through softmaxvalue iteration. Instead of the standard update rules in valueiteration:

Q∗θ(s, a) = rθ(s, a) + V ∗θ (s′) (8)V ∗θ (s) = max

aQ∗θ(s, a) (9)

where (s, a) → s′ is the next state (from deterministicdynamics), the max operation is replaced by a softmax:

Qθ(s, a) = rθ(s, a) + Vθ(s′) (10)

Vθ(s) = softmaxa

Qθ(s, a) (11)

In order to sample paths on which the target can movetowards one of the available destinations, we perform of-fline computation of one approximate value function perdestination d, denoted here by V

(d)θ (s). This denotes the

approximate value of the optimal path from location s todestination d on the map. We use similar notation for thestate-action value function Q(d)

θ (s, a).Given a value function for each destination, we can model

the distribution of paths, conditioned on their endpoints, bycomparing a path’s accumulated value to the value of theoptimal path on those endpoints:

p(τA→B |start A, dest C) ∝exp

(Rθ(τA→B) + V

(C)θ (B)

)exp

(V

(C)θ (A)

)(12)

This model is useful in order to perform destination predic-tion, given the history of a path so far:

p(dest C|start A, τA→B) ∝ p(τA→B |start A, dest C)p(dest C)(13)

where p(dest C) is the initial prior distribution on thedestinations. In our case this is uniform, and the set ofpossible destinations is already known and of small size.

Offline pre-computation of the value function for eachdestination d also gives us access to a destination-conditionalstochastic policy:

πθ(a|s, d) = exp(Q(d)θ (s, a)− V (d)

θ (s)) (14)

which amplifies the advantage of action a at the given state.If iteratively applied from any location in the map, thisstochastic policy typically leads to an attractor2 at the givendestination d. Examples of this are shown in Fig. 2. In effect

Fig. 2. 100 paths sampled by iterative application of the stochastic policyof Eqn. 14 at test time. Homotopically-distinct samples are possible, as longas the approximate value function is comparable between the two homotopyclasses. In practice, we observed that this is not a typical event.

this gives us a fast and direct way of sampling paths from anylocation to a specific destination, as opposed to performingMCMC or other sampling methods on Eqn. 12. This has acrucial effect on our probabilistic pursuit algorithm, becauseit enables the generation of possible paths without anyplanning at runtime for the purposes of prediction, whichallows the pursuer to dedicate its resources to formulating anavigation plan that will re-establish visual contact.

V. MODEL-BASED SINGLE-FOLLOWERPROBABILISTIC PURSUIT

We want to enable a single agent to persistently follow atarget that is independent of the follower and which movespurposefully in an environment with obstacles. We assumethe follower is equipped with: (a) knowledge of the map andits pose in it (b) demonstrated trajectories that the target hasexecuted in the past in similar environments, revealing itspreferences with respect to both potential destinations androutes that lead to them.

We pose the probabilistic pursuit problem as an integratedcombination of planning and prediction of the future long-term behavior of the target’s state. One way to see this is asa reinforcement learning problem, where the discrete statest of the pursuer-target system is defined as st = [xt, yt, dt]where xt and yt denote the planar poses (row, column, andyaw) of the target and the pursuer respectively, and d denotesthe latent destination of the target. The instantaneous reward

2As long as the softmax value iteration has converged and the featuremaps in the given environment are not conflicting.

is the indicator function of whether the follower sees thetarget or not. In particular, if we let FOV(yt) denote the fieldof view of the follower at time t, then the reward functionis r(st, at) = 1[xt∈FOV(yt)], where at is the action that thefollower takes at that time.

The follower does not always have full information aboutthe pose of the target, so it needs to maintain a beliefbel(st) = p(st|ht) about the system’s state st, given a historyvector ht = [z1:t, a1:t], where z1:t is the history of sensor ob-servations 3. We assume that the follower has full knowledgeof its own state as well as of the map, so the bel(st) is really4

bel(xt, dt), which factors into p(xt|ht)p(dt|ht). Therefore,this formulation allows us to describe the problem as aPartially-Observable Markov Decision Process (POMDP),more specifically as a Belief MDP, for the target’s pose anda classification problem for its future destination. We assumethat the initial distribution over destinations is uniform.

The POMDP transition model Tθ(st+1, st, at) =p(st+1|st, at, θ) is stochastic, and depends on the behaviorof the target, parameterized here by the vector θ of rewardweights, which we learn through inverse reinforcement learn-ing. The transition model factors into p(st+1|st, at, θ) =p(xt+1|xt, dt, θ)p(yt+1|yt, at)p(dt+1|dt, xt). We assume forsimplicity that the follower’s dynamics model is determinis-tic5, so p(yt+1|yt, at) = 1[(yt,at)→yt+1]. We also assume thatthe destination of the target is a discrete latent variable thatremains constant throughout the duration of the experiment,so p(dt+1|dt, xt) = 1[dt=dt+1]. This formulation is related tothe POMDP-lite [14] and the Hidden Parameter MDP [15]formulations. The POMDP transition model is thereforeTθ(st+1, st, at) = p(xt+1|xt, dt, θ) which is the learnedstochastic policy in Eqn. 14.

One of the advantages of using behavior predictionmodels based on reward parameters learned through IRL,is that we have access to a very fast generative model(xt+1, yt+1, zt+1, rt+1) ∼ Gθ(xt, yt, dt, at) that simulatesthe system. The more representative this simulator is of thetarget’s actual behavior, the more certain the follower is aboutthe value of each available action.

The POMDP transition model that we use accounts for thepossibility of false negative detection errors (not recognizingthe target when it is in the field of view of the follower),which is a common case for visual object detectors, partic-ularly at large distances. On the other hand, we assume nofalse positive detections (recognizing the target when it isin fact not there). The observation model we use has falsenegative detection probability of q:

p(zt = ∅|st) =

{q if xt ∈ FOV(yt)

0, otherwise(15)

The pursuit problem of finding the target after it has es-caped the follower’s field of view can be formulated as

3zt = ∅ indicates not seeing the target. zt = xt indicates it was seen4This assumption allows us to decouple the tracking problem from that of

Simultaneous Localization and Mapping, although this work is not affectedshould one need to model localization uncertainty.

5Note that the methods presented here do not rely on this assumption.

finding a pursuit policy π(at+1|ht) that maximizes thefollower’s value function6 V π(ht) = Eπ,Tθ [Rt|ht], whereRt =

∑t+Ti=t γ

i−tr(st, at) is the discounted cumulativereward function. Solving this problem exactly is in generalintractable, so sampling-based POMDP solvers, such as [16],[17] have been introduced in the literature.

Instead of resorting to general-purpose POMDP solversthat need to maintain some level of suboptimal exploratorybehavior, we design an algorithm that exploits domainknowledge about the problem and uses a greedy planningbehavior to determine which location to navigate to next,in order to maximize the chance of re-establishing visualcontact with the moving target. We demonstrate in theresults section that this approach outperforms state-of-theart POMDP solvers and provides a better solution to theproblem. In the sections that follow we present an analysisof the different components of this algorithm, which isdescribed in detail in Alg. 2.

A. Particle filter and Bayesian updates of the belief

We represent the belief bel(xt) about the target’s locationas a set of particles x(i), whose weights w(i) are updated by aparticle filter. When a particle is created we associate it witha fixed speed, a fixed destination d(i) and a single path γ(i) tothat destination. It travels along this path without deviation.The destination of the particle is sampled according toEqn. 13. The sequence of states on the path are samplediteratively through actions from the learned policy in Eqn. 14.

We model the motion of the particle as always travelingat that speed, and we rely on including enough particles inthe belief to represent a wide range of speed combinations,without modeling variable speed on any single particle. Thisis a crucial element because it allows us to deterministicallypredict where each particle is going to be at a specific time.

The transition model: The design choice of fixedspeed and fixed path makes the problem of predictingwhere the particle is going to be in a few time stepspurely deterministic. In our case the only stochasticityfor a particle’s transition model arises from the initialchoice of destination d(i), the reference path γ(i) to it,and the fixed speed v(i) at which the particle traversesthe path. After those stochastic samples have beendrawn the motion of the particle is deterministic. So,the particle dynamics model, p(x

(i)t+1|x

(i)t , d(i), at) =∑

γ,v p(x(i)t+1|x

(i)t , d(i), at, γ, v)p(γ, v|x(i)

t , d(i), at),which after the independence assumptions simplifiesto p(x

(i)t+1|x

(i)t , d(i), at, γ

(i), v(i))p(γ(i)|d(i))p(v(i)), dependsentirely on the sampled path and speed. The conditionaltransition model p(x(i)

t+1|x(i)t , d(i), at, γ

(i), v(i)) is completelydeterministic [18] in our system.

We model the speed distribution for particles using twocriteria: first, that we guarantee that the maximum possibletarget velocity is assigned to many particles, so that theirdiffusion can catch up with the target’s motion, even if it is

6Note that the follower’s value function and policy are different than thetarget’s value function and policy.

being evasive; second, that no particles are too slow becausethey might cause the follower to stay behind and try toclear up hypotheses that move very slowly, while the targetprogresses to its destination. Thus, we set:

p(v) =

{Uniform[vmax/2, vmax] b = 1

1[v=vmax], otherwise(16)

where b ∼ Bernoulli(0.5) and vmax is the maximum speed ofthe target. Approximately half the particles have full speedunder this model while the remaining half will be at leasthalf as fast as the target. Again, this encourages progresstowards the known destination.

The observation model of the particle filter incorporatesa maximum range and field of view limitation to modeldepth sensors such as the Kinect, but also to reflect the factthat today’s target trackers are not generally reliable at largedistances. The observation model may also encode particular-ities of the detector, such as its susceptibility to false negativeerrors as well as dependence to viewpoint, which is becomingless crucial given recent advances in object detection usingsupervised deep learning. The observation model we use inthe particle filter is the one presented in Eqn. 15.

The choice of the particle filter as the main Bayesianfiltering mechanism was made because it is able to incor-porate negative information, in other words, not being ableto currently see the target makes other locations outside thefield of view more likely. This is something that most otherfilters cannot provide.

B. Navigation

Our algorithm assumes for simplicity that navigation isdone on a 2D plane. We start by using the Voronoi diagramof the environment to find points that are equidistant fromat least two obstacles, and on top of that structure we buildthe Generalized Voronoi Graph [19]. Its edges include pointsthat are equidistant to exactly two objects, while its nodesare at the remaining points of the Voronoi diagram, either asmeetpoints of at least three edges, or as endpoints of an edgeinto a dead-end in the environment. These two structuresprovide a roadmap for navigation in the environment onwhich graph-based path planning takes place.

One of the main advantages of using the structure ofthe GVG in order to facilitate topological and shortest pathqueries is that for realistic environments its size is usuallysmall enough to be suitable for real time reasoning. Thepath obtained from executing a shortest-path algorithm on theGVG is not suitable for fast navigation, in terms of length,curvature and appearance. Shortest paths on the GVG arenot globally optimal paths in the rest of the environment. Wepartially address this issue by using an iterative refinementprocedure, where we replace parts of the path that arejoinable by a straight line with the points on that line,until little improvement is possible. This is essentially theDouglas-Peucker algorithm, referred to here as REFINE.

We use Yen’s K-shortest paths algorithm [20] on the GVGto compute multiple topologically-distinct paths to a givengoal, and then we iteratively refine each of the paths to

reduce their length. We use this as a heuristic in order toavoid the scenario where the shortest path found on theGVG belongs to a homotopy class that does not containthe globally optimal path to the goal. After executing Yen’salgorithm and refining the paths, we return the one withshortest length as an estimate of the shortest path.

Once we have the shortest path, we compute the optimalvelocity and linear acceleration controls that will traversethe path in minimum time. We assume that the followerdynamics is omnidirectional y′ = φ(y, a) = y + aδt, whichsimplifies significantly optimal control for path execution.

C. Pursuit algorithm

Our pursuit algorithm incorporates two modules, depend-ing on whether the target is currently in view or not. Ifit is, the follower uses reactive feedback PID or model-predictive control to reach a so called “paparazzi” referenceframe behind the target (or whichever the desired viewingpose happens to be). This frame of reference gets updatedas the target moves in compliance with the surroundingenvironment, so as to not hit any obstacles. This is illustratedin Fig. 3. When visual contact is lost, planning replaces

Fig. 3. Paparazzi frames around the current target pose. They describeconfigurations from which the follower can observe the target. Trees aretreated as obstacles. Yellow denotes the field of view. Better seen in color.

reactive following. We sample destinations and paths forthe particles, as explained in the previous section, and westart the sampling and resampling steps in the particle filteraccording to incoming observations. During a sequence ofincoming observations with no detections, the distribution ofthe destinations p(dt|ht) is not updated; it is only updatedwhen the target is detected.

The follower plans a path that will lead to a high-valuelocation, as outlined in Alg. 2, and executes that path. Whenthe end of path is reached, if unsuccessful, the follower plansanother one. If the target re-enters the field of view, thepath hypotheses and the particles are discarded and reactivecontrol resumes its operation.

In the previous section we insisted on being able todeterministically query where a particle is going to be ata particular time in the future. The main reason behindthis was to be able to compute the possible times andplaces at which the follower could visually intercept it. Tothis end we assume the functionality of a function called

T, τ = MIN-TIME-TO-VIEW(GVG, yt, p), which is a tra-jectory planner in space and time that enables the followerto navigate from its current state yt to the paparazzi frame pin minimum time, so as to bring itself in view of a potentialtarget location, as quickly as possible. T is the time it willtake to navigate to p, and τ is the trajectory that is planned.We implemented MIN-TIME-TO-VIEW for the case wherethe follower is an omnidirectional robot, by using iterativerefinement of the shortest path obtained from the GVG, asdescribed in the previous section7.

We restrict the set of candidate locations for visual in-terception to be points along the reference paths of theparticles, for the sake of computational efficiency. Ourpursuit algorithm computes the minimum times to reachpaparazzi frames for a set of waypoints along these referencepaths. The possible waypoints include the final destinationof the target. So, heading directly to the destination withoutintermediate stops is one of the considered strategies. Thealgorithm then selects as the next navigation waypoint thepaparazzi frame that sees the particle with the highest ratioof probability over distance , and heads over to reach thatpaparazzi frame. This is a greedy nearest neighbor algorithm,and we term it NNm for “nearest neighbor pursuit alongmultiple homotopy classes”, or more simply, “topologicalpursuit.”

VI. RESULTS

A. Setup

We set up a simulation environment in order to benchmarkour algorithm against existing MDP and POMDP solvers.This environment includes 20 different aerial images, withtop-down view, shown in Fig. 7, each of which covers areaswhere the dominant semantic labels of the terrain are: roads,vegetation, trees, buildings, and vehicles.

Each map was annotated offline by human annotators, whoalso provided examples of target trajectories to pre-specifieddestinations, from various starting points on the aerial image.The 280 target trajectories, which were demonstrated acrossthe range of all maps, expressed preference for roads andvegetation, avoiding trees and buildings, which were treatedas obstacles.

We compared our algorithm with variations on the threefollowing baseline methods: (i) full-information pursuit, (ii)UCT [21], and (iii) POMCP [16]:

(i) Full-information pursuit refers to the variant of theproblem, where the follower knows a priori the trajectoryof the target, so it can make use of techniques similar toModel Predictive Control (MPC), which recompute a controlsequence at each time step, based on known dynamics, andselect the best action at each time step, discarding the rest ofthe plan. Following similar rationale, at each time step, ourfull-information pursuit algorithm replans a trajectory fromthe current state of the follower to the closest valid paparazziframe for the target, and executes the first action prescribed

7Trajectory optimization and following is required for other types ofdynamical systems, but that is outside the scope of this paper.

Algorithm 2 NNm(zt, bel(xt) = {(x(i)t , w(i))}i=1...M , yt)

1: zt : the follower’s observation2: bel(xt) : the follower’s belief about the target’s pose3: yt : the follower’s pose4: D : set of potential destinations5: if zt = xt i.e. the target is visible then6: bel(xt+1)← PF-UPDATE(GVG, bel(xt), zt, yt, D)7: xt ← desired paparazzi frame based on xt

(e.g. behind the target)8: δθ, δr ← yt − xt9: at ← feedback control from δθ, δr

10: yt+1 ← φ(yt, at)11: Update p(dt|ht) as in Eqn. 1312: NNm(zt+1, bel(xt+1), yt+1)13: else14: for i = 1...M do15: Sample destination d(i)

t ∼ p(dt|ht) as in Eqn. 1316: Sample path γ(i) ∼ p(γ|d(i)

t ) as in Eqn. 1417: Sample speed v(i) ∼ p(v) as in Eqn. 1618: Assign path γ(i) and speed v(i) to particle x(i)

t

19: Tji, τji ← MIN-TIME-TO-VIEW(GVG, yt, pji),∀j, ifor paparazzi frames pji sampled along γ(i)

20: ∆lji,∆θji ← total length and rotation in the path πji21: I ← (i, j) such that particle x(i)

t is predicted to reachpji ∈ γ(i) close to follower’s arrival time Tji

22: i∗, j∗ ← argmax(i,j)∈I

w(i)

∆lji+∆θji

23: while not reached pj∗i∗ do24: bel(xt+1)← PF-UPDATE(GVG, bel(xt),

zt, yt, D)25: at ← next action in follower’s trajectory τj∗i∗26: yt+1 ← φ(yt, at)27: t← t+ 1

28: Go to step 5

by that trajectory, similarly to MPC. The performance ofthe full-information pursuer provides an upper bound on theperformance of pursuit methods, in which the state of thetarget may be latent.

(ii) UCT is a sampling-based MDP solver that grows aMonte Carlo Search Tree, using the Upper Confidence Bound(UCB1) rule to trade off exploration vs exploitation. Weapply UCT to the variant of our problem in which the stateof the target is revealed to the follower at each time step, butthe future states of the follower-target system are stochastic,according to the latent preferences of the target, such asdestination, route, type of terrain etc.

(iii) POMCP is one of the state-of-the-art sampling-basedPOMDP solvers. Like UCT, it also performs Monte CarloTree Search. It differs from UCT by the fact that eachstate node in the tree contains action edges that lead toobservation nodes, as opposed to other state nodes. It avoidspropagating and updating the particle filter belief at eachsimulation path through the tree, by sampling a particle fromthe belief according to its weights in the filter, and simulating

its evolution through the tree. POMCP addresses the sameversion of the problem that our method does. It is worthnoting that both in our method as well as in POMCP we usethe same particle filter settings, and the same learned IRLreward weights, so the prediction mechanism for the targetbehavior is identical among the two methods8.

We compared our methods against these baselines acrossapproximately 4500 experiment scenarios in total, each ofwhich was repeated under the exact same settings across5 different episodes, to get an estimate of variance inthe outcomes of each scenario. These experiment scenariosincluded variations on:• the start and destination• the follower’s / target’s maximum speed• the time limit allowed to UCT and POMCP to generate

a single action• the probability of false negative detections• the speed profile of the target, as it executes its trajectory

For the performance of UCT and POMCP baselines wereport the success rate for planning each action within 5seconds; this of course implies offline operation for thesemethods, whereas our topological pursuit method runs inreal-time at 1Hz.

B. Findings

Our experiments demonstrate that our method outperformsboth POMCP and UCT when the follower and the targetshare the same maximum speed. This is better shown inFig. 4, which demonstrates the duration of successful pursuit(i.e. how long the follower sees the target) relative to thefull-information pursuit, which achieves 100% success rate,managing to maintain visual contact with the target duringthe entire duration of the experiment. In Fig. 4 the probabilityof false negative observations is set to 0.5, which meansthat half the time, when the target is in the field of viewof the follower, it is not detected, in addition to other timeswhen visual contact is necessitated by the structure of theenvironment. Our method scores on average approximately40%, whereas POMCP is at 15% and UCT is under 5%.The main reason why POMCP and UCT perform poorly inthis regime of equal maximum speed for the follower andthe target is that any suboptimal pursuit actions performedearly on in an episode have critical consequences throughoutits duration, in the sense that there is little time to recovervisual contact if the target is moving at full speed. POMCPand UCT are prone to committing such suboptimal movesdue to their tendency to explore and be optimistic in the faceof uncertainty, whereas our algorithm is greedy and focusedon navigating to a single destination at a time, so it does notmake use of exploratory actions. Additionally, without anexplicit penalty term in the design of the reward function,actions produced by UCT and POMCP tend to have highvariance as well, which is problematic for deployment in areal vehicle.

8Also for both UCT and POMCP the Upper Confidence Bound explo-ration constant was set to

√2, and the default leaf expansion policy is the

uniform distribution.

0 5 10 15 20

satellite maps

0

20

40

60

80

dura

tion

ofvi

sual

cont

act

%

Topological pursuit (ours) POMCP UCT

Fig. 4. Relative duration of visual contact across the available test maps.Bars denote 1σ standard deviation. Averages were taken with respect totarget trajectories and episodes. Colors denote algorithm types: topologicalpursuit (red, our method), POMCP (purple) and UCT (green) algorithms(See text).

1.0 1.5 2.0 2.5

max follower speed / max target speed

0

10

20

30

40

50

60

70

80

dura

tion

ofvi

sual

cont

act

%

Topological pursuit (ours) POMCP

Fig. 5. Relative duration of successful pursuit as a function of maximumspeed advantage of the follower. Bars denote 1σ standard deviation. Aver-ages were taken with respect to maps, target trajectories, and episodes.

As the follower becomes faster relative to the target, sub-optimal actions for exploration become less critical becausethere is time for the pursuer to recover visual contact withthe target. This is better illustrated in Fig. 5, which showsthat the tracking performance of POMCP gradually improvesas the follower becomes twice as fast as the target. Thisis not surprising as POMCP is an anytime algorithm. Ourmethod, however, does not demonstrate the same property.One hypothesis that might explain this has to do with step21 of Alg. 2, where we are searching over points along thesampled particle paths, such that the particles are projected toarrive there approximately at the same time as the follower,which means that they are suitable to become rendezvouslocations. As the relative speed of the follower increases, therange of locations that are reachable by the target at the sametime as the follower becomes smaller and smaller, whichmeans that the number of options for the next best paparazziframe to navigate to is gradually reduced. As the numberof good hypotheses for rendezvous locations becomes morelimited the quality of pursuit deteriorates.

It is worth mentioning, however, that many practicalpursuit scenarios belong in this regime, where the pursuerdoes not need to be more than twice as fast as the target(aerial photography with vehicles carrying heavy cameraequipment, or diving assistant underwater robots). In thisneighborhood of relative speed advantage our algorithm stilloutperforms Monte Carlo Tree Search-based methods.

0.1 0.2 0.3 0.4 0.5

false negative detection probability

0

10

20

30

40

50

60

dura

tion

ofvi

sual

cont

act

%

Topological pursuit (ours) POMCP

Fig. 6. Relative duration of visual contact as a function of the false negativedetection rate. Bars denote 1σ standard deviation. Averages were taken withrespect to maps, target trajectories, and episodes.

The third axis of variation that we examined is the falsenegative detection rate, whose effect is shown in Fig. 6. Ourmethod is more robust compared to POMCP, with more than3x longer tracking on average, when the observation modelis prone to false negative errors with probability up to 0.5,which is significant, particularly in vision-based tracking anddetection at large distances, where the target occupies a smallpart of the image.

VII. CONCLUSIONS

We presented a real-time predictive pursuit algorithm thathandles the case of loss of visual contact during tracking,and plans to re-establish it. Our method uses a behaviormodel of the target, learned from navigation demonstrationsand Inverse Reinforcement Learning, in order to predict thetarget’s short-term and long-term actions. This model is easyto sample from, and we use it to maintain predictions ofthe target’s latent destination, as well as to forecast the

Fig. 7. Some of the satellite maps and the human-annotated target pathsthat we used in our test set. Red denotes destinations.

route it will take to said destination, thereby being able togenerate plausible hypotheses about where the target is goingto be and when. Our physical search method exploits thisinformation to devise a search plan, which aims to recoverthe target in the follower’s field of view. We validate ourmethod by comparing it to state-of-the art MDP and POMDPsolvers, and we show that it outperforms them by significantfactors.

REFERENCES

[1] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximumentropy inverse reinforcement learning,” in 23rd National Conferenceon Artificial Intelligence - Volume 3. AAAI, 2008, pp. 1433–1438.

[2] F. Shkurti and G. Dudek, “Topologically distinct trajectory predictionsfor probabilistic pursuit,” IEEE International Conference on IntelligentRobots and Systems, pp. 5653–5660, September 2017.

[3] J. O’Rourke, Art Gallery Theorems and Algorithms. Oxford Univer-sity Press, 1987.

[4] S. M. Lavalle, H. Gonzalez-Banos, C. Becker, and J.-C. Latombe,“Motion Strategies for Maintaining Visibility of a Moving Target,” inIEEE ICRA, 1997, pp. 731–736.

[5] S. H. Sourabh Bhattacharya, “Approximation Schemes for Two-PlayerPursuit Evasion Games with Visibility Constraints,” in Proceedings ofRobotics: Science and Systems IV, Zurich, Switzerland, June 2008.

[6] S. Bhattacharya and S. Hutchinson, “On the Existence of NashEquilibrium for a Two-player Pursuit–Evasion Game with VisibilityConstraints,” The International Journal of Robotics Research, vol. 29,no. 7, pp. 831–839, Dec 2009.

[7] V. Isler, S. Kannan, and S. Khanna, “Randomized pursuit-evasion ina polygonal environment,” IEEE Transactions on Robotics, vol. 21,no. 5, pp. 875–884, Oct 2005.

[8] N. M. Stiffler and J. M. O’Kane, “Shortest paths for visibility-basedpursuit-evasion,” IEEE ICRA, pp. 3997–4002, May 2012.

[9] D. Hsu, W. S. Lee, and N. Rong, “A point-based POMDP planner fortarget tracking,” in IEEE International Conference on Robotics andAutomation, May 2008, pp. 2644–2650.

[10] A. Somani, N. Ye, D. Hsu, and W. S. Lee, “DESPOT: Online POMDPPlanning with Regularization,” in Advances in Neural InformationProcessing Systems 26, 2013, pp. 1772–1780.

[11] K. M. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert, ActivityForecasting. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012,pp. 201–214.

[12] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A.Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa, “Planning-basedprediction for pedestrians,” in IEEE/RSJ International Conference onIntelligent Robots and Systems, 2009, pp. 3931–3936.

[13] A. Tamar, S. Levine, and P. Abbeel, “Value iteration networks,”CoRR, vol. abs/1602.02867, 2016. [Online]. Available: http://arxiv.org/abs/1602.02867

[14] M. Chen, E. Frazzoli, D. Hsu, and W. S. Lee, “POMDP-lite forRobust Robot Planning under Uncertainty,” CoRR, 2016. [Online].Available: http://arxiv.org/abs/1602.04875

[15] F. Doshi-Velez and G. Konidaris, “Hidden parameter markov decisionprocesses: A semiparametric regression approach for discovering latenttask parametrizations,” in International Joint Conference on ArtificialIntelligence. AAAI Press, 2016, pp. 1432–1440.

[16] D. Silver and J. Veness, “Monte-Carlo Planning in Large POMDPs,”in Advances in Neural Information Processing Systems 23, 2010, pp.2164–2172.

[17] G. Shani, J. Pineau, and R. Kaplow, “A survey of point-based POMDPsolvers,” AAMAS, vol. 27, no. 1, pp. 1–51, Jul 2013.

[18] A. Y. Ng and M. Jordan, “PEGASUS: A Policy Search Method forLarge MDPs and POMDPs,” in 16th Conference on Uncertainty inArtificial Intelligence, 2000, pp. 406–415.

[19] H. Choset, “Incremental Construction of the Generalized Voronoi Di-agram, the Generalized Voronoi Graph, and the Hierarchical General-ized Voronoi Graph,” in CGC Workshop on Computational Geometry,October 1997.

[20] J. Y. Yen, “Finding the K-Shortest Loopless Paths in a Network,”Management Science, vol. 17, no. 11, pp. 712–716, jul 1971.

[21] L. Kocsis and C. Szepesvari, Bandit Based Monte-Carlo Planning.Springer Berlin Heidelberg, 2006, pp. 282–293.