ARE: Augmented Reality Environmentfor Mobile RobotsMario Gianni & Federico Ferri & Fiora PirriALCOR Lab, DIAG, Sapienza University of Rome

Contact Information:ALCOR Lab, DIAGSapienza University of RomeVia Ariosto 25, 00185 Rome, ItalySite: www.dis.uniroma1.it/∼alcorPhone: +39 0677274155Email: gianni@dis.uniroma1.it

AbstractARE is a development tool which allows cognitive robotics modelers to construct,

at real-time, complex planning scenarios for robots, eliminating the need to model thedynamics of both the robot and the real environment as it would be required by wholesimulation environments. The framework also builds a world model representation thatserves as ground truth for training and validating algorithms for vision, motion planningand control. The AR-based framework is applied to evaluate the capability of the robotto plan safe paths to goal locations in real outdoor scenarios, while the planning scenedynamically changes, being augmented by virtual objects.

IntroductionAugmented Reality (AR) has recently stepped beyond the usual scope of appli-cations like machine maintenance, military training and production [1].•AR facilitates robot programming [2].•AR allowing cognitive robotics developers to design a variety of complex

scenarios involving real and virtual components, in a flexible manner [6, 3].•AR eliminates the need to model the entire environment as it supplements the

real world instead of replacing it.•AR facilitate experiments bypassing complex simulation models, expensive

hardware setup and a highly controlled environment, in the various stages ofa cognitive robot development.

Figure 1: Degree of Complexity

ObjectivesIn this work, we present an AR-based simulation framework which allows robotdevelopers to build on-line an Augmented Reality Environment (ARE) for realrobots, integrated into the visualization interface of Robot Operating System(ROS) [5]. The system goes beyond an interface for drawing objects, as the de-sign exploits a stochastic model regulating both the life-cycle and the behaviorof the artefacts. Furthermore, a degree of certainty of the existence and behav-iors of the artefacts, with respect to what the robot perceives and knows aboutits space, can be tuned according to the experiment needs.

AR-based Simulation FrameworkThe AR-based simulation framework registers virtual objects, such as robots,cars, people, pallets and other kind of obstacles into the real environment.

Figure 2: Artefacts

The AR-based simulation framework includes:• The model of the real world.• The model of the Artefacts.• The AR-builder server.

The real world modelThe real world model comprises both the 2D occupancy grid mapM2D and theoctree-based 3D mapM3D [7, 8].

Figure 3: World Model

In addition a polygonal mesh SE is used to geometrically represent the environ-ment. This model is compact and better suited for•Updating.•Detecting collisions.•Handling occlusions.

The Artefacts modelAn artefact A is a dynamic object, defined by the following components:• The properties Q = {l, b,p(., t),q(., t),Φ}.• The life-cycle model {N(t)|t ≥ 0}.• The behavioral modelHA(t).• The polygonal mesh SA.

Figure 4: Artefacts components

Artefact properties are sampled in the space Q of all possible tuples, definingan artefact type, according to a mixture of n Poisson distributions:

Pr(X = k|λ1, . . . , λn) =

n∑i=1

πiλkik!

exp(−λi)

The life-cycle of the artefacts, namely the arrivals and the leavings of the arte-facts in M2D, is ruled by a time-homogeneous irreducible, continuous-timeMarkov chain {N(t)|t ≥ 0} with state space N+, whose stationary transitionprobabilities pij(∆t) between state i and state j, for infinitesimal lapses of time∆t, are given by

pij(∆t) := P(N(t + ∆t) = j|N(t) = i) =

=

λν(M2D)∆t + o(∆t) if j = i + 1

1− (λν(M2D) + µ)∆t + o(∆t) if j = i

µ∆t + o(∆t) if j = i− 1

o(∆t) if |j − i| > 1

{N(t)|t ≥ 0} can be seen as a special case of a birth-death process [4], withbirth rates λi = λν(M2D) and death rates µi = µ, for each i ∈ N+.The behavior of an artefact is determined by a finite-horizon Markov DecisionProcess

HA(t) = {D,S,As, pt(·|s, α), rt(·|s, α) : t ∈ D, s ∈ S, α ∈ As}

•D = {0, . . . , d} set of decision epochs.• S ≡M2D × {0, π2 , π,

32π, 2π} state space.

• As set of possible actions.• α : B × Π× S 7→ (k1, . . . , km)> behavior-motion mapping.HA(t) selects the motion actions up to the time horizon d, according to theunderlying action policy.

The AR-builder server

The AR-builder server interconnects the real environment model together withthe simulation model of the artefacts. The AR-builder server relies on the tf

software library in order to suitably register the artefacts into the real environ-ment. The AR-builder server correctly places an artefact within the real envi-ronment, by both projecting its bounding box b on M2D and concatenating thevertexes of the polygonal mesh SA to the voxels of M3D.

Figure 5: The augmented model of the real world

The AR-builder server implements a collision detection algorithm, based onpairwise hit-testing. Collisions are resolved by either moving back the artefactto its last known safe pose or by allowing the artefact to move up to a safe dis-tance. The AR-builder server checks occlusion effects by implementing a raytracing version of the z-buffer algorithm. The algorithm relies of the followingstructures:• The view model Fview = {P, (W,dmax)} of the real robot.• The occlusion matrix ZS3D = {dki,j|d

ki,j ∈ [0, dmax] ,∀i, j} of the implicit

surface of the polygonal mesh S3D of the real environment.• The occlusion matrix ZA of each artefactA, populating the real environment.

The set of artefacts perceived by the real robot is

Af = {a|count((ZA − ZS3D)ij ≤ 0) > τ}

ExperimentsTwo different experiments have been settled, where ARE has been used to pop-ulate the real surroundings with artefacts, in order to• check the robot ability to replan the path towards a goal location, as the fre-

quency of the arrivals of the artefacts into the environment changes;• test the long-term capability of the robot to navigate the cluttered environ-

ment in order to reach several goal locations.

Figure 6: Numerical results

To measure the robot ability to replan the following time ratio is introduced

ρ =ρt

ρt + Gt

The long-term capability of the robot to navigate the cluttered environment hasbeen measured with respect to the space complexity of the environment. Thiscomplexity is defined by the following space ratio:

ν =nAnfree

Figure 7: The augmented reality view in ROS rviz

ConclusionsWe proposed a framework to augment the robot real world that advances thestate of the art, as it introduces, together with the augmented environment, alsothe robot perceptual model of the augmented environment and the possibility oftuning the degree of confidence and uncertainty of the robot on what it is pre-sented in the augmented scene. The framework has been used to test both theshort-term and the long-term capability of the robot to navigate the real envi-ronment, populated by artefacts, as the complexity of the environment changed,according to the parameters of ARE.

AcknowledgementsThe research has been funded by EU-FP7 NIFTI Project, Contract No. 247870.

References[1] Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon

Julier, and Blair MacIntyre. Recent advances in augmented reality. IEEEComput. Graph. Appl., 21(6):34–47, 2001.

[2] J. W. S. Chong, S. K. Ong, A. Y. C. Nee, and K. Youcef-Youmi. Robotprogramming using augmented reality: An interactive method for plan-ning collision-free paths. Robotics and Computer-Integrated Manufactor-ing, 25(3):689–701, 2009.

[3] T.H.J. Collett and B.A. MacDonald. Augmented reality visualisation forplayer. In ICRA, pages 3954–3959, 2006.

[4] Geoffrey Grimmett and Geoffrey Grimmett. Probability and Random Pro-cesses. Oxford University Press, Third Ed., 2001.

[5] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler,and A. Ng. Ros: an open-source robot operating system. In ICRA Workshopon Open Source Software, 2009.

[6] Michael Stilman, Philipp Michel, Joel Chestnutt, Koichi Nishiwaki, SatoshiKagami, and James Kuffner. Augmented reality for robot development andexperimentation. Technical report, Robotics Institute, 2005.

[7] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics.MIT Press, 2005.

[8] Kai M. Wurm, Armin Hornung, Maren Bennewitz, Cyrill Stachniss, andWolfram Burgard. Octomap: A probabilistic, flexible, and compact 3d maprepresentation for robotic systems. In Proc. ICRA Workshop on Best Prac-tice in 3D Perception and Modeling for Mobile Manipulation, 2010.