Improving Augmented Reality Visualisation for Mobile RobotDevelopment

Daniel Brooker1,2, Toby Collett1, Andrew Graham1

1Inro Technologies Limited, Auckland, New Zealand2School of Engineering, The University of Auckland, Auckland, New Zealand

{dan.brooker,toby.collett,andrew.graham} at inro.co.nz

Abstract

Developing mobile robot applications in thereal world is difficult. ARDev is an opensource tool designed to create a shared percep-tual space for humans to see the world from arobots point of view. This is achieved by aug-menting a live video feed with virtual objectsresulting in an Augmented Reality visualisa-tion toolkit used to visualise robot data. Thisproject aims to expand and improve ARDevso that it may become more widespread andubiquitous within mobile robot development.The main contributions of this work are newfeatures including runtime configuration of ob-jects and views, control over a virtual cameraand further modularisation with the additionof plug-ins. Runtime configuration allows thedeveloper to change the visualised data depend-ing on the particular sensor or visualisation ofinterest. The virtual camera allows for differentviews of the same data or scene to increase thedevelopers understanding of sensor data. Plug-ins provide a convenient extension point for fur-ther development. Overall the contributions inthis paper have greatly improved ARDev formobile robot development.

1 Introduction

Developing mobile robot applications in a real world en-vironment presents a number of challenges. The chal-lenges are a result of the researcher developing the ap-plication often being remote from the robot, real envi-ronments are dynamic and the information provided bysensors cannot easily be interpreted in real-time withclassical techniques. Augmented reality (AR) provides ameans to solve many of these problems placing the sen-sor data geographically with the robot in real-time andthe associated physical surrounding.

The application domains for AR are diverse and rang-ing from medical displays [Sielhorst et al., 2008], to en-

tertainment [Lyu et al., 2005; Kirner et al., 2006], totreating psychological disorders [Juan et al., 2004]. Fo-cusing on robotics, AR has been used in [Brujic-Okreticet al., 2003] to aid an operator when manoeuvring a ve-hicle in limited or no visibility. The design of the ARsystem is presented discussing hardware and integrationof a near-scene map building algorithm. In [Chen etal., 2008] an AR system is used to aid in commission-ing helicopter tasks for agriculture. Their work focuseson tracking the camera pose using natural features asmarkers, with results given from a mock-up simulation.

It is generally agreed that it is the developers lackof understanding of the robot’s world view that makesit difficult to code new algorithms and tasks, debugproblems in the resulting actions and commission inte-grated systems for real world work. The problem of un-derstanding can be overcome with a shared perceptualspace [Breazeal et al., 2001]. AR provides this sharedspace between developers and robots enabling the devel-oper to view the world through the robots sensors.

The Augmented Reality Visualisation Project(ARDev), originally created in 2006, was designed toallow visual debugging of robot data [Collett and Mac-Donald, 2006]. ARDev integrates with the open-sourcePlayer Project, providing intuitive visualisations ofPlayer robot data. The integration into the Playerproject makes ARDev very accessible with easy accessto existing sensors through the generic Player interfaces.

The AR visualisations used in ARDev provide devel-opers with a clear, visual representation of robot data,and can be used by the researcher to detect problemsbetween the robot and the real world. Displaying therobots world view allows developers to better understandwhere faults lie and aids development as discrepanciesin robot data visualisations can be compared to the realworld [Collett and MacDonald, 2006].

The main contributions described in this work relateto increased capabilities of ARDev adding the ability tocreate and control a virtual camera, including panningand zooming in the virtual environment, the addition of

Australasian Conference on Robotics and Automation (ACRA), December 2-4, 2009, Sydney, Australia

Figure 1: Laser data drawn in a real world environmentand vehicles position and heading

runtime calibration and manipulation of data and views,and improving general stability and usability. The workis applied in the Inro Technologies’ development environ-ment where developers were creating a robotic forkliftused to move stock around in a warehouse environment.

2 ARDev

ARDev was initially built to interface with Player andcurrently can visualise a core set of Player drivers. How-ever, with its modular software architecture, ARDev al-lows for other general augmented reality use. The ARsystem is intended to visually display robot data andprovide an insight into the world from the robots per-spective.

Player is a network oriented device server. Player pro-vides transparent network access to various sensors andactuators [Gerkey et al., 2001]. The modular architec-ture of Player allows the creation of modules for anysystem that supports TCP/IP. Player provides a visual-isation tool (playerv) which displays robot data, howeverthis tool lacks the context of data for the visualisationsand this limits its usefulness.

An image illustrative of the core functionality ofARDev can be seen in Fig. 1. It shows three lasers,coloured red, green and blue overlaid on a camera feed.The path of the robot is also shown in yellow and thearrow head indicates the main direction of movement.A vector map can also be seen around the edges of theimage.

The goals of this project were to:

• Improve extensibility by implementing a runtimeplug-in system. Plug-ins should be able to add Po-

sition, Render and Output Objects.

• Allow more runtime configuration, including selec-tion between multiple views at runtime.

• Make the system more robust to device availability.

• Rendering of abstract values such as ’health’ anderror states. This requires an interpretation of thedata and may combine a number of factors. Thereis no specific way to display such concepts, althoughin [Dragone et al., 2007] abstract data such as ’emo-tion’ is shown as a virtual character above the robot.

• Add support for IP cameras.

The following sections are broken down into the maingoals; runtime configuration, extensibility, robustnessand deployment. Each section describes the state of thesystem prior to this project, the issues or areas identifiedfor improvement followed by the development that tookplace.

3 Robustness

Part of the requirements for a practically effective ARtoolkit [Collett and MacDonald, 2006] describe that itmust be able to run continually as a permanent instal-lation in a robotics lab and be able to detect Playerserver connections and disconnections. This requirementshould also be extended to be easily deployable and runcontinuously in a real world environment, particularly toassist in commissioning robot tasks.

3.1 Player Interface

ARDev is implemented as a Player server and this pro-vides a graphics interface to other Player drivers. Driverscan then render to the standard Player graphics inter-faces and it will display as AR without the need for anyadditional setup.

These requirements for robustness were not entirelyrealised in ARDev. ARDev was holding connections andpreventing the Player server shutdowns. Additionally, ifsome interfaces were not present at runtime it wouldnot start up correctly or would not load until they werepresent.

In order to address the issues presented above, it wasnecessary to change some of the underlying ARDev ar-chitecture for Player connections. Changes allowed con-nections to drop and reconnect at a later date withoutsignificant failure. To provide additional control, GUIbuttons were added which enable and disable Player con-nections, or to forcefully disconnect all Player connec-tions.

4 Runtime Configuration

In modern systems runtime configuration is an expectedand desirable feature. The usability of the original sys-

Australasian Conference on Robotics and Automation (ACRA), December 2-4, 2009, Sydney, Australia

tem was significantly reduced as it lacked runtime con-figuration or control. Previously with ARDev, it wasnecessary to confirm any configuration before runtime ofthe visualisation. Once ARDev was running the visuali-sation needed to be stopped and completely reinitialisedfor any changes to propagate.

4.1 Event Model

ARDev had some basic keyboard driven events whichwere used to clear drawn geometry from the screen, suchas the robot’s path or the Player graphics interface. Byabstracting the keyboard event system further, it wouldbe possible to support network events from other Playerservers.

An event model with significantly more capability wasadded to ARDev to enable a great deal of flexibility andcustomisation. This allows modules to take advantage ofthe dynamic nature of ARDev objects. Events can nowbe created and subscribed to by any number of objects.A basic event contains an event type and a number ofevent arguments, which provide additional event infor-mation. Events allow for a wide variety of visualisationsto interact and communicate in various ways. This newevent model provides the groundwork for many of thenew features discussed later in this paper.

Some of the notable uses for events: camera control,environment switching and sub-positions within positionobjects. Sub-positions have been used in conjunctionwith an actuator array. This exposes further positionobjects for adding extra actuator geometry, for exampleat an actuator joint.

4.2 Output Objects

Output objects define the output for the system. Mul-tiple outputs may be defined and each contains its ownrender and object lists. Primary output objects are usu-ally a graphical display while secondary outputs can bedefined for output to images or video.

ARDev defines outputs in the terms of an environ-ment. Each environment may define its own geometry,camera position, video feed, and post and pre-processinginformation, but often it may be optimal or required toshare some components, rather than defining the sameobjects for each environment. Multiple environmentshave been used to great effect in conjunction with astereo head mounted display (HMD) for stereographicAR. In this case only the camera position and angle aredifferent between the environments.

Previously each environment was only able to be asso-ciated with a single Output object and there was no wayto tightly integrate multiple environments. One of theobvious benefits of a combined environment would be tobe able to quickly switch from a perspective AR view toa top down virtual camera or a controllable fly-through

virtual camera while keeping the render and object liststhe same. Multiple camera angles can be used to givebetter perspective to a scene, a top down camera virtualcamera is better suited for discerning spatial positioningof objects.

To integrate environments together, the base outputobject was altered to respond to events and now con-tains a list of all environments. When considering theX11 output object, events were created to cycle throughall the environments. GUI integration was also addedfor quick jumping between environments and to reducedependency on the keyboard. It is now trivial to ren-der each environment in a separate viewport in a singlewindow.

4.3 Position Objects

Position objects are the underlying geographic positionsof any rendered object defining the position and rota-tion. Position objects can be added to the render listin a hierarchical fashion, allowing for objects to remainrelative to each other. These objects can then subscribeto a Player server and are updated according to the lo-cation provided by the Player server.

For runtime control and manipulation position objectswere extended to respond to position events. Theseevents can be directed at certain objects and form thebasis for camera movement. Controllable objects are pri-marily useful for abstract objects, but in a virtual envi-ronment controllable objects could be utilised further.

4.4 Virtual Camera

In ARDev the primary concern is the AR view, but itis important not to overlook the usefulness of a purelyvirtual camera. Given enough suitable AR objects, theability to move or change the view of the camera can beincredibly useful, even if these different camera positionare not coupled with the video feed. A virtual camerais able to move to positions that are unreachable or im-practical for a real camera.

A top down view of the environment is hard to achievewith a real camera, but it is often easier to discern spa-tial information from this angle compared with that ofa lower angle associated with wall mounted cameras. Apure virtual camera view is shown in Fig. 2.

A controllable position object is used with events fromthe keyboard to zoom and pan the virtual camera. How-ever, where camera control is not appropriate, for exam-ple when overlaying a video feed, camera movement isnot controlled by user input.

5 Extensibility

For any software product, extensibility and modifiabilityare important design principles allowing developers to

Australasian Conference on Robotics and Automation (ACRA), December 2-4, 2009, Sydney, Australia

Figure 2: Pure virtual top down view of the scene dis-playing a 1m grid. The GUI on the right shows therender options for each render object

easily integrate existing tools and software or create newmodules for additional functionality.

5.1 Plug-ins

In any software product, extensibility and modifiabil-ity are desirable properties to have. Due to ARDev’smodularised architecture, a plug-in manager was quicklyadded. The ARDev object registry is used to dynami-cally add new object types, which can then be added tothe scene using the ARDev graphical configuration ed-itor. The plug-in interface currently allows for plug-insfor new Output, Render and Position Objects.

AIO

The AIO (Analogue Input and Output) player interfaceprovides a raw data stream from the robot and containswithin it a great deal of information. This informationcan describe important state data but is not in an eas-ily discernible format. The AIO plug-in maps the in-formation into the ARDev GUI and can monitor valuesfor changes. Currently the implementation is very light,but as is, can be extended to display visual informationquickly.

Grid

The grid plug-in started as simple example plug-in devel-opment and grew into a mature measuring tool. Gridsare very useful for physical calibration procedures of arobotic platform, such as verifying the location of sen-sors, or measuring real world objects and comparingthese with processed data from the robot. Positioningthe grid is critical as currently the perspective cameradoes add some parallax error. The addition of an ortho-graphic virtual camera would eliminate this issue.

Along with a standard rectangular grid as in Fig. 2,a radial grid is also developed. The grid itself can beconfigured to any size with accurate major and minor

Figure 3: Screenshot of the graphical interface

grid lines. The grid has the additional function of mea-suring the distance along the grid’s plane between twoselected points. The measurement is then displayed inthe GUI in metres. Measuring was a request functionand is provided for user convenience.

5.2 Graphical User Interface

Previously, configuration for ARDev was unable to beperformed at runtime. It was necessary to terminate theprogram and reinitialise it in order to change any set-tings. As previously mentioned, this tool should ideallybe run constantly helping to avoid the potential loss oftime and important data.

In order to provide runtime control and configurationof ARDev a graphical user interface library was added.The library AntTweakBar was chosen as a lightweightoption. The main GUI window for ARDev is small, pro-vides buttons to open further windows and also providescontrol over player connections, as shown in Fig. 1. Themain windows as shown in Fig. 3 - Display List, Environ-ment and Data - display information regarding renderedobjects, environments and environment switching, andmiscellaneous robot data, respectively.

The Display List window can be used to show and hideobjects of interest and to change the colour or trans-parency of rendered objects shown in Fig. 2, which canbe very useful at runtime when rendered objects overlapand obscure each other. For some objects extended set-tings are provided; in the case of the grid, the major andminor grid lines can be changed quickly for more or lessprecision.

Australasian Conference on Robotics and Automation (ACRA), December 2-4, 2009, Sydney, Australia

With the addition of the GUI it is now easy to changeruntime settings, manipulate the environment and viewthe state of the system. Further configurations easily beadded for any type of object using the extensible archi-tecture.

6 General

This section discusses some of the secondary improve-ments that benefit ARDev.

6.1 IP Camera

The camera in an AR visualisation is an important partof the system. The camera drivers that exist for Playerwere not able to utilise a new camera purchased foruse with ARDev. The camera produced a stream ofMJPEGs across a network connection, unlike previousUSB-based cameras. A new driver to support this IPcamera was created and added to Player. The newcamera driver could then be used in conjunction withARDev. The benefits of the new camera include higherresolution and a faster capture rate, improving the de-veloper’s AR experience.

6.2 Packaging

With any software package ease of distribution isparamount in order for software to be utilised and be-come more widespread. ARDev was packaged into aDebian (.deb) package for Debian-based Linux distribu-tions. The package installs ARDev and all required li-braries and resources, and any updates to ARDev thatoccur can be easily pushed out to those with the packageinstalled. The deb package has been tested on a numberof 32 and 64 bit Ubuntu machines.

This packaging of ARDev should help develop a fur-ther install base and hopefully encourage further devel-opment and testing.

7 Discussion

Developing robots that are mobile and interact with thereal world can be a significant challenge. The availabletools for a developer to understand complex robot datain a real-time and useful way are limited, resulting in anextended time to finding correct solutions. The real-timeworld is complex and dynamic and cannot be easily un-derstood from a robot’s perspective without placing therobot’s data in context with the developers world view.Visualisations of sensor and actuator readings can beeasily interpreted when overlaid onto an image of thereal world. Any inconsistencies between real-time robotdata and what the developer can actually see in the realworld become obvious when presented in a visual man-ner. The benefits of AR in robotics is clear and thereare many examples of its use.

The ARDev project is a work in progress and there isgreat potential for future development. Some examplesare discussed in the following paragraphs.

To improve the GUI and the ease of GUI extensionit would be easy to create a slight abstraction from theAntTweakBar library. Currently adding a single variableto the GUI requires many lines of code. This could beeasily simplified and would encourage use of the GUI.

When trying to measure distances in the virtual en-vironment an orthographic view would be useful as itwould remove the parallax error associated with perspec-tive views.

Empirical testing on the ground truth of the ARDevvisualisations would be important to undertake. Beingable to ensure that the visualisations were reliable wouldlend credibility to the ARDev project.

A user study would also be beneficial to determinethe best way to present visualisations to the user. Userscould be evaluated on aspects such as their understand-ing, clarity and usefulness of the visualisations. Com-parison could also be made to classical methods such asprinting to a standard console output.

While ARDev is quite visually impressive, from expe-rience it takes developers a while to get accustomed tothis kind of tool and the advantages it has over classicaltechniques for debugging and mobile robot development.The adaptability of ARDev to meet the developers needis vast and can be tailored to the task at hand. Theideal situation is having ARDev deployed as a ubiqui-tous tool for robotic development. This paper aims toillustrate some of the possibilities of the ARDev projectand encourage more developers to use it.

8 Conclusion

The work described here has been successful and bene-ficial to the ARDev project, with the improvements toextensibility, robustness and runtime configuration. Theusefulness of the new features has been indicated by theirquick uptake in a commercial development environment.ARDev is now maturing into a valuable tool, provid-ing developers a means of visualising mobile robot’s per-ception of the world. Some of the improvements suchas plug-ins and events have created the foundation forfurther development. The general improvement to ro-bustness increases the quality of ARDev and the aim ofbeing a long running fixture in a robotics lab. Compil-ing ARDev and required libraries into a Debian pack-age has made ARDev easier to install and thus availableto a larger install base within a commercial operation.Over the course of this project a great deal has beenachieved but there are still further improvements thatcan be made.

The ARDev project is open source and the cur-rent build ardev-0.9.0-rc1 is available for download at

Australasian Conference on Robotics and Automation (ACRA), December 2-4, 2009, Sydney, Australia

http://sourceforge.net/projects/ardev/ under theLesser General Public license (LGPL).

9 Acknowledgment

This project was supported and funded by Inro Tech-nologies Limited, New Zealand and the Foundation forResearch, Science and Technology (FRST).

References[Breazeal et al., 2001] C. Breazeal, A. Edsinger, P. Fitz-

patrick, and B. Scassellati. Active vision for sociablerobots. Systems, Man and Cybernetics, Part A: Sys-tems and Humans, IEEE Transactions on, 31(5):443–453, Sep 2001.

[Brujic-Okretic et al., 2003] V. Brujic-Okretic, J.Y.Guillemaut, L.J. Hitchin, M. Michielen, and G.A. andParker. Remote vehicle manoeuvring using aug-mented reality. In Visual Information Engineering.International Conference on, 2003.

[Chen et al., 2008] I. Y. Chen, B. MacDonald, andW. Burkhard. Markerless augmented reality forrobotic helicopter applications. In Robot Vision.Springer Berlin / Heidelberg, 2008.

[Collett and MacDonald, 2006] T.H.J. Collett and B.A.MacDonald. Augmented reality visualisation forplayer. In Robotics and Automation, 2006. ICRA2006. Proceedings 2006 IEEE International Confer-ence on, pages 3954–3959, May 2006.

[Dragone et al., 2007] M. Dragone, T. Holz, and G.M.P.O’Hare. Using mixed reality agents as social interfaces

for robots. In Robot and Human interactive Commu-nication, 2007. RO-MAN 2007. The 16th IEEE Inter-national Symposium on, pages 1161–1166, Aug. 2007.

[Gerkey et al., 2001] B.P. Gerkey, R.T. Vaughan,K. Stoy, A. Howard, G.S. Sukhatme, and M.J.Mataric. Most valuable player: a robot device serverfor distributed control. In Intelligent Robots andSystems, 2001. Proceedings. 2001 IEEE/RSJ Inter-national Conference on, volume 3, pages 1226–1231vol.3, 2001.

[Juan et al., 2004] M. C. Juan, C. Botella, M. Alcaniz,R. Banos, M. Carrion, C.and Melero, and J. A.Lozano. An augmented reality system for treatingpsychological disorders: application to phobia to cock-roaches. In Mixed and Augmented Reality. Third IEEEand ACM International Symposium on, 2004.

[Kirner et al., 2006] C. Kirner, E. R. Zorzal, and T. G.Kirner. Case studies on the development of gamesusing augmented reality. In Systems, Man and Cyber-netics. IEEE International Conference on, 2006.

[Lyu et al., 2005] M. R. Lyu, I. King, T. T. Wong,E. Yau, and P. W. Chan. Arcade: Augmented re-ality computing arena for digital entertainment. InAerospace Conference, 2005 IEEE, 2005.

[Sielhorst et al., 2008] T. Sielhorst, M. Feuerstein, andN. Navab. Advanced medical displays: A literature re-view of augmented reality. Display Technology, Jour-nal of, 4:451–467, 2008.

Australasian Conference on Robotics and Automation (ACRA), December 2-4, 2009, Sydney, Australia