Tele-existence

Autonomous Robots 11, 29–38, 2001c© 2001 Kluwer Academic Publishers. Manufactured in The Netherlands.

Tele-Existence Techniques of Heavy Work Vehicles

JUSSI SUOMELA AND AARNE HALMEAutomation Technology Laboratory, Helsinki University of Technology, FIN-02015 HUT, Finland

[email protected]

Abstract. The problem of finding a feasible and technically simple tele-existence system for teleoperation ofheavy work vehicles is considered with the aid of two experimental test series. Several test persons of different agesand experience were used in experiments in which a basic audio/video tele-existence system with head-trackingwas modified into different versions and tested. Conclusions indicate that details of the tasks and work environmenthave an important role in finding the optimal solution.

Keywords: tele-existence, teleoperation, heavy work vehicles

1. Introduction

Partially teleoperated semi-autonomous workingmachines are no doubt the next step in the developmentof automation in various worksites. Simple tasks canbe automated easily, whereas more sophisticated onesneed the hand of an experienced operator. When mostof the work cycle of working machines is automated,one operator can easily handle from 2 to 5 machines,assuming that all machines can be teleoperated fromone place. Teleoperation is also the most natural tech-nical solution in cases where robotic machines are usedfor individual and unique operations (Sheridan, 1995)like surveillance and rescue work in disaster areas orplanet exploration.

When tasks are too difficult to be automated they areprobably very demanding to teleoperate, especially incases where large, heavy machines have to be drivenin narrow spaces or unknown areas.

A teleoperation interface can be made more naturaland easier for the operator by creating a feeling of pres-ence, for example, by the use of stereo video camerasthat follow the movements of the user’s head, to givea more natural visual perception of the environment.Adding microphones and audio deepens still furtherthe feeling of presence, especially in cases where thesounds are related to the work of the machine. Another

dimension by which the operation interface can befurther expanded is virtual reality. Objects related tothe environment or to the work being done can be rep-resented in a virtual world overlaying the view of theoperator. Tele-existence is then realized by the aid ofaugmented reality.

Augmented reality can be set up and utilized in tele-operation in a variety of ways, and the objects whichare used to augment the reality can be used for severalpurposes. For example, artificial walls can be created towarn or guide the operator in bad visibility; or the workdone with the machine may be conducted with the aidof simulated guide marks, etc. In Halme et al. (1997)the authors describe and demonstrate a system whichcan assist the operator to create interactively a simpli-fied augmented reality for an unknown environment.The system utilizes cameras and a remotely aimablelaser range finder. Using the same basic idea withthe hardware and software developed further, we havemade a system by which complex human work, likediagnosis or maintenance of complex machinery, canbe supported remotely by an expert sitting in his/her of-fice. The ETALA-system utilizes Internet technologyin communication. Part of the communication path-way can go via satellites enabling global location of thedistance unit. More details can be found in our web-sitehttp://www.automation.hut.fi

30 Suomela and Halme

2. Tele-Existence Systems

The term tele-existence refers to the situation wherethe main senses of an operator, like sight and hear-ing, are transferred to the remote place by means oftelecommunication so that he/she has the “feeling” ofpresence. This feeling is supposed to help when exe-cuting teleoperation, because human senses are evolvedfor local use rather than for supporting operations at adistance. In addition to sight and hearing, other senses,like feeling and balance, may be used. One of the pi-oneers in this field is professor Tachi of Tokyo Uni-versity, who has constructed several notable systemsand published many interesting papers (see web-sitehttp://www.star.t.u-tokyo.ac.jp/).

A full tele-existence system with all trimmings istechnically demanding. The system easily becomescomplex and expensive. In the case of heavy workingvehicles, the price matters and so does the robustnessof the system. Depending on the work and the environ-ment in which it is done, a simplified system is oftensufficient, and in some cases is even better than a morecomplex one. It is not very clear, however, what themain factors affecting are. In what follows, we try toillustrate some related problems through a number offield tests.

3. Problems of Tele-Existence Systems

In heavy work vehicle applications sight and hearingare without doubt the most important senses for theteleoperator. In certain applications, in construction orforestry for example, force feeling might be also use-ful. Hearing, however, replaces in many cases directforce feeling because an experienced operator can in-terpret machine noise as sense of power or force. As tosight, full immersion in the work environment usuallyrequires stereovision and a head-tracker that turns thecamera head at the remote site in accordance with thehead motions of the operator. The vision may be dis-played for the operator in several alternative ways usingan HMD (helmet) or just glasses and normal displaywhich generate stereo impressions.

The main two problems when applying augmentedtele-existence systems in practice are technical com-plexity and the need of broad bandwidth for datatransmission. Ergonomic problems are not minor andthe human interface aspects of such problems needcareful consideration. The basic requirement—a real-time stereo image—needs 2 × 120 Mbit/s without

compressing. Especially in applications where wirelesscommunication is the only choice, the cost of the datatransmission system for long distances can be veryhigh. Complex servo systems for cameras should haveup to 7 DOF if all head and eye movements are tobe tracked and repeated. As the number of DOFs in thesystem doesn’t only increase the cost but also the faultprobability, the minimum number of DOFs (=2) whichstill might be acceptable from the operator’s point ofview is preferred in practical applications. Deteriora-tion of the visual information due to delays and toosmall a bandwidth in communication is obvious. Aug-menting the system with overlaid technique requiresgraphical software and computer power.

With the above facts in mind, we made two series ofexperimental tests to discover a feasible tele-existenceteleoperation system for the application in question. Wechose initially a system with full video/audio capabilityand minimum delays in the control loops, but with 2DOFs only in the servo head. The small number ofDOFs was chosen because the task was to teleoperate avehicle on more or less flat terrain. Several alternativesto simplify this “complete” system were then made andthe effect of quality deterioration of task performancewas assessed systematically by taking into account thecapability of the additional tools and senses installed.

4. Typical Task and Work Environment

The test site was an underground tunnel constructionsite illustrated in Fig. 1. Actually the test site and taskwere both characteristic of a typical mining environ-ment. The task was a loading task with a relativelylong transportation part. The task consisted of drivingabout 500-m to the loading place, loading the bucket,driving back and emptying the bucket. The loading endand the emptying end of the route were connected witha vertical cleft which closed the loop. The 3D sketchand the map of the test route are shown in Fig. 1. Mostof the route was located in a 5-m wide and 5-m highrock tunnel.

The tasks performed by the operator during a singlemission were threefold: 1) loading the bucket, 2) trans-porting the bucket along the transportation route, 3)emptying the bucket. The nature of the tasks was quitedifferent in each case. Loading required the operatorto load the bucket as full as possible to get maximumpayload for transportation. Although the vehicle speedwas almost zero, loading nevertheless needed accuratesimultaneous control of the bucket actuators and the

Tele-Existence Techniques of Heavy Work Vehicles 31

Figure 1. Underground test route: 3D sketch and freehand map.

vehicle. Transportation required the speed to be keptas high as possible to reduce the transportation timewhile performing the difficult task of driving safelyalong narrow and winding parts of the route. Emptyingthe bucket involved the difficult task of parking as closeas possible the dumping shaft.

Figure 2. Test vehicle for first series of experiments.

5. Experiments with Test-Bed Vehicle

Simplification of the tele-existence system was firsttested with the aid of experiments done with a small-scale outdoor vehicle, shown in Fig. 2. The vehiclewas a standard ATV which was fitted with instruments


Figure 3. Operator station for first series of experiments.

and computerized. The test route was established inan open field. More about the vehicle can be found inSchonberg et al. (1994). The experiments were plannedwith more generic tasks in mind than just the miningone. The teleoperation system was implemented witha steering wheel and pedal combination providing thefeeling of normal car driving. The operator station isshown in Fig. 3. Communication between control sta-tion and ATV was made with one pair of half-duplexradio modems. The tele-existence equipment includeda stereo-HMD, a head-tracker, two monitors, two cam-eras, a laser pointer, a 2 DOF servo-head, two pairs ofshort range video links, a half duplex radio (for noisetransmission), and pair of radio modems for transmit-ting and receiving the head-tracking data from trackerto servo-head.

The system used for driving tests was a simplifiedversion of the full tele-existence system developed pre-viously in the laboratory (Halme et al., 1997). The laserrange finder, which was installed between the cameras,was used as a pointer and a measurement stick by which

the operator can get more precise information aboutthe geometry of the environment. A 3D virtual worldsoftware environment was used to create an interactiveoverlaid augmented interface for the operator. The laserpointer was used at this interface.

The tele-existence hardware was configured to fivedifferent systems with different respective enhance-ment levels in vision and camera control:

A: stereovision, sound, and 2DOF head-trackingB: monovision, sound, and 2DOF head-trackingC: image on screen, sound, and 2DOF head-trackingD: image on screen, sound, manual camera controlE: image on screen, sound, fixed camera

Sound was present in each alternative because omit-ting it was observed to deteriorate the system radically.Using the laser pointer, the user was able to measuredistances up to 2–3 m with the optical axis of thecameras. This feature was used to provide indepen-dent depth information for the driver. The pointer couldalso be used as a tool when virtual reality objects weredefined to the augmented system.

The experiments included tasks, which simulatedreal world driving tasks in material handling and trans-portation. The test tasks were the following:

“Corridor” driving: Driving on defined routes repli-cating driving on the road, ore transportation in minetunnels, etc.

The test was conducted by following windingpath, which included narrow gates. Driving overthe path border and colliding with obstacles werecounted as errors.

Unknown terrain driving: Driving on an unknown areawhere there were many obstacles and no specificroute, a task typical for forestry machines.

The test was conducted after driving to an un-known forest area where the operator had to followa natural path after perceiving it.

Loading tasks: The vehicle had to take a load into itsmanipulator and move during the loading, a task typ-ical for different kinds of loaders. The test involvedpushing boxes from one line to another one with theaid of a beak assembled on the front of the vehicle.

Maneuvering tasks: Maneuvering in confined places,a task typical for forestry and loading machines.

The test was conducted on a “slalom” track, wherethe driver was required to dodge piles, stop on a lineand park in a given slot.


Fast driving: Driving at speeds of more than 5 m/s,this is typical for transporting tasks.

The test required the operator to drive fast and tostop the vehicle in a given position in an open field.

Off-road driving: Driving in areas where both obsta-cles and surface conditions could stop the vehicle.

The test required the operator to drive over un-even ground and cross obstacles such as ditches andstones.

Five different men aged 26–40 were used as testoperators. Two of them were classified as experienced,and the others as amateurs. In the case of each systemthe following properties were evaluated:

• Easy to drive with continuous motion• Easy to drive accurately• Easy to navigate• Perception of obstacles and unexpected objects in

the environment• Possible ergonomic drawbacks

The results of each test were evaluated by measuringoverall execution time and the number of errors duringthe test. Verbal assessments concerning the propertiesof the system were obtained from the operators. Whenrepeating a certain test several times the effect of learn-ing could be clearly seen. This effect was independentof the system used. In order to eliminate this effect,only the best results obtained after a training periodwere taken into account in further evaluation.

The test results are reported in detail in Halme et al.(1997). After making about 80 test drives with 5 differ-ent operators the overall conclusion was that no uniqueanswer can be given to the question of optimal choiceamong the systems A–E. It depends on the details ofthe task, and the fact that learning has very importantrole in tasks, which are repeatable in nature. Our overallexperiences can be summarized as follows:

• There were not many differences between the sys-tems after a repeatable task was learned well.

• The head-tracking camera helped many tasks includ-ing new and changing situations.

• For each system, training improved results.• The most difficult of all test tasks was loading, which

demands accurate estimation of distances and finecontrol of actuators.

An important conclusion was that head-tracking helpsand is a good solution especially if the task includes

unknown and changing elements. Whether it shouldbe implemented with a stereo-HMD or just a singlemono-monitor is a more difficult question. It seems thatergonomy is better in the latter case, but tasks whichdemand ultra high accuracy and distance estimation canbe done better with the stereo system. A laser rangefinder installed on the camera system can help solvethis problem.

6. Tests with Full-Scale Loader

Experiments were continued using a full-scale loaderin cooperation with the manufacturing company. Thecompany’s test site, containing the tunnel constructionillustrated in Chapter 4, provided the test environment.

The equipment used in the test consisted of a com-mercial front-end loader and tele-existence equipment.The overall system is shown in Fig. 4.

6.1. Test Vehicle

The loader used in the tests weighed 40 tons and had aloading capacity of about 5 m3. The vehicle was dieselpowered, and had mechanical power transmission withhydrodynamic torque converter. The steering was thearticulated type, i.e., with a frame divided into twoparts of about same size connected about the steeringjoint. The bucket had two degrees of freedom: lift andtilt.

6.2. Loader Control Equipment

The loader was equipped for full teleoperation. All driv-ing actions required could be performed remotely froma remote control station.

Remote Control Station. The actual user interface inthe remote control station was the control chair with asteering wheel and pedals shown in Fig. 5. The steeringcould be controlled with either the steering wheel or thejoystick on the left arm of the chair, while the throttleand brake were controlled respectively with one of twopedals. Engine start, gears, camera, etc. were controlledwith the buttons, switches and another joystick fittedto the other chair arm. The feedback data was shownin the control PC’s monitor. The video image from thevehicle was shown in a separate monitor. Also soundfrom the vehicle was available. The control PC read thedata from control devices (pedals, joysticks, buttons,etc.) and sent it to the vehicle.


Figure 4. Test equipment for a full-scale loader.

Data Transmission. The control data was transmit-ted by means of a pair of radio modems between thevehicle and the control station. A leaky feeder ca-ble system was used in order to cover the whole testarea.

Image and Sound Transmission. Two video chan-nels and one sound channel were required for the tests.The leaky feeder system used didn’t support video fre-quencies so two analog video-links with 2,4 GHz fre-quency were used. To ensure the connection in thewhole test route, 3 pairs of video receivers were lo-cated along the route. The right receiver was chosenmanually during an initial run.

6.3. Tele-Existence Equipment

The tele-existence equipment was basically the sameas that used in the previous tests. Only the servoedcamera head was redesigned to obtain better trackingproperties. Details of the equipment can be found inSuomela et al. (1999).

6.4. Test Persons

The following persons were chosen for the tests:

A (age 40): A professional loader driver of 20 years’experience, and with few years’ experience in loader


Figure 5. Control chair, steering wheel and pedals.

teleoperation. He had driven several thousands ofbuckets with a system similar to the test system withthe fixed camera configuration.

B (age 43): A professional loader driver with severalyears’ experience, and experience of other types ofworkmachines. He hadn’t any teleoperation experi-ence before the tests.

C (age 35): A professional loader driver, with no tele-operation experience.

D (age 30): A research engineer. He had no experienceof loaders, although he had teleoperated the ATV(vehicle used in the first experiments) in all tele-existence configurations. He was an amateur pilotwith extensive experience of video and computergames.

6.5. Tele-Existence Configurations

The described tele-existence system was configured tofour different enhancement levels:

Fixed cameras and monitor: The image came fromthe fixed cameras pointing straightforward and back-ward. Cameras were located on the centerline of the

vehicle. The front camera was fixed to the front partand the back camera to the back part of the vehi-cle. The image was shown on a monitor ahead ofthe driver, and was automatically switched betweenthe front and the back camera depending on gearposition. It was also possible to look in the oppo-site direction by pressing a button on the arm of theoperating chair.

Servo camera and monitor: The image came from theright camera of the servo head located on the top ofthe cabin about 1 m from the vehicle centerline. Theoperator controlled the camera movements with theADL head-tracker, while the image was shown in amonitor as in the previous configuration. The mainpointing direction of the servo head could be rotated180◦ by a button in chair arm.

Servo camera and mono HMD: As in the previous con-figuration, but with the HMD replacing the monitor.The image from the right camera is divided to theboth displays in the HMD.

Servo camera and stereo HMD: As in the previous con-figuration, but both cameras were used. The imagesfrom the cameras were transferred to both displaysof the HMD to create a stereo image.


Figure 6. Tele-existence aided driving. Driver (front) is driving by using servo camera and normal monitor. Virtual passenger (back) is lookingat the stereo image from servo cameras with HMD.

The sound feedback was in all configurations. Theoperator station is illustrated in Fig. 6.

6.6. Test and Evaluation Methods

The test was conducted in two phases. In the first,the drivers drove the test route three times using eachof the three steering configurations, but only the firsttele-existence configuration. In the second, the driversused the steering configuration they liked most, and allthree tele-existence configurations. The evaluation wasbased on the method suggested by Schloerb (1995) fortele-existence systems. In this method, objective evalu-ation is made by concluding how well the defined taskis performed, while subjective evaluation is based onthe operator’s feeling how good the presence is.

In the objective evaluation, performance was evalu-ated on the basis of the recorded performance data. Inthe subjective evaluation, the quality of the presence

was evaluated on the basis of comments of the driversboth during driving, and following each run, when eachdriver was interviewed.

In the objective evaluation, the following data weredetected: 1) Time of runs: The time of each run wasmeasured, and the e-stop interrupts subtracted. Theproblem remained that the amount of the test runs didn’tallow a proper statistical analysis. 2) Errors: All the er-rors, which were mostly hits to the walls or emergencybraking by the safetyman, were recorded 3) Loggingthe operator driving data: The steering and throttlemovements made by the operator were logged with100 Hz frequency. The data recorded shows in the timephase and, especially, in the frequency phase, very wellhow nervous the driving was. 4) Logging the vehiclepositional data: The test vehicle also had navigationequipment. Vehicle’s 2D position was calculated fromoptical gyro and velocity. Speed, motor speed, angle ofthe middle joint, time and angular speed were measuredand logged.


6.7. Summary of Results

The test results have been reported in detail in Suomelaet al. (1999). The results were partly a surprise, espe-cially when compared to those of the preliminary tests.The servo cameras controlled by head movements wereconsidered very bad by all drivers. The objective evalu-ation from the driving data also supported this. The onlyadvantage in the servo camera configurations was thestereovision. It created real tele-existence and helpedthe driver to estimate distances. This helped in loadingespecially. The servo cameras were located away fromthe centerline of the vehicle in the position where thedriver sits when driving on board. This was probablya mistake. When the view was not along the center-line the teleoperator had difficulties in driving alongtunnels, and in repositioning the camera back to thedriving position again after a glance elsewhere. How-ever, even if the control had been perfect, the resultswouldn’t have changed much. The drivers reported thatit would have been useful to have been able to controlthe cameras in some cases, but that most of the time theyshould be fixed. Also, control with the head was judgedunergonomic. This result was the opposite of what wasobtained in the preliminary tests. The only rational ex-planation lies in the different sensing environment. Thetest route of the preliminary tests was made in an openfield where the operator followed a path rather than atunnel. In a tunnel the walls cause an optical flow thatpossibly helps driving with minimal head movements.

The other problem was the HMD, which wasconsidered uncomforted and caused nausea in thedrivers. This was the same result as in the preliminarytests. The results clearly indicate that permanent work-ing with a HMD is not recommended. For the operatorit is better to be “in real life” most of the time andreceive visual information from a monitor. The HMDcan be used for short times when it’s really needed. Thestereo effect can also be created on a monitor.

7. Conclusions

The results from the two completed and one partlycompleted sets of experiments indicate that the optimalsystem for tele-existence in teleoperation of workma-chines depends very much on the tasks done with themachines and the details of the environment where themachine is working. It is quite clear that a fixed camerais enough for teleoperation of the loader in most cases.In open field tests (Halme et al., 1997) it was noticed

that when driving on an unfamiliar route without the“tunneling effect” the servo-controlled cameras haveadvantages. The vertical walls in the underground sitemake a “video game-like” driving environment which,contrary to outdoor applications, doesn’t favor head-tracking. In addition to this, the working environmentin mining is relative static and most of the time was veryfamiliar to the drivers. Learning has a strong effect andshould be taking into account when developing prac-tical applications of tele-existence for such work en-vironments. Both these facts disagree against the levelof advancement needed in the field of tele-existence.This doesn’t mean, however, that in other applicationsthe case would be the same. The preliminary tests con-ducted in a slightly different environment support thisassumption. Although not tested experimentally yet,we could say that in forestry, for example, or in con-struction site application, the case might be different.

References

Halme, A. and Rintala, N. 1997. An interactive tele-existence sys-tem augmented with VR-models of the working environment.ISMCR’97 Topical Workshop on Virtual Reality and AdvancedMan-Machine Interfaces, Tampere 4–5 June, 1997.

Halme, A., Rintala, N., Savela, M., and Suomela, J. 1997. Applyingtelepresence and augmented reality techniques in teleoperation ofmobile field robots. FSR’97, Canberra, Australia, 8. 10.12.1997(see also Robotics and Autonomous Systems Nro 26, pp. 117–125,1999).

Schloerb, D.W. 1995. A quantitative measure of telepresence.Presence, 4(1):64–80.

Sheridan, T.B. 1995. Teleoperation, telerobotics and telepresence: Aprogress report. Control Engineering Practice, 3(2):205–214.

Schonberg, T., Ojala, M., Koskimaki, E., Suomela, J., and Halme,A. 1994. A small scaled autonomous test vehicle for developingautonomous off-road applications. International Conference onMachine Automation, Tampere, February 1994.

Suomela, J., Savela, M., and Halme, A. 1999. Tele-existence technicsof different enhancement degrees in front end loader teleoperation.FSR’99, Pittsburgh, USA, 29–31, August 1999.

Tachi, S., Arai, H., and Maeda, T. 1989. Development of anthromor-phic tele-existence slave robot. In Proceedings of the InternationalConference on Advanced Mechatronics, Tokyo, 1989.

Jussi Suomela received his MSc in Electrical engineering fromHelsinki University of Technology (HUT) 1992. From 1992 he hasworked as research scientist and project manager in several mobile


robotics projects in Automation Laboratory of HUT. At the momenthe is doing his doctoral thesis concerning teleoperation of semiau-tonomous work machines.

Aarne Halme received his Ph. D. of Technology with honors fromthe Helsinki University of Technology in 1972. He subsequently

worked as associate professor of control engineering (1972–1977)at the Tampere University of Technology and then as professor ofcontrol and systems engineering (1977–1985) at the University ofOulu. Since 1985 he has been Professor of Automation Technol-ogy in the Department of Automation and Systems Engineering atthe Helsinki University of Technology. He has published extensively(more than 230 publications) in the areas of control, computer aideddesign, and robotics and his current research interests include ser-vice and field robotics, robot societies, sensor fusion and control ofbiotechnological processes.

Documents

Tele-existence