An Experimental Tele-operated Robotic System

Patrick LimAutomation Systems Team

Industrial Research24 Balfour Road, Auc~land,

New Zealand

Abstract

Automation has been introduced in variousindustries primarily to improve efficiency,productivity and worker safety. In applicationswhere fully autonomous systems are not feasible,semi autonomous systems can be applied insteadto achieve these goals. An experimental tele­operated system has been developed consisting ofa manipulator with a gripper attachment. Thispaper describes the integration of key systemcomponents and a control architecture that has theflexibility to adapt to various applications. A keycomponent is the human machine interface fromwhich the human operator controls themanipulator. Focus of this research is to improve

.the quality of control. This is achieved byincreasing the feedback channels to the user in theform of tactual stimulus in addition to vision.Control strategies were developed for thisstimulus to present critical operational and taskinformation that can be exploited for control.Preliminary experiments indicate that this conceptwas successful in the completion of a simple pickand place task. The research outcomes willprovide solutions for both tele-operated and tele­robotic systems.

1 IntroductionSemi-autonomous systems are widely used in applicationsthat usually demand some human input. This is evident incomplex tasks that cannot rely solely on machine sensorsfor full automation. These systems have enabled humans toaccomplish tasks that are often beyond their physicalabilities. In addition, they can be controlled remotely thusremoving humans from hazardous environments.

In general, a semi-autonomous system can beclassified as a teleoperated or telerobotic system. A humanoperator controls the system via a human machine interface(HMI) unit. By defmition, a teleoperated system is one that

135

Surangi Sharma and Werner FriedrichAutomation Systems Team

Industrial Research24 Balfour Road, Auckland,

New Zealand

is manually controlled. A telerobotic system incorporates ahuman operator in the execution of a task either by sharedor traded control.

Research in the area of teleoperation has beenextensively carried out. Numerous examples can be found inthe literature. Among them are applications in the area ofcomputer assisted surgery [Troccaz et aI, 1996], fuelreprocessing [Draper, 1994], virtual control [Zhai &Milgram, 1992] and space teleoperation [Tilley et aI, 1991].

One important issue concerning teleoperatedsystems is the reliability of human input. Studies haveshown that whilst humans are capable of very complexoperations, they are limited in their abilities for spaceorientation and interpretation of geometrical data·[Lumelsky, 1992]. Apart from physiological limitations,other factors such as fatigue, loss of concentration andrepetitive strain will also affect the quality of control.

A number of schemes have been proposed toaddress these limitations. Schryver [Schryver & Draper,1996] proposed a supervisory control where the user isresponsible only for strategic inputs leaving low-levelcontrol responsibilities to the robot. Draper [Draper, 1994]proposed the use of force reflection in the joystick handle toassist operators. Sundstrom [Sundstrom et aI., 1995] alsoreported on the preference of users .for force reflectingposition control. Results have been encouraging with lowererror rates, lower peak forces and more consistentapplication of forces. .

In this paper, we propose implementing forcefeedback in the forms of force reflection and forceresistance. The aim is to use this tactual stimulus to generateinformation that can be exploited for improved control.

2 OverviewThe human operator within a teleoperated system is anintegral part of the control loop and essentially closes theloop by performing three functions. Firstly, the operatorperceives the machine and task status information throughvarious perception channels such as sight, tactile andhearing.

Secondly, the brain will process relevantinformation and detennines necessary actions to achieve acertain goal. And, thirdly, is the articulation of specific bodymembers to perform those actions.

An automated sMstem can perform very similarfunctions to humans. A' sensor based system provides vitalmachine status to the main .controller. The controllerprocesses the information and detennines the necessarycourse of action or actions. These are then relayed to themachine outputs or actp,ators.

In a teleoperated system, however, an operator hasexclusive control of a robot. The system generally consistsof a HMI unit, a system controller and a programmablemachine. The system controller· supervises all operationswithin the system and serves as a link between theHMI unitand manipulator. Information between these sub-systems isbi-directional. The program receives control informationfrom the .HMI unit and actuates the manipulatoraccordingly. It also receives information from sensors anddelivers feedback information to the HMI unit. Userinteraction is through this HMI unit. .

2.1 HMlunitThe HMI unit is an essential part of the machine that isaccessible to the operator. Functionally, it can be dividedinto the input and feedback sections as shown in figure 1.The input section interprets operator's control commands

Figure 1: Interactions of the HMI unit.

and delivers instructions to the machine.Commonly, most HMI units will require some

form of physical contact by humans to input devices such askeyboards, touch screens, computer mice and joysticks.Other more sophisticated units use sensors to interpret eyemovements, speech or hand gestures. In the feedbacksection, machine sensor information relayed to the operatorin the form of visual display or force .feedback on thejoystick. Obviously, only certain HMI types are capable ofoperating with force feedback.

The HMI unit is physically divided into two majorparts; control console and status display. The function of thecontrol console is to facilitate operator control of

zY-axis rail

Load cellSteppermotor

y

x

Gripper

Figure 2: Diagram of a four axes manipulator with a gripper attachment.

136

manipulator and end effector while the status displayprovides useful operational and machine status information.

The control console consists of a four axesjoystick, control switches, mouse and computer keyboard.The main function of the joystick is to provide dynamiccontrol of manipulator. Its secondary function is to deliverfeedback to the user in the form of force feedback. Controlswitches that are strategically placed on the handle andcomputer keyboard enable selection of features andactuation of end effectors.

The status display provides vital real time data ofmachine status, operation mode and parameter settings.With the use of a keyboard, the user is able to changeparameter settings and select control options.

Joystick displacement is measured by rotarypotentiometers. Potentiometer analogue signals are in therange of 0-5V. The system controller through the analoginput port monitors these signals. A force feedbackmechanism consisting of DC motors is installed to provideforce feedback to the control handle. Linear amplifiers driveDC motors to supply feedback forces proportional to theanalogue signals generated by the system controller.

2.2 ManipulatorA four DOF manipulator was developed as shown in Figure2. It consists of three linear axes, one rotary axis, a fourcomponent load cell and a gripper unit.

An assembly of two rectangular telescopic roboticarms mounted on a horizontal track provides linear motionsin the x, y and z-axes. The robotic arms are actuatedpneumatically. The fourth axis is rotation about the x-axisactuated by a stepper motor. The joystick is used to controlthe direction and speed of each axis.

A load cell is installed to measure impact forces inthe x, y and z-axes. These forces are reproduced inproportion as force feedback on the joystick from which theoperator is aware of contact with objects. The load cell is a4-component piezo electric model that measures three linearforces and op.e moment. It uses a specialised driver to drivethe piezo crystals and a signal amplifier that producesoutput signals in the range of±10V.

The end effector is a simple on-off type pneumaticgripper. It has a pair of jaws that are fixed with foamgripping plates for better grip. In the open state, the jawshave a separation of 60 rom. The gripper is designed tograsp a rectangular object with surface dimensions of 150mmby 40 mm.

3 System controllerImplementation of the tele-operated system requires theinteraction of sub-systems on both hardware and softwarelevels. The hardware level concerns physical integration ofdevices such as sensors, controllers, drivers and actuators.The software level comprises of modules that control,

Manipulator control: •••• 1Il' •••.•••••.•.•••• 1l lO"' · "' "'•••••••••••••••••.•••••••••• :

Position control Linear actuator~

II1II..

modules .... signalsJoystickdisplacement

Input-outputI--

mappingsignals

Orientation Rotary actuator.....4 control module r signal

Force feedback control

Resistance-+ control module

Joystick actuatorL ~ signals

ReflectiveLoad cell Input-outputsignals - .. mapping ~ control module

137

signal is pre-assigned to a manipulator axis. Similarly, fourload cell signals are also mapped to the four joystick'actuators. When a different mapping is required, theoperator can simply re-configure it off-line and assumecontrol with the new mapping.

3.2.2 Normalisation I de-normalisationEach control module performs a number of functions.Firstly, all input signals are normalised to-l00% and 100%value based on the respective variable dynamic range (Eqn.1). Processing of data is then done in the normalised state.After data is' processed, all output signals are then de­normalised (Eqn. 2).

Normalising is used because all input and outputcomponents have dissimilar dynamic ranges. Using thisconcept enables any normalised input variables to beinterchanged without reprogramming.

(y:, + 1OO)(ymax - Ymin) (2)y = 200 +Ymin

3.2.3 Control moduleInput variables are coupled to respective output variables by .control modules. Thew. process normalised input data anddetermine suitable output data to be generated. Eachmapping is governed by control parameters. Controlparameters are user preset values that determines the outputvalue based on the input value. The parameters are deadband (Db) and gain. The relationship between the input andoutput is shown in Figure 4.

The region between [-Xmax' Xmax] is the fulldynamic range and remains unchanged. However, theregion between [-Ymax' Ymax] is governed by the gain settingand changes accordingly. Yn is the. normalised output anddependent on the value of ~ (Eqn. 3). Sensitivity of theswstem is evaluated bW the ratio of the output dynamic rangeover the input range. It increases with the ratio. Zero is thelowest sensitivity.

200(x - xmin )

X = 100n (x

max- x

min)

(3)

(1)

Db Db--<x <-2 - 11 - 2

X >Db11 2

X <Dbn 2

O·

Y = (x -Db') x Gain;11 11 2

(x. + ~b) x Gain;

Output (YnJ

Ym

XmafInputjnJ

..·· ·..·· ··· · · ···17····..· :-1m x

interface and synchronise these devices.

3.2 Control architectureThe system program is written in a graphical programminglanguage, LabVIEWTM and is designed to operate in realtime. Within each program loop, it must perform dataacquisition, information' processing and signal generationwith minimal time delay otherwise poor control results. Thearchitecture is designed to be flexible such that the systemfunctionality can be expanded by addition of functionmodules.

The control architecture is shown in Figure 3. Itconsists of two main sections. One section is responsible formanipulator control for both position and orientation of thetool. The other section determines the measure of forcefeedback to be exerted on the joystick.

3.2.1 Input-output mappingThe initial stage is the mapping of input components to therespective output components. For example, the joystickproduces four displacement signals, which are used tocontrol the four axes of the manipulator. Each displacement

3.1 lIardvvareinterfaceThe system controller runs on a Pentium 333MHz underWindows NT operating swstem. It interfaces with devicesthrough multipurpose I/O cards fuat are connected to the PCbus. They enable all input and output signals to bemonitored and generated respectively by the systemprogram. A.ll analog input signals are bipolar anddifferential. They are wired to I/O cards with shieldedtwisted pair cables to minimise noise transmission.

Figure 4. Relationship between normalisedinput and normalis~d output signals.

3.3 Control strategyThe strategy is to employ force feedback to provide usefulinformation to the operator while controlling the joystick byway of force reflection and force resistance. These forcesare superimposed on the joystick handle but are

138

Figure 5. Resistance force applied at controlboun4aries on each joystick axis.

distinguishable by functionality and measure. Forcereflection is a measure of the force in proportion to the forceimpacting the tool head.

By contrast, force resistance is an applied force tocertain regions within the joystick dynamic range. Theseregions are known as control boundaries. It is introduced toprovide the operator with strategic spatial information. Withthis information, the operator is able to fmd key joystickpositions rapidly that produce known results such as movingthe manipulator at a certain speed. By this, the operator cancontrol a machine with greater confidence and reliability.

transporting and releasing the object. Four pre-selectedlocations with different orientations were marked on thewhiteboard. The duster is placed on the first location.Subjects were asked to perform the task as accurately and asquickly as possible.

The fITst task is to position the gripper in the rightorientation to grip the object. When the object is picked up,it is moved to the next position. The subject can move oneor more axis at a time depending on the level of confidence.When the object is at the second position, it is released. Therobot arm is then retracted and subject rests for a fewseconds before commencing to the next position and so on.

The joystick is placed behind the robot but over onthe right hand side so that the subject's vision of the robot'swork space is not obscured. It is mounted on a platform thatis height adjustable.

Figure 6. A whiteboard and magnetic duster wereused for pick and place testing.

5.1 Handling of joystick

5 Discussion and future workA total of six subjects participated in this experiment.Although it is a relatively small number of subjects, theresults were sufficient to confrrm preliminary success of theconcept. All six subjects were able to perform the abovetasks successfully when force feedback was activated but allfailed when force feedback was deactivated.

There were a number of deficiencies in the system,which prevented further trials to be done without substantialmodifications to the system.

All subjects agreed that the resistance on the joystick assiststhem to execute proper control although it requires lesseffort to use without it. With resistance, they are aware ofcertain strategic control areas within the dynamic range. Thedead band boundary enables the user to move one axis

+h

+maxDisplacement

o

-h

-max

4 ExperimentA simple pick and place experiment was set up to test all thefunctionality and operational features of the system. Theoverall aim is to assess the control strategy that wasimplemented to improve quality of control. This experimentinvolves the control of all axes to accomplish the task. Forcereflection and force resistance have been activated tosupport joystick control.

An experimental test bed consisting of awhiteboard and a magnetic duster is shown in Figure 6. Theexperiment involved executing tasks such as gripping,

Figure 5 shows a measure of force applied at theseregions. Since the joystick is related to the manipulatorvelocity, these regions are formed to distinguish betweenzero (z), slow (s), medium (m) and high (h) speeds.

The inner boundary is a dead band region. There isno manipulator movement in this region. Once past thisregion, speed increases linearly with displacement. Thecontrol boundaries become tactile indicators that inform theuser where these positions are and subsequently the variousspeeds. The user sets the demarcation of the range into anumber of. specific categories.

139

without causing involuntary movements on the other axes.However, effort to overcome boundary regions can causethe joystick to be displaced further than necessary resultingin unwanted increases in manipulator speed.

Force reflection was another feature that enabledthe operator to "feel" forces that are impacting on the tool.This is helpful during pick up and release phases. Whilemoving the joystick, the sensing of tool impact with eitherthe whiteboard or duster confrrms that contact has beenmade. It enables the subjects to perform these tasks withgreater reliability and confidence.

5.2 Tele-operated systemOne major factor in teleoperat~on control is the time delaybetween the control and response of the system (C/R). The

.controller response and pneumatic drives contribute to thisC/R delay. Since the operator is an integral part of thesystem, a variable reaction time is also added to the delay.An appreciable delay may result in poor control. C/R delaymust be reduced to a minimum.

Another factor is the consistency of· speed. Thesystem is currently using direct proportional drives withoutposition feedback. Consistency in speed cannot beguaranteed.

5.3 Future workThe next phase is to extend the functionality of the currentsystem to a tele-robotic system. This includes installingspecific hardware, extending system. capabilities andredevelopment of the manipulator. illhese tasks include:

• install motion. control system to enable coordinatedmotions of the linear axes.

• modify HMI ooit to incorporate new features.• extend system capabilities by developing new

program modules.• redevelopment of manipulator to incorporate new

hardware.

6 ConclusionAn experimental tele-o:perated system h.as been developed.The system was successfully trialed by o:perators executingpick and place routines via a force feedback joystick. Forcefeedback was used as an additional perception channel toprovide tactile information for better control of themanipulator. On this channel, both force resistance andforce reflection were implemented concurrently. Forceresistance was used as tactile indicators to :provide strategicspatial information while force reflection enabled theoperator to feel forces that were impacting the tool head.Both were successfully trialled.

140

References

[Draper, 1994] J.V. Draper, Hand acceleration impulsebandwidth during target acquisition: Implications forteleoperator bandwidth requirements; IEEE illransactionson Systems, Man and Cybernetics, 24(6), pp 931-936,1994.

[Lee & Lee, 1992] S. Lee & H.S. Lee, Advancedteleoperator control system design with human in controlloop, Winter Annual Meeting of the American Society ofMechanical Engineering, 1992.

[Lee &' Le~, 1992] S. Lee & H.S. Lee, An advancedteleoperator control system.' Design and evaluation,Proceedings 1992 IEEE International Conference onRobotics and Automation, 1992

[Lumelsky, 1992] V. Lumelsky, On human performance intelerobotics, IEEE 'Fransactions on Systems, Man andCybernetics, 21(5), PP 971-982, 1991.

[Schryver & Draper, 1996] J.C. Schryver & J.V. Draper,Network simulation analysis of level of control for thesingle-shell tank waste retrieval manipulator system,(ORNL/'FM-12752). Oak Ridge, TN: Oak Ridge NationalLaboratory, 1996.

[Sundstrom et aI., 1995] E. Sundstrom, J.V. Draper, A.Fausz & H. Woods, Test bed control centre designconcept for tank waste retrieval manipulator systems,Proceedings of the .ANS Sixth 'Fopical Meeting onRobotics and Remote Systems, pp. 187-193. Monterey,CA: The American Nuclear Society, 1995.

['Filley et aI, 1991] S.W illilley, M.G. Hollars, C.M. Francis& A.H. 'Fadros, Space telerobotic research andapplications at Space Systems/Loral, Proceedings of the14th .Annual Rocky Mountain Guidance and ControlConference, 1991.

[Troccaz et aI, 1996] .J. 'Froccaz & Y. Delnondedieu, Semi­active guiding systems in surgery, A two-dof :prototype ofthe passive arm with dynamic constraints (PADyc),Mechatronics Vol. 6, No.4, PI' 399-421, 1996.

[Zhai & Milgram, 1992] S. Zhai & P. Milgram, Teleroboticvirtual contro/system, Cooperative Intelligent Robotics inSpace II, 1992.