Teleoperation of a 7 DOF Humanoid Robot ArmUsing Human Arm Accelerations and EMG Signals

J. Leitner, M. Luciw, A. Forster, J. Schmidhuber

Dalle Molle Institute for Artificial Intelligence (IDSIA) &Scuola universitaria professionale della Svizzera Italiana (SUPSI) &

Universita della Svizzera italiana (USI), Switzerlande-mail: juxi@idsia.ch

Abstract

We present our work on tele-operating a complex hu-manoid robot with the help of bio-signals collected fromthe operator. The frameworks (for robot vision, collisionavoidance and machine learning), developed in our lab,allow for a safe interaction with the environment, whencombined. This even works with noisy control signals,such as, the operator’s hand acceleration and their elec-tromyography (EMG) signals. These bio-signals are usedto execute equivalent actions (such as, reaching and grasp-ing of objects) on the 7 DOF arm.

1 Introduction

More and more robots are sent to distant or dangerousenvironments to perform a variety of tasks. From spaceexploration to nuclear spill clean-up are scenarios thesesystems are usually used. In most of these cases the robotsare controlled remotely. Yet these systems, to be able toperform interesting tasks, become increasingly complex.To teleoperate intricate systems requires extensive train-ing. Even then efficient operation is not possible in mostcases and it is not clear how to operate these complex,remote robots robustly and safely. Furthermore these con-trol interface systems should aim to be as intuitively to theoperator as possible. It has been hinted that intuitive userinterfaces can improve the efficiency and the science re-turn of tele-operated robotic missions in space explorationscenarios. To improve such systems an abstraction fromthe low level control of the separate motors to a higherlevel, such as, defining the action to be taken, is generallyperformed. To further improve the intuitiveness of the in-terface bio-signals from the operator have been proposed.In this case the operator’s intuition would be transferred tothe robot. In particular electromyography (EMG) [1], i.e.signals that represent the user’s muscle activations tend tobe interesting for such remote robotic control.

2 Background and Related Work

The presented research combines a variety of subsys-tems together to allow the safe and robust teleoperation ofa complex humanoid robot. For the experiments herein,we are using computer vision, together with accelerom-eters and EMG signals collected on the operator’s armto remotely control the 7 degree-of-freedom (DOF) robotarm – and the 5 fingered end-effector – during reachingand grasping tasks.

2.1 Robotic TeleoperationTelerobotics usually refers to the remote control of

robotic systems over a (wireless) link. First use of tele-robotics dates back to the first robotic space probes usedduring the 20-th century. Remote manipulators are nowa-days commonly used to handle radioactive materials innuclear power plants and waste management. In the spaceexploration scenario tele-operated mobile robots, such as,the Russian lunar and NASA’s Martian rovers, are rou-tinely used.

More complex robotic systems have only recentlyseen use in tele-operation settings. One of these system,NASA’s Robonaut, was the first humanoid robot in space.It is currently on-board the International Space Stationand used for tests on how tele-operation, as well as, semi-autonomous operation, can assist the astronauts with cer-tain tasks within the station [2].

For complicated robotic systems to perform tasks incomplex environments, a better understanding – i.e. per-ception – of the situation and a closer coordination be-tween action and perception – i.e. adapting to changes oc-curring in the scene – are required [3, 4]. For the robotto safely interact with the environment sensory feedbackis of critical importance. Even when tele-operated robotsrequire the ability to perceive their surroundings, as it isgenerally not feasible or possible to transfer all the infor-mation back to the operator.

A nice overview of the history of telerobotics and theproblems arising when tele-operating robots is the text-book by Sheridan [5]. Since then a variety of labs atNASA, ESA and JAXA have investigated how to effi-ciently control remote robots. Furthermore tests of con-trolling robots on the ground from the space station havebeen conducted to investigate how intermittant data trans-mission and the time-delay effect the operations [6, 7, 8].

Recently quite a few telepresence robot have enteredthe market, these though provide only a very limited, easy-to-control mobile robot. In addition tele-operated robotshave been proposed to support medical staff. They couldallow doctors to perform various medical check-ups in re-mote locations on Earth or space.

2.2 Electromyography (EMG)Biomedical signal generally refers to any electrical

signal acquired from any organ or part of the human bodythat represents a physical variable of interest. Like othersignals, it is considered a function of time and describedin terms of its amplitude, frequency and phase. The elec-tromyography (EMG) signal is a biomedical signal thatrepresents neuromuscular activity. An EMG sensor mea-sures electrical currents generated in a muscle during itscontraction. The signal itself stems from the control ofthe muscles by the nervous systems and is dependent onthe anatomical and physiological properties of muscles.Furthermore the signal collected contains noise acquiredwhile traversing through varying tissues. In addition thecollection of the signal by the electrode most likely con-tains the signal from multiple motor units, generating in-terference and signal interactions.

The first documented about electricity and its relationto muscles is the experiments by Francesco Redi. In themid-1600 he discovered that highly specialised musclesin the electric ray fish are generating electricity [9]. Inthe 18th century the famous experiments by Luigi Gal-vani showed that electricity could also be used to stimu-late muscle contraction [10]. The first recording of electri-cal activity during voluntary muscle contraction was per-formed by Etienne-Jules Marey, who also introduced theterm electromyography, in 1890 [11]. In the 20th centuryever more sophisticated methods of recording the signalwere invented, with the first sEMG electrodes being usedby Hardyck in the 1960s [12].

There are two methods to collect EMG signals, di-rectly in the motor units with invasive electrodes or withnon-invasive, surface attached electrodes. Both methodshave their own advantages and disadvantages. We are fo-cussing on surface EMG (or sEMG) herein. The com-paratively easy setup has the drawback of increased noise,

motion artefacts and possible readout failures when losingthe contact to the skin. EMG has previously been used inmedicine to measure the rehabilitation in case of motordisabilities, but only recently has it been identified as apossibly useful input in applications, such as, prosthetichand control, grasp recognition and human computer in-teraction (HCI) [13].

2.3 Machine Learning for EMGMachine learning techniques have already been used

to classify EMG signals from one subject or one instru-ment previously. Of interest nowadays is to compare howwell these techniques perform with multiple subjects, aswell as, collected with multiple instruments. Furthermorewe investigate how classification can still be achieved withlimited available training data. This is of interest, notjust to allow flexibility in who is controlling the robot butalso in other areas of application, such as rehabilitation orprothesis control. It is important to not exhaust (or annoy)the subjects, amputees or people with other impairments,with too much training. Reaz et al. published an overviewof techniques applied to the problem of detection, classi-fication and decoding EMG signals [14].

The NASA JPL developed BioSleeve [15] contains anarray of electrodes and an inertia-measurement unit thatcan easily be put onto a subjects forearm. The detectionof the signals is using a machine learning approach (multi-class SVM) to decode static and dynamic arm gestures.

3 Setup

Our experimental platform is an open-hardware,open-software humanoid robotic system developed withinvarious EU projects, called iCub [16] (shown in Figure 1).It consists of two arms and a head attached to a torsoroughly the size of a human child. The head and arms fol-low an anthropomorphic design and provide a high DOFsystem that was designed to investigate human-like ob-ject manipulation. It provides also a tool to investigatehuman cognitive and sensorimotor development. To al-low for safe and ‘intelligent’ behaviours the robot’s move-ments need to be coordinated closely with feedback fromvisual, tactile, and acoustic sensors. The iCub is an excel-lent experimental platform for cognitive, developmentalrobotics and embodied artificial intelligence. Due to itscomplexity it is also a useful platform to test the scalabil-ity and efficiency of various machine learning approacheswhen used together with real-life robotic systems. To ourknowledge the platform has though so far not been usedin tele-operation scenarios.

Figure 1. The iCub humanoid robot con-sisting of two 7-DOF arms, each witha 5 digit hand.

Figure 2. Grasping a variety of objects,such as, tin cans, cups and tea boxes.

The grasp of the iCub are controlled using alight-weight, easy-to-use, one-shot grasping system(LEOGrasper1) developed at IDIA. The system has beenin operation in our lab for a variety of scenarios (see Fig-ure 2). It can be configured to perform one of multiplegrasps. Each type of grasp requires the closing of the fin-gers in a coordinated fashion, may it be for a power grasp,pinch grasp or even just a pointing gesture. Instead of thenoisy touch sensors on the fingertips, the errors reportedby the PID controllers for the motors of each digit are pro-viding estimates of contact with an object.

3.1 Control and Obstacle AvoidanceTo control the complex humanoid with all-in-all 41

DOF, the Modular Behavioral Environment (MoBeE) [17]was built. It is a software infrastructure to facilitate com-plex, autonomous, adaptive and foremost safe robot ac-tions. It acts as an intermediary between three looselycoupled types of modules: the Sensor, the Agent and theController. These correspond to abstract solutions to prob-lems in Computer Vision, Motion Planning, and Feed-back Control, respectively. An overview of the system

1Source code available at: https://github.com/Juxi/iCub/

Figure 3. The Modular Behavioral Envi-ronment Architecture (MoBeE) imple-ments low-level control and enforcesnecessary constraints to keep the robotsafe and operational in real-time.

is depicted in Figure 3. The framework is built openthe robot middleware YARP [18] to be as independent ofthe hardware as possible. The behavioural componentsare portable and reusable thanks to their weak coupling.While MoBeE controls the robot constantly, other ‘forces’can be sent by the Actor to move the robot’s end effector.

The control is following the second order dynamicalsystem:

Mq(t) + Cq(t) + K(q(t) − q∗) =∑

fi(t) (1)

where q(t) ∈ Rn is the vector function representing therobot’s configuration, M, C, K are matrices containingmass, damping and spring constants respectively. q∗ de-notes an attractor (resting pose) in configuration space.Mechanical constraints and infeasibilites of the systemare implemented by forcing the system via fi(t). Themodel computes fictitious constraint forces, which repelthe robot from self-collisions, joint limits, and other infea-sibilities. These forces, fi(t), are passed to the controller,which computes the attractor dynamics that governs theactual movement of the robot. Obstacles in the world cre-ate additional forces providing a simple way to react andavoid collisions.

3.2 Obstacle DetectionThe obstacles need to be detected and tracked prior

to be avoided. This is done using our icVision [19] sub-system. It has been developed at IDSIA over the lastyears and is an open-source framework consisting of dis-tributed modules performing robot and computer visionrelated tasks, supporting a variety of applications. The

framework contains a set of modules, including subsys-tems for for the detection and identification of objects (bi-nary segmentation of the camera images), as well as, thelocalisation of the objects (in the robot’s operation space).

The main part in obstacle detection, the binary seg-mentation of the object from the background in the visualspace, is performed in separate icVision filter modules.A variety of filters have been learnt using CGP-IP [20] –which we already showed to work for Martian rock detec-tion [21] – and most of them are able to perform the objectdetection in near real-time. To avoid collision or interactwith the objects the robot also needs to know where theobject is located, wrt. to the robot’s reference frame. De-veloping an approach to perform robust localisation to bedeployed on a real humanoid robot is necessary to providethe necessary inputs for on-line motion planning, reach-ing, and object manipulation. icVision provides modulesto estimate the 3D position based on the robot’s pose andthe location of object in the camera images [22]. It hasbeen used extensively at IDSIA for various robot visionexperiments [23].

3.3 IntegrationThe sensory and motor sides establish quite a few

capabilities by themselves, yet to perform a safe tele-operation, e.g. to grasp objects successfully while avoid-ing obstacles, a close integration is necessary. The contin-uous tracking of obstacles and the target object is requiredto create a reactive reaching behaviour which adapts inreal-time to the changes of the environment.

We created interfaces between MoBeE and icVisionallowing for a continues visual based localisation of the

Figure 4. Overview of the modules and theon-going research towards a functionaleye-hand coordination system on theiCub. The modules are shown by sub-system: perception (green), action (yel-low) and memory (blue).

detected objects to be propagated into the world model.This basic eye-hand coordination allows for an adapta-tion to changing circumstances, while executing the be-haviours specified by the operator, improving the usabilityof the whole system.

Figure 4 gives an overview of the research in our lab.Various modules (grey boxes) were developed and inte-grated into a functional system. The perception side con-sists mainly of object detection and localisation (top area,depicted in green). The bottom area (yellow) shows theaction/motion side, while the blue area on the right repre-sents the memory modules, including a world model.

The forcing of the dynamical system to control therobot in operational space, allows to project the opera-tor’s motion onto the 7 DOF of the iCub. Because of thecontroller’s interconnection with MoBeE and icVision, therobot is still able to prevent collisions between itself andobstacles perceived in the environment. It is therefore safeto send the signals recorded at the operator directly to ourrobot. To avoid colliding the robot will stop before im-peding collisions, the human in the loop needs thereforenot to worry about unwanted collisions (with the table forexample) just about reaching and grasping.

3.4 EMG MeasurementThe collection of EMG signals on the operator allows

to control the robot’s grasp by performing the grasp itself.To necessary signals are collected using surface electrodesmounted on the operator’s lower arm (Figure 8). In thisfigure you can also see the accelerometer placed at theback of the hand (or the wrist) of the subject. To collectthe data we use a system called BITalino (also visible inthe picture) developed and produced by the Instituto deTelecomunicacoes in Portugal and their spin-off PLUX.It is a small stand-alone embedded system with its ownbattery, allowing to collect up to 8 data channels with-out cables running from the subject to the computer. Thechannels, which can be either EMG or ECG (electrocar-diography) signals, accelerometer data or luminosity in-formation are transferred from the system via Bluetoothto a computer in real-time. With this signal we are able todetect the intent to grasp an object. This results in a com-mand to the robot to perform the equivalent action (usingthe subsystem mentioned above).

4 Experiments and Results

We ran two separate test setups to show that the hu-manoid can be controlled safely from a remote operator.In both cases we were able to avoid self-collisions andcollisions with the environment.

4.1 Remote Control using Accelerations

We first tested our setup with a standard 6-axisgamepad to show the efficacy of our approach and con-trol system. Figure 5 shows this in action. The two con-trol sticks are used to control the two arms respectively.Up/down, left/right motions of the sticks would apply therespective forces to the robot’s end effector. To force thearms forwards (away from the body) or back, two morebuttons per side were incorporated. Various test subjectswere allowed to control the robot, without receiving toomuch information on how the system or controller works.After their initial fear that they could break the robot andthe realisation that the robot would prevent all impedingcollisions, they were enjoying to control the both arms ofthe robot simultaneously.

They also realised quickly that the robot is able to re-act to changing environments. When an obstacle is movedinto the robot’s path, the hand is repelled to avoid colli-sion (Figure 6 shows MoBeE and its information aboutimpeding collisions, and Figure 7 shows a sequence ofimages visualising the robot’s arm motion). This is evenperformed when the arm of the robot is not tele-operated.Additionally they were able to place a hand close to targetspecified by us.

This setup worked very well for when the user wasplaced behind the actual robot, when the operator was fac-ing the robot, a few test subjects had issues because fromtheir point-of-view the right stick would control the leftarm. It was also seen that the control of the arms aloneyields only a very limited reach space for our humanoidrobot. An intuitive way to also control the torso wouldclearly improve this.

Figure 5. The iCub being controlled by astandard 6-axis gamepad.

Figure 6. MoBeE during one of our experi-ments. Parts in red indicate (an imped-ing) collision with the environment. Theinset shows the actual scene.

Figure 7. The reactive control of the leftarm, permitting the iCub to stay clear ofthe static and non-static obstacles.

Figure 8. The setup for one the bio-signal(EMG) collection. The data is colle-ceted by the BITalino sensor, which thensends the accelerations (from the hand)and EMG signals via Bluetooth to theoperator’s computer.

4.2 Measuring the Operator for ControlDuring our next experimental setup the recorded ac-

celerations and the EMG signals of the operator were usedto control one arm of the iCub humanoid. Instead of thegamepad used previously a 3-axis accelerometer was con-nected to the BITalino system. It was placed on the op-erator’s hand (back) or wrist for some individuals. TheBITalino was furthermore facilitated to collect the EMGsignal measured at the extensor digitorum communis (seethe electrodes in Figure 8). The differential between thetwo electrodes is measured and sent back via Bluetooth, athird electrode is used as reference/neutral and is attachedaround the elbow.

The low-pass filtered accelerations of the arm areused directly to ‘force’ the end-effector, similar like withthe gamepad in the previous experiment. In addition theEMG readouts are used to trigger a closing or opening ofthe iCub’s hand. A simple MAV thresholding was used,which seemed to reasonable for most subjects after a quickmanual tweaking. The opening and closing though re-quired to be more forced than a regular use of the handto provide a better signal-to-noise ratio.

We tested this with a handful of test subjects. Whilethe setup is less comfortable than the gamepad, a certainsense of excitement was clearly visible in the subjects.The motion of the arm though produced a rather noisysignal. Reasons are most like the motion of electrodes onthe skin (or even detachment from the skin), some inter-ference in the cables while they are being moved around,and last but not least the change of the EMG signal due tothe change in the dynamics of the system (e.g. counter-acting gravity effects the muscle contraction therefore thesignal). A third setup is planned during which we aim touse machine learning techniques to use both the accelera-tion data and the EMG signal (probably of more channels)to allow a more robust classification and maybe the sepa-ration of different grasp types.

5 Conclusions

Our aim is to generate a tele-operation system for ouriCub humanoid robot. The operation of complex robotsremotely is of interest not just in space but also duringother tele-robotic operations, such as, handling radioac-tive materials or bomb disposal, or disaster response sce-narios (as selected for the DARPA Robotics Challenge).

For this to work, functional motion and vision sub-systems are integrated to create a closed action-perceptionloop to allow for a safe operation. The operator is able tocontrol the end-effector using either a gamepad or an ac-

celerometer attached to its hand. The robot then performsthe equivalent operation, while still avoiding obstacles andcollisions. We also showed that a simple EMG measure-ment can be used to trigger grasps.

In every tele-robot system time lag and the connec-tion bandwidth will limit a full immersion of the operator.Therefore to ensure a safe operation various modules needto take over certain aspects on the robot side. These mod-ules, mainly, the low-level control, obstacle avoidance andobstacle detection are integrated into a full system. Thisis of interest not just in space but also during other tele-robotic operations, such as, handling radioactive materi-als or bomb disposal, or disaster response scenarios (asselected for the DARPA Robotics Challenge).

5.1 Future WorkThis proof-of-concept will be extended in the future

to include more sophisticated bio-signal processing. Fur-thermore a significant amount of user testing will be doneto further improve the operator interface.

Especially the EMG and machine learning side canstill be improved to include multiple grasp types and re-duce the amount of learning required for each operator.We are confident that this will also be useful in other sce-narios, such as, prosthetics control by amputees. Dif-ferent collecting methods, different number of channels,and instruments have so already been used to collect newdatasets for a variety of grasp types. We aim to classifythese with machine learning techniques, such as, support-vector machines, artificial neural networks – including re-current variants – and genetic programming.

The most interesting problem to solve will be the ro-bust detection of the signal while the operator is movingthe arm. Here the noise is high and the signal will containartefacts from other muscle activations (e.g. to counteractgravity and the dynamics of the motion).

Acknowledgment

This research is partly sponsored by the European Com-mission’s Framework Programme under contract FP7-ICT-288551 (WAY). The authors would furthermore liketo thank the researchers from the Scuola SuperioreSant’Anna and SUPSI-DSAN for their valuable input andhelp with the EMG measurements.

References

[1] R. Merletti, P. Parker. Electromyography: physiol-ogy engineering and noninvasive applications., JohnWiley & Sons, 2004.

[2] M. Diftler, J. Mehling, M. Abdallah, N. Radford,L. Bridgwater, A. Sanders, S. Askew, D. Linn, J.Yamokoski, F. Permenter, B. Hargrave, R. Platt, R.Savely, R. Ambrose. ”Robonaut 2: The first hu-manoid robot in space”, in International Conferenceon Robotics and Automation (ICRA), 2011.

[3] R. Ambrose, B. Wilcox, B. Reed, L. Matthies, D.Lavery, D. Korsmeyer, ”Robotics, Tele-Robotics,and Autonomous Systems Roadmap”, NASA’sSpace Technology Roadmaps, National Aeronauticsand Space Administration, 2012.

[4] C. C. Kemp, A. Edsinger, E. Torres-Jara. ”Chal-lenges for robot manipulation in human envi-ronments [grand challenges of robotics]”, IEEERobotics & Automation Magazine 14(1), 2007.

[5] T. B. Sheridan. Telerobotics, automation, and humansupervisory control, MIT press, 1992.

[6] M. Bualat, T. Fong, M. Allan, X. Bouyssounouse,T. Cohen, et al. ”Surface Telerobotics: Developmentand Testing of a Crew Controlled Planetary RoverSystem”, in AIAA Space Conference, 2013.

[7] T. Fong, C. Provencher, M. Micire, M. Diftler,R. Berka, B. Bluethmann, D. Mittman. ”The Hu-man Exploration Telerobotics project: objectives,approach, and testing”, in Aerospace Conference,2012.

[8] F. B. de Frescheville, S. Martin, N. Policella, D. Pat-terson, M. Aiple, P. Steele. ”Set-up and validation ofMETERON end-to-end network for robotic experi-ments”, in ASTRA Conference, 2011.

[9] F. Redi. Esperienze intorno a diverse cose naturalie particolarmente a quelle che ci sono partat dalleIndie, 1686. (taken from J. R. Cram, M. D. Durie.The Basics of Surface Electromyography.)

[10] L. Galvani, ”De viribus electricitatis in motu mus-culari: Commentarius”, Bologna: Tip. Istituto delleScienze, 1791. (trans. R. Montraville Green, 1953)

[11] E.-J. Marey. ”Des appareils enregistreurs de lavitesse”, La Nature (878), 1890.

[12] C. D. Hardyck, L. F. Petrinovich, D. W. Ellsworth.”Feedback of speech muscle activity duringsilent reading: Rapid extinction”, Science154(3755):1467-1468, 1966.

[13] C. Cipriani, F. Zaccone, S. Micera, M. C. Carrozza.”On the shared control of an EMG-controlled pros-thetic hand: analysis of user–prosthesis interaction”,Trans. Robotics 24(1):170-184, 2008.

[14] M. B. I. Reaz, M. S. Hussain, F Mohd-Yasin, ”Tech-niques of EMG signal analysis: detection, process-ing, classification and applications”, Biological pro-cedures online 8(1):11-35, 2006.

[15] M. T. Wolf, C. Assad, A. Stoica, K. You, H. Jethani,M. T. Vernacchia, J. Fromm, Y. Iwashita. ”Decodingstatic and dynamic arm and hand gestures from theJPL BioSleeve.” In Aerospace Conference, 2013.

[16] N. G. Tsagarakis, G. Metta, G. Sandini, D. Vernon,R. Beira, et al. D. Caldwell, ”iCub: the design andrealization of an open humanoid platform for cogni-tive and neuroscience research”, Advanced Robotics(21):1151-1175, 2007.

[17] M. Frank, J. Leitner, M. Stollenga, S. Harding,A. Forster, J. Schmidhuber, ”The Modular Behav-ioral Environment for Humanoids and other Robots(MoBeE).” in International Conference on Informat-ics in Control, Automation and Robotics (ICINCO),2012.

[18] G. Metta, P. Fitzpatrick, L. Natale. ”YARP: Yet An-other Robot Platform”, International Journal of Ad-vanced Robotics Systems 1(3).

[19] J. Leitner, S. Harding, M. Frank, A. Forster, J.Schmidhuber, ”icVision: A Modular Vision Sys-tem for Cognitive Robotics Research”, InternationalConference on Cognitive Systems (CogSys), 2012.

[20] S. Harding, J. Leitner, J. Schmidhuber. ”CartesianGenetic Programming for Image Processing”, Ge-netic Programming Theory and Practice X, Springer,2013.

[21] J. Leitner, S. Harding, A. Forster, J. Schmidhuber.”Mars Terrain Image Classification using CartesianGenetic Programming”, in International Symposiumon Artificial Intelligence, Robotics and Automationin Space (i-SAIRAS), 2012.

[22] J. Leitner, S. Harding, M. Frank, A. Forster, J.Schmidhuber. ”Transferring Spatial Perception Be-tween Robots Operating In A Shared Workspace”, inInternational Conference on Intelligent Robots andSystems (IROS), 2012.

[23] J. Leitner, S. Harding, P. Chandrashekhariah, M.Frank, A. Forster, J. Triesch, J. Schmidhuber.”Learning Visual Object Detection and LocalisationUsing icVision”, Biologically Inspired Cognitive Ar-chitectures 5, 2013.