Tele-Operation of Robot Using Gestures

Farhan Mohammad Electronics & Communication Engineering

Padmasri Dr. B.V.Raju Institute of technology Narsapur, Andhrapradesh, India

farhan.fm92@gmail.com

Kishore Reddy Sudini Electronics & Communication Engineering

Padmasri Dr.B.V.Raju Institute of technology Narsapur, Andhrapradesh, India Sudinikishore25d@gmail.com

Varun Puligilla

Computer science and Engineering Padmasri Dr.B.V.Raju Institute of Technology

Narsapur, Andhrapradesh, India varunrocksup@gmail.com

Prabhakara Rao Kapula

Electronics & Communication Engineering Padmasri Dr. B.V.Raju Institute of technology

Narsapur, Andhrapradesh, India prabhakar.kapula@bvrit.ac.in

Abstract—This paper proposes an idea to develop a tele-

operated humanoid robot, which has capability to perform assigned task at a long distance based on the gestures given by the human. Tele-operation of robots is the field that bridges the transition between manual control and complete autonomous control of robots. Controlling robots at anyplace in the world by accessing the sensory data from the robot gives the humans the capability to perform several tasks. To start with this idea a framework is implemented in which the user can send commands to control a humanoid (a Robosapien toy) from anywhere in the world over Transmission control protocol. An extension is being developed to control the robot remotely using gestures. The goal is to make the robot perform the same gestures as the operator at a different place. A communication channel to transfer image feedback data from the robot via a TCP socket is also being developed. Users just need to use the most natural body gestures to interact with the robot. Microsoft Kinect is applied in this system to recognize different body gestures and generate visual Human-Robot interaction interface, then the controlling signals of different body gesture modules are sent to the humanoid robot thorough a infrared transmitter, which can stimulate the robot to complete tasks. This kind of system aims to enrich the interactive way between human and robots and help non-expert users to control the robot freely, making Human-Robot interaction much easier.

Keywords—Tele-operation, humanoid robot, gesture recognition, TCP socket, kinect.

I. INTRODUCTION

To communicate with computers through human interaction by using gestures and movements is fascinating. The data acquired from a motion capturing system can be used in the field of robotics for creating virtual interfaces and to make characters in computer games move realistically. The motion capture suit used in the movie industry has multiple references attached to it, which aids the software to acquire correct data, and

analyzes where each reference-point is positioned. Motion capturing suits are often used by the game and movie industry to create realistic movements for computer generated �gures. Positioning and motion analyzing of other objects are often done by lasers. In the production industry, laser-positioning systems are together with industrial robots used for locating objects and determine its position and movement. Both of these existing technologies are expensive, and therefore the opportunity to create an alternative solution arises. Is it possible to make an open source competitive motion tracking system that is easy to use and is addressed to people and companies with limited resources? Research in the �eld of computer vision is not a new area. It has existed for a long time (by computer standards), and progress is being made continuously. One of the most common computer vision libraries is OpenCV [1] which was originally developed by Intel cooperation. It includes features like shape recognition, a big collection of di erent tracking-algorithms, and motion-analysis techniques [2]. With this speci�c library, and a few others, all the necessary tools are provided to implement a robust and accurate motion tracking application [3]. The only thing required is a camera with depth sensors. Microsoft released the Kinect input-device in November 2010. It provides depth and color amongst other features. It did not take a long time before the Kinect was hacked and libraries to communicate with the device were released. The source community quickly embraced this new technology, and all sorts of hacks circulated on the Internet. A framework for motion capture of an arbitrary object could be built using the Kinect input device and various computer vision libraries. We would like to extend this criterion and compare humans and robots by looking at their actions. This regards the abilities to perceive the world, to learn from what happen around themselves, and the way they move with respect to the rest of the environment, that is the hallmark of an intelligent system. This project aims to

2013 7th Asia Modelling Symposium

978-0-7695-5101-2/13 $26.00 © 2013 IEEE

DOI 10.1109/AMS.2013.15

67

2013 7th Asia Modelling Symposium

978-0-7695-5101-2/13 $26.00 © 2013 IEEE

DOI 10.1109/AMS.2013.15

67

provide a robot the Capability to perform a task, recognizing actions from human demonstrators and learning from their movements. A lot of work has been done from the 1950 in these fields, in particular in computer vision and robotics research. Computer vision goal is to extract information from a scene and structure it, and action recognition is assuming more and more importance in this field for its correlation with wide range of application such as biometrics, animation and interactive environments, video analysis and surveillance. The identification process can be divided into three stages: feature extraction, feature representation and action classification [4]. Two different approaches characterize features extraction: model-based and with no-model assumption. The first technique takes advantage of the prior knowledge of the body model to extract the selected features from the scene. The second one, instead, uses a common features extraction method. The features representation is strictly connected to the extraction phase. The model-based methods usually keep track of position and orientation of each part that composes the model. No-model methods, on the other hand, maintain the features characterize themselves: surface normals, RGB feature time-space information of each point and so on. Robotics was defined as the science which studies the intelligent connection between perception and action[8] and mixes together mechanics, controls, computers and electronics, so that it benefits from advances in the different technologies. Recently, robots have been used in more and more areas such as service robotics, field robotics, or human augmentation. The human ability to imitate behaviors is already present in even 12 - 21 day old infants [5]. Despite some hypothesis about the translation between visual and motion features [6], the mechanisms that express a demonstration to a set of internal motor primitives remain largely unknown. Several works in robotic imitation suggest that robots can perform new skills by learning key information from a demonstration [7]. More recently, this idea has been applied on humanoid robots as a natural way to teach how to perform a task in a dynamic environment.

Figure 1. Kinect detecting various Joints

II. METHODOLOGY The aim of this project is to make a robot that can navigate based on the gestures given by the user (Ex: Raising a hand). To recognize the user’s gestures, kinect is used. Kinect is Microsoft’s motion sensor add-on for the Xbox 360 gaming console. The device provides a natural user interface (NUI) that allows users to interact intuitively and without any intermediary device, such as a controller. A depth camera, which sees in 3-D, creates a skeleton image of a player and a motion sensor detects their movements. Kinect gathers the color and depth information using an RGB and Infra-Red camera respectively. It then creates an approximate skeleton of all the limbs of the blob detected. From this, the Kinect can make out the location of the various joints (hands, neck, head, etc.) of the human body. We have interfaced the kinect to the computer with Linux based operating system. The kinect is programmed in such a way that it can understand any kind of manual command given by the human so as to produce the desired output. This concept is nothing but gesture recognition. Generally, the concept of gesture recognition is used to process the information in some of the conditions where there is no possibility of transferring the data or information through speech or type by the human. This gesture recognition is achieved by using different types of mathematical algorithms. Some of the algorithms are 3D model –based algorithm, skeletal model-based algorithm etc., here in this project the human gesture is captured using skeletal-tracking as shown in figure 2. The basic concept of this project is to manipulate a humanoid robot at a far place using the human gestures which is captured using kinect. The gestures captured using kinect are then sent to the humanoid robot in the form of signal. Whenever this signal is received by the humanoid robot it makes the movement. The movements of the humans are imitated by the humanoid robots as the gestures are being sent to the robot which was captured using kinect. But in order to transmit the human gestures to the humanoid robot in the form of signal we need to interface the robot with an interfacing device which can help to transmit the appropriate signals to the robot for making the movement. For interfacing the humanoid robot with the gestures in the system we make use of a serial communication USB-UIRT(Universal infrared receiver and Transmitter). Through this serial communication module the connection between the computer and humanoid robot is established. The gestures which are captured in the computer using kinect are then transferred parallel to the humanoid robot. In the next part we are going to manipulate the humanoid robot at a far place with the gestures captured using Kinect. This technique is called Teleoperation. Tele operated robots or tele robots are devices remotely operated from a distance by a human operator rather than

6868

following a predetermined sequence of movements. They are used when a human cannot be present on site to perform a job because it is dangerous, far away, or inaccessible.

Figure 2. Tracking human body using kinect

The robot may be in another room or another country, or may be on a very different scale to the operator. It will be making movements based on the gestures given by the human rather than following its own pre-defined sequence. With this technique the robot can be made to do tasks which a human being can do but with more effective and efficient way. They will also perform assigned tasks accurately and precisely. Moreover they can be operated in any environment and any place In this project we will manipulate a humanoid robot with help of a computer using serial communication later connect two computers using socket connection and by manipulating the humanoid robot which is at remote place by sending the gestures from the second computer.

II. IMPLEMETATION OF ALGORITHMS In this interactive demonstration system, Robosapien Humanoid robot is controlled by the body gestures. Kinect is used as the vision part to collect the controller’s body gestures. It is able to capture the surroundings in 3D by combining the information collected from the depth

sensors and the standard RGB camera. The result of this combination is an RGBD image with 640x480 resolutions, where each pixel is assigned certain color information as well as depth information. In ideal conditions the resolution of the depth information can be 3 mm in height, using 11 bits resolution. Kinect works with the frequency of 30 Hz for both RGB and depth camera. On the left side of the Kinect is a laser infrared light source which generates electromagnetic waves with the wavelength of 830 nm. Information is encoded in light patterns which are deformed as the light reflecting from objects in front of the Kinect. In this system, the ideal interactive distance is given as 2 meters after a number of experiments. Kinect can not only offer natural interaction interface, generate the 3D model of objects, but also track the skeleton in 3D space. We can get the x-y-z coordinates of the interested pointed through Kinect which are used to defined gesture modules. Skeleton tracing is the core technology of Kinect and because of this the Human-Robot interactive demonstration system can be achieved. Kinect can track 20 skeleton points of human body at most, and these specific points are shown in Figure4. Here, the client connects to server by using the IP address of it and a PORT number is used for transferring the data which has to be sent from client to server. If both the port numbers match with each other the client connects to the port and the server accepts the connection then the server creates a new temporary port that is used for the actual send or receives communication with the server during the transaction as shown in figure 3. For interfacing the humanoid robot with the gestures in the system we make use of a serial communication USB-UIRT. The USB-UIRT is a unique IR (infra red) receiver and transmitter. It connects to the computer through a USB port. Through this serial communication module the connection between the computer and humanoid robot is established. The gestures which are captured in the computer using Kinect are then transferred parallel to the humanoid robot. The goal is to make the robot perform the same gestures as the operator at a different place. This technique is called Tele-operation.

Figure 3 Block Diagram of Tele-operation of Robot

6969

Figure 4. Joints of a human body

. Tele operated robots or tele robots are devices remotely operated from a distance by a human operator rather than following a predetermined sequence of movements. They are used when a human cannot be present on site to perform a job because it is dangerous, far away, or inaccessible. The robot may be in another room or another country, or may be on a very different scale to the operator. In this project we will manipulate a humanoid robot with help of a computer using serial communication later connect two computers using socket connection and by manipulating the humanoid robot which is at remote place by sending the gestures from the second computer. Finally, the Robot will act according to the gesture detected by the Kinect given by the user which is at far place. To create a socket between two different computers (server, client) which are located at long distance we must write a server program that creates the server socket, associating it with a port. We then listen to the server socket for incoming requests. We must then write a client program that creates the client socket, associating it with a port. The humanoid robot used in our project is a Robosapien toy. The Robosapien is preprogrammed with moves, and also can be controlled by an infrared remote control included with the toy, or by either a infrared transmitter device. In our project we have replaced the infrared remote control with USB-UIRT interfaced with the server system so that the robot can be manipulated using the gestures given by the user. Finally, the formation of the output will be like the gesture detected by the Kinect (which is interfaced with client) is first sent into the computer and then to the client’s port and from there it is received by server’s port and after receiving data is transmitted to the robot in the form of signal through UIRT. In the project we have replaced the infrared remote control with USB-UIRT interfaced with the server system so that the robot

can be manipulated using the gestures given by the user in the form of a signal to USB-UIRT so that robot acts to it.

Figure 5. Interfacing Robot using UIRT

IV. RESULTS AND APPLICATIONS

Figure 6. Calibration of the human gestures at Client system

Figure 7. Server program transmitting the signals to robosapien.

Humanoid robots can perform various tasks autonomously which a human being cannot do. Humanoids will interact socially with people in typical, everyday environments. There are already robots which can do tedious, repetitive labor for specialized

7070

environments and tasks. Instead, humanoids will be designed to act safely alongside humans, extending our capabilities in a wide variety of tasks and environments. In the project as we are concerned with the movements and actions of humanoid robots performed and controlled by imitating human gestures they can be helpful to society in many ways. A. Military, security and law enforcement Military, Security and law enforcement applications for robots that look and act like humans are plentiful. These humanoid robots can be of much help. We can operate a humanoid robot to guard any area from any place which requires special military enforcements. They can guard the place for any terror activity and we can monitor the place using cameras mounted in the robot. With this technique we can reduce human efforts. If we can remove human soldiers, fireman and police officers (almost) completely and replace them with humanoid robots it would be a great achievement. B. Medical research and training Humanoid robots can be used for medical research and training. A humanoid robot can be trained adapt themselves in hospitals and as they are capable of imitating human actions they are also ideal in performing operations. Humanoids can also be trained to act as nurses. Humanoid nurses will take care of patients and their routine tasks. And since robots can perform more accurate and precise actions, they will also be ideal to perform very delicate surgery. C. In Manufacturing and Construction Humanoid robots have the ability to do the most hazardous and mundane jobs in manufacturing and construction. Humanoid robots with their structure resembling the human being they can be made to do factory jobs like transporting and transferring tools, etc.

V. CONCLUSION Most human interactions with the environment depend on our ability to navigate freely and to use our hands and arms to manipulate objects. Developing natural means of controlling these abilities in humanoid robots can significantly broaden the usability of such platforms. It may have applications ranging from household works, medical, welfare, security, education, search and rescue and many more. An ideal interface for humanoid robot teleoperation will be inexpensive, person-independent, require no wearable equipment, and will be easy to use, requiring little or no user training.

ACKNOWLEDGEMENT We are highly indebted to Randy Pausch Robotics Engineering Centre of Padmasri Dr.B.V.Raju Institute of Technology for giving us this opportunity to develop and implement the project. We wish to express our sincere gratitude to Col. Dr.T.S.Surendra, Director, R&D and Dr. I A Pasha, Head of Electronics and Communication Engineering, BVRIT for their kind support and encouragement to take up the project work. We sincerely thank our project mentors Ms. Sirisha Kallakuri, Tennessee state university, USA and Mr. Srikanth kallakuri, Research Engineer, ATR International, Japan for their guidance as well as for providing necessary information regarding the project. REFERENCES [1] OpenCV Reference Manual, v2.2 edition, December 2010. [2] Aroh Barjatya. Block matching algorithms for motion estimation.

Technical report, Blekinge [3] Institute of Technology, IEEE, 2004. [4] Libfreenect, https://github.com/OpenKinect/libfreenect June 2012. [5] Wei, W., Yunxiao, A.: Vision-based human motion recognition: A

survey. In: Proceedings of the 2009 Second International Conference on Intelligent Networks and Intelligent Systems. pp. 386{389. ICINIS '09, IEEE Computer Society, Washington, DC, USA (2009)

[6] Meltzo, A.N., Moore, M.K.: Imitation of Facial and Manual Gestures by Human Neonates. Science 198(4312), 75{78 (Oct 1977)

[7] Wolpert, D.M., Ghahramani, Z., Jordan, M.I.: An internal model for sensorimotor integration. Science 269, 1880{1882 (1995)

[8] Schaal, S.: Learning from demonstration. In: Advances in Neural Information Processing Systems 9 (1997)

[9] Siciliano, B., Khatib, O. (eds.): Springer Handbook of Robotics. Springer (2008)

[10] Michael Van den Bergh, Daniel Carton, Roderick De Nijs, Nikos Mitsou, Christian Landsiedel,Kolja Kuehnlenz, Dirk Wollherr, Luc Van Gool, Martin Buss, Real-time 3D Hand Gesture Interaction with a Robot for Understanding Directions from Humans IEEE RO-MAN, 357-362, 2011

[11] Michal Tolgyessy, Peter Hubinsk, The Kinect Sensor in Robotics Education.

[12] Hee-Deok Yang, A-Yeon Park, Seong-Whan Lee, Human-Robot Interaction by Whole Body Gesture Spotting and Recognition, 18th International Conference on Pattern Recognition, Vol.4, 774-777, 2006. Radu Muresan, Stefano Gregori, Natural User Interface for Robot Armature Control, February 7, 2011.

[13] Aaron F. Bobick, James W. Davis, and IEEE Computer Society. The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23:257–267, 2001.

[14] Gary Bradski and James Davis. Motion segmentation and pose recognition with motion history gradients. In Machine Vision and Applications, pages 238–244, 2000.

[15] T. Kanda, H. Ishiguro, M. Imai, T. Ono, and K. Mase, “A constructive approach for developing interactive humanoid robots,” in Proc IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2002, pp. 1265–1270.

7171