Guiding the Helping Hand Swagat Kumar Rajesh Sinha › ... › Guiding-the-Helping-Hand.pdfThe market is opening up for robots. According to current market research1,2, the global

Reimagining Research 2018

Guiding the Helping HandInnovations in Robotics for Nimble Tasks

Robots are getting smarter and more complex, with rapid progress in mechatronics, machine learning, and artificial intelligence (AI). Robots are learning to perceive better, do nimble tasks skillfully, and interact effectively with people around them. The advent of collaborative robots that can work alongside humans without cages is opening up new paradigms for teaching new skills to robots through demonstration.

These transformational developments are giving rise to new business opportunities in the space of flexible assembly lines, smarter automated warehouses and distribution centers, as well as intelligent retail stores. New business models, such as Robotics-as-a-Service (RaaS), are emerging with the ability to scale up and down on demand, in place of a CAPEX-oriented acquisition of such systems.

IN B

RIE

F

The largest e-commerce businesses are delivering millions of packages a day. However, there are periodic reports about poor working conditions at fulfilment centers, some even decrying the fact that humans are being worked like robots. There are other jobs where humans can profit from a helping hand; truck loaders and baggage handlers everywhere are prone to musculoskeletal problems, apart from accidents and other hazards.

These jobs are not going away, as we demand that more goods be delivered at our doorsteps in a matter of hours.

The market is opening up for robots.

According to current market research1,2, the global industrial robotics market will increase from about $40 billion to $70 billion during 2016–2023 at a compound annual growth rate (CAGR) of about 10%. The collaborative robot (cobot)

1 International Federation of Robotics (IFR), Executive Summary World Robotics 2017 Industrial Robots, 2017.

2 Markets and Markets, Industrial Robotics Market by Type (Articulated, Cartesian, SCARA, Parallel, Collaborative Robots), Industry (Automotive, Electrical & Electronics, Metals & Machinery, Pharmaceuticals & Cosmetics), and Geography - Global Forecast to 2023, July 2017.

Swag

at K

umar

R

ajes

h Si

nha

Data_Section_TCS-Checked0315.indd 36 29/01/1941 Saka 15:27

Guiding the Helping Hand

Fact File

TCS Research: Robotics

Outcomes: Automatic Truck Loading System, MoPicker, Cloud Robotics Platform. Among Top Four at the Amazon Robotics Challenge (ARC) 2017.

Principal Investigator: Swagat Kumar & Rajesh Sinha

Academic Partners: Professors at IIT Kanpur, IIT Delhi, IIT Jodhpur, and IIIT Hyderabad.

Techniques used: Machine Learning, Computer Vision, and AI

Industries benefited: All industries that use robots for automation

Patents: 12 filed, 1 granted in US, 2 granted in China, Japan, and Europe

Papers: 4 journals, over 20 conferences (6 in Tier 1)

market is also increasing—at a rate of about 56% (CAGR)—and will reach about $4 billion by 2023, indicating aggressive adoption of cobots.3

With these emerging trends, we at TCS Research have focused on the following specific areas related to robotics:

Robots for warehouses and shop floorsAutomated pick-and-place robots for e-commerce fulfillment centers: Based on the order list received from a user, this robot is capable of identifying and picking the items directly from the rack and putting them into the packing box, which is then shipped to the user. Some of the core technologies developed for this project include a deep learning-based system for recognizing and segmenting thousands of stock-keeping units (SKUs) in a clutter, computing

necessary motion trajectories for robot end-effectors to reach targets while avoiding obstacles en route, and space optimization within shipping boxes. Alternatively, several e-tailers use a highly autonomous and dense goods-to-picker model.4 The process starts when the robots send item totes to picking stations where autonomous robots pick and pack the final order by undertaking semantic and/or instance segmentation, optimal grasping, and final packing into shipping boxes.

Automated Truck Loose-Loading Systems (ATLS): We explore the automation of manual processes involved in loose loading or unloading trucks and palletizing and de-palletizing goods by using robots in a closed-loop operation. Some of the challenges include identifying the size and shape of packets in a cluttered environment and poor illumination. High throughput is essential while

3 Markets and Markets, Collaborative Robots Market by Payload Capacity (Up to 5 kg, Up to 10 kg, Above 10 kg), Industry (Automotive, Electronics, Metals & Machining, Plastics & Polymer, Food & Agriculture, Healthcare), Application, and Geography - Global Forecast to 2023, September 2017.

4 Intermodal Transport, Cargo Loading and Unloading Efficiency Analysis in Multimodal Transport, July 8, 2014.



ensuring closed-space packing with minimum failures. Dealing with dynamic and unstructured environments is a key requirement. The ATLS solves a huge problem: efficient utilization of transport by dead-pile or loose loading (95%), as compared to standard pallets (60%), provided the robotic loading and unloading matches forklift-driven operations speed.

Fleet Management Systems (FMS) for Autonomous Mobile Robots (AMR): We are looking into problems of Multi-Robot Task Allocation (MRTA) and Multi-Agent Path Finding (MAPF). Significant costs are incurred over internal transport, which carry parcels, parts, and totes within factories, terminals, and fulfillment centers. The task allocation problem is solved by using an auction-based system with dependencies, where each agent bids for tasks and the one with the best bid gets the task. The bidding mechanism is based on a utility function, one for each agent that best describes its ability to execute the task at hand. The second problem (MAPF) aims at finding paths for all agents that minimize deadlock and livelock situations and avoid collision with other agents. These processes will be further extended for distributing work between robot- and human-driven machines working together in shops and warehouses. Further extensions done by our academic partners will provide the ability for more than one robot to collaborate and manoeuver large items in constrained spaces, as a team.

Smart factories: Smart factories represent digitalization, reconfigurability, and agility of assembly lines, mass customization,

higher Overall Equipment Efficiency (OEE), and deep ecosystem linkages across Original Equipment Manufacturers (OEMs), suppliers, and customers. Automation and robotics play a key role in smart factories. Rather than fixing open-loop pick places, painting, welding, machining, parts presentments, and material resupply, increasing cobots and autonomous flexible robots will allow uncaged robots to work alongside humans. We are also deploying cognitive robotic systems, which do deep learning–based visual quality control and line balancing in the food industry. Projects that reimagine the current working cells that require substantial changeover time, are being ‘roboticized’ to react to a variety of inputs without large changeover time.

Smart warehouse and factory showcaseKeeping up with the requirements of Industry 4.0, we are working towards setting up a smart warehouse with people-less operations on the shop floor. A schematic of this warehouse is shown in Figure 1. Once an order is placed by a user, it will be processed by an order fulfillment system, which will feed the necessary data to a cloud robotics system that will manage the operation of multiple robots. The cloud platform will carry out compute-heavy tasks such as image processing, task allocation among robots, path planning, and real-time status checks of the robots. Individual robot agents will interact with each other either directly or through cloud mediation to get information about the goals to be accomplished, dynamic map updates, and changing order priorities.



Looking ahead: smarter, cheaper, and in the cloudIn order to realize the full potential of robotics opportunities in multiple industries, it is important to address some of the research challenges related to computer vision, hand-eye coordination, imitation learning, and hardware-agnostic control design. We present some developments that are already happening now and will gain widespread adoption in the coming years.

Programming by demonstration: Robots can be endowed with the ability to learn from their human operators quickly through a paradigm called ‘programming by demonstration’ (PbD)5 or ‘imitation learning’6, without undergoing time-consuming programming-debugging-testing cycles. The operators will be able to effectively interact with these robots through natural modes of human communication, such as gestures, body language, gaze, physical contact (kinesthetic teaching), and even emotions.

Cloud Robotics Platform:• Deep-learning-based Image Processing• Multi-robot path planning, task allocation• Sharing of object models and action recipes among machines

ERP SystemConveyors

StationaryRoboticArms

Shipped to the User

Packing

Mobile ManipulatorSystems

Racks

Online User

Figure 1: The Warehouse of the future

5 Springer Handbook of Robotics, Robot programming by demonstration, 2008.

6 Advanced Robotics, Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input, 2011.



Flexible assembly lines: Smart robots will eventually reduce the overall cost of robot-based automation, thereby making it accessible and affordable to small and medium enterprises (SMEs) who so far could not pay for complex assembly lines that were required to guide the conventional robots currently being used in factories. One such example is the project ‘Factory-in-a-day’7,8 that aims at reducing installation time and cost, thereby allowing smaller enterprises to rent or lease robots and set up their factories within a single day.

Cloud robotics platforms: These will leverage the advances made in the areas of cloud computing and communication technologies to manage a large number of robot swarms spread densely or sparsely over a large area. Such a framework will make it easier for robots to learn from humans as well as other machines, collaborate and cooperate with each other, provide reliability of operations, in the event of failures, to one or more individual units, and increase efficiency and productivity of operations through optimal planning, task allocation, and decentralized decision-making.

Combining cloud robotics,

vendor neutral robot skills and action recipes, reconfigurability of work environment and goals, deep learning and symbolic AI with IT-enabled technologies, it is possible to offer various robot-based services through a RaaS9,10 model. A RaaS vendor delivers business process fulfillment on a pay-as-you-use model, leasing robots, machines, and software, as necessary. The key to this model is effective utilization of base robots across industries, domain-specific End of Arm Tooling (EOAT), and rapid synthesis of task-specific controllers with a fleet of cheap robots.

7 International Joint Conference on Neural Networks (IJCNN), Learning Stable Movement Primitives by Finding a Suitable Fuzzy Lyapunov Function from Kinesthetic Demonstrations, July, 2018.

8 Factory-in-a-day, Plug&Work Robots, 2013.

9 IEEE Transactions on Automation Science and Engineering, Rapyuta: A cloud robotics platform, 2015.

10 Tata Consultancy Services, Robotics-as-a-Service: Transforming the Future of Retail, 2015.



Amazon Robotics has been conducting robotics challenges for the past few years, to encourage innovation in the area of warehouse robotic automation. TCS partnered with the Indian Institute of Technology, Kanpur (IITK) and participated in the 2016 and 2017 editions of this challenge.

In these competitions, the participant robots are expected to pick and stow items autonomously from and to a given rack. The picking task involves moving items from a rack and placing them in a

tote, while the stowing task involves moving items from the tote to the rack. The objects to be picked or stowed are general household items most commonly ordered through the Amazon web store. They vary greatly in size, shape, appearance, hardness, and weight. Since there is no constraint on how the products are organized on the rack or the tote, there are several possibilities of configuration one might encounter during the actual operation. This uncertainty that may arise due to factors such as occlusion, variation in illumination, pose, viewing angle, and so on, makes the problem of autonomous picking and stowing extremely challenging.

The 2017 edition of the challenge had a packing task: to pick and pack in three most used Amazon shipping boxes. Unlike the 2016 challenge, where all the objects to be picked and stowed were known in advance, in 2017, the teams were given 20 new items 45 minutes prior to the start of the challenge in which they had to train their software for object recognition. The rack designs were left to the teams, with only volume constraints given, which in effect made the clutter even more dense.

Mr. Pick-and-Stow



Some of the major innovations by the TCS-IITK team for the ARC 2017 event were as follows:

• An automated data collection system that could generate thousands of labeled images for training deep networks

• A deep learning-based object detection and segmentation system that was capable of recognizing objects in extreme clutter with, high (>95%) accuracy

• An online motion planning system capable of generating trajectories with variable speed while avoiding collisions with obstacles on its way

• A hybrid EOAT that combined suction with gripping action to pick or grasp various kinds of objects, including deformable objects

• Preemptive state machine architecture and suitable parallelization, to finish all picks in half the allotted time

The results of the 2017 ARC challenge at Nagoya yielded rich rewards for the IITK-TCS team. The team had a podium finish securing third place in the pick event, qualifying for finals and finishing in fourth position in the final round. The IITK-TCS team was ahead of several well-known industrial houses (Panasonic, Mitsubishi, and Toshiba) and leading academic institutes (the Massachusetts Institute of Technology, Princeton University, Carnegie Mellon University, the KTH Royal Institute of Technology, and Duke University).



Swagat KumarSwagat Kumar is a Scientist at TCS Research and Innovation and heads the Robotics Research Group at the TCS Innovation Lab, New Delhi. His research is focused on robotics and computer vision. Prior to joining TCS, Swagat worked as a post-doctoral researcher at the Kyushu University in Japan and as an assistant professor at IIT Jodhpur. He has a Ph.D. in Electrical Engineering from IIT Kanpur. He has authored several publications for international conferences and journals.

Rajesh SinhaRajesh Sinha is a Chief Scientist at TCS Research and Innovation and heads the Smart Machines and Autonomous Infrastructure Research Program. He has over 20 years of experience in building engineered software and hardware solutions for Transportation, Logistics, Retail, and Government applications for TCS and for start-ups. His areas of interest are robots, drones, augmented reality platforms, and smart vision. Rajesh holds a degree in Electrical and Electronics Engineering from BITS Pilani and a Masters in Comparative Religion.



© G

lasb

erge

n gl

asbe

rgen

.com


All content / information in Guiding the Helping Hand is the exclusive property of Tata Consultancy

Services Limited (TCS) and/or its licensors. This publication is made available to you for your

personal, non-commercial use for reference purposes only; any other use of the work is strictly

prohibited. Except as permitted under the Copyright law, this publication or any part or portion thereof

may not be copied, modified, adapted, translated, reproduced, republished, uploaded, transmitted,

posted, created as derivative work, sold, distributed or communicated in any form or by any means

without prior written permission from TCS. Unauthorized use of the content/information appearing

here may violate copyright, trademark and other applicable laws, and could result in criminal or civil

penalties.

TCS attempts to be as accurate as possible in providing information and insights through this

publication, however, TCS and its licensors do not warrant that the content/information of this

publication, including any information that can be accessed via QR codes, links, references or

otherwise is accurate, adequate, complete, reliable, current, or error-free and expressly disclaim any

warranty, express or implied, including but not limited to implied warranties of merchantability or

fitness for a particular purpose. In no event shall TCS and/or its licensors be liable for any direct,

indirect, punitive, incidental, special, consequential damages or any damages whatsoever including,

without limitation, damages for loss of use, data or profits, arising out of or in any way connected with

the use of this publication or any information contained herein.

©2019 Tata Consultancy Services Limited. All Rights Reserved.

Tata Consultancy Services (name and logo), TCS (name and logo), and related trade dress used in

this publication are the trademarks or registered trademarks of TCS and its affiliates in India and other

countries and may not be used without express written consent of TCS. All other trademarks used in

this publication are property of their respective owners and are not associated with any of TCS’

products or services. Rather than put a trademark or registered trademark symbol after every

occurrence of a trademarked name, names are used in an editorial fashion only, and to the benefit of

the trademark owner, with no intention of infringement of the trademark.

Documents

Guiding the Helping Hand Swagat Kumar Rajesh Sinha › ... › Guiding-the-Helping-Hand.pdfThe market is opening up for robots. According to current market research1,2, the global