Demonstrating Everyday Manipulation Skills in RoboCup@Home

•By J€org St€uckler, Dirk Holz, and Sven Behnke

As benchmarking robotics research isinherently difficult, robot competi-tions are increasingly popular.They bring together researchers,

students, and enthusiasts who arein the pursuit of a technological challenge. Prom-inent examples for such competitions include theDefense Advanced Research Projects Agency’s(DARPA’s) Grand and Urban Challenges [1], theInternational Aerial Robotics Competition (IARC)[2], the European Land-Robot Trial (ELROB) [3],and, last but not least, RoboCup [4], [5].

Such competitions provide a standardized testbed for different robotic systems. All participatingteams are forced to operate their robots outside theirown lab in an uncontrolled environment at a sched-uled time. This makes it possible to directly comparethe different approaches used for robot construction,environment perception, and control.

While the annual RoboCup competitions are bestknown for their soccer leagues, they also feature two leaguesin other domains—the RoboCup Rescue league for robotssupporting first responders and RoboCup@Home address-ing service robot applications in domestic environments.

In RoboCup@Home, different disciplines of roboticsresearch, such as mobile manipulation and human–robotinteraction (HRI), are tightly coupled, i.e., approaches areintegrated systems, and benchmarking of individual compo-nents becomes less suitable. Benchmarking is done by demon-strating (and comparing) the performance and reliability ofcomplete systems in a realistic setup and in an integrated way.

In this article, we present the contributions of our team NimbRoto the RoboCup@Home league. We describe the challenges in

34 • IEEE ROBOTICS & AUTOMATION MAGAZINE • JUNE 2012 1070-9932/12/$31.00ª2012 IEEE

Digital Object Identifier 10.1109/MRA.2012.2191993

Date of publication: 6 June 2012

© ISTOCKPHOTO/CHENG EN LIM

this league and detail our approaches to the competition.Our team achieved first place in the 2011 competition.While we successfully participated in many standard tests,we also demonstrated some of the novel capabilities of theleague, such as real-time tabletop segmentation, flexiblegrasp planning, and real-time tracking of objects. We alsodescribe our approach to human–robot cooperativemanipulation using compliant control.

The RoboCup@Home LeagueThe RoboCup@Home league [6], [7] was established in2006 to foster the development and benchmarking of dex-terous and versatile service robots that can operate safelyin everyday scenarios. The robots have to exhibit a widevariety of skills: object recognition and grasping, safeindoor navigation, and HRI. In 2011, 19 internationalteams competed in the RoboCup@Home league. It is cur-rently one of the strongest growing leagues in RoboCup.

Competition DesignThe competition is organized into two preliminary roundsor stages and a final stage [8]. The preliminary stages consistof predefined test procedures as well as open demonstra-tions. The predefined tests include skills for domestic servicerobots that must be solved with state-of-the-art approaches.The time to complete the tests is limited, which forces theteams to implement time-efficient approaches. During thetests, the robots must operate autonomously. Helping withphysical interaction or remote control is not allowed. Therules also include extra scores for specific skills that requiresolutions to research questions and rewarding scientific sol-utions that go beyond the fulfillment of the basic require-ments of a test. In open demonstrations, the teams canchoose their own task for the robot to demonstrate theresults of their own research.

After the first stage, the top 50% of the teams (with respectto score) advanced to the second stage, where they have toperform more complex tasks. The top 50% of the teams (withrespect to score) after the second stage (including the pointsscored in the first stage) further advanced to the final stage,which is conducted as an open demonstration.

While the rules and the tests are announced severalmonths before the competition, the details of the competi-tion environment are not known to the participants inadvance. During the first two days of the competition, theteams can map the competition arena, which resembles anapartment, and train object recognition on a set of about 20smaller objects that are used as known objects with namesthroughout the recognition and manipulation tests. Thearena is subject to minor and major changes during thecompetition and also contains previously unknown objects.

Performance EvaluationIn the predefined tests, each subtask is assigned a certainnumber of points, which are awarded upon successfulcompletion. This allows an objective evaluation of the

overall system performance as well as the assessment ofindividual components.

The performances of the teams in the open demonstra-tions vary greatly and are hence harder to compare. Opendemonstrations are evaluated by juries for their technicaland scientific merits. To provide a fair assessment, thesejuries are formed from leaders of other teams or frommembers of the technical and executive committees of theleague. The jury in the final is formed from members of theleague’s executive committee and distinguished externalrepresentatives of science, industry, and the media.

The juries evaluate the teams by specific criteria that aredefined in the rules of the competition. In the final stage,for example, the external jury assesses originality andpresentation, usability of HRI, difficulty and success, andrelevance for daily life.

Tests and SkillsThe tests in the RoboCup@Home league are designed toreflect (and test for) the large diversity of problemsaddressed in service robotics research.

Tests in Stage IAll teams participate in the first stage, which tests basicmobile manipulation and HRI capabilities. In the RobotInspection and Poster Session test, the robots have to navigateto a registration desk, introduce themselves, and get inspectedby the league’s technical committee. Meanwhile, the teamgives a poster presentation that is evaluated by the leaders ofthe other teams. In Follow Me, the robots must demonstrateperson tracking and recognition capabilities in an unknownenvironment. The robot is guided by a previously unknownuser who can command the robot either by speech or ges-tures. At several checkpoints, the robustness of the ap-proaches is tested by applying different disturbances. Mobilemanipulation and HRI capabilities have to be integrated forGoGetIt. Here, the robot has to retrieve the correct objectamong others from a room. The room is specified to therobot by a human user using speech input. Person detectionand recognition in the home environment is tested withinWhoIsWho. The robot has to learn the identity of two personsand must later find the persons among others in a differentroom. In the General Purpose Service Robot I test, the robotsmust understand and act according to complex speech com-mands that consist of three subtasks, such as moving to alocation, retrieving a specific object, and bringing it back tothe user. The last test in this stage is the Open Challenge inwhich the teams can demonstrate their system in a 5-min slot.

Tests in Stage IIThe teams that advance to the second stage are tested inmore complex scenarios. Enhanced WhoIsWho extendsWhoIsWho toward a robotic butler scenario. A user tellsthe robot to bring beverages to three out of five persons.The robot has to fetch the beverages and deliver them tothe correct person. Again, the robot is introduced to the

JUNE 2012 • IEEE ROBOTICS & AUTOMATION MAGAZINE • 35

persons at the beginning. This test has to be solved within10 min. In General Purpose Service Robot II, commandswith missing or erroneous information are given to therobot. The robot has to ask for missing information, or, ifit detects erroneous task specifications during the execu-tion of the task, it must react accordingly, report back tothe user, and propose alternative solutions. Shopping Malltests the abilities of the robots to operate in previouslyunknown environments. A human user guides the robotthrough a real shopping mall and shows the location ofseveral objects. Afterward, the robot must fetch a subset ofobjects specified by the human user. Stage II concludeswith the Demo Challenge. This 7-min open demonstrationfollows a theme that is defined before the competition. In2011, the theme was cleaning the house.

Related Research on Integrated SystemsThe lean rules in the RoboCup@Home league facilitatediverse approaches. Some teams construct new and inno-vative robot hardware while others resort to off-the-shelfhardware to focus on algorithmic problems.

The Chinese team WrightEagle [9] has competed in theRoboCup@Home league since 2009. In 2011, they intro-duced the KeJia-2 robot platform that supports omnidirec-tional driving and is equipped with two seven degrees offreedom (7 DoF) manipulators for humanlike reach, similarto our robots. In the competition, KeJia made popcorn in amicrowave oven. For this demonstration, the robot had topress buttons to open and close the microwave door.

The German team b-it-bots [10] introduced their robotJenny in the 2011 competition. Jenny consists of a modifiedCare-O-Bot 3 platform from Fraunhofer IPA with a 7-DoFKuka lightweight robot arm and a three-finger Schunk hand.

The Australian team RobotAssist [11] competes with arobot that combines a Segway RMP 100 base with an ExactDynamics iArm manipulator. For manipulator control,they apply an optimization method that finds collision-free

arm configurations for the object to manipulate. RobotAs-sist also demonstrated person detection, identification, andsocial skills with their robot.

Besides RoboCup@Home, many research groups devel-oped integrated systems for mobile manipulation in every-day environments. Demonstrations of these systems areperformed in isolated settings in labs or at trade fairs.

A prominent example is the Personal Robot 2 (PR2)developed by Willow Garage. Bohren et al. [12] demonstratedan application in which PR2 fetches drinks from a refrigera-tor and delivers them to human users. Both the drink orderand the location at which it has to be delivered are specifiedby the user in a Web form. In Beetz et al. [13] a PR2 and acustom-built robot cooperatively prepare pancakes. In thehealth-care domain, Jain and Kemp [14] present EL-E, amobile manipulator that assists motor-impaired patients byperforming pick and place operations to retrieve objects. Sri-nivasa et al. [15] combined object search and retrieval in dif-ferent demonstrations in their lab. Their autonomous servicerobot HERB navigates around a kitchen, searches for mugs,and brings them back to the kitchen sink. Xue et al. [16] dem-onstrated grasping and handling of ice cream scoops with atwo-armed robot standing at a fixed position. The robotRollin’ Justin of the German Aerospace Center (DLR) pre-pared coffee in a pad machine [17]. The robot grasped coffeepads and inserted them into the coffee machine, whichinvolved opening and closing the pad drawer.

In the RoboCup 2011 competition, our team NimbRoparticipated with the robot Dynamaid and its successorCosero. In the tests, the robots showed their HRI and mobilemanipulation capabilities. We introduced many new devel-opments, such as grasp planning to extend the range ofgraspable objects, real-time scene segmentation and objecttracking, and human–robot cooperative carrying of a table.

System Overview

Robot DesignWe focused the design of our robots Dynamaid [18] andCosero [19] (see Figures 1 and 2) on typical requirementsfor autonomous operation in everyday environments.While Cosero still retains the lightweight design principlesof Dynamaid, we improved its construction andappearance significantly and made it more precise andstronger. Cosero’s mobile base has a small footprint of59 3 44 cm and drives omnidirectionally. This allows therobot to maneuver through the narrow passages found inhousehold environments. Its two anthropomorphic armsresemble average human body proportions and reachingcapabilities. A yaw joint in the torso enlarges the work-space of the arms. To compensate for the missing torsopitch joint and legs, a linear actuator in the trunk can movethe upper body vertically. This enables the robot to manip-ulate on similar heights like humans, even on the floor.

We constructed our robots from lightweight aluminumparts. All joints are driven by Robotis Dynamixel actuators.

(a) (b)

Figure 1. Cognitive service robot Cosero (a) grasps a spoon and (b)pours milk into a bowl of cereals at RoboCup GermanOpen 2011.

UN

IVE

RS

ITY

OF

BO

NN

36 • IEEE ROBOTICS & AUTOMATION MAGAZINE • JUNE 2012

These design choices allow for a lightweight and inexpen-sive construction, compared with other domestic servicerobots. While each arm of Cosero has a maximum payloadof 1.5 kg and the drive has a maximum speed of 0.6 m/s,the low weight (in total, approximately 32 kg) requires onlymoderate actuator power. The robot’s main computer is aquadcore notebook with an Intel i7-Q720 processor.

Both robots perceive their environment with a varietyof complementary sensors. The robots sense the environ-ment three-dimensionally with a Microsoft Kinect RGB-Dcamera using its pan–tilt head. For obstacle avoidance andtracking in farther ranges and larger field of views than theKinect, the robots are equipped with multiple laser-rangescanners, from which one can be rolled and one can bepitched. The sensor head of the robots also contains a shot-gun microphone for speech recognition. By placing themicrophone on the head, the robots point the microphonetoward human users and, at the same time, direct theirvisual attention to them.

Perception and Control FrameworkThe autonomous behavior of our robots is generated usinga modular control architecture. We employ the interpro-cess communication infrastructure and tools of the robotoperating system (ROS) [20].

We implement task execution, mobile manipulation,and motion control in hierarchical finite state machines.The task execution level is interwoven with HRI modal-ities. For example, we support the parsing of naturallanguage to understand and execute complex commands.

Tasks that involve mobile manipulation trigger andparameterize subprocesses on a second layer of finite statemachines. These processes configure the perception ofobjects and persons, and they execute motions of bodyparts of the robot. The motions themselves are controlledon the lowest layer of the hierarchy and can also adapt tosensory measurements.

Everyday Manipulation SkillsOne significant part of the competition in the RoboCup@Home league tests mobile manipulation capabilities. Therobots should be able to fetch objects from various locationsin the environment. To this end, they must navigatethrough the environment, perceive objects, and grasp them.

We implement navigation with state-of-the-art meth-ods. Cosero localizes and plans paths in a two-dimensional(2-D) occupancy grid map of the environment [21]–[23].For three-dimensional (3-D) collision avoidance, we inte-grate measurements from any 3-D sensing device, such asthe tilting laser in the robot’s chest. Because of limitedonboard computing power of our robots, we focused onefficient and lightweight implementations.

In mobile manipulation, the robot typically estimates itspose with reference to walls, objects, or persons. For example,when the robot grasps an object from a table, it first approachesthe table roughly within the reference frame of a static map.

Then, it adjusts its height and distance to the table. Finally, italigns itself to bring the object into the workspace of its arms.

Our robots grasp objects on horizontal surfaces, such asthe floor, tables, and shelves in a height range from the floor toapproximately 1 m. They carry the objects and hand them tohuman users. We also developed solutions to pour from con-tainers, place objects on horizontal surfaces, dispose objects incontainers, and receive objects from users. We implementedthese capabilities by parameterized motion primitives andaccount for collisions during grasping motions.

Compliance ControlFrom differential inverse kinematics, we derived a methodto limit the torque of the joints, depending on how muchthey contribute to the achievement of the motion in taskspace [24]. Our approach not only allows adjustment ofcompliance in the null space of the motion but also in theindividual dimensions of the task space. This is very usefulwhen only specific dimensions in task space shall be con-trolled in a compliant way.

We applied compliant control for the opening and clos-ing of doors that can be moved without the handling of anunlocking mechanism. Refrigerators or cabinets are com-monly equipped with magnetically locked doors that can bepulled open without special manipulation of the handle. SeeFigure 2 for an example. Several approaches exist to manip-ulate doors when no precise articulation model is known[25], [26]. Our approach does not require feedback fromforce or tactile sensors. Instead, the actuators are backdriv-able and measure the displacement due to external forces.

To open a door, our robot drives in front of it, detects thedoor handle with the torso laser, approaches the handle, andgrasps it. The drive moves backward while the grippermoves to a position to the side of the robot in which theopening angle of the door is sufficiently large to approachthe open fridge or cabinet. The gripper follows the motionof the door handle through compliance in the lateral and the

(a) (b)

Figure 2. Domestic service robot Dynamaid (a) opens and (b)closes the refrigerator during the RoboCup@Home Final 2010 inSingapore.

JUNE 2012 • IEEE ROBOTICS & AUTOMATION MAGAZINE • 37

yaw directions. The robot moves backward until the gripperreaches its target position. For closing a door, the robot hasto approach the open door leaf, grasp the handle, and moveforward while it holds the handle at its initial grasping poserelative to the robot. When the arm is pulled away from thispose by the constraining motion of the door leaf, the drivecorrects for the motion to keep the handle at its initial poserelative to the robot. The closing of the door can be detectedwhen the arm is pushed back toward the robot.

Real-Time Tabletop SegmentationIn household environments, objects are frequently locatedon planar surfaces such as tables. Therefore, we base ourobject-detection pipeline on fast planar segmentation ofthe depth images of the Kinect [19]. Figure 3 shows anexemplary result for a tabletop scene. Our approach proc-esses depth images with a resolution of 160 3 120 at framerates of approximately 20 Hz on the robot’s maincomputer. This enables our system to extract informationabout the objects in a scene with a very low latency fordecision-making and planning stages. For object identifi-cation, we utilize texture and color information [18].

Similar to Rusu et al. [27], we segment point clouds intoobjects on planar surfaces. To process the depth imagesefficiently, we combine rapid normal estimation with fastsegmentation techniques. The normal estimation methoduses integral images to estimate surface normals in a fixedimage neighborhood in constant time [28]. Overall, therun-time complexity is linear with the number of pixels forwhich normals are calculated. Since we search for horizon-tal support planes, we find all points with vertical normals.We segment these points into planes using random sampleconsensus algorithm (RANSAC) [29]. We find the objectsby clustering the measurements above the convex hull ofthe points in the support plane.

Efficient Grasp PlanningWe investigated grasp planning to enable our robots tograsp objects that they typically encounter in RoboCup. To

grasp objects flexibly from shelves and in complex scenes,we consider obstructions by obstacles [19].

Related approaches measure grasp quality, e.g., thegrasp wrench space [30], and virtually test grasps in physi-cal simulation [31] in a time-expensive process. Weobserved, however, that a well-designed gripper, simplegrasp strategies, and a compliant robot mechanism oftensuffice to grasp a large variety of household objects. Mostrelated to our method is the approach by Hsiao et al. [32].They use a time-consuming sampling-based motion plan-ner to find collision-free reaching motions. In many situa-tions, though, the direct reach toward the object is collisionfree, or only few obstacles obstruct the motion. We thusapply parameterized motion primitives and take a conserv-ative but efficient approach that checks simplified geomet-ric constraints to detect collisions.

In our approach, we assume that the object is rigidand symmetric along the planes spanned by the principal axesof the object, e.g., cylindrical or box shaped. We found that ourapproach also frequently yields stable grasps when an objectviolates these assumptions. Figure 4 illustrates the main stepsin our grasp planning pipeline and shows example grasps.

We consider two kinds of grasps: A side grasp thatapproaches the object horizontally and grasps the objectalong the vertical axis in a power grip. The complementarytop-grasp approaches the object from the top and grasps itwith the finger tips along horizontal orientations. Ourapproach extracts the object’s principle axes in the hori-zontal plane and its height. We sample pregrasp posturesfor top and side grasps, which we examine for feasibilityunder kinematic and collision constraints. In detail, weconsider the following feasibility criteria:l Grasp width: We reject grasps if the object’s width

orthogonal to the grasp direction does not fit intothe gripper.

l Object height: Side grasps are likely to fail if the objectheight is too small.

l Reachability: We do not consider grasps that are outsideof the arm’s workspace.

(a) (b) (c)

Figure 3. Tabletop segmentation. (a) Example setting. (b) Raw-colored point cloud from Kinect. (c) Each detected object is markedwith a distinct color.

UN

IVE

RS

ITY

OF

BO

NN

38 • IEEE ROBOTICS & AUTOMATION MAGAZINE • JUNE 2012

l Collisions: We check for collisions during the reachingand grasping motion.The remaining grasps are ranked according to efficiency

and robustness criteria:l Distance to object center: We favor grasps with a smaller

distance to the object center.l Grasp width: We reward grasp widths closer to a pre-

ferred width (0:08m).l Grasp orientation: Preference is given to grasps with a

smaller angle between the line toward the shoulder andthe grasping direction.

l Distance from robot: We prefer grasps with a smallerdistance to the shoulder.The best grasp is selected and finally executed with a

parameterized motion primitive.

Real-Time Object TrackingThe location of many household objects such as tables orchairs is subject to frequent changes. A robot must hencebe able to detect objects in its current sensor view and esti-mate the relative pose of the objects.

We developed methods for real-time tracking of objectswith RGB-D cameras [33]. We train full-view multiresolu-tion surfel maps of objects (see Figure 5) and track thesemodels in RGB-D images in real time. Our method oper-ates on 160 3 120 images at frame rates of approximately20 Hz on the robot’s onboard computer.

Our maps represent the normal distribution of points,including their color in voxels at multiple resolutions usingoctrees. Instead of comparing the image pixelwise to themap, we build multiresolution surfel maps with colorinformation from new RGB-D images.

We register these maps to the object map with an effi-cient multiresolution strategy. To this end, we measure theobservation likelihood of the current image under the nor-mal distributions of the surfels in both maps and deter-mine the most likely pose through optimization of thislikelihood. To cope with illumination changes, we ignoreminor luminance and color differences.

We associate surfels between maps using efficient near-est neighbor lookups in the octree. To determine the corre-spondences between surfels in both maps, we apply acoarse-to-fine strategy that selects the finest resolutionpossible. We establish a correspondence only if the surfelsmatch the color cues. Our association strategy not onlysaves redundant comparisons on coarse resolution but alsomatches surface elements at coarser scales if shape andcolor cannot be matched on finer resolutions. By this, ourmethod allows the object to be tracked from a wide rangeof distances.

Human–Robot InteractionA service robot, in everyday environments, not only needsmobile manipulation abilities but also closely interactswith humans, even physically. This interaction should benatural and intuitive such that laymen can operate therobot and understand its actions.

To be aware of potential interaction partners, ourrobots detect and keep track of the persons in their sur-roundings [34]. Users can utter complex sentences to therobots, which the robots recognize and parse for semantics.Our robots also synthesize humanlike speech. Further-more, we equipped our robots with nonverbal communi-cation cues. The robots can perform several gestures suchas pointing or waving. They can also perceive gestures suchas pointing, showing of objects, or stop gestures [35] withan RGB-D camera.

Semantic Speech InterpretationWe rely on the commercial Loquendo system for speechrecognition and synthesis. Loquendo’s speech recognitionis grammar based and speaker independent. Its grammardefinition allows rules to be tagged with semantic attrib-utes. For instance, one can define keywords for actions orattributes such as unspecific for location identifiers such asroom. When Loquendo recognized a sentence that fits tothe grammar, it provides a recognized set of rules togetherwith a semantic parse tree. Our task execution module

(b) (c)(a)

Figure 4. Grasp planning. (a) Object shape properties. The arrows mark the principal axes of the object. (b) We rank feasible, collision-freegrasps (red, size proportional to score) and select the most appropriate one (large, RGB coded). (c) Example grasps on box-shaped objects.

UN

IVE

RS

ITY

OF

BO

NN

JUNE 2012 • IEEE ROBOTICS & AUTOMATION MAGAZINE • 39

then interprets the resulting semantics and generatesappropriate behavior.

Human–Robot Cooperative ManipulationWe compiled mobile manipulation, object perception, andHRI capabilities in a cooperative manipulation task [33].In our scenario, the human and the robot cooperativelycarry a table. For successful performance of this task, therobot must keep track of the table and the actions of thehuman. To accurately approach the table, the robot tracksits pose in real time. The user can then lift and lower thetable, which the robot simply perceives through the motionof the table. The robot follows the pulling and pushing onthe table by the user through compliant control of its arms.

Experiences at RoboCup 2011With Dynamaid and Cosero, we competed in the RoboCup@Home 2011 competition in Istanbul. Our robots partici-pated and performed well in all tests of stages I and II. Weaccumulated the highest score of all 19 teams in bothstages. Our final demonstration was also awarded the bestscore. Hence, we achieved first place in the competition.

Competition PerformanceIn Stage I, Cosero and Dynamaid (Figure 6) registeredthemselves in the Robot Inspection and Poster Session test,while we presented our work in a poster session. The robotsgenerated speech and gestures and handed over the registra-tion form. The leaders of the other teams awarded us thehighest score in this test. In Follow Me, Cosero met a previ-ously unknown person and followed him reliably throughan unknown environment. Cosero could show that it distin-guishes this person from others and recognizes stop ges-tures. In the WhoIsWho test, two previously unknownpersons introduced themselves to Cosero. Later in the test,our robot found one of the previously unknown persons,two members of our team, and one unknown person andrecognized their identity correctly. In the Open Challenge,Cosero fetched a bottle of milk, opened it, and poured it intoa cereal bowl. Then Cosero grasped a spoon using ourapproach to grasp planning and placed it next to the bowl.Cosero understood a complex speech command partiallyand went to the correct place in the General Purpose ServiceRobot I test. In GoGetIt, Cosero found the correct object anddelivered it. After Stage I, we were leading the competition.

In the second stage, Cosero par-ticipated in Shopping Mall. It learneda map of a previously unknown areaand navigated to a shown location.Taking a shopping order was hin-dered by speech-recognition failuresin the unknown acoustic environ-ment. In the General Purpose ServiceRobot II test, Cosero first under-stood a partially specified commandand asked questions to obtain miss-ing information about an object andits location. It executed the task suc-cessfully. In the second part of thetest, it worked on a task with errone-ous information. It detected that theordered object was not at the speci-fied location, went back to the user,

(a) (b) (c)

Figure 5. Learning object models. (a) During training, the user selects points (red dots) to form a convex hull around the object. (b)Color and shape distribution modeled at 5 cm resolution. Lines indicate surface normals (color coded by orientation). (c) Color andshape distribution modeled at 2.5 cm resolution.

UN

IVE

RS

ITY

OF

BO

NN

(a) (b)

Figure 6. (a) Cosero and Dynamaid register themselves for the RoboCup@Home 2011competition. (b) Cosero opens a bottle of milk during the Open Challenge at RoboCup 2011.

UN

IVE

RS

ITY

OF

BO

NN

40 • IEEE ROBOTICS & AUTOMATION MAGAZINE • JUNE 2012

and reported the error. In the Demo Challenge, we demon-strated pointing gestures by showing the robot in whichbaskets to put colored and white laundry. The robot thencleaned the apartment, picked white laundry from thefloor, and placed them into the correct basket. It thenpicked carrots and tea boxes from a table. The objects couldbe chosen and placed by a jury member. The technical com-mittee awarded us the highest score. We reached the finalstage with 8,462 points, followed by Wright Eagle fromChina with 6,625 points.

In the final stage, we demonstrated the cooperative car-rying of a table by Cosero and a human user (see Figure 7).Afterwards, a user showed Cosero where it could find abottle of omelet mixture. Our robot went to the cookingplate and partially succeeded in turning the plate on. Then,it drove to the location of the mixture and grasped it. Atthe cooking plate, it opened the bottle and poured it intothe pan. We applied our real-time object tracking methodto approach the cooking plate. Meanwhile, Dynamaidopened a refrigerator, grasped a bottle of orange juice outof it, and placed it on the breakfast table. Our performancereceived the best score from the high-profile jury.

Lessons LearnedOur experiences at RoboCup 2011 clearly demonstrate oursuccess in designing a balanced system that incorporatesnavigation, mobile manipulation, and intuitive HRI. Thedevelopment of the system gave us many insights into therequirements and future steps toward complex domesticservice scenarios.

Since the competition setting is unknown in advance, wehave to develop methods that robustly work in a wide rangeof environments. We are also forced to implement means ofadapting our approaches to new scenarios easily and quickly.For example, it is important to develop tools that allow maps,objects, and persons to be enrolled quickly. Such robust andfast-adaptable methods will be enablers for practical use.

In the typical manipulation scenarios that we encounterin the competition, our efficient grasping strategy seems

more practical than traditional plan-ning approaches with respect to timeefficiency and robustness in the pres-ence of uncertainty. For complexmanipulation settings such as grasp-ing objects out of drawers and boxes,it will be necessary to develop efficientgrasp and motion planning techni-ques that reason out uncertainties.

We have demonstrated that com-plex high-level behavior can begenerated by semantic parsing ofnatural language and by a well-designed hierarchical state machine.It will be fruitful to push the com-plexity of the tasks with the versatil-ity in skills. Then, new requirements

will arise on reasoning capabilities for task execution andon semantic perception.

ConclusionsThe RoboCup@Home league is a competition for servicerobots in domestic environments. It benchmarks mobilemanipulation and HRI capabilities of integrated robotic sys-tems. In this article, we presented the contributions of ourwinning team NimbRo. We detailed our methods for real-time scene segmentation, object tracking, and human–robotcooperative manipulation. In the predefined tests, we coulddemonstrate that our robots Cosero and Dynamaid solvemobile manipulation and HRI tasks with high reliability. Ouradvanced mobile manipulation and HRI skills have been wellreceived by juries in the open demonstrations and the final.

In future work, we aim to further advance the versatilityof the skills of our robots. We constantly enhance ourapproaches to object and person perception. To extend themanipulation skills of our robots, we will improve thedesign of the grippers. We plan to construct thinner fingerswith touch sensors. Then, we can devise new methods tograsp smaller objects or to use tools.

AcknowledgmentsThis research has been partially funded by the FP7 ICT-2007.2.2 project ECHORD (grant agreement 231143)experiment ActReMa. We thank the members of teamNimbRo Kathrin Gr€ave, David Droeschel, Jochen Kl€aß,Michael Schreiber, and Ricarda Steffens for their dedicatedefforts before and during the competition.

References[1] (2012). DARPA grand challenge [Online]. Available: http://archive.

darpa.mil/grandchallenge

[2] (2012). IARC: International Aerial Robotics Competition [Online].

Available: http://iarc.angel-strike.com

[3] (2012). ELROB: The European Robot Trial [Online]. Available:

http://www.elrob.org

[4] (2012). The RoboCup Federation [Online]. Available: http://www.robocup.org

(a) (b)

Figure 7. Cosero cooperatively (a) carries a table with a user and (b) cooks an omeletduring the 2011 RoboCup@Home Final in Istanbul.

UN

IVE

RS

ITY

OF

BO

NN

JUNE 2012 • IEEE ROBOTICS & AUTOMATION MAGAZINE • 41

[5] H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, and E. Osawa,

“RoboCup: The robot world cup initiative,” in Proc. 1st Int. Conf. Autono-

mous Agents, New York, 1997, pp. 340–347.

[6] T. van der Zant and T. Wisspeintner, “RoboCup X: A proposal for a new

league where RoboCup goes real world,” in Proc. RoboCup 2005: Robot Soccer

World Cup IX (LNCS 4020). New York: Springer-Verlag, 2006, pp. 166–172.

[7] T. Wisspeintner, T. van der Zant, L. Iocchi, and S. Schiffer,

“RoboCup@Home: Scientific competition and benchmarking for domestic

service robots,” Interact. Stud., vol. 10, no. 3, pp. 393–428, 2009.

[8] D. Holz, A.-L. Jouen, M. Rajesh, J. Savage, K. Sugiura, L. Iocchi, J. R.

del Solar, and T. van der Zant. (2011). RoboCup@Home: Rules & regula-

tions. [Online]. Available: http://purl.org/holz/rulebook.pdf

[9] X. Chen, G. Jin, J. Ji, F. Wang, and J. Xie, “KeJia project: Towards inte-

grated intelligence for service robots,” in Proc. RoboCup Symp., Istanbul,

Turkey, 2011. [CD].

[10] F. Hegger, C. M€uller, Z. Jin, J. A. A. Ruiz, G. Giorgana, N. Hoch-

geschwender, M. Reckhaus, J. Paulus, P. Ploeger, and G. K. Kraetzschmar,

“The bit-bots RoboCup@Home 2011 team description paper,” in Proc.

RoboCup Symp., Istanbul, Turkey, 2011. [CD].

[11] A. Alempijevic, S. Carnian, D. Egan-Wyer, G. Dissanayake, R. Fitch,

B. Hengst, D. Hordern, N. Kirchner, M. Koob, M. Pagnucco, C. Sammut,

and A. Virgona, “RobotAssist—RoboCup@Home 2011 team description

paper,” in Proc. RoboCup Symp., Istanbul, Turkey, 2011. [CD].

[12] J. Bohren, R. Rusu, E. Jones, E. Marder-Eppstein, C. Pantofaru, M.

Wise, L. M€osenlechner, W. Meeussen, and S. Holzer, “Towards autono-

mous robotic butlers: Lessons learned with the PR2,” in Proc. IEEE Int. Conf.

Robotics and Automation (ICRA), Shanghai, China, 2011, pp. 5568–5575.

[13] M. Beetz, U. Klank, I. Kresse, A. Maldonado, L. M€osenlechner, D.

Pangercic, T. R€uhr, and M. Tenorth, “Robotic roommates making pan-

cakes,” in Proc. IEEE-RAS Int. Conf. Humanoid Robots (Humanoids),

Bled, Slovenia, 2011, pp. 529–536.

[14] A. Jain and C. C. Kemp, “EL-E: An assistive mobile manipulator that

autonomously fetches objects from flat surfaces,” Auton. Robots, vol. 28,

no. 1, pp. 45–64, 2010.

[15] S. Srinivasa, D. Ferguson, C. Helfrich, D. Berenson, A. Collet, R. Dia-

nkov, G. Gallagher, G. Hollinger, J. Kuffner, and J. M. Vandeweghe, “HERB:

A home exploring robotic butler,” Auton. Robots, vol. 28, no. 1, pp. 5–20, 2010.

[16] Z. Xue, S. Ruehl, A. Hermann, T. Kerscher, and R. Dillmann, “An

autonomous ice-cream serving robot,” in Proc. IEEE Int. Conf. Robotics

and Automation (ICRA), Shanghai, China, 2011, pp. 3451–3452.

[17] B. B€auml, F. Schmidt, T. Wimb€ock, O. Birbach, A. Dietrich, M.

Fuchs, W. Friedl, U. Frese, C. Borst, M. Grebenstein, O. Eiberger, and G.

Hirzinger, “Catching flying balls and preparing coffee: Humanoid

Rollin’ Justin performs dynamic and sensitive tasks,” in Proc. IEEE Int. Conf.

Robotics and Automation (ICRA), Shanghai, China, 2011, pp. 3443–3444.

[18] J. St€uckler and S. Behnke, “Integrating indoor mobility, object manipu-

lation, and intuitive interaction for domestic service tasks,” in Proc. IEEE-RAS

Int. Conf. Humanoid Robots (Humanoids), Paris, France, 2009, pp. 506–513.

[19] J. St€uckler, R. Steffens, D. Holz, and S. Behnke, “Real-time 3D per-

ception and efficient grasp planning for everyday manipulation tasks,” in

Proc. European Conf. Mobile Robots (ECMR), €Orebro, Sweden, 2011,

pp. 177–182.

[20] S. Cousins, B. Gerkey, K. Conley, and W. Garage, “Sharing software

with ROS,” IEEE Robotics Automat. Mag., vol. 17, no. 2, pp. 12–14, 2010.

[21] D. Fox, “Adapting the sample size in particle filters through KLD-

sampling,” Int. J. Robot. Res., vol. 22, no. 12, pp. 985–1003, 2003.

[22] P. Hart, N. Nilson, and B. Raphael, “A formal basis for the heuristic

determination of minimal cost paths,” IEEE Trans. Syst. Sci. Cybernetics,

vol. 4, no. 2, pp. 100–107, 1968.

[23] G. Grisetti, C. Stachniss, and W. Burgard, “Improved techniques for

grid mapping with Rao-Blackwellized particle filters,” IEEE Trans. Robot.,

vol. 23, no. 1, pp. 34–46, 2007.

[24] J. St€uckler and S. Behnke, “Compliant task-space control with back-driv-

able servo actuators,” in Proc. RoboCup Symp., Istanbul, Turkey, 2011. [CD].

[25] G. Niemeyer and J.-J. E. Slotine, “A simple strategy for opening an

unknown door,” in Proc. IEEE Int. Conf. Robotics and Automation

(ICRA), Albuquerque, NM, 1997, pp. 1448–1453.

[26] A. Jain and C. C. Kemp, “Pulling open doors and drawers: Coordi-

nating an omni-directional base and a compliant arm with equilibrium

point control,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA),

Anchorage, AK, 2010, pp. 1807–1814.

[27] R. B. Rusu, N. Blodow, Z. C. Marton, and M. Beetz, “Close-range

scene segmentation and reconstruction of 3D point cloud maps for mobile

manipulation in human environments,” in Proc. IEEE/RSJ Int. Conf. Intelli-

gent Robots and Systems (IROS), St. Louis, MO, 2009, pp. 1–6.

[28] D. Holz, S. Holzer, R. B. Rusu, and S. Behnke, “Real-time plane seg-

mentation using RGB-D cameras,” in Proc. RoboCup Symp., Istanbul,

Turkey, July 2011. [CD].

[29] M. A. Fischler and R. C. Bolles, “Random sample consensus: A para-

digm for model fitting with applications to image analysis and automated

cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395, 1981.

[30] C. Borst, M. Fischer, and G. Hirzinger, “Efficient and precise grasp

planning for real world objects,” in Multi-Point Interaction with Real and

Virtual Objects (Springer Tracts in Advanced Robotics, vol. 18). New

York: Springer-Verlag, 2005, pp. 91–111.

[31] A. Miller and P. Allen, “GraspIt! A versatile simulator for robotic

grasping,” IEEE Robot. Automat. Mag., vol. 11, no. 4, pp. 110–122, 2004.

[32] K. Hsiao, S. Chitta, M. Ciocarlie, and E. G. Jones, “Contact-reactive grasp-

ing of objects with partial shape information,” in Proc. IEEE/RSJ Int. Conf.

Intelligent Robots and Systems (IROS), Taipei, Taiwan, 2010, pp. 1228–1235.

[33] J. St€uckler and S. Behnke, “Following human guidance to coopera-

tively carry a large object,” in Proc. IEEE-RAS Int. Conf. Humanoid Robots

(Humanoids), Bled, Slovenia, 2011, pp. 218–223.

[34] J. St€uckler and S. Behnke, “Improving people awareness of service robots

by semantic scene knowledge,” in Proc. RoboCup 2010: Robot Soccer World Cup

XIV (Springer Lecture Notes in Computer Science, vol. 6556), pp. 157–168.

[35] D. Droeschel, J. St€uckler, D. Holz, and S. Behnke, “Towards joint

attention for a domestic service robot—Person awareness and gesture

recognition using time-of-flight cameras,” in Proc. IEEE Int. Conf.

Robotics and Automation (ICRA), Shanghai, China, 2011, pp. 1205–1210.

J€org St€uckler, Autonomous Intelligent Systems Group,Computer Science Institute VI, University of Bonn, 53113Bonn, Germany. E-mail: stueckler@ais.uni-bonn.de.

Dirk Holz, Autonomous Intelligent Systems Group,Computer Science Institute VI, University of Bonn, 53113Bonn, Germany. E-mail: dirk.holz@ieee.org.

Sven Behnke, Autonomous Intelligent Systems Group,Computer Science Institute VI, University of Bonn, 53113Bonn, Germany. E-mail: behnke@cs.uni-bonn.de.

42 • IEEE ROBOTICS & AUTOMATION MAGAZINE • JUNE 2012