Dynamic User Task Scheduling for Mobile Robots

Brian Coltin and Manuela VelosoSchool of Computer ScienceCarnegie Mellon University

5000 Forbes AvenuePittsburgh, PA 15213

{bcoltin, veloso}@cs.cmu.edu

Rodrigo VenturaVisiting Scholar

Institute for Systems and RoboticsInstituto Superior Tecnico

Lisboa, Portugalyoda@isr.ist.utl.pt

Abstract

We present our efforts to deploy mobile robots in of-fice environments, focusing in particular on the chal-lenge of planning a schedule for a robot to accomplishuser-requested actions. We concretely aim to make ourCoBot mobile robots available to execute navigational-based tasks requested by users, such as telepresence,and picking up and delivering messages or objectsat different locations. We contribute an efficient web-based approach in which users can request and schedulethe execution of specific tasks. The scheduling problemis converted to a mixed integer programming problem.The robot executes the scheduled tasks using a syntheticspeech and touch-screen interface to interact with users,while allowing users to follow the task execution online.Our robot uses a robust Kinect-based safe navigation al-gorithm, moves fully autonomously without the need tobe chaperoned by anyone, and is robust to the presenceof moving humans, as well as non-trivial obstacles, suchas legged chairs and tables. Our robots have already per-formed 15km of autonomous service tasks.

Introduction and Related WorkWe envision a system in which autonomous mobile robotsrobustly perform service tasks in indoor environments. Therobots perform tasks which are requested by building resi-dents over the web, such as delivering mail, fetching coffee,or guiding visitors. To fulfill the users’ requests, we mustplan a schedule of when the robot will execute each task inaccordance with the constraints specified by the users.

Many efforts have used the web to access robots, includ-ing the early examples of the teleoperation of a roboticarm (Goldberg et al. 1995; Taylor and Trevelyan 1995)and interfacing with a mobile robot (e.g, (Simmons et al.1997; Siegwart and Saucy 1999; Saucy and Mondada 2000;Schulz et al. 2000)), among others. The robot Xavier (Sim-mons et al. 1997; 2000) allowed users to make requests overthe web for the robot to go to specific places, and othermobile robots soon followed (Siegwart and Saucy 1999;Grange, Fong, and Baur 2000; Saucy and Mondada 2000;Schulz et al. 2000). The RoboCup@Home initiative (Visserand Burkhard 2007) provides competition setups for indoor

Copyright © 2011, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Figure 1: CoBot-2, an omnidirectional mobile robot for in-door users.

service autonomous robots, with an increasingly wide scopeof challenges focusing on robot autonomy and verbal inter-action with users.

In this work, we present our architecture to effectivelymake a fully autonomous indoor service robot available togeneral users. We focus on the problem of planning a sched-ule for the robot, and present a mixed integer linear program-ming solution for planning a schedule. We ground our workon the CoBot-2 platform 1, shown in Figure 1. CoBot-2 au-tonomously localizes and navigates in a multi-floor officeenvironment while effectively avoiding obstacles (Biswasand Veloso 2010). The robot carries a variety of sensingand computing devices, including a camera, a Kinect depth-camera, a Hokuyo LIDAR, a touch-screen tablet, micro-phones, speakers, and wireless communication.

CoBot-2 executes tasks sent by users over the web, and wehave devised a user-friendly web interface that allows usersto specify tasks. Currently, the robot executes three typesof tasks: a GoToRoom task where the robot visits a loca-tion, a Telepresence task where the robot goes to a location

1CoBot-2 was designed and built by Michael Licitra, mlici-tra@cmu.edu, as a scaled-up version of the CMDragons small-sizesoccer robots, also designed and built by him.

Server side Robot side

interfaceagent

executingmanager

agent

web interface

schedulingagent

KBase:• scheduled tasks• robot states

robots state

feasible?bookingrequest robots state scheduled

tasks

new tasks

scheduledtasks

execute task

robot state

tasks cancelation

onbo

ard

user

inte

rfacerobot manager

agent

navigation

telepresence

behavior interaction planner

Figure 2: Users to Mobile Robots (UMR) architecture.

and allows someone to control it remotely, and a Transporttask where the robot picks up an object at one location andbrings it to another. When placing a reservation for a task,the user selects when the task should be completed. Tasksare scheduled taking into account feasibility constraints in-volving other previously scheduled tasks as well as the es-timates of both the navigation time and of the interactiontime. If the task cannot be completed, the user may be askedto loosen the constraints or is told that the task cannot becompleted.

Once tasks are scheduled, a behavior plan is executed onthe robot to interact with the users at the task locations andto incidentally interact with other humans in the environ-ment (Fong, Nourbakhsh, and Dautenhahn 2003). The planmay need to be adjusted when tasks do not complete in theexpected time.

In the remainder of this paper, we will present our com-plete users to mobile robots architecture. We will then dis-cuss task scheduling and the behavior planner. We presentthe web-based interface for users to interact with the robot,and show illustrative examples of the effectiveness of thesystem.

Users to Mobile Robots ArchitectureDeploying a mobile robot to general users poses severalchallenges: (1) the task-request interface must be easily ac-cessible and user-friendly, (2) the scheduling algorithm mustplan feasible schedules, (3) navigation must be safe and re-liable in office environments, and (4) the robot must interactintuitively with humans.

To address these challenges, we contribute a Users to Mo-bile Robots (UMR) architecture (see Figure 2), with differ-ent modules to connect the user requests to the executorrobot, namely:

• interface agent — manages the interaction with users us-ing the web interface, serving two main purposes: manag-ing bookings, including placing new ones and cancelingpreviously booked tasks, and showing the robot’s locationin real-time;

• scheduling agent — receives booking requests from theinterface agent, checks whether they are feasible with re-spect to previously scheduled tasks, stores them if feasi-ble, or proposes an alternative otherwise;

• KBase — stores all scheduled tasks, including the onesalready executed, the one being executed, and the onesscheduled in the future, as well as the robot’s state (suchas location and battery level);

• executing manager agent — handles the communicationwith the robots, sending them tasks to be executed, andreceiving their current state;

• robot manager agent — handles the communication ofthe robot with the server;

• behavior interaction planner — executes tasks bybreaking them down into sub-tasks (e.g., the Transporttasks comprises a sequence of interleaved navigation andinteraction sub-tasks);

• navigation — handles the navigation of the robot to alocation given by the behavior interaction planner, usingthe robot sensors for localization, navigation, and obstacleavoidance (Biswas and Veloso 2011);

• onboard user interface — manages human-robot inter-action, using synthetic speech and touch-screen display.

This architecture interacts with users in two distinct ways:through the web interface, for managing bookings and fol-lowing the robot’s state, and directly with the robot throughits onboard user interface.

The web-based booking interface addresses challenge (1),to the extent that web-based booking systems are awidespread and familiar scheme for reserving services, be-ing found in numerous services such as in hotel reservations,car rental, and more recently in ZipCar™. Challenge (2) isaddressed by the scheduling agent presented in section . Thisagent plans a feasible schedule of when the robot can com-plete all of the tasks, or informs the interface agent that itcannot. A robust navigation method based on the Kinectdepth camera and capable of effectively navigating in anoffice environment, while avoiding moving and fixed ob-stacles addresses challenge (3) (Biswas and Veloso 2011).The robot’s face-to-face interaction with its users is basedon a joint synthetic-voice and touch-screen onboard user in-terface. Messages are both spoken by a synthetic voice anddisplayed on the touch-screen, while the user can respond tothese messages using buttons displayed on the touch-screen.This interface is simple and easy to use, thus addressingchallenge (4).

(a) book a robot (b) confirmation (c) view bookings

Figure 3: Screenshots of the web interface, showing (a) the web interface to perform a booking, (b) the confirmation screencontaining the start and (estimated) end times, and (c) the list of current and past bookings performed by the user.

To illustrate the functionality of the UMR architecture, wepresent next a running example of the booking and executionof a task requested by a user:

1. At 6:35PM, Alice uses a web browser to request a robot totransport a bottle of water, from room 7705 to room 7005,as soon as possible (Figure 3a);

2. The web interface passes this new request to the schedul-ing agent, which plans the robot’s arrival at the pick uplocation (7705) at 6:38PM (Figure 3b);

3. After confirmation, CoBot-2 starts executing this task im-mediately: it navigates to room 7705, while displayingand speaking the message “Going to 7705 to pick up abottle of water and bring it to 7005” on the onboard userinterface;

4. Upon arrival to 7705, CoBot-2 displays and speaks themessage “Please place a bottle of water on me to deliver”,and awaits someone to click the ‘Done’ button displayedon the touch-screen;

5. Once this button is pressed, the robot starts navigating toroom 7005;

6. upon arrival to 7005, CoBot-2 displays and speaks themessage “Please press ‘Done’ to release me from mytask”, and awaits the user to press the ‘Done’ button;

7. Once this button is pressed, the task is marked as success-fully executed, and the robot navigates back to its homelocation.

After the task has been booked, Alice can check the bookingon the web (Figure 3c) and cancel it if necessary. During thetask execution, Alice could follow the progress of the robotnavigation, either on the map view (Figure 5a) or throughthe camera view (Figure 5b).

Scheduling AgentOnce users have placed requests for the robot over the web, aschedule of tasks must be formed with times for the robot tocomplete them. Tasks that the robots could perform include

coming to a specific location for a demonstration, guidinga visitor from one room in the building to another, fetchingcoffee or printouts, delivering mail, or delivering a spokenmessage. When the user makes a request, a time is specifiedfor when the task is to be carried out. This may either be“as soon as possible”, an exact time, or a window of timewith a beginning and an ending time. Internally, all theserepresentations are converted to windows of time. The servermay choose to either accept, reject, or ask to refine a request.

We first look at the batch problem, where the schedulingagent is given a list of n task requests, T , that the robotsmust fulfill. The goal is to find a starting time, ti, for eachtask such that the tasks do not overlap and the requests arefulfilled as soon as possible. Each time ti must fall withina window of times [si, ei], and the task has a duration di.Furthermore, each task has a starting and ending locationlsi , l

ei ∈ L (they will be different for the Transport task), and

there is a function dist : L × L → R giving an estimate ofthe time to travel between two locations.

We solve this scheduling problem using mixed integerprogramming. We solve for the variables ti, and also intro-duce some helper indicator variables prei,j which indicatewhether task i precedes task j. Our first set of constraints isthat each time ti must fall within the start and end times ofthe window.

∀i si ≤ ti ≤ ei

Then, we constrain the times so that they do not overlap.First, we define the prei,j to be indicator variables.

∀i, j 0 ≤ prei,j ≤ 1 intThen, we add constraints which ensure that the executiontimes of two tasks do not overlap, handling both the case thattask i comes before task j or the reverse order depending onthe value of prei,j .

∀i, j ti + di + dist(lei , lsj)− tj ≤ |ej − si|(1− prei,j)

∀i, j tj + dj + dist(lej , lsi )− ti ≤ |ei − sj |prei,j

The objective that we choose to minimize is the total dif-ference in time from the start of the scheduling window forall the tasks. This objective ensures that user tasks are com-pleted as soon as possible.

min∑i

(ti − si) = min∑i

ti

Alternatively, we could minimize the total distance travelledby the robot to prolong battery life, or we could optimizea constant to more quickly find any feasible schedule. Sowe have

(n2

)+ n variables, and a proportional number of

constraints.However, we can reduce the problem further. Some pairs

i and j may never even have a possibility of overlap-ping depending on their time windows. If either ei + di +dist(li, lj) ≤ sj or ej + dj + dist(lj , li) ≤ si, then there isno possibility of the two tasks conflicting and we can elimi-nate the constraint entirely. In practice, we expect that manyconstraints will not overlap at all.

Although solving the MIP is an NP-hard problem, wehave found that in practice it can be solved quickly for cer-tain problems. We generated a thousand random problem in-stances, each of 15 tasks with two minutes to half an hourdurations, with time windows over the course of four hours.This is the type of input we expect the robot to receive. Thescheduler solved (either found a schedule or found that noschedule existed) 99% of the problems in under two seconds.In the cases where a schedule is not found quickly, ratherthan waiting, we can declare that there is no valid schedule.

In practice, the task requests are not processed in a batch,but come in an online fashion over the web. We rescheduleeverything whenever a new request is made. If a schedulecannot be found, the user’s request is rejected and the userhas an opportunity to relax the constraints.

Since the tasks are running on actual robots, the actualtime to complete a task will often differ from the expectedtime. In these cases, a new schedule is built, and some of thetasks may need to be abandoned.

After a task is scheduled, the executing manager agentsends the robot-specific scheduled task set to the corre-sponding robot manager agent to execute. The robot’s Be-havior Interaction Planner plans the sequence of action tocomplete each task.

Behavior Interaction PlannerTypically, task planners only plan the autonomous actions tocomplete a task and a separate dialog manager interacts withhumans to receive the task requests. However, a robot cannotalways perform its actions autonomously and relies on hu-mans in the environment to help it complete tasks. Addition-ally, as a robot performs actions, humans in the environmentmay want to know what the robot’s goals are. Our BehaviorInteraction Planner therefore reasons about a robot’s inca-pabilities (Rosenthal, Biswas, and Veloso 2010) and humaninterest in the robot and plans for human interactions in ad-dition to the autonomous actions. As the robot executes theplan, it reports a descriptive message back to the server foronline users to follow the robot’s progress in the web inter-face.

Actions and InteractionsWe define actions and interactions that are required to com-plete a task along with their preconditions and effects. Forask interactions, for example, there are no preconditions,the robot speaks the defined text, and the effect is the re-quired human response (e.g. clicking a ‘Done’ button onCoBot’s user interface). For navigate actions, the pre-condition is that the robot speak aloud its new goal to hu-mans in the area, the robot then sends a desired location tothe navigation module, and the effect is that the robot is inthe location that it should navigate to. The separate naviga-tion module controls the low level motor control and obsta-cle avoidance for navigation. Any other actions needed for atask can be defined similarly.

Autonomous Planning and ExecutionGiven a new task, the robot plans the sequence ofactions necessary to complete it. For example, in theTransport(s, lp, ld,m) task, the Behavior Interaction Plan-ner plans the following sequence of actions (illustrated inFigure 4): navigate to location lp, ask for the object m,navigate to ld, and ask for task completion confirma-tion.

The Behavior Interaction Planner can also plan for arobot’s incapabilities. For example, if CoBot (with no arms)must navigate between different floors of the building, thisrequires not only navigate actions, but also human inter-action to ask for help with pressing buttons and recognizingwhich floor the robot is on. In these cases, the Behavior In-teraction Planner plans:

• navigate to elevator,

• ask for help pressing the up/down button,

• navigate into the elevators,

• ask for help pressing the floor number and recognizingthat floor,

• navigate out of the elevator,

• navigate to goal

Upon arriving at goal locations, the robot may also need helppicking up objects and plans for these additional ask inter-actions accordingly.

Web-based User InteractionThe interface agent and the scheduling agent are imple-mented as a web application, running the web-based userinterface, and connecting to the central scheduling databaseand the robot. The web interface is used for scheduling newtasks, examining and modifying existing tasks, and monitor-ing or controlling the robot through telepresence.

Web Booking InterfaceAfter successful authentication with a username and pass-word, each user is presented with a web interface, composedof four sections.

• Book a Robot. A user specifies the task to book togetherwith the desired time (Figure 3a). The booking interface

(a) (b) (c) (d) (e) (f) (g) (h) (i)

Figure 4: (a,b,c) After CoBot-2 receives a Transport task request, it autonomously navigates to the location lp to pick up a bottleof water. (d,e) Upon arriving to lp, CoBot-2 asks the user to place the bottle of water and press ‘Done’. (f,g) Then, CoBot-2navigates to location ld to deliver the bottle of water. (h,i) When the user presses ‘Done’ CoBot-2 navigates back to its homelocation. (The complete example sequence is submitted as a video with this paper.)

supports two ways of setting the start time of a task: userscan request the task to be executed as soon as possible, forwhich the soonest feasible time slot is provided, or at aspecific time, in 10 minutes intervals. The actual bookingis preceded by a confirmation screen (Figure 3b).

• View Bookings. This section displaying both the currentand the past bookings (Figure 3c). Current bookings canbe canceled at any time. If the robot is already execut-ing a task, it is immediately aborted. The past bookingslist shows both the scheduled and the actual start and endtimes for each task, as well as whether its execution wassuccessful or an unexpected event canceled it (e.g., lowbattery).

• Monitor Robots. Users can follow the location of therobot in real-time on a map. This interface also displaysthe task currently being executed by the robot, as well asits battery level.

• Telepresence. Users can view the state of the robot andoperate it remotely if they have scheduled a telepresencetask.

The web booking interface enables users to intuitively in-teract with the robot.

TelepresenceIn addition to performing tasks fully autonomously, one ofthe tasks users may request a robot for is to control CoBot-2in a semi-autonomous “telepresence” mode in the Telepres-ence task. In telepresence mode, live sensory informationand camera images are streamed and displayed directly inthe user’s web browser on a page linked to from the webscheduler.

The telepresence interface, shown in Figure 5, displaysthe camera image, the text-to-speech interface, and the con-trols for both the robot navigation and camera pan-tilt-zoomsettings. The telepresence interface provides three controlmodalities with increasing levels of autonomy. In all modal-ities, the robot autonomously avoids obstacles. In additionto controlling the robot with the interface buttons, users mayclick directly on the image to point the camera or to navigatethe robot to the point clicked. The interface map displays therobot’s current location and orientation, and highlights de-tected obstacles to help the user to navigate safely. The usermay click on the map to send the robot autonomously to a

(a) map view (b) telepresence view

Figure 5: Screenshots of the telepresence interface, show-ing (a) the map view of CoBot-2 location with its navigationpath, and (b) the telepresence interface, showing the robot’scamera view, together with camera and robot motion con-trols.

location. We have found that users utilize all of these controlmodalities depending on the situation.

Illustrative ResultsIn order to illustrate the effectiveness of our system we con-ducted a set of experiments comprising 41 tasks requestedon the web interface (21 Transport, 20 Go-To-Room), wascarried out using CoBot-2. The locations used were all ex-isting 88 offices at the 7th floor of the Gate-Hillman Centerin Carnegie Mellon University. The execution of these tasksresulted in 138 minutes of autonomous navigation, along atotal distance of 4.9 km. The location(s) used in each taskwas randomly chosen from all of the 88 offices. Of the 41tasks, 37 were successfully completed (> 90%), three weremanually aborted due to blocked paths, and one was auto-matically aborted due to a low battery condition.

Figure 6 shows all the trajectories traveled by CoBot-2 during these experiments, spanning almost all navigablespace on the floor. The time taken by each navigation sub-task as a function of traveled distance is plotted in Figure 7.The relation between distance traveled and time is roughlylinear, except for a few outlier points. Some of these out-liers correspond to cases when the task was aborted due toa blocked path, while others can be explained by the robottrying to navigate around people blocking its way, possibly

Figure 6: Union of all trajectories traveled by CoBot-2 onthe 7th floor of the Gates-Hillman Center (CMU).

0

50

100

150

200

250

300

0 20 40 60 80 100 120 140

navi

gatio

n tim

e (s

)

navigation length (m)

Figure 7: Time taken by each navigation sub-task.

driven by curiosity. This shows the robot safely navigatesaround walls, chairs, and people throughout the entire officeenvironment.

ConclusionThis paper presented the Users to Mobile Robots (UMR) ar-chitecture, addressing the goal to deploy mobile robots togeneral users. This architecture allows users to request tasksand monitor their execution, using an easy to use web inter-face. Our scheduling agent finds a feasible schedule, takinginto account time constraints imposed by the mobile natureof robots. The Behavioral Interaction Planner autonomouslyplans and executes the scheduled tasks. We demonstratedthat the navigation method is robust; CoBot-2 has traveledan accumulated distance of 15 kilometers without beingchaperoned. The onboard user interface is intuitive, as anyperson was able to appropriately respond to it.

ReferencesBehnke, S.; Stuckler, J.; and Schreiber, M. 2009. Nimbro@home 2009 team description paper. In RoboCup 2009.Biswas, J., and Veloso, M. 2010. WiFi localization and nav-igation for autonomous indoor mobile robots. In IEEE In-ternational Conference onRobotics and Automation (ICRA),4379–4384.Biswas, J., and Veloso, M. 2011. Depth camera based indoormobile robot autonomy. In submission.

Breuer, T.; Giorgana, G.; Hegger, F.; Muller, C.; Jin, Z.;Reckhaus, M.; Paulus, J.; Hochgeschwender, N.; Awaad, I.;Hartanto, R.; Ploeger, P.; and Kraetzschmar, G. 2010. Theb-it-bots robocup@home 2010 team description paper. InRoboCup 2010.Fong, T.; Nourbakhsh, I.; and Dautenhahn, K. 2003. A sur-vey of socially interactive robots. Robotics and AutonomousSystems 42:143–166.Goldberg, K.; Mascha, M.; Gentner, S.; Rothenberg, N.; Sut-ter, C.; and Wiegley, J. 1995. Desktop teleoperation via theworld wide web. In Proceedings of the IEEE InternationalConference on Robotics and Automation, volume 1, 654–659.Grange, S.; Fong, T.; and Baur, C. 2000. Effective vehi-cle teleoperation on the world wide web. In Proceedingsof the IEEE International Conference on Robotics and Au-tomation, volume 2, 2007–2012.Rosenthal, S.; Biswas, J.; and Veloso, M. 2010. An effec-tive personal mobile robot agent through symbiotic human-robot interaction. In Proceedings of the 9th InternationalConference on Autonomous Agents and Multiagent Systems,volume 1, 915–922.Saucy, P., and Mondada, F. 2000. KhepOnTheWeb: openaccess to a mobile robot on the internet. IEEE Robotics Au-tomation Magazine 7(1):41–47.Schulz, D.; Burgard, W.; Fox, D.; Thrun, S.; and Cremers,A. 2000. Web interfaces for mobile robots in public places.IEEE Robotics Automation Magazine 7(1):48–56.Siegwart, R., and Saucy, P. 1999. Interacting mobile robotson the web. In Proceedings of the IEEE International Con-ference on Robotics and Automation.Simmons, R.; Goodwin, R.; Haigh, K. Z.; Koenig, S.; andO’Sullivan, J. 1997. A layered architecture for office deliv-ery robots. In Proceedings of the first international confer-ence on Autonomous agents (AGENTS’97), 245–252.Simmons, R.; Fernandez, J.; Goodwin, R.; Koenig, S.; andO’Sullivan, J. 2000. Lessons learned from xavier. IEEERobotics Automation Magazine 7(2):33–39.Taylor, K., and Trevelyan, J. 1995. Australia’s telerobot onthe web. In Proceedings of the International Symposium onIndustrial Robots, volume 26, 39–44.Visser, U., and Burkhard, H.-D. 2007. RoboCup: 10years of achievements and future challenges. AI Magazine28(2):115–132.