Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee

Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky

Brett Browning, M. Bernardine Dias

Students: Thomas Harris, Brenna Argall, Gil Jones

Satanjeev Banerjee

Dynamic Human-Robot Teams Engaged in Complex Adversarial Tasks Using Language-

Based Communication

Carnegie Mellon School of Computer ScienceBoeing Human-Robot Teams Project

2Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project

Project Goals

Treasure hunt with two or more teams of humans and robots competing to locate target objects as they explore an unknown space

Research Goals/Questions: Specify team member’s roles and capabilities to

perform tasks Rapidly form ad-hoc heterogeneous teams of robots

and humans Robots and humans executing synchronized action

as a team, while communicating via speech Improve team performance with experience


Alignment with Boeing’s Objectives

Our project develops component technologies relevant to:

FCS – Force multiplication for human-robot teams NASA Code T – Emphasis on robots assisting and augmenting

humans in complex tasks Space activities (Other than Code T)

Automating launch sites Automating escape systems Automating maintenance inspections/repair Ice inspections

Aircraft maintenance Automating maintenance inspections/repair Ice inspections

Automated baggage handlers

In collaboration with Phillip Koons, Boeing


STP: Overview

“Skills” for low-level control, “Tactics” for single robot, “Plays” for teamwork

APPLICABLE offenseDONE aborted !offense

ROLE 1 pass 3 mark best_opponentROLE 2 blockROLE 3 pos_for_pass R B 1000 0 receive_pass shoot AROLE 4 defend_lane


STP: Implementation

Play selection from playbook Dynamic role assignment Active tactic determines

transition Execution monitoring Reward, and adaptation

Play 1Play 2

Play 3Role 1 Search Retrieve

Selection

ExecutionMonitoring,Adaptation

Robot 1TacticRobot 1

TacticRobot NTactic


Relevant Research Interests

1. Methods for pickup teams

2. Extend play-based coordination to distributed execution

3. Effective human-robot teamwork with pickup teams


TraderBots: Overview

Robots are organized as an economy

Team mission is to maximize production and minimize costs

Robots exchange virtual money for tasks to maximize individual profit

System is designed to align local and global profit maximization

$

$

$$

$


TraderBots: Implementation

Operator

OpTrader

Robot 1

Robot 2

Robot 3

X X

X

XTask 1 Task 2 Task 3

Task 4

Bids

Bids

BidsRoboTrader

Auction

Announce and clear auction

OpTraderGreedy Auction

Announce and clear auction


Relevant Research Interests

1. Human-multi-robot interaction

2. Role assignment for highly heterogeneous teams

3. Improving robustness and adaptivity


Speech: Overview

Communication in mobile environments Natural spoken language-

based interaction Supporting high semantic

content Communication in multi-

participant domains Human(s) and robots(s)

interacting on the same channel


Speech: Implementation

Sphinx ASR engine Speaker-independent recognition Speaker-adaptive capability

Phoenix semantic parser Concept extraction

RavenClaw dialogue engine Task-based representations Full mixed-initiative dialogue

Festival/Theta synthesis engine Rosetta generation engine

Portable/Wearable platforms

ROBOT (A)RavenClaw

ROBOT (B)RavenClaw

Comms Channel

. . .

Sphinx

Phoenix

Theta

Rosetta


Relevant Research Interests1. Developing conversational

algorithms for human-robot polylogue

2. Communicating navigational information between humans and robots in unstructured environments

3. Cooperative grounding of objects, locations, and tasks in novel environments


Current Status Wireless network-based (UDP) communication

protocol for human-robot interaction designed and implemented

Closed loop integration between speech-based command system, a Segway robot, and a Pioneer robot accomplished

2-D kinematic simulation tool for testing interface to Pioneer robots

Video demonstration of speech-based command of a Segway robot and a Pioneer robot

Plan for “year 1 scenario” and relevant technology development and integration to accomplish this scenario

14

Treasure Hunt Scenario

First Year Objectives


Basic Scenario 2 Segways, 2 Pioneers, and 1 human Discover and return to base as much treasure as

possible within a 20 minute period Treasure will be identified by landmarks currently used

by the Segway soccer team

Mapper (M) Seeker (S) Handler (H) Deliverer (D) Leader (L)

Humans X X X

Segways X X X

Pioneers X X X

Gators X X X

AIBOs X


Detailed Scenario Goal

To combine TraderBots for negotiation and role assignment with plays for synchronization

Low-level software remains independent Challenges

How do we decide which Play to adopt? How do we handle sub-teams? How do we generate “leaders”?


Step 1: Form Sub-Teams

Lets form a sub-team

Fixed sub-teams for May 2005


Step 2: Command to Search

Team A, search area 1

1. Build map


Step 3: Explore and Search

1. Follow 1. Go to search area 2. Build map


Step 4: Search

1. Follow and search for treasure

1. Execute search pattern2. Build map


Step 5: Found!

We found it!

We are at <x,y>


Step 6: Recovery

Human aided loading


Step 7: Return

1. Follow2. Unload

1. Return home2. Unload

Human aided unloading


Post Year 1 Goals Failure detection and recovery Outdoor environments Adversaries Larger more diverse teams Adaptation and learning

Documents

Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee