Potential Scaling Effects for Asynchronous Video in Multirobot Search

Potential Scaling Potential Scaling Effects for Effects for

Asynchronous Video Asynchronous Video in Multirobot Searchin Multirobot Search

Prasanna VelagapudiPrasanna Velagapudi11, Huadong Wang, Huadong Wang22, Paul , Paul ScerriScerri11, Michael Lewis, Michael Lewis22 and Katia Sycara and Katia Sycara11

11Carnegie Mellon University, USACarnegie Mellon University, USA22University of Pittsburgh, USAUniversity of Pittsburgh, USA

Urban Search and Rescue Urban Search and Rescue (USAR)(USAR)

► Location and rescue Location and rescue of people in a of people in a structural collapse structural collapse

► Urban disastersUrban disasters LandslidesLandslides EarthquakesEarthquakes TerrorismTerrorism

Credit: NIST

USAR RobotsUSAR Robots

► Robots can helpRobots can help Unstable voidsUnstable voids Mapping/clearingMapping/clearing

► Want them to be: Want them to be: SmallSmall CheapCheap PlentifulPlentiful

Credit: NIST

Urban Search and Rescue Urban Search and Rescue (USAR)(USAR)

►Now: One operator Now: One operator one robot one robot Directly teleoperatedDirectly teleoperated Victim detection through synchronous videoVictim detection through synchronous video

►Future: One operator Future: One operator many robots many robots Manufacturing robots is easyManufacturing robots is easy Training operators is hardTraining operators is hard

►Need to scale navigation and Need to scale navigation and searchsearch

Synchronous VideoSynchronous Video

►Most common form of camera Most common form of camera teleoperationteleoperation High bandwidthHigh bandwidth Low latencyLow latency

►ApplicationsApplications SurveillanceSurveillance Bomb disposalBomb disposal InspectionInspection

Credit: iRobot

Synchronous VideoSynchronous Video

►Does not scale with team sizeDoes not scale with team size

Synchronous VideoSynchronous Video

►Does not scale with team sizeDoes not scale with team size

Synchronous VideoSynchronous Video

►Does not scale with team sizeDoes not scale with team size

Asynchronous ImageryAsynchronous Imagery

► Inspired by planetary robotic solutionsInspired by planetary robotic solutions Limited bandwidthLimited bandwidth High latencyHigh latency

► Multiple photographs from single locationMultiple photographs from single location Maximizes coverageMaximizes coverage Can be mapped to virtual pan-tilt-zoom camera Can be mapped to virtual pan-tilt-zoom camera

HypothesisHypothesis

►Asynchronicity may improve performanceAsynchronicity may improve performance Helps guarantee coverageHelps guarantee coverage Can review imagery on demandCan review imagery on demand

►Asynchronicity may reduce mental Asynchronicity may reduce mental workloadworkload Only navigation must be done in real-timeOnly navigation must be done in real-time Search becomes self-pacedSearch becomes self-paced

USARSimUSARSim

[http://www.sourceforge.net/projects/usarsim]

► Based on Based on UnrealEngine2UnrealEngine2

► High-fidelity physicsHigh-fidelity physics► ““Realistic” Realistic”

renderingrendering CameraCamera Laser scanner Laser scanner

(LIDAR)(LIDAR)

MrCSMrCSMMulti-ulti-rrobot obot CControl ontrol SSystemystem

MrCSMrCSMMulti-ulti-rrobot obot CControl ontrol SSystemystem

Waypoint Waypoint NavigationNavigation

TeleoperationTeleoperation

Video/ Video/ Image Image ViewerViewer

Status Status WindowWindow

Map Map OverviewOverview

Pilot StudyPilot Study

► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map

► Control 4 robotsControl 4 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation

► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners

Experimental ConditionsExperimental Conditions

10 Victims10 Victims

Arena 2

Arena 1

Experimental ConditionsExperimental Conditions

► Streaming ModeStreaming Mode ► Panorama ModePanorama Mode

Panoramas stored for later viewingPanoramas stored for later viewingStreaming live videoStreaming live video

Experimental ConditionsExperimental Conditions(Streaming Mode)(Streaming Mode)

Experimental ConditionsExperimental Conditions(Panorama Mode)(Panorama Mode)

SubjectsSubjects

►21 paid participants21 paid participants 9 male, 12 female9 male, 12 female No prior experience with robot controlNo prior experience with robot control Frequent computer users: 71%Frequent computer users: 71% Played computers games > 1hr/week: Played computers games > 1hr/week:

28%28%

MethodMethod

►Written instructionsWritten instructions►20 min. training session20 min. training session

Both streaming and panoramas enabledBoth streaming and panoramas enabled Encouraged to find and mark at least one Encouraged to find and mark at least one

victimvictim

►20 min. testing session (Arena 1)20 min. testing session (Arena 1)►20 min. testing session (Arena 2)20 min. testing session (Arena 2)

MetricsMetrics

►Switching timesSwitching times►Number of victimsNumber of victims

Thresholded accuracyThresholded accuracy

Victims FoundVictims Found

Average Average # of # of

victims victims foundfound

Accuracy ThresholdAccuracy Threshold

11

22

33

44

55

66

Within Within 0.75m0.75m

Within 1mWithin 1m Within 1.5mWithin 1.5m Within 2mWithin 2m00

PanoramaStreaming

11

22

33

44

55

66

77

First SessionFirst Session Second SessionSecond Session

Panorama First

Streaming First

< 1.5m< 2m

< 1.5m< 2m

00

Trial Order InteractionTrial Order Interaction

AveragAverage # of e # of victimvictim

s s foundfound

Switching TimeSwitching Time (Streaming Mode) (Streaming Mode)

AveragAverage # of e # of reportreport

ed ed victimvictim

ss

1122

2020

Number of SwitchesNumber of Switches

1100

88

66

44

22

4040 6060 8080 100100 12012000

00

p=0.064

Switching TimeSwitching Time (Panorama Mode) (Panorama Mode)

AveragAverage # of e # of reportreport

ed ed victimvictim

ss

1122

2020

Number of SwitchesNumber of Switches

1100

88

66

44

22

4040 6060 8080 100100 12012000

00

SummarySummary

►Streaming is better than panoramicStreaming is better than panoramic Perhaps not by as much as expectedPerhaps not by as much as expected Conditions favorable to streaming videoConditions favorable to streaming video

►Asynchronous performance has Asynchronous performance has potentialpotential May avoid forced pace switchingMay avoid forced pace switching May scale with team sizeMay scale with team size

Synchronous ScalingSynchronous Scaling

► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map

► Control 4, 8, 12 robotsControl 4, 8, 12 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation

► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners

Experimental ConditionsExperimental Conditions

8

4

12

Experimental ConditionsExperimental Conditions(Streaming Mode)(Streaming Mode)

SubjectsSubjects

►15 paid participants15 paid participants 8 male, 7 female8 male, 7 female No prior experience with robot controlNo prior experience with robot control Most were frequent computer usersMost were frequent computer users

MethodMethod

►Written instructionsWritten instructions►20 min. training session20 min. training session

Encouraged to find and mark at least one Encouraged to find and mark at least one victimvictim

►20 min. testing session (4 robots)20 min. testing session (4 robots)►20 min. testing session (8 robots)20 min. testing session (8 robots)►20 min. testing session (12 robots)20 min. testing session (12 robots)

MetricsMetrics

►Explored regionsExplored regions►Number of victimsNumber of victims►Neglect toleranceNeglect tolerance►Switching timesSwitching times►Number of missionsNumber of missions►NASA-TLX workloadNASA-TLX workload

Explored RegionExplored Region

Area explore

d

Victims FoundVictims Found

Number of

Victims

Victims Found per RobotVictims Found per Robot

Number of

Victims

Neglected RobotsNeglected Robots

Number of Robots

Totally

Initial Move

Switch TimesSwitch Times

Number of Switches

Mission NumbersMission Numbers

Number of Missions

NASA-TLX WorkloadNASA-TLX Workload

Workload

(Interaction Time)

(Neglect Tolerance)

Fan-outFan-out

SummarySummary

►Bounded number of directly Bounded number of directly controllable robots between 8 and 12controllable robots between 8 and 12 Diminishing returns as robots are addedDiminishing returns as robots are added Performance drops above 8 robotsPerformance drops above 8 robots

►Fan-out parallels the number of robots Fan-out parallels the number of robots operator controlsoperator controls Operators using satisficing strategyOperators using satisficing strategy

Asynchronous Scaling Asynchronous Scaling (Proposed)(Proposed)

► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map

► Control 4, 8, 12 robotsControl 4, 8, 12 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation

► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners

Experimental ConditionsExperimental Conditions

8

4

12

Experimental ConditionsExperimental Conditions(Panorama Mode)(Panorama Mode)

MethodMethod

►Written instructionsWritten instructions►20 min. training session20 min. training session

Both streaming and panoramas enabledBoth streaming and panoramas enabled Encouraged to find and mark at least one Encouraged to find and mark at least one

victimvictim

►20 min. testing session (4 robots)20 min. testing session (4 robots)►20 min. testing session (8 robots)20 min. testing session (8 robots)►20 min. testing session (12 robots)20 min. testing session (12 robots)

MetricsMetrics

►Explored regionsExplored regions►Number of victimsNumber of victims►Neglect toleranceNeglect tolerance►Switching timesSwitching times►Number of missionsNumber of missions►NASA-TLX workloadNASA-TLX workload

Expected ContributionsExpected Contributions

►Determine when asynchronicity is Determine when asynchronicity is usefuluseful Advantages for larger team sizesAdvantages for larger team sizes Simultaneous search is not viableSimultaneous search is not viable

►Establish performance baselines for Establish performance baselines for asynchronous searchasynchronous search