"Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation...

Jet Propulsion LaboratoryCalifornia Instituteof Technology

Using Vision to Enable Autonomous Land, Sea, Air,

and Space Vehicles

Larry MatthiesComputer Vision Group

Jet Propulsion LaboratoryCalifornia Institute of Technology

lhm@jpl.nasa.gov

Gale Crater

Application Domains andMain JPL Themes

Land: all-terrain autonomous mobility; mobile manipulation

Sea: USV escort teams; UUVs for subsurface oceanography

Space: assembling large structures in Earth orbit

Air: Mars precision landing; rotorcraft for Mars and Titan; drone autonomy on Earth

Basic Taxonomy ofPerception Capabilities and Challenges

Capabilities•  Localization

–  Absolute, relative•  Obstacle detection

–  Stationary, moving–  Obstacle type–  Terrain trafficability

•  Other scene semantics–  Landmarks, signs,

destinations, etc–  Perceiving people and their

activities•  Perception for grasping

Challenges•  Sensors for observability•  Fast motion•  Lighting conditions

–  Low light, no light–  Very wide dynamic range

•  Atmospheric conditions–  Haze, fog, smoke–  Precipitation

•  Difficult object/terrain types–  Featureless, specular,

transparent–  Obstacles in grass; water, snow,

ice; mud•  Computational cost vs processor

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyLocalization

•  Relevant use cases:–  Wheeled, tracked, legged vehicles

indoors and outdoors–  Drones–  Mars rovers–  Mars landers

•  Key challenges:–  Appearance variability: lighting,

weather, season–  Moving objects–  Fail-safe performance

•  Examples I’ll describe:–  Day/night relative localization with IMU,

leg odometry, and NIR active stereo for DARPA LS3 program

–  Map relative localization for Mars precision landing

1976 Viking174 x 62 mi

•  Some central themes:–  Fusion of IMU with vision or lidar

now commonplace for relative localization

–  Map matching is a very active topic for absolute localization

Day/Night LS3 Relative Navigationwith Visual, Inertial, and Leg Odometry

•  Array of NIR LEDs with different lenses/power to achieve more uniform illumination with dense stereo out to ~ 4.5 m

SENSORS:1)  Bumblebee Stereo (1024x768)2)  Tactical-grade IMU (600Hz)3)  Optional: Nav-grade IMU (600Hz)4)  Optional: Leg Odometry (200Hz)

asphalt dirt road snow forest

-  Position error over a moving window of 50m-  Position error < 0.5m 95% of runs-  Position error < 0.75m 100% of runs

•  How far can the lookahead scale with illuminators?

•  Can it work at night with thermal images (stereo or monocular) – with more noise, motion blur, and rolling shutter readout?

Mars Airbag-based Landers (1997, 2004):Horizontal Velocity at Impact

impactvelocity

Pathfinder

Pathfinderwith TIRS

Pathfinderwith TIRS ��and DIMES

Descent Image Motion Estimation System for 2004 Mars Landers

•  Horizontal velocity estimation during last 2 km of descent

vh11, vh12

vh21, vh22

~ 20 sec with 20% of 20 MHz RAD6000 flight computer

Map-Relative Localizationfor Mars Precision Landing

Backshell Separation

Powered Descent

Sky Crane

Flyaway

Prime MLEs

Radar Data Collection

Divert Maneuver

Safe Target Selection

Lander Vision System

TRN increases the ��probability of safe landing

Hazards in Landing��Ellipse without TRN

Hazards in Landing��Ellipse with TRN

Map-Relative Localizationfor Mars Precision Landing

LVS IMU

LVS Compute Element

Processor Navigation Filter

Data Flow

Virtex 5 FPGA Image Processing Sensor Interfaces

Memory for map

Spacecraft Flight Computer

LVS Camera

Image 1 Image 2

Image 3

IMU IMU

Image 4 Image 5

Coarse Landmark Matching Remove Position Error (3km)

Fine Landmark Matching Improve Accuracy (40m)

State Estimation Fuse inertial measurements with landmark

matches and complete in 10 seconds

spacecraft attitude, altitude

map relative ��position

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyObstacle Detection

•  Relevant use cases:–  Land vehicles indoors and outdoor

on-road and off-road–  Drones: flying and landing–  Boats on and under water–  Robot manipulators

•  Key challenges:–  Appearance variability:

•  Lighting, weather, season•  Surface reflectance, transparency

–  Terrain variability–  Moving objects–  Fail-safe performance

8 HSMSTs (Teleoperated) 5 USVs

(Autonomous)

•  Issues I’ll discuss:–  Sensors and phenomenology–  3-D representations–  Land, air, and sea examples

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyStereo Vision in a Marina

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyStereo + 360-degree Monocular EO/IR

JPL 360(monocular)

JPL Hammerhead (stereo)

Stereo

IR Deck

EO Deck Color

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyUnmanned Surface Vehicles

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyProcessing Blocks

Almost crossed à “crossing” rule

not applied USV must give way

1 knot

10 knots

30 knots

10 knots

COLREGS Illustration

Crossing from left Crossing from right Overtaking Head-on

Your boat

Traffic boat

Need more than the geometry to

determine COLREGS situaHons

Jet Propulsion LaboratoryCalifornia Instituteof TechnologySeeing through Atmospheric Obscurants

Smoke from a

controlled burn

Photon 640LWIR camera

Color cameraVisible vs SWIR

in haze

Uncooled LWIR Stereo:More Challenging

LBM SAD5 SGBM

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyNight Operation with Thermal Stereo Cameras

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyWhy are Negative Obstacles so Hard to Detect?

H θp h

θ p ≈hR

θn ≈Hw

R(R + w )

Heat Transfer Characteristics:Negative Obstacle Detection

Color crosswise view

Color lengthwise view

MWIR image 1 hr after sundown

Weatherproof sensor enclosure

Heat Transfer Characteristics:After Sundown, Holes Cool More Slowly than Surface

• Radiation

•  Evapotranspiration (ignored here)

qnegobs1

2negobsq

terrainq

1sideT 2sideT

terrainT

airTterrainT

terrainq

negobsq

negobsT

C5020 °−=diurnalTterrainT

terrainq

negobsq

negobsT

• Convection

• Conduction

Heat Transfer Characteristics:Negative Obstacle Detection

0.53 m

9 pm 7 am

9 am 9 pm

5 pm 5 pm

10 pm 7 am

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyDetection Results using Thermal Signature

Rectified thermal infrared intensity image.

After intensity

difference thresholding.

Closed contours

overlaid on intensity image.

After geometry

based filtering.

Trench 3 pixels wide at first detection. Trench first detected at 18.2m range.

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyWater Detection: Why is it Useful and is it Hard?

Water Body Detection with Reflections in Stereo:Works with Visible and Thermal Images

100m map

Stereo range imageLeft rectified image

Jet Propulsion LaboratoryCalifornia Instituteof TechnologySome Water Detection and Mapping Results

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyAbsorption Coefficient of Light in Pure Water

Foliage Classification andObstacle Detection in Foliage

lhm - 32

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyPolar Grid Maps

C-space-like obstacle expansion of disparity map

Architecture for obstacle avoidance

Image-based Obstacle Representation

MAV Obstacle Avoidance:Test Site Near JPL

MAV Obstacle Avoidance:Test Results

C-space-like obstacle expansion

of disparity map

Obstacle points and path projected on ground plane

Upright view

Research Approach:Challenges and InnovationGetting Very Wide Field of Regard

Stereo-OF fusion:

“egocylinder”

•  Range from stereo and OF

•  Scale propagation from stereo in overlap region

•  Projection into common cylinder representation

•  C-space expansion in image space

Egocylinder Representation

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyEgocylinder with Real Data

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyOnboard Implementation

Update Rates

Stereo 384x240 @ 5 Hz

SFM (LSD-SLAM)

384x240 @ 10 Hzboth cameras

Egocylinder 5 Hz

C-space 5 Hz

Planning5 Hz updates (1 ms

verification, several ms searching)

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyOnboard Implementation

Autonomous Landing:Problems and Solution Characteristics

Gale Crater, Mars

•  Problem characteristics •  Variable potentially complex 3-D structure •  Variable appearance •  Variable altitude for approach •  Need for very lightweight hardware

•  Solution characteristics •  Desire dense 3-D perception •  Must work from variable altitude •  Must work in direct sunlight •  Hardware as light as possible – just a camera?

Autonomous Safe Landing:Variable Baseline Dense Motion Stereo

Onboard Implementationwith Smartphone Processor

Autonomous Rooftop Landing of Micro Air Vehicles for Recon Applications

Real-time Onboard Mappingfor Safe Landing in Unknown Terrain

Raw image from camera at ~15m height.

Computed elevation map

Landing confidence map (dark blue: highest confidence)

river bed

Notional Future Directions for Space Exploration

Pre-Decisional Information – For Planning and Discussion Purposes Only

•  Mars sample return•  Accessing recurring slope lineae,

caves, and vertical/microgravity

•  Mars sample return•  Accessing recurring slope lineae•  Comet sample return, Ocean Words, Titan

Perception and Planning for Robots in Human Environments, Interacting with People

Base reachability Arm

reachability

Desired end-effector goal

Perception and planning for mobile manipulation

Deep learning-based object class labeling/pose estimation; human articulate body pose estimation

Power grasp opportunities

Pinch grasp opportunities

Some Thoughts About Back on Earth:Better Perception for Human-Robot Interaction

Integrate facial expressions, head pose, and body pose

into robot perception of people for more intelligent

human robot interaction

S_1 S_2 S_3 S_4 S_5 S_6 S_7 S_8 S_9

S_13S_12 S_15S_14 S_17S_16S_11S_10

Electromyography sleeve with forearm IMU and magnetometer for recognizing arm and

hand gestures

Jet Propulsion LaboratoryCalifornia Instituteof TechnologyA Few Recent Highlights of Non-NASA Work

Sunset at Gusev Crater, Mars, from Spirit Rover

lhm@jpl.nasa.gov

Questions?

"Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation...

Technology

JPL Bolt Torque

jpl 2013.pdf

COSMIC and Space Weather Anthony Mannucci, JPL Brian Wilson, JPL Vardan Akopian, JPL, USC George Hajj, JPL, USC Lukas Mandrake, JPL Xiaoqing Pi, JPL, USC

Drupal @ JPL

Jpl coding standard_c

The JPL Autonomous Helicopter Testbed: A Platform for ...stergios/papers/JFR_UAV_2006.pdf · Technology Research and Development ... lhm} @jpl.nasa.gov. S. I ... Inspiron 8200 laptop

Spark at NASA/JPL-(Chris Mattmann, NASA/JPL)

Jpl Iitb.pres1

Daily Operation and Validation of a Global Assimilative Ionosphere Model Brian Wilson, JPL George Hajj, JPL, USC Lukas Mandrake, JPL Xiaoqing Pi, JPL,

KEYNOTE ADDRESS: Embracing the Autonomous Supply Chain and Rethinking Innovation

JPL Studies in Support of the SWOT Mission Studies in Support of the SWOT Mission Ernesto Rodriguez JPL/CalTech Washington, October 28, 2007 JPL Internal Investment on SWOT • JPL

Bottom Leads - JPL

Jpl Iitb.pres2

Jpl presentation

Fusion Catalogue JPL

presentation AD19 - Inria...Dinesh Manocha, Department of Computer Science, University of Maryland, USA Keynote title: Autonomous Driving: Simulation and Navigation Abstract: Autonomous

Discover - JPL Technologyscienceandtechnology.jpl.nasa.gov/sites/default/files/documents/JPL... · JpL has a long history of working with educational insti-tutions. We believe that

bage. are on each c»late tar ootirnurn viewing o' cornowigon. We avolied a uniformea 20urn thick, the standard thickness luster base. tar the coating JPL-OOI JPL-002 JPL-003 JPL-004

The JPL Autonomous Helicopter Testbed: A Platform …stergios/papers/JFR_UAV_2006.pdf · To date it has been used for planetary landing technology development. Specifically, image-based

JPL Institutional Coding Standard - JPL Laboratory for Reliable