Estlin aegissoyajpl 2012

Autonomous Exploration for Gathering Increased Science

AEGIS

Tara EstlinBenjamin Bornstein, Daniel Gaines, David R. Thompson, Rebecca Castano, Robert C. Anderson, Michael Burl, Charles de Granville and Michele Judd

2011 NASA Software of the Year

Consider the following problem…• You are a robotic explorer

on another planet

• You only talk to Earth once a day

• You are in a hostile environment

• You have limited power and computing abilities

• You are constantly on the move exploring different terrains

• As you move, you need to quickly determine if you see objects that are interesting to scientists

• If you do, you want to acquire data on these objects, before the rover moves past 2

This is why we developed AEGIS

AEGIS:• Is a new paradigm for in‐situ science using onboard autonomy

• Provides intelligent targeting and data acquisition by

– analyzing images of the rover scene– identifying high‐priority science targets (e.g., rocks), and

– taking high quality data of these targets completely autonomously with no ground interaction required

3

AEGIS Video

4

How is AEGIS being used?

AEGIS:• Is in regular operational use onboard

the Mars Exploration Rover (MER) Mission Opportunity rover for the past two years

• Excels in automated targeting with narrow field‐of‐view (FOV) remote sensing instruments, such as:

– MER Panoramic Cameras (in current use)

– MER Mini‐Thermal Emission Spectrometer

– Mars Science Laboratory (MSL) Rover ChemCam Spectrometer

– Before AEGIS, had to manually select targets, based on ground analysis

5

Mini-TES Mosaic

AEGIS Process for MER

Pancam pointing

Pancam acquisition

Navcam acquisition

Target feature extraction

Process fully automated!Advanced image processing technique enables reliable, rapid identification of candidate targets.

Top score for large size

Scientists can prioritize important properties for each run

Algorithms quantify key intuitive target properties such as brightness, size, and shape.

Robust approach to pointing selection maximizes data of target.

High-quality, 13 color filter, quarter-frame Panoramic camera image

Target pointing determination

Target prioritization

Target detection

Benefit of AEGIS for Rover Drive

Manually‐targeted remote sensing as specified by science team(taken before drive)

XAutonomously‐targeted remote sensing taken mid‐drive by AEGIS

X

Autonomously‐targeted remote sensingtaken end of drive by AEGIS

7

Targeted data selected manually

Targeted data with AEGIS

AEGIS identifies rock targets

8

Scientist target profile: large rocks of high reflectance.Markers show top ten prioritized targets.


9

Area covered by MER Panoramic Camera.


10

Area covered by MER Panoramic Camera.

AEGIS delivers 13F Pancam image

11

after autonomously targeting the top priority rock

MER before AEGIS

Sol 3Sol 2Sol 1Perform

MANUALLY TARGETED remote sensing of current

rover areaDrive rover 100m to new stopping

pointAcquire wide‐angle images of new terrain area

Perform untargeted remote

sensing of local area

Perform untargeted remote


12

Pancam Sol 2

Pancam Sol 3

Multi-Sol Plan

MER after AEGIS

13

AEGIS Pancam 1

AEGIS Pancam 2

Sol 3Sol 2Sol 1Perform

MANUALLY TARGETED remote sensing of current

rover areaDrive rover 100m to new stopping

pointAcquire wide‐angle images of new terrain area

Perform autonomously

targetedremote


Perform autonomously

targetedremote


Multi-Sol Plan

SIGNIFICANCEScientists trust AEGIS to make intelligent decisions about collecting new science.

Significance: Aerospace• Fully operational and used regularly on MER mission– Saves valuable time every targeted

data collection– Has been used more than any other

new technology on the Opportunity rover

• Currently infusing into Mars Science Laboratory (MSL) Mission

• Attracting strong interest for – 2018 Mars rover and other future in‐situ missions (e.g., Titan, Venus, Europa)

– Military applications (e.g., UAVs)15

Opportunity Today

Significance: Science/Technology• Enables the collection of science data that would

otherwise not be possible– During or right after drives– Different times of day and temps

• Saves significant time and cost for targeted data collection– Gets data into the hands of scientists twice as fast

(or more) than standard operations

• Enables scientists to easily use and interact with autonomy software – Parameters chosen after significant

consultation with scientists

• 25+ science and technology publications

• Application to large number of problems in industry and academia (e.g., underwater robotics)

16

Significance: Humanitarian• Directly contributing to humanity goal of finding life on other planets– Mars program theme of “Follow the water…”

• Significant outreach vehicle; over 35 media articles since release in 2010– “Mars Rover Getting Smarter As It Gets Older”– “NASA upgrades Mars rover brain”

17

Significance: Humanitarian

18

AustraliaUnited Kingdom

Over 46 Amateur Astronomy Clubs, Schools and Teacher Organizations around

the world featured our “rock hound” software in their newsletters

Inspiring the next generation in STEM:Science, Technology, Engineering and Mathematics

DEVELOPMENTAEGIS is flight proven and fully operational

Development Status• TRL Level 9: Flight Proven

– Software fully operational – In regular use on MER Opportunity rover

• MER and MSL Scientists have already asked for extensions, which are in progress:– Enabling multiple targeted observations– Triggering on single filter color images– Identifying novel targets– Identifying representative targets– “Soil only” detector

20

“AEGIS is a true success story for the Mars Technology Program”

Dr. Samad HayatiManager of Mars Technology Program

ASSESSMENTof USE

AEGIS technology is being applied in a wide range of applications.

NASA Use – MERAEGIS is considered everymulti‐sol plan.

22

Sol 2138 Sol 2172 Sol 2221 Sol 2247

Sol 2278

Sol 2304

Sol 2290

Sol 2312

Sol 2313

“AEGIS is a significant enhancement for the mission and the scientific community. MER is the first mission to implement the capability that AEGIS provides – and it has really paid off.”

Dr. John CallasMars Exploration Rover Mission Project Manager

NASA Use – MSL Rover• The MSL Rover ChemCam Team has requested

AEGIS (PI: Roger Wiens)

• AEGIS is ideal for ChemCam’s narrow field‐of‐viewLaser‐Induced Breakdown Spectrometer (LIBS)– Samples rocks from a distance of 1 to 7 meters – Able to rapidly identify rock elemental composition

• AEGIS enables multiple autonomously targetedChemCam measurements throughout the day

• MSL flight software integration in progress

23

Video clip from Dr. Roger WiensPrincipal InvestigatorMSL Rover Mission Chemcam Instrument

24

NASA Use – MSL

NASA Use – MSL

25

NASA Use – Mars 2018 Rover

• 2018 Rover Mission will have limited time to core and store up to 30 rock samples – Will need to drive up to

20 kilometers– Will need to consider targets

from distinct areas

• Strong interest from Mars 2018 Mission Program Office (Charles Whetsel, Chris Salvo) in using AEGIS to collect data on potential targets– Get data to science team faster– More targets could be considered

26

Future Use – Other Missions• AEGIS system can enable a wide spectrum of

missions: – Collect valuable science more often– Enhance onboard autonomy capabilities

• Strong application to in‐situ missions to Titan, Europa, Venus, Mars, the Moon, and small bodies

• Science autonomy listed as critical capability in Titan Prebiotic Explorer Mission Study– Helps address challenges such as extremely limited

communication, high platform mobility, etc.

27

“Onboard science algorithms will analyze the image data to detect trigger conditions such as science events, interesting features, changes relative to previous observations, …”

TiPEx mission study team

Industry, Government, Research Use

28

AEGIS

Autonomous Underwater

Vehicles (AUVs)

Unmanned Aerial Vehicles

Multi‐core Processor

Benchmarking

Search and Rescue Robotics

Commercial Spectroscopy

Unmanned Sea Surface Vehicles (USSVs)

Lunar Exploration

AEGIS is transferable to a wide range of

application domains

29

AEGIS


Vehicles (AUVs)



Benchmarking




Lunar Exploration

Unmanned Aerial Vehicles (UAVs)

• Developing automated cueing capability for UAV surveillance platforms.

• Lower-resolution wide area imagery used to trigger higher-resolution follow-up on selected areas.

• Proof-of-concept completed for ships using satellite imagery

• Evaluating for use identifying ground vehicles on imagery from AFRL "Angel Fire" aerial asset


30

AEGIS


Vehicles (AUVs)



Benchmarking




Lunar Exploration

“Moon Express is developing a lunar lander and mobility system for exploration of platinum group metals on the surface of the moon as well as compete for the Google Lunar X-Prize.

AEGIS could be a great asset to this quest by autonomously recognizing rocks from iron-rich asteroids that might contain platinum.”

-- Moon Express, Inc.


31

AEGIS


Vehicles (AUVs)



Benchmarking




Lunar Exploration

Robotic Underwater Vehicles• WHOI Nereus vehicle

– Performs deep ocean scientific survey and sampling

– Used to locate hydrothermal systems, volcanic processes, etc.

• CMU/Pittsburgh Aquarium Reefbot– Automatically detect,

classify, and count fish in their natural habitat

• AEGIS could save days/weeks of exploration time through autonomous data collection


32

AEGIS


Vehicles (AUVs)



Benchmarking




Lunar Exploration

Robotic Underwater Vehicles• WHOI Nereus vehicle

– Performs deep ocean scientific survey and sampling

– Used to locate hydrothermal systems, volcanic processes, etc.

• CMU/Pittsburgh Aquarium Reefbot– Automatically detect,

classify, and count fish in their natural habitat

• AEGIS could save days/weeks of exploration time through autonomous data collection


“AEGIS makes important advances in automatic data analysis. .. It has direct relevance to work at CMU in underwater vehicles for detecting and cataloging fish in deep water reefs.”

D. WettergreenCMU Robotics Institute

IMPACT“AEGIS usage on the MER Opportunity Rover showcases how it could be extremely beneficial for the Mars 2018 Mission.”

Charles Whetsel, Manager, Advanced Concepts2018 Mars Program Office

MER Impact: Increased Science• Before AEGIS, all targeted data required:

– Manual evaluation of images– One to several communication cycles – The rover to remain stationary and sometimes backtrack

• After AEGIS, targeted data can be collected:– Without ground analysis of context images– Without communication cycles– Any time during a rover drive – Any time of day 34

By the time the “Block Island” meteorite was noticed in an image, the Opportunity rover was already 200 meters past. The rover had to turn around and backtrack (costing 25 additional sols).

Video clip from Professor Steve SquyresPrincipal InvestigatorMars Exploration Rover Mission

35

MER Impact: Increased Science

MER Impact: Increased Science

36

MSL 2011 Rover Impact

MSL Impact: Increased Science

• 2008 study on ChemCam target selection using 65 MER Panoramic camera images

• Top 5 targets evaluated– If random sampling, 10% chance of being rock– If chosen by AEGIS, 92% chance of being rock

37

9m

2018 Rover Impact

Mars 2018 Rover Impact: Increased Science• Previous study on AEGIS application to 2018 Mission• AEGIS can be used to collect additional targeted

science data– Increased close‐contact measurements by 50%– Increased remote‐sensing measurements by 500%– Provides scientists more targets to choose from

for coring

2018 Rover Impact: Cost Benefit• Study showed number of sols required to investigate

and core a target could be decreased from 7 to 4 sols

38

CREATIVITYAEGIS is a pioneering flight software system that provides scientists with sophisticated control over targeted data collection

Creativity: Innovation• AEGIS provides new paradigm for

surface data acquisition– Scientist provides description of target– System can collect data whenever target

detected

• Flight challenges– Image processing performed on RAD6000

(orders of magnitude slower than standard desktop machine)

– AEGIS limited to < 4 MB of memory– Large performance optimizations made!

• Inventive approach to flight software change– Full flight software upload not possible– AEGIS uploaded as standalone module– Loaded into memory whenever want to use

(< 30 secs)40

Memory reduced 16x62 MB to 3.75 MB

Performance improved 7X

Runtim

e (secon

ds)

Benchmark Images

62

3.75

Mem

ory Usage

(megab

ytes)

Original Optimized

• Parameters defined through collaboration with scientists– Describe attributes of candidate targets– Express diverse and evolving science goals

• System did not require extensions to MER command dictionary or telemetry

• Training materials – Web interface for creating commands– User’s guide for software usage and

sequencing– Standard terrain profiles available– Result message (EVR) interpreter

Creativity: Usability

41

Albedo, shape, and size

“One of the key aspects that has made the AEGIS team successful is their long track record of working with the scientists.”

Dr. Jack StockyNew Millennium Program Manager

Creativity: Quality Factors

• Reliable target detection– Find rock targets in diverse terrain– Is resilient to dust‐covered or shadowed rocks– Works under strict computation constraints

• Risk control through resource limits and time deadlines

• Validation and Verification– Extensive MER testing procedure and code reviews– Nightly build, static analysis, unit and regression tests

42

SUMMARY“This autonomous science breakthrough is really changing expectations for future science mission operations.”

Raymond E. ArvidsonDeputy Principal InvestigatorMars Exploration Rover

Summary• Significance: Far Reaching

– Aerospace: Routinely used on MER– Science/Technology:

• Enables science that could not be previously collected• Applications in military, commercial and research fields

– Humanitarian:• Contributing to goal of finding life on other planets• Inspiring next generation in STEM areas

• Development: Flight Proven• Assessment of use:

– Planned for Infusion into New Missions and Applications

• Creativity: Pioneering/Deeply Innovative– Innovation: New paradigm for in‐situ data acquisition– Usability: System parameters designed through direct

collaboration with scientists– Quality: Reliable target detection under strict computation

constraints 44

For more information and surface results, visit the AEGIS website: http://aegis.jpl.nasa.gov/45

For more information…

[email protected]

Questions?

We’d like to acknowledge our sponsors: New Millennium Program, Mars Technology Program, JPL Research and Technology Development Program,

and the IND Technology Program

and thank you to the:The Mars Exploration Rover Mission

EXTRA

AEGIS Target Detection

Edge detection Edge detection

Flood fill +Flood fill +

ContoursContours

Morphology ops

Contours can be further filtered

49

Operator Filtering Rules

Operator Filtering Rules

Rover Body MaskingRover Body Masking

Target Feature Extraction

Shape– Eccentricity– Ellipse fit error– Roundness– Ruggedness– Angularity

Reflectance

Size– Inscribed circle– Pixel area

– Mean– Variance– Skew– Kurtosis

Rounded

Light Dark

Angular

Target Prioritization / Top Target Selection

Near the top of the list of“round” rocks

Near the bottom of the list of“round” rocks

• Scientists can prioritize different feature values and combinations of two– e.g., prefer large, high

albedo rocks– Can also support MER

cobble campaign, outcrop finder, soil finder, etc.

• Priority specification is part of command sequencing

• Can be easily changed as rover enters different terrain areas

Images from MER field trial

AEGIS Code Details

• AEGIS is 7968 SLOC (C)• Limited to less than 4 MB of memory• Requires only 232 KB of disk space• Regular static analysis using Coverity

PreventTM

• Formal code reviews– Internal AEGIS Team– Other JPL AI/machine-learning developers not

members of AEGIS Team– MER Team

52

Meteorite Detector

Devin Island

Meteorite Detector

Marquette Island

AEGIS Target Detections• Target detections are consistent with AEGIS selection

profiles – 90% of top targets meet the selection profile – Confirmed by evaluation of context Navcam imagery– All results reviewed with MER Science Team

• The MER Science Team is very happy with AEGIS and continues to request it regularly

55

OASIS Framework • OASIS: Onboard Autonomous Science

Investigation System• Objective: Maximize science returned on surface mission

– Identify and respond to science opportunities– Data prioritization for downlink– Maximize utilization of onboard resources

• Approach– Data segmentation and feature

extraction for multiple instruments– Science Data Analysis

• Prioritize targets and/or data• Summarize data

– Automated Planning and Scheduling• Adjust rover activities to collect new

data• Ensure operation within rover

resource and operation constraints

AEGIS in the OASIS Framework

• AEGIS is a flight software system derived from the larger OASIS framework

• Developed by same team of people• AEGIS includes a subset of OASIS capabilities

selected for MERRelevant - Instruments available on MERDesired - Requested by scientistsFeasible - Fit within memory and time limits

AEGIS Results from B Sol 2138

AEGIS Result from B Sol 2221