Image Processing and Cartography with the NASA Vision Workbench

Intelligent Systems Division NASA Ames Research Center

Image Processing and Cartography with the NASA

Vision Workbench

Matthew D. HancherIntelligent Systems Division

NASA Ames Research CenterSeptember 26, 2007


Talk Overview

• Who We Are

• Introduction to the Vision Workbench

• Example Applications

• FOSS and NASA


NASA Ames Research Center

• NASA’s Silicon Valley research center

• Small spacecraft

• Supercomputers

• Intelligent Systems

• Human Factors

• Thermal protection systems

• Aeronautics

• Astrobiology


GIS & Imaging at Ames

NASA World Wind MASTER(MODIS/ASTER simulator)

NASA/GooglePlanetary Content

Western StatesFire Monitoring Mission


IRG & ACES

Intelligent RoboticsGroup

Adaptive Control &Evolvable Systems Group


Intro to the Vision Workbench


NASA Vision Workbench

• Open-source image processing and machine vision library in C++

• Developed as a foundation for unifying raster image processing work at NASA Ames

• A “second-generation” C++ image processing library, drawing on lessons learned by VXL, GIL, VIGRA, etc.

• Designed for easy, expressive coding of efficient image processing algorithms


Open-Source VW Modules

• Core: Low-level types & platform support

• Math: General-purpose mathematical tools

• Image: Basic image operations, filters, etc.

• FileIO: Simple, flexible image file IO layer

• Camera: Camera models & related tools

• Cartography: Geospatial image manipulation

• Mosaic: Image mosaicing & multi-band blending

• HDR: High-dynamic-range imaging

(Open source as of now)

VW “Foundation”

Modules


VW Modules Underway

• InterestPoint: Interest point detection & matching

• Stereo: Stereo correlation & 3D reconstruction

• Python: Python bindings to many VW capabilities

• GPU: GPU-accelerated image operations

• Texture: Texture analysis & matching

• Display: Image display and user interaction

(The first four to be released later this year)


Design Goals & Approach

• A simple, clean API for easy hacking

• Simple syntax: Write what you mean!

• Easy to manipulate arbitrarily large images

• Automatic memory management

• Generates high-performance code

• Optimized processing via lazy evaluation

• Function inlining via “generic” (template-based) C++ style


API Philosophy

• Simple, natural, mathematical, expressive

• Treat images as first-class mathematical data types whenever possible

• Example: IIR filtering for background subtraction

background += alpha * ( image - background );

• Direct, intuitive function calls

• Example: A Gaussian smoothing filter

result = gaussian_filter( image, 3.0 );


Image Module Basics


Under the Hood: Image Views

• The core “image view” concept:• Can be evaluated at a location to return a pixel value

• Has a width and height in pixels

• Cannonical example: the ImageView class•ImageView<PixelRGB<uint8> > image(1024,768);

• Data processing represented as views

•image2 = gaussian_filter(image1, 3.0);

• Lazy container for arbitrary views

•ImageViewRef<PixelRGB<uint8> > image3 = gaussian_filter(image1, 3.0);


Image Views II

• Eliminates unnecessary temporaries• background += alpha * ( image - background );

• Supports procedurally generated images•image2 = fixed_grid(10,10,white,black,1024,768);

• Allows greater control over processing•image2 = block_rasterize( gaussian_filter(image1, 3.0) );

• Views of images on disk

•DiskImageView<PixelRGB<uint8> > disk_image(filename);


Applications & Modules


GigaPan Panorama Stitcher

(As featured in the GigaPan layer in Google Earth.)


Mosaic Module

• ImageComposite

• Composite an arbitrary number of arbitrarily large images

• It’s “just another image view”

• Supports multi-band blending for seamless composites

• QuadTreeGenerator

• Generates a tiled pyramid representation of an arbitrary image view on disk

• Great for building e.g. KML superoverlays or TMS maps


Cartographic Reprojection

(As seen in the newly updated Google Moon.)


Cartography Module

• GeoReference• Uses PROJ.4 for standard projections, GDAL to read/write

• GeoTransform• Reprojects image data between GeoReferences

• Makes “just another image view”

• OrthoImageView• Ortho-rectifies an aerial or satellite image against an

arbitrary DEM (in conjunction with the Camera module).

• Also “just another image view”


Automated Image Alignment

• Problem: Given two images, find and align the overlap region.


Image Alignment w/ Interest Points

Images to be alignedLocate interest points in first imageLocate interest points in second imagePoint correspondences determine image alignment


Interest Point Module

• Interest point detectors, descriptors, and matching

ScaledInterestPointDetector<LoGInterest> detector;InterestPointList ip1 = interest_points( image1, detector );InterestPointList ip2 = interest_points( image2, detector );

PatchDescriptor descriptor;compute_descriptors( image1, ip1, descriptor );compute_descriptors( image2, ip2, descriptor );

DefaultMatcher matcher(threshold);InterestPointList matched1, matched2;matcher.match( ip1, ip2, matched1, matched2 );

Matrix2x2 homography = ransac( matched1, matched2, SimilarityFittingFunctor(), InterestPointErrorMetric() );


The Ames Stereo PipelineFast, high quality, automated stereogrammetric surface reconstruction originally developed for

Mars Pathfinder science operations

Disparity

Now a Vision Workbench application.


Primary Image Secondary Image

RegistrationEphemeris or

Automated Interest Points

Fast Stereo Correlation

Outlier Rejection / Hole Filling / Smoothing

Disparity Map

Camera Model (e.g. Linear Pushbroom)

Mesh Generation

Point Cloud/DTM

3D Mesh

Mask / Sign of Laplacian of Gaussian

The Ames Stereo Pipeline

Surprise: It’s all just Vision Workbench image views!


Mars Stereo: MOC NA

MGS MOC-Narrow Angle• Malin Space Science Systems• Altitude: 388.4 km (typical) • Line Scan Camera: 2048 pixels• Focal length: 3.437m• Resolution: 1.5-12m / pixel• FOV: 0.5 deg


NE Terra Meridiani

Upper Left: This DTM was generated from MOC images E04-01109 and M20-01357 (2.38°N, 6.40°E). The contour lines (20m spacing) overlay an ortho-image generated from the 3D terrain model. Lower Right: An oblique view of the corresponding VRML model.

!!"""#$

!!"""$

!%""#$

!%""#$

!%""#$


Preliminary MOLA Comparison

Scanline Capture Time (s)

Elev

atio

n at

bor

esig

ht p

ixel

(m

)


Lunar Stereo: Apollo Orbiter Cameras

ITEK Panoramic Camera• Focal length: 610 mm (24”)• Optical bar camera• Apollo 15,16,17 Scientific

Instrument Module (SIM)• Film image: 1.149 x 0.1149 m• Resolution: 108-135 lines/mm


Apollo 17 Landing Site

Top: Stereo reconstruction

Right: Handheld photo taken by an orbiting Apollo 17 astronaut


Public Outreach: Hayden Planetarium


Public Outreach: Hayden Planetarium


Application: Image Matching

• Problem: Given an image, find others like it.

Example database: Apollo Metric Camera images


Texture-Based Image Matching

Texture bank filtering (Gaussian 1st derivative and LOG)

Grouping to remove orientationEnergy in a window

E-M Gaussian mixture modelIterative tryouts, MDL

Max vote

GroupingMean energy in segment

Euclidian distance

Summarization

Post-processing

Output Representation

Filtering

Model image

Segmentation

Vector Comparison

Matched image


Image Matching: Results


FOSS and NASA


The NOSA

• The NASA Open Source Agreement, an OSI-approved non-viral open source license

• Intended to protect users from contributor patent licensing issues.

• Yes, we know: The current version (1.3) has several well-known peculiarities.


U.S. Contractor Rights

• The University and Small Business Patent Procedures Act of 1980, a.k.a. “Bayh-Dole”.

• A university, small business, or non-profit can claim patent ownership of a federally-funded invention before the government.

• The government must actively promote and attempt to commercialize the invention.

• Severely complicates open-source initiatives within the government that involve universities, small businesses, or non-profits.


The Open Source Process

• Open-source approval stages include:

• Invention disclosure

• Copyright assignment (all parties)

• Legal review (copyright & patent issues)

• Export control review (e.g. ITAR)

• Computer security review

• more....


Signs of Improvement

• The old model: (e.g. VW 1.0)• Seek approvals after code completion

• Long, slow, high-latency release cycle

• The new model: ?? (e.g. WV 2.0 ??)• Seek periodic approval for upcoming development

• Allows regular updates within prescribed bounds

• On the horizon: ??• User contribution process ?

• Publicly-accessible subversion repository ???


Free and Open Data

• Free and open data has received much less attention than free and open software.

• The National Aeronautics & Space Act:

• The Administration, in order to carry out the purpose of this Act, shall... provide for the widest practicable and appropriate dissemination of information concerning its activities and the results thereof.

• Alas, NASA does not own much of what is often imagined to be “NASA data”.


Outreach: Google Earth

MODIS CoveragesAstronaut Photography

• Make more datasets publicly available as KML (and soon WMS) for mash-ups.

• Increase the visibility of existing public repositories of NASA data and imagery.


Outreach: Google Moon

Data coming soon via KML and WMS from NASA.


Obtaining the Vision Workbench

• VW version 1.0.1 available now.

• VW version 2.0 coming this fall!

• To contact me:

[email protected]

http://ti.arc.nasa.gov/visionworkbench/

Technology

Image Processing and Cartography with the NASA Vision Workbench