Scott Sawyer (scott.sawyer@ll.mit.edu) MIT Lincoln Laboratory

HPEC ’12 12 September 2012

Cluster-based 3D Reconstruction of Aerial Video

This work is sponsored by the Assistant Secretary of Defense for Research and Engineering under Air Force contract FA8721-05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by the United States Government.

Cluster SfM - 2 SMS 09/12/12

Given a collection of photos…

Cluster SfM - 3 SMS 09/12/12

…determine their 3D structure

Cluster SfM - 4 SMS 09/12/12

…determine their 3D structure

Cluster SfM - 5 SMS 09/12/12

•  Feature-based 3D Reconstruction •  Reconstructing Aerial Video •  Cluster-based Structure from Motion •  Results

Outline

Cluster SfM - 6 SMS 09/12/12

•  Feature-based 3D Reconstruction •  Reconstructing Aerial Video •  Cluster-based Structure from Motion •  Results

Outline

Cluster SfM - 7 SMS 09/12/12

“Structure from Motion” (SfM)

Feature Extraction

Feature Matching

Sparse Reconstruction

Dense Reconstruction

Photo Collection

3D Point Cloud

Applications:

3D Maps Robotic Navigation Video Geo-registration

Cluster SfM - 8 SMS 09/12/12

Feature Extraction

Feature Extraction

Feature Matching

Sparse Reconstruction

Dense Reconstruction

Photo Collection

3D Point Cloud

Scale Invariant Feature Transform (SIFT)

•  Identifies distinctive keypoints in an

image and provides 128-dimensional descriptor vector

•  Based on finding local extrema in scale space

•  Invariant to scale, orientation and lighting conditions (to an extent) and aspect angles up to 30 degrees

•  Published by David Lowe in 1999 and widely used in computer vision

Cluster SfM - 9 SMS 09/12/12

Feature Matching

Feature Extraction

Feature Matching

Sparse Reconstruction

Dense Reconstruction

Photo Collection

3D Point Cloud

Euclidean Distance Approximate Nearest

Neighbors (ANN)

•  Euclidean distance between SIFT keypoint descriptors used to determine matches

•  Tree-based approximation reduces computation

•  SIFT points matched between each pair of images – O(n2)

Cluster SfM - 10 SMS 09/12/12

Sparse Reconstruction

Feature Extraction

Feature Matching

Sparse Reconstruction

Dense Reconstruction

Photo Collection

3D Point Cloud

Projective Structure from Motion

•  Given m images and n 3D points

(with point correspondences)

•  Estimate m projection matrices (Pi, i = 1, …, m) and n 3D points (Xj, j = 1, …, n) from the mn correspondences (xij)

•  Iteratively minimize re-projection error (“bundle adjustment”):

( )2

1 1,),( ∑∑

= =

=m

i

n

jjiijDE XPxXP

x1j

x2j

x3j

Xj

P1

P2

P3

Cluster SfM - 11 SMS 09/12/12

Dense Reconstruction

Feature Extraction

Feature Matching

Sparse Reconstruction

Dense Reconstruction

Photo Collection

3D Point Cloud

Identify Patches of Rigid Structure

•  Mature software (“PMVS2”)

forms patched-based reconstruction (open source, parallel)

•  Removes spurious and non-rigid points

•  This step not included in benchmark analysis

Cluster SfM - 12 SMS 09/12/12

•  Feature-based 3D Reconstruction •  Reconstructing Aerial Video •  Cluster-based Structure from Motion •  Results

Outline

Cluster SfM - 13 SMS 09/12/12

Lincoln Laboratory performed a campaign of aerial data collections in 2011: •  Aerial platform circles ground

subject of interest •  Camera gimbal remains fixed on

ground point •  Video frames synchronized to

GPS •  Challenge: Processing must keep

up with rate of collection (e.g. one-hour mission timeline)

Reconstructing Aerial Video

Cluster SfM - 14 SMS 09/12/12

Sample Video

Cluster SfM - 15 SMS 09/12/12

Bundler

Bundler (v. 0.4) by Noah Snavely

•  Works on unordered image collections (no assumptions about input)

•  Sequential implementation (single-core)

•  Optimization solution considered baseline (truth) 0

20

40

60

100 200 300 400 500

Run

time

(hou

rs)

Image Set Size

Bundler (single-core)

Feat. Extr. Feat. Match Bundle Adj.

Cluster SfM - 16 SMS 09/12/12

VisualSFM

VisualSFM (v. 0.5.15) by Changchang Wu

•  Restructures optimization to better enable parallelization of dense matrix kernels

•  Feature extraction and bundle adjustment use GPU acceleration (CUDA)

•  Multi-threaded feature matching (pThreads) 0

10

20

30

100 200 300 400 500

Run

time

(hou

rs)

Image Set Size

VisualSFM (multi-core & GPU)

Feat. Extr. Feat. Match Bundle Adj.

Run on quad-core Intel Xeon E5620 (2.4 GHz), 6 GB RAM, nVIDIA Quadro FX1800 (64 compute cores)

Cluster SfM - 17 SMS 09/12/12

•  Feature-based 3D Reconstruction •  Reconstructing Aerial Video •  Cluster-based Structure from Motion •  Results

Outline

Cluster SfM - 18 SMS 09/12/12

LLGrid Computing Environment

Scheduler assigns each job to a single

core (no threading or shared memory)

Job submission

Jobs may or may not be on the same node

Inter-process communication supported via file system or network

Number of cores used varied experimentally up to 64

Custom Python framework for

submitting jobs, mapping data and

synchronizing between steps

Master Node

Scheduler

Compute Nodes

Network File System (Lustre)

Cluster SfM - 19 SMS 09/12/12

Cluster-based SfM Overview

1. Feature Extraction

2. Feature Matching

3. Reconstruction

4. Point Cloud Fusion

Job 1 Job 2 Job Np

JPEG JPEG Photo

Remap

…

Single Job

Partial Point Clouds

Final Point Cloud

Matched Feature Graph

SIFT keypoints

Remap

Job 1 Job 2 Job Np …

Job 1 Job M …

Runtime Growth

O(N)

O(N2)

O(N2/M)

JPEG JPEG Photo JPEG JPEG Photo

Remap

Job 1 Job 2 Job Np …

Combine Results

Step

O(P2/M)

N: Input photo count; M: Partition count; P: Sparse point count

Cluster SfM - 20 SMS 09/12/12

Feature Matching

•  Matching is most expensive step, growing O(N2) with number of photos

•  Results of matching represented as graph G:

•  Vertices V represent SIFT points

•  Edges E represent SIFT matches

Photo ID

1 2 3 4 N

P

hoto

ID

1 0 1 1 1 … 1 2 0 0 1 1 … 1 3 0 0 0 1 … 1 4 0 0 0 0 1

…

N-1 0 0 0 0 1 N 0 0 0 0 0

Matching Pair Matrix

If (i,j) = 1, then match photo i to photo j

Matching Graph (notional)

(1,10)

(2,12) (3,15)

(2,20)

(4,75)

(1,50) (5,81)

Vertices described by tuple (Photo ID, Keypoint ID)

Cluster SfM - 21 SMS 09/12/12

Flight path partitioned using block-cyclic mapping:

•  N photos mapped to M partitions

•  Assume photo set covers one revolution

•  Angular gap between photos should be less than 30 degrees (to help with SIFT matching)

•  Photos per partition (N/M) must be greater than 25

•  Block size and partition count set automatically

Bundle Adjustment Data Mapping

Reconstructing partial photo sets independently creates challenges: •  Point clouds are in different arbitrary coordinate systems

•  Many points are duplicated across the partial point clouds

•  3D points have shorter view lists (fewer iterative improvements) Flight P

ath Less than 30°

Example: Block size: 2; M=3

Cluster SfM - 22 SMS 09/12/12

Each 3D point has a view list comprised of SIFT keypoints (vertices in G). View lists from different partitions will always be disjoint because photos are mapped to exactly one partition.

Identifying Duplicate Points

(1,10)

(2,12)

(3,15)

(4,75)

(5,81)

View list A

View list B

Subgraph of G

3D points are considered duplicates if any vertices in B are in the neighborhood of A.

Cluster SfM - 23 SMS 09/12/12

Determine rotation, scale and translation based on duplicate points:

•  Robustly eliminate outlier 3D points

•  Solve for transformation matrix that minimizes error

•  Combine duplicate points as a weighted average

Transforming to a Common Space

Sparse point clouds after independent bundle

adjustment on 4 partitions

Fused sparse point cloud after transformation

Cluster SfM - 24 SMS 09/12/12

Final Point Cloud

Cluster SfM - 25 SMS 09/12/12

•  Feature-based 3D Reconstruction •  Reconstructing Aerial Video •  Cluster-based Structure from Motion •  Results

Outline

Cluster SfM - 26 SMS 09/12/12

Runtime Performance

0

20

40

60

100 200 300 400

Run

time

(hou

rs)

Image Set Size

Bundler (single-core)

Feat. Extr. Feat. Match Bundle Adj.

0

10

20

30

100 200 300 400

Run

time

(hou

rs)

Image Set Size

VisualSFM (multi-core & GPU)

Feat. Extr. Feat. Match Bundle Adj.

0

0.2

0.4

0.6

0.8

1

100 200 300 400

Run

time

(hou

rs)

Image Set Size

Cluster-based SfM (64 cores)

Feat. Match & Extr. Bundle Adj. Fuse

Cluster SfM - 27 SMS 09/12/12

Speed-Up & Scalability

Cluster SfM Scalability (400 photos)

Cluster Size 8 16 32 64

Partitions 8 16 16 16 Frame rate (per min.) 1.2 2.4 4.3 7.6

Points per Frame (x103) 2.0 .87 .87 .87

0

5

10

15

20

25

30

35

40

45

0 16 32 48 64

Spee

d-up

(t(1

)/t(Np)

)

Number of Processors (Np)

Cluster-based SfM Speed-up

N=75

100

200

300

400

Cluster SfM - 28 SMS 09/12/12

Reconstruction Quality

RMS Error of Reconstructed Points (meters) Input size (N) 100 200 300 400

VisualSFM 0.20 7.15 1.05 0.44

Cluster-based SfM

4 Partitions 0.10

8 Partitions 0.27

12 Partitions 0.26

16 Partitions 0.24

0

50

100

150

200

250

300

50 100 150 200 250 300 350 400

Poin

ts (x

1,00

0)

Image Set Size

Sparse Point Cloud Density

Bundler

VisualSFM

Cluster (N/M=25)

Cluster SfM - 29 SMS 09/12/12

•  Creating 3D reconstructions of photo collections scales well to the cluster –  64-core cluster shows 14x speedup over multi-core workstation with

GPU

•  Our algorithm effectively parallelizes reconstruction for the case of aerial video around a ground point

•  For algorithms that aren’t “pleasantly parallel”, deep knowledge or application and computing environment necessary to parallelize

•  Future work: –  Reduce feature matching complexity for different types of input sets –  Point cloud geo-registration by matching against reference data

(e.g. LIDAR or satellite imagery)

Summary

Cluster SfM - 30 SMS 09/12/12

Scott Sawyer (scott.sawyer@ll.mit.edu)

SIGMA Team:

•  Karl Ni

•  Alex Vasile

•  Nick Armstrong-Crews

•  Ethan Phelps

•  Nadya Bliss

•  Prof. Noah Snavely (Cornell University)

Acknowledgements