Recognising Panoramas M. Brown and D. Lowe, University of British Columbia

Recognising Panoramas

M. Brown and D. Lowe, University of British Columbia

Introduction

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°

Introduction

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°

Introduction

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°

Why “Recognising Panoramas”?

• 1D Rotations ()– Ordering matching images

• 2D Rotations (, )– Ordering matching images

Overview

• Feature Matching• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions

Overview

• Feature Matching– SIFT Features– Nearest Neighbour Matching

• Image Matching• Bundle Adjustment• Multi-band Blending• Results• Conclusions

Overview

Invariant Features

• Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002

SIFT Features

• Invariant Features– Establish invariant frame

• Maxima/minima of scale-space DOG x, y, s• Maximum of distribution of local gradients

– Form descriptor vector• Histogram of smoothed local gradients• 128 dimensions

• SIFT features are…– Geometrically invariant to similarity transforms,

• some robustness to affine change

– Photometrically invariant to affine changes in intensity

Overview

Nearest Neighbour Matching

• Find k-NN for each feature– k number of overlapping images (we use k = 4)

• Use k-d tree– k-d tree recursively bi-partitions data at mean in the

dimension of maximum variance– Approximate nearest neighbours found in O(nlogn)

Overview

• Feature Matching• Image Matching

– RANSAC for Homography– Probabilistic model for verification

• Bundle Adjustment• Multi-band Blending• Results• Conclusions

Overview

RANSAC for Homography

Overview

Probabilistic model for verification

• Compare probability that this set of RANSAC inliers/outliers was generated by a correct/false image match

– ni = #inliers, nf = #features

– p1 = p(inlier | match), p0 = p(inlier | ~match)

– pmin = acceptance probability

• Choosing values for p1, p0 and pmin

Finding the panoramas

Overview

• Feature Matching• Image Matching• Bundle Adjustment

– Error function

• Multi-band Blending• Results• Conclusions

Overview

– Error function

Error function

• Sum of squared projection errors

– n = #images– I(i) = set of image matches to image i– F(i, j) = set of feature matches between images i,j

– rijk = residual of kth feature match between images i,j

• Robust error function

• Parameterise each camera by rotation and focal length

• This gives pairwise homographies

Homography for Rotation

Bundle Adjustment

• New images initialised with rotation, focal length of best matching image

Bundle Adjustment

• New images initialised with rotation, focal length of best matching image

Overview

– Error function

Overview

Multi-band Blending

• Burt & Adelson 1983– Blend frequency bands over range

Low frequency ( > 2 pixels)

High frequency ( < 2 pixels)

2-band Blending

Linear Blending

2-band Blending

Overview

Results

Overview

Conclusions

• Fully automatic panoramas– A recognition problem…

• Invariant feature based method– SIFT features, RANSAC, Bundle Adjustment, Multi-

band Blending– O(nlogn)

• Future Work– Advanced camera modelling

• radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash …

– Full 3D case – recognising 3D objects/scenes in unordered datasets

http://www.cs.ubc.ca/~mbrown/panorama/panorama.htmlhttp://www.cs.ubc.ca/~mbrown/panorama/panorama.html

Questions?

http://www.cs.ubc.ca/~mbrown/panorama/panorama.htmlhttp://www.cs.ubc.ca/~mbrown/panorama/panorama.html

Analytical computation of derivatives

Levenberg-Marquardt

• Iteration step of form

Recognising Panoramas M. Brown and D. Lowe, University of British Columbia

Documents

Pulpa Panoramas Concentrados Abril 2011

Vienna Animated Panoramas - widescreen

360 Photo & Video Panoramas

Upright Adjustment of 360 Spherical Panoramas...Spherical panoramas (360 panoramas) are images with 360 degrees of horizontal and 180 degrees of vertical ﬁeld of views. In the past,

Fotos de Panoramas 1

AR Panoramas - 3D Panoramas - Virtual/Augmented Reality

Fotos de Panoramas

Multi-perspective Panoramasgraphics.cs.cmu.edu/courses/15-463/2010_fall/Lectures/MultiPerspe… · Panoramas Registration: Brown & Lowe, ICCV’05 Blending: Burt & Adelson, Trans

Homographies and Panoramas - inst.eecs.berkeley.edu

AUTOMATIC ADJUSTMENT OF WIDE -BASE GOOGLE STREET VIEW PANORAMAS · 2016-06-06 · AUTOMATIC ADJUSTMENT OF WIDE -BASE GOOGLE STREET VIEW PANORAMAS KEY WORDS: panoramas, wide baseline,

Homographies and Panoramas - Brown University

Green Computing - Panoramas et perspectives

Shape defined Panoramas

Lecture 15: Panoramas

Light Field based 360o Panoramas

Panoramas 1

Multi-perspective Panoramas

Panoramas vu

Cylindrical panoramas

Panoramas using acrylics - GEORGIA MANSUR