Lecture: RANSAC and feature detectorsvision.stanford.edu/teaching/cs131_fall1718/files/06_ransac.pdf · RANSAC Line Fitting Example • Task: Estimate the best line ... 5 4 6 12 17

Lecture 6 -Stanford University

Lecture:RANSACandfeaturedetectors

JuanCarlosNiebles andRanjayKrishnaStanfordVisionandLearningLab

12-Oct-171


Whatwewilllearntoday?

• Amodelfittingmethodforedgedetection– RANSAC

• Localinvariantfeatures– Motivation– Requirements,invariances

• Keypoint localization– Harriscornerdetector

12-Oct-172


Whatwewilllearntoday

• Amodelfittingmethodforlinedetection– RANSAC



12-Oct-173


FittingasSearchinParametricSpace

• Chooseaparametricmodeltorepresentasetoffeatures

• Membershipcriterionisnotlocal– Can’ttellwhetherapointbelongstoagivenmodeljustbylookingatthatpoint.

• Threemainquestions:– Whatmodelrepresentsthissetoffeaturesbest?– Whichofseveralmodelinstancesgetswhichfeature?– Howmanymodelinstancesarethere?

• Computationalcomplexityisimportant– Itisinfeasibletoexamineeverypossiblesetofparametersandeverypossiblecombinationoffeatures

Source:L.Lazebnik

12-Oct-174


Example:LineFitting• Whyfitlines?Manyobjectscharacterizedbypresenceofstraightlines

• Wait,whyaren’twedonejustbyrunningedgedetection?Slide credit: Kristen Grauman

12-Oct-175


DifficultyofLineFitting• Extraedgepoints(clutter),multiplemodels:– Whichpointsgowithwhichline,ifany?

• Onlysomepartsofeachlinedetected,andsomepartsaremissing:– Howtofindalinethatbridgesmissingevidence?

• Noiseinmeasurededgepoints,orientations:– Howtodetecttrueunderlyingparameters?

Slide credit: Kristen Grauman

12-Oct-176


Voting

• It’snotfeasibletocheckallcombinationsoffeaturesbyfittingamodeltoeachpossiblesubset.

• Votingisageneraltechniquewhereweletthefeaturesvoteforallmodelsthatarecompatiblewithit.– Cyclethroughfeatures,castvotesformodelparameters.– Lookformodelparametersthatreceivealotofvotes.

• Noise&clutterfeatureswillcastvotestoo,but typicallytheirvotesshouldbeinconsistentwiththemajorityof“good”features.

• Okifsomefeaturesnotobserved,asmodelcanspanmultiplefragments.

Slide credit: Kristen Grauman

12-Oct-177


RANSAC[Fischler &Bolles 1981]

• RANdom SAmple Consensus

• Approach:wewanttoavoidtheimpactofoutliers,solet’slookfor“inliers”,anduseonlythose.

• Intuition:ifanoutlierischosentocomputethecurrentfit,thentheresultinglinewon’thavemuchsupportfromrestofthepoints.

Slid

e cr

edit

: Kr

iste

n G

raum

an

12-Oct-178


RANSACloop:1. Randomlyselectaseedgroup ofpointsonwhichtobase

transformationestimate(e.g.,agroupofmatches)2. Computetransformationfromseedgroup3. Findinlierstothistransformation4. Ifthenumberofinliersissufficientlylarge,re-compute

least-squaresestimateoftransformationonalloftheinliers

• Keepthetransformationwiththelargestnumberofinliers

Slid

e cr

edit

: Kr

iste

n G

raum

an

12-Oct-179

RANSAC[Fischler &Bolles 1981]


RANSACLineFittingExample

• Task:Estimatethebestline– Howmanypointsdoweneedtoestimatetheline?

Slid

e cr

edit

: Ji

nxia

ngCh

ai

12-Oct-1710



• Task:Estimatethebestline

Sampletwopoints

Slid

e cr

edit

: Ji

nxia

ngCh

ai

12-Oct-1711




Fitalinetothem

Slid

e cr

edit

: Ji

nxia

ngCh

ai

12-Oct-1712




Total number of points within a threshold of line. Sl

ide

cred

it:

Jinx

iang

Chai

12-Oct-1713




Total number of points within a threshold of line.

“7 inlier points”

Slid

e cr

edit

: Ji

nxia

ngCh

ai

12-Oct-1714




Repeat,untilwegetagoodresult.

Slid

e cr

edit

: Ji

nxia

ngCh

ai

12-Oct-1715




Repeat,untilwegetagoodresult.

“11 inlier points”

Slid

e cr

edit

: Ji

nxia

ngCh

ai

12-Oct-1716

Lecture 6 -Stanford University 12-Oct-1717


RANSAC:Howmanysamples?• Howmanysamplesareneeded?

– Supposew isfractionofinliers(pointsfromline).– n pointsneededtodefinehypothesis(2forlines)– k sampleschosen.

• Prob.thatasinglesampleofn pointsiscorrect:

• Prob.thatallk samplesfailis:

Þ Choosek highenoughtokeepthisbelowdesiredfailurerate.

nwknw )1( -

Slide credit: David Lowe

12-Oct-1718


RANSAC:Computedk(p=0.99)Sample

size

n

Proportion of outliers

5% 10% 20% 25% 30% 40% 50%

2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 1177

Slide credit: David Lowe

12-Oct-1719


AfterRANSAC• RANSACdividesdataintoinliersandoutliersandyieldsestimatecomputedfromminimalsetofinliers.

• Improvethisinitialestimatewithestimationoverallinliers(e.g.withstandardleast-squaresminimization).

• Butthismaychangeinliers,soalternatefittingwithre-classificationasinlier/outlier.

Slid

e cr

edit

: D

avid

Low

e

12-Oct-1720


RANSAC:ProsandCons

• Pros:– Generalmethodsuitedforawiderangeofmodelfittingproblems

– Easytoimplementandeasytocalculateitsfailurerate• Cons:

– Onlyhandlesamoderatepercentageofoutlierswithoutcostblowingup

– Manyrealproblemshavehighrateofoutliers(butsometimesselectivechoiceofrandomsubsetscanhelp)

• Avotingstrategy,TheHoughtransform,canhandlehighpercentageofoutliers

12-Oct-1721






12-Oct-1722

Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004


Imagematching:achallengingproblem

12-Oct-1723


by Diva Sian

by swashford

Slid

e cr

edit

: St

eve

Seit

z

12-Oct-1724

Imagematching:achallengingproblem


HarderCase

by Diva Sian by scgbt

Slid

e cr

edit

: St

eve

Seit

z

12-Oct-1725


HarderStill?

NASA Mars Rover images

Slid

e cr

edit

: St

eve

Seit

z

12-Oct-1726


AnswerBelow(Lookfortinycoloredsquares)

NASAMarsRoverimageswithSIFTfeaturematches(FigurebyNoahSnavely) Sl

ide

cred

it:

Stev

e Se

itz

12-Oct-1727


Motivationforusinglocalfeatures• Globalrepresentationshavemajorlimitations• Instead,describeandmatchonlylocalregions• Increasedrobustnessto

– Occlusions

– Articulation

– Intra-categoryvariations

θqφ

dq

φθ

d

12-Oct-1728


GeneralApproachN

pixe

ls

N pixels

Similarity measureAf

e.g. color

Bf

e.g. color

A1

A2 A3

Tffd BA <),(

1. Find a set of distinctive key-points

3. Extract and normalize the region content

2. Define a region around each keypoint

4. Compute a local descriptor from the normalized region

5. Match local descriptors

Slid

e cr

edit

: Ba

stia

n Le

ibe

12-Oct-1729


CommonRequirements• Problem1:

– Detectthesamepointindependently inbothimages

No chance to match!

We need a repeatable detector! Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

12-Oct-1730


CommonRequirements• Problem1:

– Detectthesamepointindependently inbothimages

• Problem2:– Foreachpointcorrectlyrecognizethecorrespondingone

We need a reliable and distinctive descriptor!

?

Slid

e cr

edit

: D

arya

Fro

lova

, D

enis

Sim

akov

12-Oct-1731


Invariance:GeometricTransformations

Slid

e cr

edit

: St

eve

Seit

z

12-Oct-1732


LevelsofGeometricInvariance

Slid

e cr

edit

: Ba

stia

n Le

ibe

12-Oct-1733

CS131 CS231a


Invariance:PhotometricTransformations

• Oftenmodeledasalineartransformation:– Scaling+Offset

Slid

e cr

edit

: Ti

nne

Tuyt

elaa

rs

12-Oct-1734


Requirements• Regionextractionneedstoberepeatable andaccurate

– Invarianttotranslation,rotation,scalechanges– Robust or covarianttoout-of-plane(»affine)transformations– Robust tolightingvariations,noise,blur,quantization

• Locality:Featuresarelocal,thereforerobusttoocclusionandclutter.

• Quantity:Weneedasufficientnumberofregionstocovertheobject.

• Distinctiveness:Theregionsshouldcontain“interesting”structure.

• Efficiency:Closetoreal-timeperformance.

Slid

e cr

edit

: Ba

stia

n Le

ibe

12-Oct-1735


ManyExistingDetectorsAvailable

• Hessian&Harris [Beaudet ‘78],[Harris‘88]

• Laplacian,DoG [Lindeberg ‘98],[Lowe‘99]

• Harris-/Hessian-Laplace [Mikolajczyk &Schmid ‘01]

• Harris-/Hessian-Affine [Mikolajczyk &Schmid ‘04]

• EBRandIBR [Tuytelaars &VanGool ‘04]

• MSER [Matas ‘02]

• SalientRegions [Kadir &Brady‘01]

• Others…

• ThosedetectorshavebecomeabasicbuildingblockformanyrecentapplicationsinComputerVision.

Slid

e cr

edit

: Ba

stia

n Le

ibe

12-Oct-1736






12-Oct-1737

Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004


Keypoint Localization

• Goals:– Repeatabledetection– Preciselocalization– InterestingcontentÞ Lookfortwo-dimensionalsignalchanges

Slid

e cr

edit

: Ba

stia

n Le

ibe

12-Oct-1738


FindingCorners

• Keyproperty:– Intheregionaroundacorner,imagegradienthastwoormoredominantdirections

• Cornersarerepeatable anddistinctive

C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“Proceedings of the 4th Alvey Vision Conference, 1988.

Slid

e cr

edit

: Sv

etla

na L

azeb

nik

12-Oct-1739


CornersasDistinctiveInterestPoints• Designcriteria

– Weshouldeasilyrecognizethepointbylookingthroughasmallwindow(locality)

– Shiftingthewindowinany direction shouldgivealargechange inintensity(goodlocalization)

“edge”:no change along the edge direction

“corner”:significant change in all directions

“flat” region:no change in all directions Sl

ide

cred

it:

Alyo

sha

Efro

s

12-Oct-1740


Cornersversusedges

12-Oct-1741

Large

Large

Small

Large

Small

Small

Corner

Edge

Nothing


Cornersversusedges

12-Oct-1742

??

??Corner


HarrisDetectorFormulation

• Changeofintensityfortheshift[u,v]:

E(u,v) = w(x, y) I (x +u, y + v)− I (x, y)"# $%2

x ,y∑

IntensityShifted intensity

Window function

orWindow function w(x,y) =

Gaussian1 in window, 0 outside

Slid

e cr

edit

: Ri

ck S

zelis

ki

12-Oct-1743


HarrisDetectorFormulation• Thismeasureofchangecanbeapproximatedby:

whereM isa2´2matrixcomputedfromimagederivatives:

úû

ùêë

é»

vu

MvuvuE ][),(

M

Sumoverimageregion– theareawearecheckingforcorner

Gradient with respect to x, times gradient with respect to y

2

2,( , ) x x y

x y x y y

I I IM w x y

I I Ié ù

= ê úê úë û

å

Slid

e cr

edit

: Ri

ck S

zelis

ki

12-Oct-1744


HarrisDetectorFormulation

IxImage I IxIyIy

whereM isa2´2matrixcomputedfromimagederivatives:

M

Sumoverimageregion– theareawearecheckingforcorner

Gradient with respect to x, times gradient with respect to y

2

2,( , ) x x y

x y x y y

I I IM w x y

I I Ié ù

= ê úê úë û

å

Slid

e cr

edit

: Ri

ck S

zelis

ki

12-Oct-1745


WhatDoesThisMatrixReveal?• First,let’sconsideranaxis-alignedcorner:

• Thismeans:– Dominantgradientdirectionsalignwithx ory axis– Ifeitherλ iscloseto0,thenthisisnotacorner,solookforlocationswherebotharelarge.

• Whatifwehaveacornerthatisnotalignedwiththeimageaxes?

úû

ùêë

é=

úúû

ù

êêë

é=

åååå

2

12

2

00l

l

yyx

yxx

IIIIII

M

Slid

e cr

edit

: D

avid

Jac

obs

12-Oct-1746


WhatDoesThisMatrixReveal?• First,let’sconsideranaxis-alignedcorner:

• Thismeans:– Dominantgradientdirectionsalignwithx ory axis– Ifeitherλ iscloseto0,thenthisisnotacorner,solookforlocationswherebotharelarge.

• Whatifwehaveacornerthatisnotalignedwiththeimageaxes?

úû

ùêë

é=

úúû

ù

êêë

é=

åååå

2

12

2

00l

l

yyx

yxx

IIIIII

M

Slid

e cr

edit

: D

avid

Jac

obs

12-Oct-1747


GeneralCase

• SinceM issymmetric,wehave

• WecanvisualizeM asanellipsewithaxislengthsdeterminedbytheeigenvaluesandorientationdeterminedbyR

RRM úû

ùêë

é= -

2

11

00l

l

Direction of the slowest change

Direction of the fastest change

(lmax)-1/2

(lmin)-1/2

adap

ted

from

Dar

ya F

rolo

va,

Den

is S

imak

ov

(Eigenvalue decomposition)

12-Oct-1748


InterpretingtheEigenvalues• ClassificationofimagepointsusingeigenvaluesofM:

l1

“Corner”l1 andl2 arelarge,l1 ~ l2;E increasesinalldirections

l1 andl2 aresmall;E isalmostconstantinalldirections “Edge”

l1 >> l2

“Edge”l2 >> l1

“Flat”region

Slid

e cr

edit

: Kr

iste

n G

raum

an

l2

12-Oct-1749


CornerResponseFunction

• Fastapproximation– Avoidcomputingthe

eigenvalues– α:constant

(0.04to0.06)

l2

“Corner”θ > 0

“Edge”θ < 0

“Edge”θ < 0

“Flat”region

θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2

l1Slid

e cr

edit

: Kr

iste

n G

raum

an

12-Oct-1750


WindowFunctionw(x,y)

• Option1:uniformwindow– Sumoversquarewindow

– Problem:notrotationinvariant

• Option2:SmoothwithGaussian– Gaussianalreadyperformsweightedsum

– Resultisrotationinvariant

1 in window, 0 outside

2

2,( , ) x x y

x y x y y

I I IM w x y

I I Ié ù

= ê úê úë û

å

Gaussian

2

2,

x x y

x y x y y

I I IM

I I Ié ù

= ê úê úë û

å

2

2( ) x x y

x y y

I I IM g

I I Is

é ù= *ê ú

ê úë û

Slid

e cr

edit

: Ba

stia

n Le

ibe

12-Oct-1751


Summary:HarrisDetector[Harris88]

• Computesecondmomentmatrix(autocorrelationmatrix)

1. Image derivatives

Ix Iy

2. Square of derivatives

Ix2 Iy2 IxIy

3. Gaussian filter g(sI) g(Ix2) g(Iy2) g(IxIy)

R

2

2

( ) ( )( , ) ( )

( ) ( )x D x y D

I D Ix y D y D

I I IM g

I I Is s

s s ss s

é ù= *ê ú

ê úë û

2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g Ia= - - +

θ = det[M (σ I ,σ D )]−α[trace(M (σ I ,σ D ))]2

4. Cornerness function – two strong eigenvalues

5. Perform non-maximum suppression

Slid

e cr

edit

: Kr

ysti

anM

ikol

ajcz

yk

12-Oct-1752


HarrisDetector:Workflow

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

12-Oct-1753


HarrisDetector:Workflow- computercornerresponsesθ

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

12-Oct-1754


Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

HarrisDetector:Workflow- Takeonlythelocalmaximaofθ,whereθ>threshold

12-Oct-1755


HarrisDetector:Workflow- ResultingHarrispoints

Slid

e ad

apte

d fr

om D

arya

Fro

lova

, D

enis

Sim

akov

12-Oct-1756


HarrisDetector– Responses[Harris88]

Effect: A very precise corner detector.

Slid

e cr

edit

: Kr

ysti

anM

ikol

ajcz

yk

12-Oct-1757



Slid

e cr

edit

: Kr

ysti

anM

ikol

ajcz

yk

12-Oct-1758



Slid

e cr

edit

: Kr

iste

n G

raum

an

• Resultsarewellsuitedforfindingstereocorrespondences

12-Oct-1759


HarrisDetector:Properties

• Translationinvariance?

Slid

e cr

edit

: Kr

iste

n G

raum

an

12-Oct-1760



• Translationinvariance• Rotationinvariance?

Ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner response θ is invariant to image rotation

Slid

e cr

edit

: Kr

iste

n G

raum

an

12-Oct-1761



• Translationinvariance• Rotationinvariance• Scaleinvariance?

Not invariant to image scale!

All points will be classified as edges!

Corner

Slid

e cr

edit

: Kr

iste

n G

raum

an

12-Oct-1762


Whatwearelearnedtoday?




12-Oct-1763

Documents

Lecture: RANSAC and feature detectorsvision.stanford.edu/teaching/cs131_fall1718/files/06_ransac.pdf · RANSAC Line Fitting Example • Task: Estimate the best line ... 5 4 6 12 17