View
9
Download
1
Category
Preview:
Citation preview
Lecture 6 - Fei-Fei Li
Lecture6:FindingFeatures(part1/2)
Dr.JuanCarlosNieblesStanfordAILab
ProfessorFei-FeiLiStanfordVisionLab
10-Oct-161
Lecture 6 - Fei-Fei Li
Whatwewilllearntoday?
• Localinvariantfeatures– MoOvaOon– Requirements,invariances
• KeypointlocalizaOon– Harriscornerdetector
• ScaleinvariantregionselecOon– AutomaOcscaleselecOon– Difference-of-Gaussian(DoG)detector
• SIFT:animageregiondescriptor
10-Oct-162
Nextlecture(#7)
Lecture 6 - Fei-Fei Li
Whatwewilllearntoday?
• Localinvariantfeatures– MoOvaOon– Requirements,invariances
• KeypointlocalizaOon– Harriscornerdetector
• ScaleinvariantregionselecOon– AutomaOcscaleselecOon– Difference-of-Gaussian(DoG)detector
• SIFT:animageregiondescriptor
10-Oct-163
Somebackgroundreading:RickSzeliski,Chapter4.1.1;DavidLowe,IJCV2004
Lecture 6 - Fei-Fei Li
by Diva Sian
by swashford
Slid
e cr
edit
: St
eve
Seit
z
10-Oct-165
Imagematching:achallengingproblem
Lecture 6 - Fei-Fei Li
AnswerBelow(LookforOnycoloredsquares)
NASAMarsRoverimageswithSIFTfeaturematches(FigurebyNoahSnavely) Sl
ide
cred
it:
Stev
e Se
itz
10-Oct-168
Lecture 6 - Fei-Fei Li
MoOvaOonforusinglocalfeatures• GlobalrepresentaOonshavemajorlimitaOons• Instead,describeandmatchonlylocalregions• Increasedrobustnessto
– Occlusions
– ArOculaOon
– Intra-categoryvariaOons
θqφ
dq
φ
θ
d
10-Oct-169
Lecture 6 - Fei-Fei Li
GeneralApproachN
pix
els
N pixels
Similarity measure Af
e.g. color
Bf
e.g. color
B1B2
B3A1
A2 A3
Tffd BA <),(
1. Find a set of distinctive key- points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1610
Lecture 6 - Fei-Fei Li
CommonRequirements• Problem1:
– Detectthesamepointindependentlyinbothimages
No chance to match!
We need a repeatable detector! Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1611
Lecture 6 - Fei-Fei Li
CommonRequirements• Problem1:
– Detectthesamepointindependentlyinbothimages
• Problem2:– Foreachpointcorrectlyrecognizethecorrespondingone
We need a reliable and distinctive descriptor!
?
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1612
Lecture 6 - Fei-Fei Li
Invariance:GeometricTransformaOons
Slid
e cr
edit
: St
eve
Seit
z
10-Oct-1613
Lecture 6 - Fei-Fei Li
LevelsofGeometricInvariance
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1614
CS131 CS231a
Lecture 6 - Fei-Fei Li
Invariance:PhotometricTransformaOons
• OfenmodeledasalineartransformaOon:– Scaling+Offset
Slid
e cr
edit
: Ti
nne
Tuyt
elaa
rs
10-Oct-1615
Lecture 6 - Fei-Fei Li
Requirements• RegionextracOonneedstoberepeatableandaccurate
– InvarianttotranslaOon,rotaOon,scalechanges– Robustorcovarianttoout-of-plane(�affine)transformaOons– RobusttolighOngvariaOons,noise,blur,quanOzaOon
• Locality:Featuresarelocal,thereforerobusttoocclusionandcluier.
• QuanOty:Weneedasufficientnumberofregionstocovertheobject.
• DisOncOveness:Theregionsshouldcontain“interesOng”structure.
• Efficiency:Closetoreal-Omeperformance.
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1616
Lecture 6 - Fei-Fei Li
ManyExisOngDetectorsAvailable
• Hessian&Harris [Beaudet‘78],[Harris‘88]• Laplacian,DoG [Lindeberg‘98],[Lowe‘99]• Harris-/Hessian-Laplace [Mikolajczyk&Schmid‘01]• Harris-/Hessian-Affine [Mikolajczyk&Schmid‘04]• EBRandIBR [Tuytelaars&VanGool‘04]• MSER [Matas‘02]• SalientRegions [Kadir&Brady‘01]• Others…
• Thosedetectorshavebecomeabasicbuildingblockformanyrecentapplica8onsinComputerVision.
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1617
Lecture 6 - Fei-Fei Li
KeypointLocalizaOon
• Goals:
– RepeatabledetecOon– PreciselocalizaOon– InteresOngcontent
�Lookfortwo-dimensionalsignalchanges
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1618
Lecture 6 - Fei-Fei Li
FindingCorners
• Keyproperty:– Intheregionaroundacorner,imagegradienthastwoormoredominantdirecOons
• Cornersarerepeatableanddis8nc8ve
C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“ Proceedings of the 4th Alvey Vision Conference, 1988.
Slid
e cr
edit
: Sv
etla
na L
azeb
nik
10-Oct-1619
Lecture 6 - Fei-Fei Li
CornersasDisOncOveInterestPoints• Designcriteria
– Weshouldeasilyrecognizethepointbylookingthroughasmallwindow(locality)
– Shifingthewindowinanydirec8onshouldgivealargechangeinintensity(goodlocaliza8on)
“edge”: no change along the edge direction
“corner”: significant change in all directions
“flat” region: no change in all directions Sl
ide
cred
it:
Alyo
sha
Efro
s
10-Oct-1620
Lecture 6 - Fei-Fei Li
HarrisDetectorFormulaOon
• Changeofintensityfortheshif[u,v]:E(u,v) = w(x, y) I (x +u, y + v)− I (x, y)"# $%
2
x ,y∑
Intensity Shifted intensity
Window function
or Window function w(x,y) =
Gaussian 1 in window, 0 outside
Slid
e cr
edit
: Ri
ck S
zelis
ki
10-Oct-1621
Lecture 6 - Fei-Fei Li
HarrisDetectorFormulaOon• Thismeasureofchangecanbeapproximatedby:
whereMisa2�2matrixcomputedfromimagederivaOves:
⎥⎦
⎤⎢⎣
⎡≈
vu
MvuvuE ][),(
M
Sumoverimageregion–theareawearecheckingforcorner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
Slid
e cr
edit
: Ri
ck S
zelis
ki
10-Oct-1622
Lecture 6 - Fei-Fei Li
HarrisDetectorFormulaOon
Ix Image I IxIy Iy
whereMisa2�2matrixcomputedfromimagederivaOves:
M
Sumoverimageregion–theareawearecheckingforcorner
Gradient with respect to x, times gradient with respect to y
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
Slid
e cr
edit
: Ri
ck S
zelis
ki
10-Oct-1623
Lecture 6 - Fei-Fei Li
WhatDoesThisMatrixReveal?• First,let’sconsideranaxis-alignedcorner:
• Thismeans:– DominantgradientdirecOonsalignwithxoryaxis– Ifeitherλiscloseto0,thenthisisnotacorner,solookforlocaOonswherebotharelarge.
• Whatifwehaveacornerthatisnotalignedwiththeimageaxes?
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
∑∑∑∑
2
12
2
00λ
λ
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
10-Oct-1624
Lecture 6 - Fei-Fei Li
WhatDoesThisMatrixReveal?• First,let’sconsideranaxis-alignedcorner:
• Thismeans:– DominantgradientdirecOonsalignwithxoryaxis– Ifeitherλiscloseto0,thenthisisnotacorner,solookforlocaOonswherebotharelarge.
• Whatifwehaveacornerthatisnotalignedwiththeimageaxes?
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎦
⎤
⎢⎢⎣
⎡=
∑∑∑∑
2
12
2
00λ
λ
yyx
yxx
IIIIII
M
Slid
e cr
edit
: D
avid
Jac
obs
10-Oct-1625
Lecture 6 - Fei-Fei Li
GeneralCase
• SinceMissymmetric,wehave
• WecanvisualizeMasanellipsewithaxislengthsdeterminedbytheeigenvaluesandorientaOondeterminedbyR
RRM ⎥⎦
⎤⎢⎣
⎡= −
2
11
00λ
λ
Direction of the slowest change
Direction of the fastest change
(�max)-1/2
(�min)-1/2
adap
ted
from
Dar
ya F
rolo
va,
Den
is S
imak
ov
(Eigenvalue decomposition)
10-Oct-1626
Lecture 6 - Fei-Fei Li
InterpreOngtheEigenvalues• ClassificaOonofimagepointsusingeigenvaluesofM:
�1
“Corner” �1and�2arelarge,�1 ~ �2;EincreasesinalldirecOons
�1and�2aresmall;EisalmostconstantinalldirecOons “Edge”
�1 >> �2
“Edge” �2 >> �1
“Flat”region
Slid
e cr
edit
: Kr
iste
n G
raum
an
�2
10-Oct-1627
Lecture 6 - Fei-Fei Li
CornerResponseFuncOon
• FastapproximaOon– AvoidcompuOngtheeigenvalues
– α:constant(0.04to0.06)
�2
“Corner” θ > 0
“Edge” θ < 0
“Edge” θ < 0
“Flat”region
θ = det(M )−α trace(M )2 = λ1λ2 −α(λ1 +λ2 )2
�1 Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1628
Lecture 6 - Fei-Fei Li
WindowFuncOonw(x,y)
• OpOon1:uniformwindow– Sumoversquarewindow
– Problem:notrotaOoninvariant
• OpOon2:SmoothwithGaussian– Gaussianalreadyperformsweightedsum
– ResultisrotaOoninvariant
1 in window, 0 outside
2
2,( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
Gaussian
2
2,
x x y
x y x y y
I I IM
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
2
2( ) x x y
x y y
I I IM g
I I Iσ
⎡ ⎤= ∗⎢ ⎥
⎢ ⎥⎣ ⎦
Slid
e cr
edit
: Ba
stia
n Le
ibe
10-Oct-1629
Lecture 6 - Fei-Fei Li
Summary:HarrisDetector[Harris88]• Computesecondmomentmatrix
(autocorrelaOonmatrix)1. Image derivatives
Ix Iy
2. Square of derivatives
Ix2 Iy2 IxIy
3. Gaussian filter g(sI) g(Ix2) g(Iy2) g(IxIy)
R
2
2
( ) ( )( , ) ( )
( ) ( )x D x y D
I D Ix y D y D
I I IM g
I I Iσ σ
σ σ σσ σ
⎡ ⎤= ∗⎢ ⎥
⎢ ⎥⎣ ⎦
2 2 2 2 2 2( ) ( ) [ ( )] [ ( ) ( )]x y x y x yg I g I g I I g I g Iα= − − +θ = det[M (σ I ,σ D )]−α[trace(M (σ I ,σ D ))]
2
4. Cornerness function – two strong eigenvalues
5. Perform non-maximum suppression
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
10-Oct-1630
Lecture 6 - Fei-Fei Li
HarrisDetector:Workflow
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1631
Lecture 6 - Fei-Fei Li
HarrisDetector:Workflow-computercornerresponsesθ
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1632
Lecture 6 - Fei-Fei Li
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
HarrisDetector:Workflow-Takeonlythelocalmaximaofθ,whereθ>threshold
10-Oct-1633
Lecture 6 - Fei-Fei Li
HarrisDetector:Workflow-ResulOngHarrispoints
Slid
e ad
apte
d fr
om D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1634
Lecture 6 - Fei-Fei Li
HarrisDetector–Responses[Harris88]
Effect: A very precise corner detector.
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
10-Oct-1635
Lecture 6 - Fei-Fei Li
HarrisDetector–Responses[Harris88]
Slid
e cr
edit
: Kr
ysti
an M
ikol
ajcz
yk
10-Oct-1636
Lecture 6 - Fei-Fei Li
HarrisDetector–Responses[Harris88]
Slid
e cr
edit
: Kr
iste
n G
raum
an
• Resultsarewellsuitedforfindingstereocorrespondences
10-Oct-1637
Lecture 6 - Fei-Fei Li
HarrisDetector:ProperOes
• TranslaOoninvariance?
Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1638
Lecture 6 - Fei-Fei Li
HarrisDetector:ProperOes
• TranslaOoninvariance• RotaOoninvariance?
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response θ is invariant to image rotation
Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1639
Lecture 6 - Fei-Fei Li
HarrisDetector:ProperOes
• TranslaOoninvariance• RotaOoninvariance• Scaleinvariance?
Not invariant to image scale!
All points will be classified as edges!
Corner
Slid
e cr
edit
: Kr
iste
n G
raum
an
10-Oct-1640
Lecture 6 - Fei-Fei Li
Whatwehavelearnedtoday?
• Localinvariantfeatures– MoOvaOon– Requirements,invariances
• KeypointlocalizaOon– Harriscornerdetector
• ScaleinvariantregionselecOon– AutomaOcscaleselecOon– Difference-of-Gaussian(DoG)detector
• SIFT:animageregiondescriptor
10-Oct-1641
Nextlecture(#7)
Somebackgroundreading:RickSzeliski,Chapter14.1.1;DavidLowe,IJCV2004
Lecture 6 - Fei-Fei Li
ApplicaOon:ImageSOtching
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1643
Lecture 6 - Fei-Fei Li
ApplicaOon:ImageSOtching
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1644
• Procedure:– Detectfeaturepointsinbothimages– Findcorrespondingpairs– Usethesepairstoaligntheimages
Lecture 6 - Fei-Fei Li
ApplicaOon:ImageSOtching
Slid
e cr
edit
: D
arya
Fro
lova
, D
enis
Sim
akov
10-Oct-1645
• Procedure:– Detectfeaturepointsinbothimages– Findcorrespondingpairs– Usethesepairstoaligntheimages
Recommended