Image Matching Fundamentals
Presented by:Dr. Hamid Ebadi
Historical Remarks
Terminology, Working Definitions
Conjugate EntityMatching EntitySimilarity MeasureMatching MethodMatching Strategy
Relationship between Matching Methods and Matching Entities
Symbolic descriptionCost function symbolic
Edges, RegionsCost function Feature-based
Gray levelsCorrelation,Least-squares
Area-based
Matching EntitiesSimilarity MeasureMatching Method
Problem Statement
1- Select a matching entity in one image2- Find its conjugate entity in the other image3- Compute the 3d location of the matched entity in object space4- Assess the quality of the match
Fundamental Problems of Image Matching
Search Space, Uniqueness of Matching EntityCombinatorial ExplosionAmbiguity
Approximations, Constraints and Assumption
Geometric Distortion of Matching EntitiesNoiseChanging illuminationReflection propertiesCentral projectionRelief
Approximations, Constraints and Assumption
Pull-in range
SecondaryMinimum
Geometric Distortion of Matching Entities
a
b
Geometric Distortion Due to Orientation Parameters
a) Scale Difference Between the two Images
b) Difference Rotation Angles Between the two Images
Scale Difference Between the two Images
a b
Difference Rotation Angles Between the two Images
a b c
Effect of Tilted Surface on Geometric Distortion
a b
Effect of Relief on Geometric Distortions
Solutions to the Fundamental Problems
Search Space and Approximations
Epipolar LinesVertical Line LocusHierarchical Approach
Epipolar Lines
Epipolar Lines
ppp pxxx −= '''
fZH
BpxpD
p −=
pD
Dopp ZH
Hbxx−
−= '''
""LU xxS −=
Epipolar Lines
UD
DoUU ZH
Hbxx−
−= '"
LD
DoLL ZH
Hbxx−
−= '"
( )( )UDLDDo ZHZH
ZHbs−−
=δ
( )2pD
oZH
ZHbs D
−≈
δ
Steps for Implementing MatchingScheme Along Epipolar Lines
Select Matching entity in one image, p’Estimate the Elevation, ,of the matching entity and its uncertainty
rangeCompute approximate location P’’Compute search interval, sPerform matching within search interval Analyze the similarity measures obtained in previous step for
determining conjugate location p’’
pZZδ
Vertical Line Locus
Concept of Matching Along the Vertical Line Loci v’ and V”
Combining Vertical Line Locus and Epipolar Line Method
Implementation procedure for VLL
1-Select the X-Y position of a point P in object space and estimate the elevation Zp and it lower and upper bounds ZL, ZR2-Begin the matching process by projecting Xp,Yp,ZLback to both images3-Perform a similarity measure at the image positions X’L, Y’L, and X’’L, Y’’L4-Move along the vertical line in suitable steps dZ=(Zu-ZL)/n5-Project the object points Xp,Yp,ZL+dz back to the images and repeat the similarity measure6-Repeat step 4 until Zu is reached7-Analyze the similarity measures obtained in every step and determine the maximum/minimum for the conjugate position
Hierarchical Approach
Uniqueness of Matching Entity
Variance
Autocorrelation
Entropy
Aera-Based Matching
Template Search Window
Matching Window
Area-Based Matching
Location of TemplateSize of Template
Location and Size of Search WindowAcceptance Criteria Quality Control
Area-Based Matching
CorrelationCross-Correlation FactorMaximum Correlation Factor
Least Squares Matching
Cross-Correlation Factor
RL
LR
σσσρ =
LRσ Covariance of image patches L and R
Lσ Standard deviation of image patch L
Rσ Standard deviation of image patch R
Cross-Correlation Factor
( )mn
yxgg
n
i
m
j jiLL .
,1 1∑ ∑= =
−
=( )
mn
yxgg
n
i
m
j jiRR .
,1 1∑ ∑= =
−
=
( )1.
,1 1
2
−
−
=∑ ∑= =
−
mn
gyxgn
i
m
j LjiL
Lσ( )1.
,1 1
2
−
−
=∑ ∑= =
−
mn
gyxgn
i
m
j RjiR
Rσ
( ) ( )1.
,,1 1
−
−
−
=∑ ∑= =
−−
mn
gyxggyxgn
i
m
j RjiRLjiL
LRσ
Maximum Correlation Factor
a b c
Steps of Area-Based Matching with Correlation
Steps of Area-Based Matching with Correlation
Select the centre of the template in one imageDetermine approximate locations for the conjugate position in the other imageFor both the template and the correlation window, determine the minimum size which passes the uniqueness criterion, select the larger of the two values as the window size for the current matching locationCompute the correlation coefficient Pr,c for all positions r,c of the correlation window within the search windowAnalyze the correlation factors. A minimum threshold must be reached for a valid match, Apart from the maximum, determine the distinctness as a quality measureRepeat above steps for new template location until all positions to be matched are visitedAnalyze the matching results on a global base for consistency and/or compatibility with a priori scene knowledge
Least Squares Matching
To minimize the gray level differences between the template and the matching windows whereby the position and the shape of the matching window are parameters to be determined in adjustment process
Least Squares Matching
Mathematical Model
Adjustment Procedure
Least Squares Matching
Factors affecting gray level differences between template and matching windows
Illumination and reflectance differences between the two imagesthe photographic development process and scanner in case of digitized photographsGeometric distortions
Mathematical ModelTo compensate for differences in brightness and contrast, a transformation of the matching window is introduced
( ){ } ( )jimrrjimTR ,., 10 +=
Mathematical Model
( ) ( ) jiTmyxm G ,, =
( )jiTx x ,= ( )jiTy y ,=
Mathematical Model
Since the two image patches are related via the common surface patch by the central projection, central projection model should be usedMatching windows and surface are smallIt may be approximated by a plane
Mathematical Model(Affine Transformation)
Affine Transformation:
Observation Equation:
jtittx .. 210 ++=
jtitty .. 543 ++=
( ) ( ) ( )yxmjitjir ,,, −=
Mathematical Model(Affine Transformation)
Linearization of the model:
( ) ( ) ( ) ...,,, 22
11
00
0 +
∂∂
+∂∂
+∂∂
∂∂
+= ttTt
tTt
tT
Tyxmyxmyxm xxx
x
δδδ
( )
∂∂
+∂
∂+
∂
∂
∂∂
55
44
33
, ttTt
tT
ttT
Tyxm xyy
y
δδδ
Mathematical Model(Linearization process)
( )x
x
gTyxm=
∂∂ ,
( )y
y
gTyxm=
∂∂ ,
10
=∂∂tTx
xtTx =∂∂
1
ytTx =∂∂
21
3
=∂∂tTx
xtTx =∂∂
4
ytTx =∂∂
5
( ) ( )jimyxm ,, 00 = ( ) ( )jimjitc ,, −=
Mathematical Model(Linearization process)
Adjustment procedure
ctAr +=∧
( ) cAPAAt TT 1−∧
=
6.Pr2
−=
mnrTσ
Adjustment Procedure(Resampling)
LSM procedure
Select the centre of the template in one image (Rt,Ct)Determine approximate locations for the matching window (RM,CM)Determine minimum size of template and matching window which passes the uniqueness criteria, select the larger of the two values as the window sizeStart first iterations with matching window at location RM,CMTransform matching window and determine the gray values for the tessellation (Resampling)Repeat adjustment and resampling sequence until termination criteria is reached. Assess quality of conjugate pointRepeat above steps for new position of template
References
T. Schenk, “ Digital Photogrammetry”, Terra Science, 1999M. Kasser and W. Egels, “ Digital Photogrammetry”, Taylor and Francis, 2002H. Ebadi, “ Advanced Analytical Aerial Triangulation”, Lecture Note, K.N.ToosiUniversity of Technology, 1999T.C.Tang, “Digital Image Correlation”, UCSEm Report, 1988