Image Matching Fundamentals

Presented by:Dr. Hamid Ebadi

Historical Remarks

Terminology, Working Definitions

Conjugate EntityMatching EntitySimilarity MeasureMatching MethodMatching Strategy

Relationship between Matching Methods and Matching Entities

Symbolic descriptionCost function symbolic

Edges, RegionsCost function Feature-based

Gray levelsCorrelation,Least-squares

Area-based

Matching EntitiesSimilarity MeasureMatching Method

Problem Statement

1- Select a matching entity in one image2- Find its conjugate entity in the other image3- Compute the 3d location of the matched entity in object space4- Assess the quality of the match

Fundamental Problems of Image Matching

Search Space, Uniqueness of Matching EntityCombinatorial ExplosionAmbiguity

Approximations, Constraints and Assumption

Geometric Distortion of Matching EntitiesNoiseChanging illuminationReflection propertiesCentral projectionRelief

Approximations, Constraints and Assumption

Pull-in range

SecondaryMinimum

Geometric Distortion of Matching Entities

a

b

Geometric Distortion Due to Orientation Parameters

a) Scale Difference Between the two Images

b) Difference Rotation Angles Between the two Images

Scale Difference Between the two Images

a b

Difference Rotation Angles Between the two Images

a b c

Effect of Tilted Surface on Geometric Distortion

a b

Effect of Relief on Geometric Distortions

Solutions to the Fundamental Problems

Search Space and Approximations

Epipolar LinesVertical Line LocusHierarchical Approach

Epipolar Lines

Epipolar Lines

ppp pxxx −= '''

fZH

BpxpD

p −=

pD

Dopp ZH

Hbxx−

−= '''

""LU xxS −=

Epipolar Lines

UD

DoUU ZH

Hbxx−

−= '"

LD

DoLL ZH

Hbxx−

−= '"

( )( )UDLDDo ZHZH

ZHbs−−

=δ

( )2pD

oZH

ZHbs D

−≈

δ

Steps for Implementing MatchingScheme Along Epipolar Lines

Select Matching entity in one image, p’Estimate the Elevation, ,of the matching entity and its uncertainty

rangeCompute approximate location P’’Compute search interval, sPerform matching within search interval Analyze the similarity measures obtained in previous step for

determining conjugate location p’’

pZZδ

Vertical Line Locus

Concept of Matching Along the Vertical Line Loci v’ and V”

Combining Vertical Line Locus and Epipolar Line Method

Implementation procedure for VLL

1-Select the X-Y position of a point P in object space and estimate the elevation Zp and it lower and upper bounds ZL, ZR2-Begin the matching process by projecting Xp,Yp,ZLback to both images3-Perform a similarity measure at the image positions X’L, Y’L, and X’’L, Y’’L4-Move along the vertical line in suitable steps dZ=(Zu-ZL)/n5-Project the object points Xp,Yp,ZL+dz back to the images and repeat the similarity measure6-Repeat step 4 until Zu is reached7-Analyze the similarity measures obtained in every step and determine the maximum/minimum for the conjugate position

Hierarchical Approach

Uniqueness of Matching Entity

Variance

Autocorrelation

Entropy

Aera-Based Matching

Template Search Window

Matching Window

Area-Based Matching

Location of TemplateSize of Template

Location and Size of Search WindowAcceptance Criteria Quality Control

Area-Based Matching

CorrelationCross-Correlation FactorMaximum Correlation Factor

Least Squares Matching

Cross-Correlation Factor

RL

LR

σσσρ =

LRσ Covariance of image patches L and R

Lσ Standard deviation of image patch L

Rσ Standard deviation of image patch R

Cross-Correlation Factor

( )mn

yxgg

n

i

m

j jiLL .

,1 1∑ ∑= =

−

=( )

mn

yxgg

n

i

m

j jiRR .

,1 1∑ ∑= =

−

=

( )1.

,1 1

2

−

−

=∑ ∑= =

−

mn

gyxgn

i

m

j LjiL

Lσ( )1.

,1 1

2

−

−

=∑ ∑= =

−

mn

gyxgn

i

m

j RjiR

Rσ

( ) ( )1.

,,1 1

−

−

−

=∑ ∑= =

−−

mn

gyxggyxgn

i

m

j RjiRLjiL

LRσ

Maximum Correlation Factor

a b c

Steps of Area-Based Matching with Correlation

Steps of Area-Based Matching with Correlation

Select the centre of the template in one imageDetermine approximate locations for the conjugate position in the other imageFor both the template and the correlation window, determine the minimum size which passes the uniqueness criterion, select the larger of the two values as the window size for the current matching locationCompute the correlation coefficient Pr,c for all positions r,c of the correlation window within the search windowAnalyze the correlation factors. A minimum threshold must be reached for a valid match, Apart from the maximum, determine the distinctness as a quality measureRepeat above steps for new template location until all positions to be matched are visitedAnalyze the matching results on a global base for consistency and/or compatibility with a priori scene knowledge

Least Squares Matching

To minimize the gray level differences between the template and the matching windows whereby the position and the shape of the matching window are parameters to be determined in adjustment process

Least Squares Matching

Mathematical Model

Adjustment Procedure

Least Squares Matching

Factors affecting gray level differences between template and matching windows

Illumination and reflectance differences between the two imagesthe photographic development process and scanner in case of digitized photographsGeometric distortions

Mathematical ModelTo compensate for differences in brightness and contrast, a transformation of the matching window is introduced

( ){ } ( )jimrrjimTR ,., 10 +=

Mathematical Model

( ) ( ) jiTmyxm G ,, =

( )jiTx x ,= ( )jiTy y ,=

Mathematical Model

Since the two image patches are related via the common surface patch by the central projection, central projection model should be usedMatching windows and surface are smallIt may be approximated by a plane

Mathematical Model(Affine Transformation)

Affine Transformation:

Observation Equation:

jtittx .. 210 ++=

jtitty .. 543 ++=

( ) ( ) ( )yxmjitjir ,,, −=

Mathematical Model(Affine Transformation)

Linearization of the model:

( ) ( ) ( ) ...,,, 22

11

00

0 +

∂∂

+∂∂

+∂∂

∂∂

+= ttTt

tTt

tT

Tyxmyxmyxm xxx

x

δδδ

( )

∂∂

+∂

∂+

∂

∂

∂∂

55

44

33

, ttTt

tT

ttT

Tyxm xyy

y

δδδ

Mathematical Model(Linearization process)

( )x

x

gTyxm=

∂∂ ,

( )y

y

gTyxm=

∂∂ ,

10

=∂∂tTx

xtTx =∂∂

1

ytTx =∂∂

21

3

=∂∂tTx

xtTx =∂∂

4

ytTx =∂∂

5

( ) ( )jimyxm ,, 00 = ( ) ( )jimjitc ,, −=

Mathematical Model(Linearization process)

Adjustment procedure

ctAr +=∧

( ) cAPAAt TT 1−∧

=

6.Pr2

−=

mnrTσ

Adjustment Procedure(Resampling)

LSM procedure

Select the centre of the template in one image (Rt,Ct)Determine approximate locations for the matching window (RM,CM)Determine minimum size of template and matching window which passes the uniqueness criteria, select the larger of the two values as the window sizeStart first iterations with matching window at location RM,CMTransform matching window and determine the gray values for the tessellation (Resampling)Repeat adjustment and resampling sequence until termination criteria is reached. Assess quality of conjugate pointRepeat above steps for new position of template

References

T. Schenk, “ Digital Photogrammetry”, Terra Science, 1999M. Kasser and W. Egels, “ Digital Photogrammetry”, Taylor and Francis, 2002H. Ebadi, “ Advanced Analytical Aerial Triangulation”, Lecture Note, K.N.ToosiUniversity of Technology, 1999T.C.Tang, “Digital Image Correlation”, UCSEm Report, 1988