..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.

......Point Correspondence Performance Evaluation

Robert M. Haralick

Computer Science, Graduate CenterCity University of New York

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Types of Performance Characterization

White BoxEvaluate Each Component: Component Transfer FunctionNeeds Appropriate Random Perturbation Models for

Algorithm InputsAlgorithm Outputs

Black BoxEmpirical EvaluationNo Knowledge of Component Transfer Functions

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Performance Characterization

An Algorithm hasInputs a and their data typesOutputs b and their data typesA Relationship Between Input and Output

Output b as a function of input aGiven a, b maximizes F (a, b)

Random Perturbation Model for Input aRandom Perturbation Model for Output bGiven Random Perturbation Distribution Acting on aDetermine Random Perturbation Distribution Acting on bDetermine Robustness

Do large perturbations on a small fraction of the input datacause a small perturbation on the output data?

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Performance Characterization

Performance characterization has to do with establishing thecorrespondence of the random variations and imperfectionswhich the algorithm produces on the output data caused by therandom variations and the imperfections on the input data.

(Haralick, 1994)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. System Performance Characterization

A system performance characterization has a scoring functionthat evaluates the goodness of the output. The systemperformance characterization gives the distribution of thescoring function value as a function of the parametersdescribing the input perturbation and the tuning parameters.

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Experimental Protocols

Population of ideal inputsSimple Random SamplingStratified Sampling

Parameters of random perturbation distribution affectinginputsTuning Parameter SettingsScoring FunctionFix Tuning Parameters

Estimate Scoring Function Distribution as a function ofPerturbation Parameters

Fix inputEstimate Scoring Function Distribution as a function ofTuning Parameters

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Protocols

ModelingAnnotatingEstimatingValidatingPropagatingOptimizing

Haralick

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Finding Points To Correspond

There are many methods that are used for finding pointcorrespondences from images of multiple cameras. Amongthem are:

Corner PointsInterest PointsDense Subimage MatchingImage PyramidsCorrelationDistance

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Finding Matching PointsFinding Matching points is often posed as an optimizationproblem and uses sensor projection geometry constraints

Determine a Window Size

Maximize Normalized Cross-correlation

Minimize Normalized Distance

Minimum Description Length

Swarming

Simulated Annealing

Gradient Descent

Expectation Maximization

Mutual Information

Total Least Squares

Random Walks with Restart

SoftPosit

Energy Minimization

Dynamic Programming

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Performance Evaluation

Optimization and triangulation do not give a performanceevaluation.Performance Evaluation includes:

Estimating the Covariance of the position of each 3D pointThe rule for deciding whether or not to accept thecorrespondences associated with an estimated 3D pointThe resulting False Alarm - Misdetect Rate

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Kinds of Optical Sensor Models

(a) Pinhole (b) Orthographic (c) Pushbroom

(d) Cross-Slit (e) Pencil (f) Twisted Ortho-graphic

Figure: From Yu,McCillan, and Sturm, 2010

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Stereo Correspondence Problem

(a) Epipolar Geometry

(b) Uncertainty

Figure: From Unger and Stojanovic, 2013

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Variety of Related Point Correspondence Problems

Simultaneous Pose and CorrespondenceSensors calibratedStructure from MotionRigidity CheckingWide Baseline Stereo CorrespondenceSelf-Consistency

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Calibrated Sensors

Ground Truth Point CorrespondencesPerspective Geometry

Standard Photogrammetric ProcedureProjective Bundle AdjustmentInterior OrientationExterior Orientation

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. The Multi-Image Point Correspondence Problem

There are N > 1 calibrated sensorsTrue but unknown sensor parameters θ1, . . . , θN

3D Point q whose position is not known2D Corresponding points x1, . . . , xN , the sensor projectionsof q to the N sensorsξ1, . . . , ξN random perturbations of 2D sensor projectionpointsEstimated sensor parameters θ̂1, . . . , θ̂N

Model:xn = Pn(q, θn)x̂n = Pn(q, θn) + ξn, n = 1, . . . ,Nξn has N(0,Σξn)

θ̂n has multivariate uniformθ̂1, . . . , θ̂N , ξ1, . . . , ξN are independent

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Bayesian Setting

In the Bayesian setting, true sensor parameters are consideredas random variables with independent a priori densities

p1(θ1) . . . , pN(θN)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. The Bayesian Estimation Problem

Estimate q = (x , y , z) to maximize

p(q | x̂1, . . . , x̂N , θ̂1, . . . , θ̂N)

This is equivalent to estimate q to maximize

p(x̂1, . . . , x̂N , θ̂1, . . . , θ̂N ,q)

(Bedekar and Haralick, 1995)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. The Bayesian Estimation Problem

p(x̂1, . . . , x̂N , θ̂1, . . . , θ̂N ,q) = p(x̂1, . . . , x̂N | θ̂1, . . . , θ̂N ,q)×p(θ̂1, . . . , θ̂N ,q)

Given θ̂1, . . . , θ̂N and q, the sensor projections x̂1, . . . , x̂N areconditionally independent. Hence,

p(x̂1, . . . , x̂N | θ̂1, . . . , θ̂N ,q) =N∏

n=1

pn(x̂n, | θ̂1, . . . , θ̂N ,q)

The sensor projection x̂n only depends on the 3D point q andits associated sensor parameters θn. Hence,

N∏n=1

pn(x̂n, | θ̂1, . . . , θ̂N ,q) =N∏

n=1

pn(x̂n | θ̂n,q)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Conditional Independences

The calibration that established the estimates θn are certainlyindependent of each other and independent of the 3D point q.Hence,

p(θ̂1, . . . , θ̂N ,q) = p(q)N∏

n=1

pn(θ̂n)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. The Optimization

p(x̂1, . . . , x̂N , θ̂1, . . . , θ̂N , q) = p(q)N∏

n=1

pn(x̂n | θ̂n, q)pn(θ̂n)

log p(x̂1, . . . , x̂N , θ̂1, . . . , θ̂N , q) = log p(q)

−12

N∑n=1

(x̂n − Pn(q, θ̂n))′Σ−1

x̂n(x̂n − Pn(q, θ̂n))

+N∑

n=1

log pn(θ̂n)

where

Σx̂n (q, θ̂n) = Σξn +∂Pn

∂θ(q, θ̂n)Σθ̂n

∂Pn

∂θ(q, θ̂n)

′

p(q) is the prior for 3D point q, taking into account all the 3D points that havealready been triangulated and pn(θ̂n) is a multivariate uniform.

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Robustification

Objective Function

log p(q)− 12

N∑n=1

(x̂n − Pn(q, θ̂n))′Σxn (q, θ̂n)

−1(x̂n − Pn(q, θ̂n)) +N∑

n=1

log pn(θ̂n)

The perturbations are smallThe optimization is only correct if in fact each x̂n doescorrespond to the 3D point qBut sometimes the correspondence is not correctRobustify the objective function

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Covariance Propagation

The optimization provides an estimate q̂ of qThe covariance Σq needs to be estimatedThe consistency of q̂ with respect to x̂1, . . . , x̂N has to bechecked

(Haralick, 1994)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Covariance Propagation

X = (x1, . . . , xN)

X̂ = (x̂1, . . . , x̂N)

F (q,X ) = 0Minimize F (q̂, X̂ )

G(q,X ) = ∂F∂q

Σq =

(∂G∂q

)−1 ∂G∂X

ΣX (∂G∂X

)′(∂G∂q

)′−1

∂G∂q (q̂, X̂ )

∂G∂X (q̂, X̂ )

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Covariance Propagation

x̃n reprojection of estimated 3D point

x̃n = Pn(q̂, θ̂n)

Σx̃n(q̂, θ̂) =∂Pn

∂θΣθ̂n

(∂Pn

∂θ

)′+

∂Pn

∂qΣq̂θ̂

(∂Pn

∂θ

)′

+∂Pn

∂θΣθ̂q̂

(∂Pn

∂q

)′+

∂Pn

∂qΣq̂

(∂Pn

∂q

)′

∂Pn∂q (q̂, θ̂)∂Pn∂θ (q̂, θ̂)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Self Consistency Check

Empirically measure the predictive power of a score withrespect to a given algorithm, population of scenes and imagingconditions.Decide x̂n is a corresponding point if

(x̃n − x̂n)′(Σx̃n +Σξn)

−1(x̃n − x̂n) < τn

Acceptance rate: the fraction of corresponding point sets thatare close enough to the sensor projection of their estimated 3Dpoints.

If x̂n is decided as not a corresponding point, look near x̃ for acorresponding point.(Leclerc and Luong, 2003)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Scene and Image Populations

Textureless RegionsTextured RegionsUnoccluded CornersAmount of NoiseDepth Discontinuities

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Performance Characterization

Specify a populationLabel the true corresponding points: Ground TruthUse automatic procedure for finding corresponding pointsEstimate the 3D pointsDetermine the projection of the 3D pointsConsider only accepted pointsMatching Error compared to Ground Truth

Cumulative Distribution of distance to true positionThreshold defining when distance is close enoughMisdetect and False alarm rate

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Misdetect False Alarm Rate

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Tuning Parameters

Every algorithm has tuning parameters.How the Tuning Parameters are set influencesperformanceSome Tuning Parameters are set internal to programSome Tuning Parameters can be user setDefault Settings vs Tuned SettingsWhat is the sensitivity of the result to the settings of thetuning parameters?

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Performance Surface vs Tuning Parameters

Ruggedness of SurfaceSmoothness of SurfaceNumber of Local OptimaRatio of Local Optima values to Global Optima

Crossley, Nisbet, and Amos (2013)

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Performance Surface vs Tuning Parameters

Set a threshold θ of minimum acceptable performanceDetermine a hyperbox having the property

The fraction f of tuning parameter values in hyperbox yieldperformance > θ

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Estimating Performance Hyperbox Boundaries

N ExperimentsChoose tuning parameter M-tuples at randomEvaluate PerformanceDetermine Hyperbox Boundaries

Tuning Parameters (α1n, . . . , αMn), n = 1, . . . ,NGoodness Function ΨAcceptable Set A(θ) = {n | Ψ(α1n, . . . , αMn) > θ}bm min = minn∈A αmnbm max = maxn∈A αmnH = ×M

m=1[bm min, bm max ]

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. Goodness Fraction

N ExperimentsChoose Tuning Parameter M-tuples at random in H

(α1n, . . . , αMn), n = 1, . . . ,N

Evaluate Goodness Ψ

Estimate Goodness Fractionf = |{n | Ψ(α1n,...,αMn)>θ}|

N

Estimate Worst GoodnessΨworst = minn=1,...,N Ψ(α1n, . . . , αMn)

Find largest Hypercube HC ∈ H such that(γ1, . . . , γM) ∈ HC implies Ψ(γ1, . . . , γM) < θ

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. References

M. Unger and A. Stojanovic, “A new Evaluation Criterion For PointCorrespondences in Stereo Images”, Analysis, Retrieval and Delivery ofMultimedia Content, Lecture Notes in Electrical Engineering, Volume 158, 2013,pp. 183-202.J. Yu, L. McMillan and P. Sturm, “Multi-Perspective Modelling, Rendering andImaging,” Computer Graphics Forum, Volume 29, 2010, pp. 227-246.E. Juhàsz, A. Tanàcs, Z. Kato, “Evaluation of Point Matching Methods forWide-baseline Stereo Correspondence on Mobile Platforms,” 8th InternationalSymposium on Image and Signal Processing and Analysis. September, 2013,Trieste, ItalyY. Leclerc and Q. Luong, “Self-Consistency and MDL: A Paradigm for evaluatingPoint-Correspondence Algorithms and Its Application to Detecting Changes inSurface Elevation,” International Journal of Computer Vision, Volume 51, 2003,pp. 63-83.R. Haralick “Propagating Covariance In Computer Vision,” InternationalConference on Pattern Recognition,Vol 1, Jerusalem, Israel, 1994, pp. 493-498.A. Bedekar and R. Haralick, “A Bayesian Method for Triangulation and ItsApplication to Finding Corresponding Points,” International Conference on ImageProcessing, 1995, pp. 362-365.S. Yi, R. Haralick, L. Shapiro, “Error Propagation in Machine Vision,” MachineVision and Applications, Volume 7, pp 93-114.R. Haralick, “Performance Characterization Protocol In Computer Vision”,DARPA Image Understanding Workshop, Monterey CA 1994.

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. ReferencesR. Szeliski, Computer Vision: Algorithms and Applications, Springer-Verlag,London, 2011.y. Xiuxiao M. Yang, “A Novel Method for Multi-image Matching SynthesizingImage and Object-Space Information”, Geo-spatial Information Science, Volume12, 2009, pp. 157-164.P. Tipwai and S. Madarasmi, “A Coarse-and-Fine Bayesian Belief Propagationfor Correspondence Problems in Computer Vision, MICAI 2007, LNAI 4827, pp.683-693.S. Anvar, W. Yau, E Teoh, “Finding the Correspondence Poings in Images ofMulti-Views,” 8th International Conference on Signal Image Technology andInternet Based Systems, 2012, pp. 275-280.A. Usumezbas and B. Kimia, “Generating Dense Point CorrespondenceGround-Truth Across Multiple Views”, 2nd Joint 3DIM/3DPVT Converence: 3Dimaging Modeling, Processing, Visualization and Transmission, 2012, pp.214-221.F. Dellaert, S. Seitz, C. Thorpe, S. Thrun, “EM MCMC, and Chain Flipping forStructure from Motion with Unknown Correspondence”, Machine Learning,Volume 50, 2003, pp. 45-71.C. Bodensteiner, W. Huebner, K. Juengling, J. Mueller, and M. Arens, “LocalMulti-modal Image matching Based on Self-Similarity”, 17th InternationalConference on Image Processing, 2010, pp. 937-940.A. Ericsson and J. Karlsson, “Measures for Benchmarking of AutomaticCorrespondence Algorithms”, Mathematical Imaging Vision, Volume 28, 2007,pp. 225-241.

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. ReferencesM. Irani, “Multi-Frame Correspondence Estimation Using Subspace Constraints”,International Journal of Computer Vision, Volume 48, 2002, pp. 173-194.S. Chaudhuri and S. Chatterjee, “Performance Analysis of Total Least SquaresMethods in Three-Dimensional Motion Estimation”, IEEE Transactions onRobotics and Automation, Volume 7, 1991, pp. 707-714.T. Kim, K. Lee, and S. Lee, “A Probabilistic Model For Correspondence ProblemsUsing Random Walks With Restart”, ACCV, 2009, pp. 16-425.X. Li and Z. Hu, “Rejecting Mismatches by Correspondence Function”,International Journal Computer Vision, Volume 89, 2010, pp. 1-17.R. Sàra, “Robust Correspondence Recognition For Computer Vision”,Computational Statistics of 17th ERS-IASC Symposium, 2006.P. David, D. Dementron, R. Duraiswami, and H. Samet, “SoftPosit: SimultaneousPose and Correspondence Determination”, International Journal Of ComputerVision, Volume 59, 2004, pp. 259-284.D. Scharstein and R. Szeliski, “A Taxonomy and Evaluation of Dense Two-FrameStereo Correspondence Algorithms”, International Journal of Computer Vision,Volume 47, 2002, pp. 7-42.M. Crossley, A. Nisbet, and M. Amos, “Quantifyhing the Impact of ParameterTuning on Nature-Inspired Algorithms”, Bioinspired Learning and Optimization,2013, pp. 925-932.M. Greiffenhagen, D. Comaniciu, H. Niemann, V. Ramesh, “ Design, Analysisand Engineering of Video Monitoring Systems: An Approach and Case Study”,Proceedings of the IEEE, Volume 89, 2001, pp. 1498-1517.R. Haralick, “Propagating Covariance In Computer Vision”, International Journalof Pattern Recognition and Artificial Intelligence, Volume 10, 1996, pp. 561-572.

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

.. References

R. Haralick, “Propagating Covariance In Computer Vision”, International Journalof Pattern Recognition and Artificial Intelligence, Volume 10, 1996, pp. 561-572.

Slides can be found at:

http://haralick.org/conferences/point_correspondence_metrics.pdf