Image Image RegistrationRegistration

Advanced DIP ProjectAdvanced DIP ProjectGroup #3:Group #3:

Dave GrimmDave Grimm

Joe HandfieldJoe Handfield

Mahnaz MohammadiMahnaz Mohammadi

Yushan ZhuYushan Zhu

OutlineOutline Image RegistrationImage Registration Point MappingPoint Mapping Problem StatementProblem Statement GCP Selection MethodsGCP Selection Methods

ManualManual Contour Based GCPContour Based GCP Corner and Edge BasedCorner and Edge Based

Evaluation MethodsEvaluation Methods Group PlanGroup Plan

Time TableTime Table Responsibilities Responsibilities

Image RegistrationImage Registration Spatial matching the pixels of two (or Spatial matching the pixels of two (or

more) images of the same area or scenemore) images of the same area or scene

Relates the geometric coordinate system in Relates the geometric coordinate system in one image to anotherone image to another

Transforms one of the images so that the Transforms one of the images so that the two images share a common coordinate two images share a common coordinate system system

Image Registration Image Registration (Cont.)(Cont.)

Applications of Image Applications of Image RegistrationRegistration

Remote SensingRemote Sensing Extract information from images of the same Extract information from images of the same

region taken at different times or in different region taken at different times or in different spectral bandsspectral bands

Color ScienceColor Science Creating a mutispectral imagesCreating a mutispectral images

MedicalMedical Pathology analysisPathology analysis

Image FusionImage Fusion Image Mosaicking Image Mosaicking

Point MappingPoint Mapping

Widely usedWidely used Standard technique for registering images Standard technique for registering images

misaligned by an misaligned by an unknownunknown transformation transformation

Requires ground control points (GCPs Requires ground control points (GCPs or primitives) to be found in the imagesor primitives) to be found in the images Intrinsic or extrinsicIntrinsic or extrinsic Can be done either manually or Can be done either manually or

automaticallyautomatically

Point Mapping (cont)Point Mapping (cont) Mathematically relates the coordinate Mathematically relates the coordinate

systems of the imagessystems of the images

aa00 and b and b00 are needed for a simple shift of origin, are needed for a simple shift of origin, and the first two terms are needed for a and the first two terms are needed for a combined scale adjustment and shifting of the combined scale adjustment and shifting of the originorigin

Higher order equations for more complicated Higher order equations for more complicated transforms are possible, such as rotation, skew, transforms are possible, such as rotation, skew, and perspective differencesand perspective differences

x a0 a1 x a2 y a3 x y a4 x 2 a5 y 2...x

y b0 b1 y b2 x b3 y x b4 y 2 b5 x 2 ...y

Point Mapping StepsPoint Mapping Steps Select GCPs from Select GCPs from

each imageeach image Match GCPs to Match GCPs to

from point pairs from point pairs (points that are (points that are spatially the same spatially the same in the two images)in the two images)

Register images Register images via point mappingvia point mapping

Our focus is on Our focus is on Steps 1 & 2Steps 1 & 2

Problem StatementProblem Statement Comparison of GCP selection algorithmsComparison of GCP selection algorithms

ManualManual

AutomatedAutomated Area-basedArea-based Feature-basedFeature-based

Contour MappingContour Mapping Corner and Edge DetectionCorner and Edge Detection

Manual RegistrationManual Registration

Left: the reference image Right: the image to be registered

Automated RegistrationAutomated Registration Area-based algorithmsArea-based algorithms

A small window of points in the sensed image, A small window of points in the sensed image, correlation kernel, is compared statistically with correlation kernel, is compared statistically with windows of the same size in the reference image. windows of the same size in the reference image. The measure of similarity is usually the normalized The measure of similarity is usually the normalized cross correlation. The location of maximum in the cross correlation. The location of maximum in the normalized correlation image is a pair of GCP. normalized correlation image is a pair of GCP.

Disadvantage : the correlation value is sensitive to scale Disadvantage : the correlation value is sensitive to scale and rotationand rotation

g i, j 1

K1K2

f i, j h i, j

Automated Registration Automated Registration (Cont.)(Cont.)

Feature-based algorithmFeature-based algorithm Spatial features usually include edges, Spatial features usually include edges,

boundaries, intersections, etc boundaries, intersections, etc The general feature representations The general feature representations

are:are: Chain codeChain code Moment invariantsMoment invariants Fourier descriptorFourier descriptor Shape signaturesShape signatures

Advantage: invariant to scaling, Advantage: invariant to scaling, rotation, and translationrotation, and translation

Feature- based algorithmFeature- based algorithmContour- basedContour- based

Image segmentationImage segmentation

Image matchingImage matching

Image SegmentationImage Segmentation

Producing closed-edged contours by Producing closed-edged contours by convolving the original image with Laplacian convolving the original image with Laplacian of Gaussian (LoG) operatorof Gaussian (LoG) operator Find zero crossing points, in which the convolved Find zero crossing points, in which the convolved

image is scanned to detect pixels that have zero image is scanned to detect pixels that have zero value or pixels at which a change of sign has value or pixels at which a change of sign has occurred. Staring with a pixel, its neighboring occurred. Staring with a pixel, its neighboring pixels were expanded until a sign change occurred.pixels were expanded until a sign change occurred.

Drawbacks:Drawbacks: Discontinuity at the weak edge pixelsDiscontinuity at the weak edge pixels Thick edgesThick edges

Image segmentation (Cont.)Image segmentation (Cont.)Thin and Robust Zero- Thin and Robust Zero-

CrossingCrossing

Mark as an edge point every pixel Mark as an edge point every pixel that satisfies the following conditions:that satisfies the following conditions: The pixel is a zero-crossing pointThe pixel is a zero-crossing point The pixel lies in the direction of the The pixel lies in the direction of the

steepest gradient change (edge strength)steepest gradient change (edge strength) The pixel is the closest pixel to the virtual The pixel is the closest pixel to the virtual

zero plane of the LoG image among its zero plane of the LoG image among its eight neighborseight neighbors

Image Image SegmentationSegmentation

Thin and Robust Thin and Robust Zero- CrossingZero- Crossing

(Cont.)(Cont.)

Discarded the noisy edge Discarded the noisy edge points using points using Edge sorting Edge sorting

Edge refinementEdge refinement

Image MatchingImage Matching

Invariant momentInvariant moment Produce a set of scaled moment-based Produce a set of scaled moment-based

descriptor of planar shapes, that are descriptor of planar shapes, that are scale, rotation, and translation invariantscale, rotation, and translation invariant

Improved chain-coded Improved chain-coded representation of regionsrepresentation of regions

Image Matching (Cont.)Image Matching (Cont.)Chain codingChain coding

A way to represent a boundary by a A way to represent a boundary by a connected sequence of straight-line connected sequence of straight-line segments of specific length and segments of specific length and direction based on 4- or 8- direction based on 4- or 8- connectivityconnectivity

Image Matching (Cont.) Image Matching (Cont.) Draw backs of Standard Draw backs of Standard

Chain CodingChain Coding

The resulting chain codes tend to be The resulting chain codes tend to be quite longquite long

Any small disturbance along Any small disturbance along boundary due to noise or imperfect boundary due to noise or imperfect segmentation causes changes in the segmentation causes changes in the code that may not be related to the code that may not be related to the shape of the boundaryshape of the boundary

Improved Improved Chain-CodeChain-Code

1.1. Shift Operation:Shift Operation:

2.2. Smoothing: GaussianSmoothing: Gaussian3.3. Normalization: Normalization:

Demean Demean 4.4. Resampling OperationResampling Operation

minimized is

08mod)(int,

,

1

11

ii

iii

ii

bq

aqq

qbab

Image Matching (Cont.)Image Matching (Cont.)

Invariant-Moment Distance MatrixInvariant-Moment Distance Matrix

The pairs are accepted as candidate The pairs are accepted as candidate matches if their invariant-moment matches if their invariant-moment values are below the defined values are below the defined thresholdsthresholds

dij rk i s

k i 2

k1

7

Image Matching (Cont.)Image Matching (Cont.)

Chain-code Matching MatrixChain-code Matching MatrixContour A and B selected as matched pair if:

1) DAB≥DAB’ where B’ includes all the contours with similar shapes to A

2) DAB≥T3 where T3 is a preset threshold which can eliminate matches with poor correlation

Image Matching (Cont.)Image Matching (Cont.)

The smaller the Invariant-Moment The smaller the Invariant-Moment distance, the more similar the shapes distance, the more similar the shapes of two regionof two region

The greater the chain-code matching The greater the chain-code matching coefficient, the more contours coefficient, the more contours resemble each other in the shaperesemble each other in the shape when Dwhen Dklkl=1, there is a perfect match=1, there is a perfect match

The centroid of the matched contours The centroid of the matched contours are used as GCPsare used as GCPs

Summary of Contour Based Summary of Contour Based AlgorithmAlgorithm

Corner and Edge Corner and Edge DetectionDetection

The corners and The corners and edges present in each edges present in each image are locatedimage are located Harris and Stephens, Harris and Stephens,

19881988

A local window is A local window is placed in the image and placed in the image and changes due to shifting changes due to shifting the window are the window are consideredconsidered

Corner and Edge Detection Corner and Edge Detection (cont)(cont)

An edge produces large changes An edge produces large changes when the window is shifted when the window is shifted perpendicular to the edge direction perpendicular to the edge direction and small changes when shifted and small changes when shifted parallel parallel

A corner produces large changes A corner produces large changes when the window is shifted either when the window is shifted either perpendicular or parallelperpendicular or parallel

Insignificant points (noise) are Insignificant points (noise) are removed via thresholdingremoved via thresholding

Corner and Edge Detection Corner and Edge Detection (cont)(cont)

Each found point in the image to be registered Each found point in the image to be registered (image B) is then compared to each found point (image B) is then compared to each found point in the reference image (image A) to determine in the reference image (image A) to determine which pairs matchwhich pairs match Any point without a match is considered an outlier Any point without a match is considered an outlier

(slack)(slack)

Test ImagesTest Images

Images that we can introduce known Images that we can introduce known transformationstransformations

Various images with unknown Various images with unknown transformationstransformations

Limit testing to grayscale images for nowLimit testing to grayscale images for now

Statistical Assessment of Statistical Assessment of Image Registration Image Registration

ResultsResults

Assessment would be done on Assessment would be done on unknown GCP pairs using calculated unknown GCP pairs using calculated transformation matrix obtained by transformation matrix obtained by different point mapping algorithmsdifferent point mapping algorithms

Statistical AssessmentStatistical Assessment

Calculation of statistical distance for Calculation of statistical distance for each pair of points, P(x1,y1), each pair of points, P(x1,y1), Q(x2,y2)Q(x2,y2)

d P,Q x1 x2 2

s11

y1 y2 2

s22

P x1, y1 &

Q x2, y2

Sik 1

nx ji x x jk x k

j1

n

i 1,2,..., p k 1,2,..., p

X x1 y1

x2 y2

Statistical AssessmentStatistical Assessment

Take maximum, standard deviation, Take maximum, standard deviation, and average error between the group and average error between the group of GCPs using different of GCPs using different transformation matrixtransformation matrix

Evaluate variability of predicted Evaluate variability of predicted points along x and y axis using scatter points along x and y axis using scatter plotplot

Statistical AssessmentStatistical Assessment

Calculate the confidence intervalsCalculate the confidence intervals

H0 : 0 versus H1 : o

x i tn 1 2 sii

nii x i tn 1 2 sii

n

Registration Time TableRegistration Time Table

Week 4 – Plan presentation, programmingWeek 4 – Plan presentation, programming

Week 5 – ProgrammingWeek 5 – Programming

Week 6 – At least manual code done, present Week 6 – At least manual code done, present preliminary preliminary resultsresults

Week 7 – Evaluation methods done Week 7 – Evaluation methods done

Week 8 – Both automated methods finishedWeek 8 – Both automated methods finished

Week 9 – Gathering image results, complete write-Week 9 – Gathering image results, complete write-upup

Week 10 – Final presentation & reportWeek 10 – Final presentation & report

Plan FlowchartPlan Flowchart

Manual

Contour

C&E

Point-Mapping Registration Algorithm

Methods:

Correlations

Confidence Interval

Evaluation:Method flow:

GCPs

Task Assignment ListTask Assignment List

Joe - Manual pixel detection methodJoe - Manual pixel detection method

Yushan - Image contour detection Yushan - Image contour detection method method

Dave - Corner and edge detection Dave - Corner and edge detection

Mahnaz - Statistical evaluation (with Mahnaz - Statistical evaluation (with Joe)Joe)