A Multi-Sensor Image Registration Approach Based On Edge-Correlation

Niu Li-pi, Yang Ying-yun, Zhang Wen-hui, Jiang Xiu-hua Electronic Information Engineering School

Communication University of China Beijing, China, 100024

niulp@163.com

Abstract—In this paper we propose a multi-sensor image registration approach based on edge-correlation. Firstly, edges of images are extracted and the longer edges are coded by modified Freeman. Secondly initializing correlation of chain code is calculated by integrating correlation coefficient and other methods, then the consistency detection succeeds by relative distance ratio histogram clustering detector based on center of edge correlation and angle histogram clustering detector based on average directional angle difference of edge correlation. During the calculation of average directional angle difference of edge correlation, modified histogram approach in this paper is much better than line-fitting approach. Finally accurate edge correlation pair is obtained, and then image registration parameter is attained by applying least square algorithm (LSM) in interrelated parts. The approach presented in this paper can register images of wide translational and rotary range.

Keywords- image registration; multi-sensor; edge extract; correlation

I. INTRODUCTIONS Multi-sensor image registration is the prerequisite of

image fusion. Features frequently used in image registration mainly include: point (line endpoint, intersection point, large curvature point, and wavelet coefficient extreme point, etc.), line (edge, contour, and region boundary, etc.), and plane ( gray scale for a region or the whole image). Point is sensitive to noise, and it usually relies on the line where it is and/or the gray scale of the image (or region) nearby to confirm its corresponding relationship. Although gray scale of an image is sound in the application of image registration of homogeneous sensor, not for that of inhomogeneous sensors, especially when bands of sensors are so different that variation of image gray scale is great, so it’s hard to be used directly. Line tells the shape and contour of objects and other features in image and it indicates certain consistent features between images despite of great different gray scale. Moreover, in a sense, noise is restrained by sorts of algorithms when extracting edges. Therefore line (edge, contour, and region boundary, etc.) is widely used in multi-sensor image registration.

Featuring closed contour and contour center, Hui Li etc. [1] and Xiaolong Dai etc. [2] register images by matching closed contour. However, as images don’t always have abundant closed contours, this registration approach can be very inaccurate or even fail when there is few or no closed contour. Xiangjie Yang etc. [3] make it by extracting edges

and matching all the edges obtained using partial Hausdorff approach. However, features of multi-sensor images usually vary greatly. The proportion of those inconsistent features could be so large that partial Hausdorff approach would fail registering. For the specific situation of tank application, V. Randrianarisoa etc. [4] extract the horizon as main feature, and eliminate other inconsistent features, so as to ensure partial Hausdorff approach to take effect. Wu.Y. etc. [5] first detect the coastline based on contrast, and register the images taking the long coastline as main feature. But it’s a shame that long horizons or long coastlines don’t always appear in images.

This paper presents a multi-sensor registration approach based on edge-correlation, by which images can be registered even with lacking in closed contour and/or great proportion of inconsistent features and that there is no long edge like horizon or coastline. In this approach, first edges are extracted with a threshold set for length of edge and using larger Gaussian parameter, and only longer edges are retained and coded by modified Freeman. Then initial correlation of chain code is calculated by integrating Hui Li’s correlation method and correlative coefficient method. As regards that there are certain error ones among the initial correlation pairs, this paper presents a set of consistency detection method integrating relative distance ratio histogram clustering detector based on center of edge correlation and angle histogram clustering detector based on average directional angle difference of edge correlation. During the calculation of average directional angle difference of edge correlation, modified histogram approach in this paper is much better than line-fitting approach. Finally accurate edge correlation pair is obtained then image registration parameter is attained by applying LSM in interrelated parts. The approach presented in this paper can not only register images of wide translational and rotary range, but also has the advantage of rapid computational speed.

II. EDGE EXTRACTION AND INITIAL CORRELATION Among the edge extraction algorithms, Canny algorithm

shows better performance, mainly because the coefficient σ in Gaussian filter in Canny algorithm can adjust the edge features extracted. Usually, larger σ is taken to extract smooth and continuous long edges. But σ can not be too large, otherwise, firstly details of the edges will be lost, which is not conducive to accurate correlation; secondly, edge matching will be inaccurate and consequently the

2010 2nd International Conference on Signal Processing Systems (ICSPS)

V2-36 978-1-4244-6893-5/$26.00 2010 IEEEC

registration will not be precise. σ normally takes 2 to 3. Set a threshold of length “T” for the edges extracted and eliminate all the edges shorter than threshold “T”, and then coded by eight-orientation Freeman [7].

As to the matching of closed contour, Hui Li uses the chain code correlation algorithm in modified Freeman. For the Freeman code obtained { naaa ...21 }, it can be modified as follows:

==

ii qbab 11 (1)

iq is a integer number and meets 08mod)( =− ii aq and makes

1−− ii bq least To modified the chain code is to make it smoother, i.e.

{707070} becomes {787878}. Moreover, when the curve rotates, shape of the code almost keeps the same except for the average value of chain code (interval of taking images is not considered here). After the transaction, the chain code is modified again by Gaussian filter {0.1 0.2 0.4 0.2 0.1}.

Hui Li’s contour matching algorithm is following:

−

=++ −=

1

0

'' )(4

cos1 n

jjljkkl ba

nD π (2)

Where:

−

=+++ −=

1

0mod)(mod)(

' 1 n

jNjkNikik AA

an

aa ni <≤0 (3)

−

=+++ −=

1

0mod)(mod)(

' 1 n

jNjlNjlil BB

bn

bb ni <≤0 (4)

{ ia } is expressed as chain code of point AN in contour

A, and { ib } is chain code of point BN in contour B.

(a)Long edge from 84 band 5 (b)Long edge from 86 band 7

(c) result of Hui Li’s (d) correlative coefficient

Fig.1 comparison of edge correlation methods

This approach is sound to the correlation of closed contours, but not for the open contours, especially the flatter ones, as their correlative peak value is not clear. This is

mainly caused by the correlative nonlinear (cosine function is applied).

As to modified freeman code, in this paper edges are correlated by correlative coefficient and good result is obtained. Correlative coefficient is to figure out the similarity between two curves, to calculate the correlation within an absolute range [-1,1], and to provide a linear instruction to similarity level. So its correlation peak value is clearer (shown in Fig.1d), and its computational speed is more rapid than Hui Li’s.

The formula of correlative coefficient is following:

−−

−−=

++

++

j jbjlajk

jbjlajk

klubua

ubuaD

22 )()(

))(( (5)

Where: {ia } is expressed as chain code of point

AN in contour A, and {

ib } is chain code of point BN in contour B. au is the average value of interrelated part in chain code

{ia }, and bu is the average value of interrelated part in

{ib }. lk , is respectively the initial point. There are three

kinds of status in edge correlation: two edges are completely correlative; one edge is correlative to part of the other; part of one edge is correlative to part of the other. So usually edges are correlated by fixing the longer one and slip the shorter one from one direction. Thus the correlation coefficient is a 1-dimentional function

kD of the slipping point k.

In this paper, whether two edges are correlative or not is decided by Hui Li’s contour matching method combined with Freeman chain code correlation coefficient method. The specific rules are: when the peak value by Hui Li’s method is larger than the set threshold 1T , and the peak value by Freeman is larger than threshold

2T , and the distance from the slipping point 1k corresponding to the peak value in Hui Li’s to the slipping point 2k corresponding to the peak value in Freeman is less than threshold

3T ( normally take 3 points for 3T ), besides, the

interrelated part is larger than half of the shorter edge, the two edges are deemed to be initially correlative. Thus initial correlative aggregate is obtained. In the above edge extracting process, for the initial directions of two edges are different, the opposite direction of chain code of the edges may be taken. Therefore the correlation calculation should be done twice by one order and the opposite. Order the edges by length, respectively take longest N curves (normally N=20~30) and do correlative search according to the conditions above, thus obtain the aggregate of initial correlative edges.

III. CONSISTENCY DETECTION OF THE CORRELATIVE EDGES AND IMAGES REGISTRATION

It’s difficult to avoid some error correlative pairs in the aggregate of initial correlative edges so that consistency

2010 2nd International Conference on Signal Processing Systems (ICSPS)

V2-37

detection must be done, which includes three steps, initial detection, distance ratio detection and directional angle difference detection:

A. Initial detection Search in the initial correlative aggregate and condense

it when: a) when both two edges in positive and reverse order are in the aggregate, compare the correlation peak values of positive order and reverse order and delete the error correlative pair that is of less peak value; b) when one edge is correlative to more than one edge, compare their correlative point numbers and correlation peak values: if correlative points of correlative pair i are more than that of correlative pair j and correlation peak value of i is larger than that of j , delete correlative pair j , otherwise retain it.

B. Distance Ratio Detection Calculate the gravity center of the remaining correlative

edges X ,Y :

=

= N

i ixNX1

)1( (6)

=

= N

i iyNY1

)1( (7)

Where: {( ii yx , )} Ni ,...2,1= , is coordinate of point in interrelated part of correlative edges; N is number of point in interrelated part.

Assume that edge iA is correlative with edge iB , then correlative gravity center of iA A

iX , AiY is corresponding

to correlative gravity center of iB BiX , B

iY , thus distance

ji AA is corresponding to jiBB . Where:

ji AA = )()( Aj

Ai

Aj

Ai YYXX −+− (8)

ji BB = )()( Bj

Bi

Bj

Bi YYXX −+− (9)

Calculate the ratiojiji BBAA , and get 2/)1( −mm

ratios when there are m pairs of correlative edges. Make use of those ratios to form a histogram which should construct a cluster around the accurate ratio variation. The ratios near the cluster are corresponding to the accurate correlative edge pairs.

C. Directional angle difference detection (angle difference histogram clustering detector) In this part, directional angle difference value (that is

angle difference) of interrelated part of all the correlative edges is detected. Make use of the angle difference values to form a histogram which should construct a cluster around the rotation angle between the two images. The angle difference values near the cluster are corresponding to the accurate correlative edge pairs.

The traditional method of measuring the angle difference is following: first fit the curve with a straight line, and take

the direction of the straight line as the average direction of the curve; then calculate the angle difference of the two fitting lines.

The slope of the fitting line k is:

=

=

−⋅

−⋅= N

i i

N

i ii

xXXN

yxYXNk

12

1 (10)

Where: X and Y is the center coordinate point of formula (6) to (7); {( ii yx , )} Ni ,...2,1= is coordinate of interrelated part of the fitting curves; N is length of the interrelated part.

Thus the angle difference of the two curves i and j is:

)()( ijij karctgkarctg −=θ (11)

Given that the rotation of two images in Fig.2 is 30o. Result of calculation by traditional curve fitting method is15.11o. The reason is the correlative edges extracted from two images have been affected by noise, sampling interval, and different sensors, etc. which makes the two edges are not completely the same. This kind of effect could be slight when the curve is flat, whereas it could be great when the curvature of the curve is large as result it makes the angle of the fitting line seriously inaccurate.

Our method is to reduce the points of the fitting curve N, that is to fit the corresponding angle difference of a small piece, then put this tiny window of calculation on the entire edge and slipping calculate, so that (M-N+1) angle difference values are obtained. Use the (M-N+1) angle difference values to form a histogram and get )(θH , then smooth it with length of )12( +n and get )(θH , as follows:

−=

+=n

ni

iHH )()( θθ (12)

Thus the angle corresponding to the peak value in the histogram is the closest to the rotation angle of the two correlative edges.

Fig.2 Correlative edges extracted from two images (the rotational angle of

them is 300)

Fig.2 is images of edge with rotational angle of 30o. For the correlative edges in it, the comparison between modified histogram approach in this paper and line-fitting approach is shown as the following Table 1:

2010 2nd International Conference on Signal Processing Systems (ICSPS)

V2-38

TABLE 1 Angle difference calculated by modified histogram approach and line-fitting approach (the accurate result is 30o, space of the histogram

takes1o)

line-fitting approach ( ο ) Modified histogram approach ( ο ) a-a” 15.1 ο 26 b-b” 30.8 29 c-c” 23.4 27 d-d 22.6 26 e-e” 14.3 26 f-f” 22. 9 29 g-g” 17.3 29 h-h” 24.5 26 i-i” 17.1 27

After getting the angle difference of all the correlative edges, apply them to form a histogram, which should construct a cluster around some point, and this point is the rotation angle of image registration. The correlative edges corresponding to this point are accurate, then calculate the registration parameters with LSM.

IV. EXPERIMENTAL RESULT The following are respectively registration experiments

of images of large rotation angle, great translation and different bands, and all the registrations attain good results. In these experiments, edge extraction by Canny algorithm takes σ =2.5; the minimum length of edge takes T=30; correlative edges being searched takes N=30; number of fitting points in curve fitting takes M=19, correlative point takes 1/3 of correlative length. Time of registration is based on the result of operating program MATLAB in a computer of AMD2500+CPU (the time is calculated by inner functions of MATLAB)

Fig. 3 is two images of homogeneous sensors intercepted from optical image sequence of American Mojave Desert. Size of Fig.(a) and (b) is 512*512; Fig.(c) and (d) are 13 pairs of consistent edges by algorithm of this paper; Fig.(e) is the result of edge registration and Fig.(f) is

(a) optical image(512*512) (b) optical image(512*512)

(c)Consistent edges from a (d)Consistent edges from b

(e) Result of matching edges (f) Registration result

Fig.3 registration of images of large rotation angle

the final result of image registration. Registration parameters obtained are: proportion s is 1.0028; rotation angle θ is

ο93.29− ; translation Tx is 275.33 pixels and Ty is -188.78 pixels. Time of registration is 38.6250 seconds.

(a) band 1 TM(256*256) (b) band 8 TM(256*256)

(c) Consistent edges from a (d) Consistent edges from b

(e) Result of matching edges (f) Registration result

Fig.4 registration of images of large translation

Fig.4 is two images selected from a series of mountain images of aerial photograph range from band 1 to band 12 and the overlap is less than half of the image. Size of Fig.(a) and (b) is 256*256; Fig.(c) and (d) are 10 pairs of consistent edges detected by algorithm of this paper, and one edge in Fig.(c) is correlative to two edges in Fig.(d). Fig.(e) is the result of edge registration, where we can see the edges are perfectly matched. Fig.(f) is the final result of image registration. Registration parameters obtained are: proportion s is 0.9986; rotation angle θ is ο10.15− ; translation Tx is -3.21 pixels and Ty is 124.88 pixels. Time of registration is 16.5340 seconds.

2010 2nd International Conference on Signal Processing Systems (ICSPS)

V2-39

(a) infrared image(300*500) (b)SAR image(384*534)

(c) Consistent edges from a (d) Consistent edges from b

(e) Result of matching edges (f) Registration result

Fig.5 registration of infrared image and SAR image

Fig.5 is infrared image and SAR image of some place in Shanghai. Fig.(a) is infrared image of 300*500; Fig.(b) is SAR image of 384*534; Fig.(c) and (d) are 6 pairs of consistent edges by algorithm of this paper. Fig.(e) is the result of edge registration and Fig.(f) is the final result of image registration. Registration parameters obtained are: proportion s is 0.9999; rotation angle θ is 0.22o; translation Tx is -56.06 pixels and Ty is 11.21 pixels. Time of registration is 20.8820 seconds.

Among the above examples, the registration result of the experimental images in Fig. 3 and Fig. 4 can be found in HuiLi’s paper [1]on the IEEE in 1995. Regard the result in HuiLi’s paper as accurate to detect the accuracy of this paper:

TABLE 2 the registration accuracy of algorithm in this paper

Method Proportion Rotation angle( ο )

XΔpixel

YΔpixel

Fig. 3 512*512

HuiLi’s algorithm 1.0035 -29.95 275.74 -188.21

result of this paper 1.0028 -29.87 275.33 -188.78

Inaccuracy 0.0007 0.08 0.41 0.57 Fig. 4

256*256 HuiLi’s

algorithm 1.0000 -15.00 -3.50 125.00

result of this paper 0.9986 -15.10 -3.21 124.88

Inaccuracy 0.0014 0.10 0.29 0.12

Shown in Table 2, most precisions of registration parameters by algorithm of this paper is within half pixel. HuiLi’s approach can only apply to registering images with sufficient closed contour, whereas approach in this paper isn’t limited by this, thus is more widely applicable.

V. SUMMERIES This paper presents a multi-sensor image registration

approach based on edge-correlation, and the approach has the following characteristics: firstly, it can apply to registration of images of greatly different bands and multi-sensor images of many inconsistent features; secondly, it can be adapted to images of lacking in closed contour and no long edge like horizon or coastline; thirdly, it can register images of wide translational and rotary range; fourthly, the algorithm is highly automatic. Many registration algorithms need manual intervention while registering, for example, in HuiLi’s contour matching algorithm, the two thresholds need to be adjusted when extracting edges according to different images. While regarding the 3 pairs of images in the experiments, the registration results are obtained accurately and totally automatically. Fifthly, computational speed is rapid: in the experiment of registering images of 256*256, the registration parameters (including Time of registration) are obtained in about 20 seconds by Matlab program working on computer of AMD2500+CPU. If the algorithm is accomplished by VC++, the speed will be strikingly improved.

However this approach quite relies on attaining smooth and continuous edges. Thus the research in future will focus on extracting smooth and continuous edges for different σ value; besides, how to obtain reliable correlation of open curves when there is a proportional difference will be research direction as well.

Automatic multi-sensor image registration is a very complicated task, and it needs an artificial intellectual system consisting of various kinds of registration algorithms to complete. The multi-sensor image registration approach based on edge-correlation presented by this paper can be a part of the system.

ACKNOWLEDGMENT This work is supported in part by the “211 Project” of

Communication University of China.

REFERENCES [1]. Hui Li, A contour-based approach to multisensor image registration.

IEEE transactions on image processing 1995 [2]. Xiaolong Dai and Siamak Khorram, A feature-based image

registration algorithm using improved chain-code representation combined with invariant moments, IEEE Trans, pp2351-2362, 1999

[3]. Xiangjie Yang, Adaptive hill climbing and iterative closest point algorithm for multisensor image registration with partial Hausdorff distance. SPIE, pp99-109, 2000

[4]. V Randrianarisoa, Robust automatic ground image feature extraction for multisensor image registration. SPIE, 129-139, 2001

[5]. Wu.Y. and H.Maitre, A multiresolution approach for registration of a SPOT image and a SAR image, Proceedings of international geoscience and remote sensing symposium,635-638, 1990

[6]. Heath M D, A robust visual method for assessing the relative performance of edge-detection algorithms, IEEE 1997

[7]. Rafael C.Gonzalez, Digital Image Processing, Prentic Hall 2002

2010 2nd International Conference on Signal Processing Systems (ICSPS)

V2-40