A NEW TECHNIQUE FOR SOLVING A JIGSAW PUZZLE

M Makridis and N. Papamarkos *

*Image Processing and Multimedia LaboratoryDepartment of Electrical & Computer Engineering

Democritus University of Thrace67100 Xanthi, Greece, papamark(ee.duth.gr

ABSTRACT

A new technique for solving jigsaw puzzles is proposed,which takes advantage of both geometrical and colorfeatures. It is considered that an image is being divided intoa number of pieces (sub-images). The proposed technique isbased on extraction of a set of boundary characteristic pointsand on a Kohonen Self-Organized Feature Map (KSOFM)color reduction technique. For each characteristic point a setof color and geometrical features are extracted. Thetechnique compares these sets of features and decideswhether two sub-images match or not. When a matchingpair has been found, a corrective procedure is applied inorder for these sub-images to fit exactly. Next, the proposedtechnique creates a new sub-image, which consists of thetwo matched sub-images. The whole matching procedure isbeing repeated until only one sub-image remains or no morematching sub-images can be found.

Index Terms-Image boundary analysis, Image matching,Image shape analysis, Image restoration.

1. INTRODUCTION

The aim of this paper is to provide an automatic method forjigsaw puzzle solving. Related problems includereconstructing archeological artifacts [1]-[2] and or evenfitting a protein with known amino acid sequence to a 3Delectron density map [3]. Automatic solution of jigsawpuzzles by shape alone goes back to 1964. Freeman andGardner [4] first faced with the problem. Since thennumerous papers have been written, yet few take advantageof color information. The majority of the proposedtechniques work on curve matching [5]-[9]. Some of themdivide the contour boundary of each sub-image into partialcurves through breakpoints [5], [8].Wolfson et al. [6] propose a technique that is based on

curve matching. However, this algorithm does not take intoaccount color information. Additionally, their proposedalgorithm can handle only puzzles that obey some certainrules that all "toy-stored" puzzles obey. These rules arementioned below in detail.Another work proposed by Goldberg et al. [7], is based

upon Wolfson's et al. [6] algorithm. The main difference is

that the Goldberg's et al. algorithm does not use curvematching, yet it finds indents, outdents and straight sides inevery sub-image. However, the disadvantages here are thesame as in the Wolfson's et al. technique. Finally thistechnique, although it is recent, does not embody aprocedure to detect possible matching errors.The proposed technique deals with the majority ofjigsaw

puzzles. A jigsaw puzzle consists of a number of sub-imagesand each sub-image may be misshaped due to the rotationangle and the translation. The sub-images are beingexamined in pairs. Initially the proposed technique performsbinarization, in order to distinguish foreground frombackground. To do this we assume that the background has afixed white color. Binarization helps in extracting accuratelythe contour boundary upon which it is being estimated theposition of the characteristic points (CPs). CPs are foundaccording to an algorithm named IPAN99, which isproposed by Chetverikov and Szabo [10]. This algorithmassigns to each CP a value corresponding to an angle aroundthis point.The technique achieves color reduction by using a

KSOFM technique proposed by Papamarkos et al. [11]-[ 13].Color and geometrical features are used to compare pairs ofsub-images and to decide whether these sub-images shouldbe merged or not. The final restoration stage includes a skewcorrection algorithm, which rotates the image to horizontalorientation and fills empty regions produced due to rotation.The proposed technique was tested in a variety of types of

jigsaw puzzles and has been implemented in a visualenvironment using Delphi7.

2. DESCRIPTION OF THE METHOD

Although standard toy-store jigsaw puzzles obey certainrules that make the problem more tractable than it wouldotherwise be, the innovation of the proposed technique isthat solves the jigsaw puzzle problem, without taking intoaccount these specifications:> The puzzle has a rectangular outside border.> Every sub-image has at most four primary neighbors

(left, right, above and below).> Sub-images interlock with their primary neighbors by

tabs, consisting of an "indent" on one sub-image matingwith an "outdent" on its neighbor.

1-4244-0481-9/06/$20.00 C2006 IEEE 2001 ICIP 2006

Each sub-image has no neighbors except its primaryneighbors.

The proposed technique takes as inputs a number of sub-images and leads to the solved puzzle. The entire techniqueis based on the examination of the shape of the boundarycontour of all sub-images in pairs. Since a matching pair hasbeen found these two sub-images are being merged and thewhole procedure is being repeated for the rest of the sub-images plus the new one. This procedure is being repeateduntil one sub-image remains, which is the final key of thepuzzle.

Initially, a number of CPs is obtained on the boundarycontour, of every sub-image by using the IPAN99algorithm. Afterwards it is performed color reduction with aKSOFM based technique proposed by Papamarkos et al.[ 1]-[1 3]. Then a set of comparisons between CPs of everypair of sub-images is performed in order to discard all pointsthat do not satisfy some merging conditions. Since somecandidate CPs have been discarded, this technique takes allthe possible five-point sets among the rest of candidate CPsfor each sub-image and performs a final comparison in orderto decide whether two sub-images match or not. If two setssatisfy comparison conditions, the two sub-images can bematched. Afterwards, it is applied another procedure thatcalculates the exact rotation angle and translation. Thisalgorithm finds the minimum possible error, related to theempty regions or overlaps that are being created, due torotation and translation.

2.1 Binarization

In order to extract the contour boundary of a sub-image,each sub-image should be first converted into a binary one.Binarization is based on detecting two sets of connectedcomponents, the background set and the foreground one. Itis apparent though that there may be background pixels thatactually belong to foreground and vice versa. In order toachieve successful binarization, it has been developed amethod, which decides whether a connected componentshould remain to its set or should be transferred to the other.

Stage 1. During the first stage, all pixels with color closeto white (which is the background color) are converted towhite pixels. The rest of them are converted to black pixels.tb

(a) (b)

Figure 1. (a) Original sub-image, (b) the same sub-image aftersecond stage of binarization.

Stage 2. At the second stage the pixels of the backgroundfor all the objects that consist of less than fifty pixels arebeing converted to black. Then, all pixels of the backgroundfor all the objects that consist of less than fifty pixels are

being converted to white. Finally, it considers the pixels ofthe biggest object as pixels of the foreground (see Fig. 1).

2.2 Contour Boundary Extraction

To achieve the extraction of the contour boundary it hasbeen used the Chain code algorithm with 8-th neighborhood[14]. Around a sub-image this technique considers that thereare two contours. The first one consists of pixels that belongto the foreground of the sub-image and it is called "innercontour". The other one consists of pixels that belong to thebackground and it is called "outer contour". The "innercontour" is being enveloped in the "outer contour" andevery pixel of "outer contour" is a neighbor of at least onepixel of "inner contour" and vice-versa. The basic idea isthat if two sub-images match together then the "outercontour" of the first sub-image matches with the "innercontour" of the other and vice-versa. In Fig. 2, "innercontour" is colored black and "outer contour" is colored red.

flftw,~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...f'itshn

Figure 2. "Inner" and "Outer" Contour of a puzzle's sub-image.2.3 Corner Detection

In order to detect CPs among the points of the contour, theproposed technique uses a corner detection algorithm, calledIPAN99 [10]. IPAN99 is a two-pass algorithm, whichdetects high curvature points and is rotationally and scaleinvariant. After performing IPAN99 some points of thecontour have been selected as CPs and IPAN99 assigns tothem two values. Every CP has a value ang(p), which iscalled sharpness and another one Cang(p), which decideswhether the corner with peak the point p is convex or

concave and assigns to point p the proper value.

2.4 Color Reduction

To achieve proper color reduction, our technique uses thecolor reduction method proposed by Papamarkos et al. [11]-[13]. This technique uses a KSOFM neural network, whichis fed with RGB color values. Color reduction procedurehelps in decreasing the number of the candidate CPsaccurately and therefore speeds up rapidly the entire jigsawsolving process.

In order to achieve stable color reduction results, theneural network is being fed with samples, the number ofwhich is proportional to the size of each sub-image andthese samples are only among foreground pixels (seeBinarization). Color reduction is being performed only in

2002

the beginning of the technique and the samples are takenfrom all the sub-images. This way, the dominant colorsobtained after performing color reduction are unique for allsub-images. Due to space limitations more informationabout color reduction can be found in the relative references.

2.5 Merging Procedure

After extracting the features for each sub-image, these sub-images are being compared, so as to decide whether theymatch together or not. During the comparing stage all cornerpoints that fail to satisfy a set of comparing conditions arebeing discarded. Comparing conditions include:Angle ComparisonThe angle Cang(pli) of every CP of the first sub-image is

being compared with the angle Cang(p20) of everycandidate CP of the second sub-image.

3500 < Caxng(pli) + Cxng(p2j)| < 3700 (1)If condition (1) is not satisfied, then the corresponding

candidate point will be discarded and will not be examinedany more.Color ConditionThe remaining candidate points are compared according

to their color. Since color reduction has been performed, allthe candidate points that have different colors, will bediscarded as well.Color DistanceThis feature expresses the Euclidean distance of color

values between a connected sequence of points from the firstimage and another from the second one. If{icl,j-5, ..., I, Pj,, icl,j+1 . icCl+5 } is a sequence of pointsof the first image around Pl,i and {iC2 j-1O * -, iC2,j-1P2,1j,ic2 j ,..., ic2 j0} another sequence of points around

P2,j of the other image, where ic are pixels of the "inner

contour" of a sub-image and Cic their color valuesrespectively, then:

Fk

E = minimum (Fk),

5

CpI -CP2,j + t5 1,cl±1 2,j+±+k1tO

11

point of the first image and p li-1, pj j+j are its nearest CPs,from both sides of P1j and P2,j is a candidate and

P2,j-1, P2,j+1 are its nearest CPs, from both sides of P2,j .At this point, all the possible combinations of the five-

point sets deriving from the first image's CPs are beingcompared with the five-point sets deriving from the secondimage's candidate CPs. Each five-point set of the secondimage consists of candidate points that correspond to thepoints of the first image's set. Every corner that is beingcreated from first set's point should not differ more than 4degrees, which is a suitable threshold, from thecorresponding corner of the second set and the norm of eachside of the first set's corner, should not differ more than 4pixels, which is a suitable threshold, from the correspondingnorm of the second set. If the proposed technique finds twomatching five-point sets then ends.

2.6 Finding exact position of matching points

The proposed technique considers the position of the first setof points constant and tries to find the exact position of thesecond set of points, in order for the merging image to havethe less possible gaps and overlaps.For each of the five points belonging to the second sub-

image's set, this technique finds all points that lie within arange of three pixels and belong to the "outer contour" ofthe second sub-image. Now this procedure forms all thepossible pairs between first sub-image's and theircorresponding second sub-image's pixels and it creates allthe possible five-point sets. Lets {(P1,1, P2,1)'1,2 P2,2)(1N,3, P2,3), (N1,4, P2,4), (Ns,5, P2,5 )} a candidate matching pair offive-point sets. The angle and the translation of the secondsub-image are being calculated by the segments

('1j, P1,5 ) and (p2,1, P2,5 ) Since the rotation angle and thetranslation of the second sub-image have been applied, theproposed technique calculates the sum of distances betweenthe other three sets of points (P1I 2, P2,2 ), (PI 3, P2,3 )

(2) (N,4 P2,4 and finds the total error D .

D = (PI,2 P2,2)+(PI,3 P2,3)+(PI,4,P2,4) (4)

(3)

where k = -5, ..., 5 represents the steps of the algorithm.The distance E is being calculated for all candidates left

of each point of the first image and we keep thosecandidates who correspond to the three minimum values ofE. All the other candidates are being discarded.Furthermore, the following set (pj,j-j, pj,i Xp,i+ ) is

compared with (P2,j-1, P2,j, P2,j+l ) as far as their angles andthe color is concerned. If (1) is not satisfied or the color isdifferent, the candidate P2,j is discarded, where plj is a

(a) (b)Figure 3. (a) After correcting the final location of the matchingpoints. (b) Before correcting the final location of the matchingpoints.

If D approaches zero, the method considers that these twosub-images match. The proposed technique examines all thepossible pairs of five-point sets and it concludes to the one

2003

with minimum total error. Fig. 3 shows the result with andwithout applying this algorithm.

3. EXPERIMENTAL RESULT

Although the proposed technique has been tested for manypuzzles, only one experiment will be presented, due to spacelimitation. In this experiment, the technique deals with aseven sub-images jigsaw puzzle. The extraction of everysub-image has been made randomly and also each one hasbeen rotated randomly. This puzzle has a rectangular outfit.Fig. 4(a) depicts the sub-images, and Fig. 4(b) shows theresult. The gaps that have been created between the mergedpieces in Fig. 4(b) are due to quantization errors throughrotation.

(a) (b)Figure 4. (a) All the sub-images of the puzzle (b) The resultwithout skew correction.

After merging the sub-images a skew correction techniqueis applied in order to transform the final image to itshorizontal orientation. The skew correction technique isbased on horizontal and vertical projection of the imagearea. That is, if the image has a rectangular outfit, verticalprojections reveal the X-coordinates of the four peaks of therectangle. Additionally horizontal projections reveal the Y-coordinates of the four peaks. This procedure is depicted inFig. 5. Actually, only points (X2, Y2) and (X4, Y4) areneeded here to find the angle and to correct the skew of theimage as it is shown in Fig. 5(b).However, it is possible rotation to produce additional

gaps. For this reason, a gap elimination procedure is appliedlocally to gaps regions. To do this, simple smoothing filters(like mean, Gaussian or min) can be applied.

iEBl _,

X------

(a) (b)Figure 5. (a) Skew correction with projections (b) The final resultof the technique after skew correction and min filter

4. CONCLUSIONS

The proposed technique, which deals with color jigsawpuzzles, was tested in a variety of jigsaw puzzles. Some ofthem were puzzles without a rectangular outfit, whose sub-

images are being rotated and generally were puzzles that donot satisfy classical conditions. It is a general method whosemain advantage is that it can deal with most types ofpuzzles.

Color reduction process helps the technique to exploiteffectively color information in order to improve the speedof the algorithm. Finally, even if two matching points, onefrom each sub-image, do not fit exactly, the entire mergingprocedure works satisfactory and hence the final image doesnot have gaps or overlaps between the merged sub-images.

REFERENCES

[1] H.C.G. Leitao and J. Stolfi, "Automatic reassembly ofirregular fragments", Tech. Report IC-98-06, Univ. ofCampinas, 1998.[2] H. Leitao and J. Stolfi, "Information Contents ofFracture Lines", Tech. Report IC-99-24, Univ. of Campinas,1999.[3] C. Wang, "Determining the Molecular Conformationfrom Distance or Density Data", PhD thesis, Department ofElectrical Engineering and Computer Science, MIT, 2000.[4] H. Freeman and L. Gardner, "Apictorial jigsaw puzzles:The computer solution of a problem in pattern recognition",IEEE Trans. on El. Computers, Vol. 13, pp. 118-127, 1964.[5] H. Wolfson, "On curve matching", IEEE Trans. onPattern An. and Machine Int., Vol 12, pp. 483-489, 1990.[6] H. Wolfson, E. Schonberg, A. Kalvin and Y. Lamdan,"Solving jigsaw puzzles by computer", Annals ofOperations Research, Vol. 12, pp. 51-64, 1988.[7] D. Goldberg, C. Mallon, and M. Bern, "A GlobalApproach to Automatic Solution of Jigsaw Puzzles", Proc.of the Eighteenth Annual Symposium on ComputationalGeometry, pp. 82-87, Barcelona, 2002.[8] W. Kong and B.B. Kimia, "On solving 2D and 3Dpuzzles using curve matching", Proc. IEEE ComputerVision and Pattern Recognition, Vol. 2, pp. 583-590, 2001.[9] F.H. Yao and G.F. Shao, "A shape and image mergingtechnique to solve jigsaw puzzles", Pattern RecognitionLetters, Vol. 24, pp. 1819-1835, 2003.[10] D. Chetverikov and Z. Szabo, "A Simple and EfficientAlgorithm for Detection of High Curvature Points in PlanarCurves", In M. Vincze, Editor, Robust Vision for IndustrialApplications 1999, Vol. 128 of Schriftenreihe derOsterreihicshen Computer Gesellschaft, pp. 175-184, 1999.[11] N. Papamarkos, A. Atsalakis and C. Strouthopoulos,"Adaptive Color Reduction", IEEE Trans. on Systems, Man,and Cybernetics-Part B, Vol. 32, pp. 44-56, Feb. 2002.[12] N. Papamarkos, "Color reduction using local featuresand a SOFM neural network", Int. Journal of ImagingSystems and Technology, Vol. 10, pp. 404-409, 1999.[13] T. Kohonen, Self-Organizing Maps, 2nd edition,Springer, Berlin, 1997.[14] H. Freeman, "On the encoding of arbitrary geometricconfigurations", IRE Trans. Electron. Comput. EC, Vol. 10pp. 260-268, 1961.

2004