Patchwise scaling method for content-aware image resizing

Contents lists available at SciVerse ScienceDirect

Signal Processing

Signal Processing 92 (2012) 1243–1257

0165-16

doi:10.1

n Corr

culturalnn Cor

Sun Yat

E-m

sdliangy

journal homepage: www.elsevier.com/locate/sigpro

Patchwise scaling method for content-aware image resizing

Yun Liang a,b,c,n, Zhuo Su b,c, Xiaonan Luo b,c,nn

a College of Informatics, South China Agricultural University, Guangzhou 510642, Chinab School of Information Science and Technology, Sun Yat-sen University, Guangzhou 510006, Chinac National Engineering Research Center of Digital Life, Guangzhou 510006, China

a r t i c l e i n f o

Article history:

Received 14 October 2010

Received in revised form

21 September 2011

Accepted 19 November 2011Available online 2 December 2011

Keywords:

Image resizing

Patch division

Patchwise scaling

Image distance

84/$ - see front matter Crown Copyright & 2

016/j.sigpro.2011.11.018

esponding author at: College of Informatics,

University, Guangzhou 510642, China. Tel.:

responding author at: School of Information Scie

-sen University, Guangzhou 510006, China. Tel.:

ail addresses: [email protected],

[email protected] (Y. Liang), [email protected]

a b s t r a c t

Image resizing becomes more and more important in content-aware image displaying.

This paper proposes a patchwise scaling method to resize an image to emphasize the

important areas and preserve the globally visual effect (smoothness, coherence and

integrity). This method for resizing image is based on optimizing the image distance

presented in this paper. The image distance is defined based on so-called local

bidirectional similarity measurement and smoothness measurement to quantify the

quality of resizing outputs. The original image is divided into small important patches

and unimportant patches based on an important map. The important map is generated

automatically using a novel combination of image edge and saliency measurement. A

scaling factor is computed for each small patch. The resized image is produced by

iteratively optimizing, which is based on our image distance, the scaling factor for each

small patch. Experiments of different type images demonstrate that our method can be

effectively used in image processing applications to locally shrink and enlarge

important areas while preserving image quality.

Crown Copyright & 2011 Published by Elsevier B.V. All rights reserved.

1. Introduction

Image resizing is very crucial for effectively displayingimages on media devices with different resolutions andaspect ratios. A suitable resizing method should preserveimportant areas and globally visual effect (smoothness,coherence and integrity) when deforming images to agiven size. Although some interpolating methods havebeen proposed to change image resolution [1], theycannot simultaneously change their aspect ratios. Manyresizing methods have been proposed recently, yet mostof them have difficulty in simultaneously keeping objectappearances and emphasizing important objects while

011 Published by Elsevier

South China Agri-

þ86 13760698353.

nce and Technology,

þ86 13602858179.

su.edu.cn (X. Luo).

efficiently avoiding distortion and discontinuity. Based onan image distance presented in this paper, we propose anew resizing method by locally deforming the originalimage to emphasize important objects, preserve imagefeatures and avoid distortions.

The simple resizing methods, i.e., scaling and cropping,cannot provide ideal results, since scaling could result thatthe important objects is squeezed too small to be recognized(see Fig. 1(b)) and cropping could destroy image integrityfor losing information out of the clipping window (seeFig. 1(c)). Three main approaches have been proposed forcontent-aware image resizing. The first approach [2–5] isbased on removing or inserting unimportant pixels, such asthe seam carving method (SC) [2] and the shift map method(SM) [4]. The SC method is firstly proposed by Avidan et al.[2] and improved in the improved seam carving (ISC) [3]. Aseam of SC based methods is a path of 8-connected pixelsfrom top to bottom or from left to right with the leastenergy. The SC based methods greedily remove or insertseams passing through less important areas so they work

B.V. All rights reserved.

www.elsevier.com/locate/sigpro

www.elsevier.com/locate/sigpro

dx.doi.org/10.1016/j.sigpro.2011.11.018

mailto:[email protected]



dx.doi.org/10.1016/j.sigpro.2011.11.018

Original image

SM

Scaling Cropping ISC

OSS Dong’s BS Ours

Fig. 1. Comparison of different image resizing methods, we resize the original image (a) from 343�242 to 242�242.

Y. Liang et al. / Signal Processing 92 (2012) 1243–12571244

well for images with simple or blurred background. How-ever, these methods could not enlarge important areasenough in magnifying image and could induce new artifactsin shrinking image, especially on image edges (see Fig. 1(d)).The SM method works well for shrinking images withsimple textures but could remove important objects (seeFig. 1(e)).

The second approach [6–10] deforms the mesh of theinput image to produce the output image by minimizingsome distortion energies, such as the optimized scale-and-stretch method (OSS) [6]. But these methods areinsensitive to image features to produce some obviousdistortions, such as distortions around the straight linesor the contours of important objects (see Fig. 1(f)).

The third approach [11–13] produces a resizing resultthat is most similar to the input image among all theresizing outputs, such as the method of Dong (Dong’s)[12] and the bidirectional similarity method (BS) [13].Any one of this kind of methods is associated with adistance function defined to measure the similaritybetween the input image and its resizing output. As beingthe most similar to the input image, the resizing resultcontains as much as possible visual information from theinput and introduces as few as possible new artifacts.Using the distance function, Rubinstein et al. [11] com-bined scaling, cropping and seam carving to propose amulti-operator resizing method, and Dong et al. [12]advised another multi-operator resizing method by usingseam carving and scaling. These multi-operator resizingmethods do better than using single operator. But theycannot overcome the inherent shortage of its singleoperator such as losing information or inducing artifacts(see Fig. 1(g)). In addition, for [13], it could induce someunpredictable distortion and blurs (see Fig. 1(h)).

There are also some other image retargeting methodssuch as methods based on recomposing [14,15], saliency-

based cropping [16–21], patch transforming [22,23], fish-eye warping [24] and so on. These methods are proposedto shrink images. Recently, some video resizing methodshave been proposed, such as the motion-aware videoresizing [25] and the motion-based video retargeting[26]. The problem of image resizing still has not beenwell solved, although each of the methods mentionedabove works well for some specific cases.

In this paper, we present a new content-aware imageresizing method by patchwise deforming the sourceimage with differently scaling factors. The kernel of ourmethod is to divide the original image into small patchesand to compute the most suitable scaling factor for eachpatch to preserve important objects and avoid distortions.First, we present a scheme to automatically divide theoriginal image into small important patches and unim-portant patches based on an important map; the impor-tant map is computed by a novel combination of imageedge and saliency measures [27]. For computing the mostsuitable scaling factors, we define an image distancemeasurement to quantify the quality of the resizing out-put. The most suitable scaling factors are obtained byminimizing the image distance to preserve importantobjects, and to keep the smoothness and coherence ofresizing result. Compared with the existing methods, ourmethod is better in preserving the important objectswithout inducing more distortions (see Fig. 1(i)).

In summary, our main contributions are as follows:

(1)
Present a novel patchwise image resizing method toemphasize the important objects without inducingmore distortions because every important object isentirely included in a small important patch and thescaling factors of all patches can be patchwise assigned.
(2)
Combining a bidirectional similarity measurementand a smoothness measurement, we present a new

Fig.in (c

(e–i)

dista

colo

Y. Liang et al. / Signal Processing 92 (2012) 1243–1257 1245

image distance to quantify the preservation of themain details in the resizing result.

(3)
Present a new method to make it possible that onecan simultaneously shrink and/or enlarge differentparts of an image while keeping the distortion anddiscontinuity as little as possible.
2. Image resizing by patchwise scaling

This section describes our method to resize image bypatchwise scaling. We first compute the important map ofthe original image and identify its important areas. We covereach important area by a rectangle and its edges are parallelto the boundaries of image. Then, we divide the image intosmall rectangles based on the rectangles of important areassuch that any important area is covered by one smallrectangle patch. Still, the edges of each small rectangle patchare parallel to the boundaries of image. Thus, it is possible topatchwise assign a scaling factor for each small rectanglepatch. After each patch is assigned a scaling factor, we shrinkor enlarge the patch by some interpolation methods such asthe bilinear interpolation. By this way, we can well preservethe important objects by assigning suitable scaling factors oftheir covering rectangles. Then, the important areas will beproportionally deformed since each of them is applied byonly one scaling factor while other resizing methods cannotconsistently deform them. In addition, we can control thediscontinuity and distortion as little as possible by introdu-cing a new image distance. The final resizing result, whichhas the most suitable scaling factors for each patch, isobtained by minimizing the image distance.

The process of our scheme can be described in Fig. 2.We first compute the important map of the original image(see Fig. 2(b)) and normalize it to identify the important

Original image

Factor = 0.5

Important map

Factor = 1.1 Factor =

2. Process of our algorithm. We compute the important map (b) of the origi

)) of its important areas, then divide (a) into patches (d), and get five outpu

are 455.2�103, 245.6�103, 929.6�103, 447.1�103 and 429.2�103 an

nce. The numbers in (e–i) are the scaling factor for the important patch

ur in this figure legend, the reader is referred to the web version of this ar

areas (see Fig. 2(c)) by the method described in Section 2.1.By computing the boundary points of important areas, suchas the green points in Fig. 2(c), we divide the image intosmall important patches which covers an important area(see the patch covering flower in Fig. 2(d)), and unimportantpatches that covers unimportant areas. Then we resizeeach patch using a scaling factor, where we assign a scalingfactor for each important patch and the scaling factors ofunimportant patches are determined by the factors of theimportant patches. In fact, we obtain an optimal resizedimage by independently assigning different scaling factorsto each important patch (see Fig. 2(e–i)). The optimization isbased on an image distance that consists in evaluating thequality of the outputs, and the optimal output is mostsimilar to the original image while inducing the leastpossible discontinuity and deformation.

The optimal output is obtained by orderly implementingthree steps: divide the original image into small patchesbased on its important map; independently assign differentscaling factors to each important patch (the factors ofunimportant patches are consequently computed) andpatchwise deform the small patches; finally select theoptimal resizing result based on our image distance.

2.1. Image dividing based on important map

Previous image resizing methods have used variousmethods to compute the important map of an image.While Avidan et al. [2] and Wolf et al. [10] computed theimportance of each pixel based on its gradient value,Wang et al. [6] calculated it by multiplying the gradientand saliency [28]. Here, we propose a new measure ofimage importance that can better detect prominent imageobjects by combining image edge and saliency [27]. Withimage edges we emphasize pixels that are sensitive to the

Important area Patch division

1.4 Factor = 1.7 Factor = 2.0

nal image (a) (400�300), and identify the boundary points (green points

t images (e–i) (300�300) by patchwise scaling. The image distances for

d the optimal output of the five outputs is (f) for having least image

which covers the yellow flower. (For interpretation of the references to

ticle.)


contours of important objects. And with saliency weemphasize the areas that attract visual attention. Wecompute importance map Iimp (see Fig. 3(d)) as the sumof image edge Ie (see Fig. 3(b)) and saliency Is (seeFig. 3(c)) by Eq. (1). Here, Ie is computed with the SobelOperator; Is is calculated by the saliency of Harel [27]which is more powerful to predict human fixations onnatural images than Itti’s [28]; a, a constant coefficient, isset to 0.5 in our experiments. Usually, the mean value of Is

is smaller than that of Ie. In order to strengthen theinfluence of saliency to importance, we time it by aconstant. According to many experiments, we specifythe constant to be 10 as described in Eq. (1). Comparedwith the importance in [6], which is calculated bymultiplying saliency of Itti’s and gradient, our importantmap puts more emphasis on important areas, and useimage edges to reduce the effect of noise in gradientcomputation.

Iimp ¼ 10aIsþð1�aÞIe ð1Þ

We normalize the important map by specifying athreshold value to identify important areas. An importantarea is a connected big area such as the area covering theballoon in Fig. 3(e). For example, if there is only oneimportant area, it is the area with the biggest areas ofnormalized important map. We compute the boundarypoints of each important area such as the green points inFig. 3(e). Each important area is covered by a rectanglewhose edges are formed by the boundary points of thisarea and parallel to the boundaries of the image. Then, theimage is divided into small rectangle patches determinedby the rectangles that cover the important objects and theedges of each small rectangle patch are still parallel to theboundaries of the image, respectively (see Fig. 3(f)). Wewould like to emphasize that each important area is

Original image

Patch division

Image edge Saliency

{1.0,1.0} {1.0,0.6} {

Fig. 3. Patch division by the important map and patchwise resizing with differe

the image edge (b) and saliency map (c), and then normalize it to identify imp

divide image (a) (1511�1200) into small patches (f), and assign different scal

outputs (g–l) (900�1200). The numbers in (g–l) are the two scaling factors fo

always included in only one patch. By this way, one canpatchwise assign a scaling factor for each small patch toobtain different outputs (see Fig. 3(g–l)). We assume thatthe original image divided into m�n small patches,denoted by Pi,j(0r irm�1, 0r jrn�1). The scaling fac-tor of Pi,j is denoted by Si,j. We obtain the resizing outputby arranging the resized Pi,j using factor Si,j, together inthe same order of Pi,j.

If the image has only one important object, we divide itinto no more than 9 patches (see Fig. 2(d)). If the imagehas more than one important object, the patch divisionbecomes more complicated (see Fig. 3(f), Figs. 4(d) and6(a)). We specify that the number of small importantpatches will be no more than three, since people cannotfocus on too many different places at the same time whenhe/she views a picture.

2.2. Scaling factor and patchwise deforming

We consider one directional resizing, say the widthdirection resizing. In fact, we resize width first and thenheight if we need change both of them, instead of in twodirections simultaneously. To reduce the shearing effect,we assign only one scaling factor to each column of smallpatches. Similarly, we assign one scaling factor to eachrow of patches when do height direction resizing. We onlyconsider the width direction resizing to describe ourpatchwise scaling method. For each small patch, we willdeform it to fit in the target image by the way that thepatch is distorted as little as possible if it covers someimportant objects while diffusing the deformation intounimportant patches. In this section, we give out themethod to distribute the scaling factors of the smallpatches covering important areas, to obtain the restscaling factors, and to deform each patch.

map Important map Important areas

0.6,1.0} {1.3,0.7} {0.6,0.6} {1.2,1.2}

nt scaling factors. We first compute the important map (d) by combining

ortant areas as (e). After covering the important areas by rectangles, we

ing factors to each small important patch to compute different resizing

r the important patches covering the left balloon and the right balloon.

Original image

{0.63,0.63,0.63}

Important map Important areas Patch division

{0.9,0.5,0.6} {0.45,0.9,0.5} {0.5,0.5,1.0} {0.7,0.8,0.35} {0.7,0.5,0.7} {0.5,0.5,0.5}

Fig. 4. Resize image by distributing scaling factors to small important patches. There are three important patches in (d), which covers fish. We resize the

original image (a) from 700�438 to 400�438. The numbers in the outputs (e–k) are the scaling factors for the three important patches covering the left

fish, the middle fishes and the right fish, respectively.

Original image

{0.9,0.7}

Important map Important areas Patch division

{1.0,1.0} {0.6,0.95} {0.725,0.725} {1.2,1.2}

Fig. 5. Resize an image after computing the scaling factors of unimportant patches. We resize (a) from 400�290 to 290�290. Two important patches

are selected and each covers a dog as (d). The numbers of (e–i) are the effective combinations of the scaling factors for the important columns covering

the left dog and the right dog, respectively.


2.2.1. Distributing scaling factors of important patches

Since the quality of the final resized image is mostlydetermined by the resizing of the important objects, wefirst resize the patches covering important objects. Tokeep image integrity and to preserve the important areas,the scaling factors of important patches should be neithertoo small, nor too big. The scaling factor Si,j of eachimportant patch Pi,j should be bounded above by the ratioSmaxi,j of the target size over the size of Pi,j, and the lowerbound Smini,j of the scaling factor should be big enoughsuch that the resizing of important object is not too smallto be recognized. We specify Smini,j to be the ratio of thetarget size over the original size. To produce a series ofresizing outputs from which we select the optimal resiz-ing result, we assign Ni, jþ1 scaling factors to Pi,j which is

defined by

Si,jðtÞ¼ Smini,jþtðSmaxi,j�Smini,jÞ=Ni,j t¼ 0,1,. . .,Ni,j: ð2Þ

We ask Ni,jZ20. In addition, two important patchesshould be assigned the same scaling factor if they are on thesame column. For example, if Pi1,j1 and Pi2,j2 lies on the samecolumn, their Smax is the smaller of Smaxi1,j1 and Smaxi2,j2,and their Smin is the bigger of Smini1,j1 and Smini2,j2. InFig. 4, we give an example with three important patchesand seven resizing outputs by assigning seven scalingfactors to each important patch, respectively.

If we treat each important patch as an independentaxis and each scaling factor of this important patch as thecoordinate of this axis, then considering all the importantpatches together, we obtain a linear space called scaling

Original image with patch division

{0.65,0.6,0.62}

Important map Important areas

{0.58,0.93,0.9} {0.95,0.5,0.89} {0.95,0.92,0.5} {0.9,0.9,0.85}

Fig. 6. Resize image giraffe from (a) (600�380) to (380�380). The red lines in (a) is used to divide the original image into patches. There are three

important patches in (a) and each covers a giraffe. The important patch covering the left giraffe is adjacent with the important patch covering the middle

giraffe. The numbers of (d–h) are the scaling factors for the three important columns covering the left giraffe, the middle giraffe and the right giraffe,

respectively. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)


factor space. Each component of a point in this linearspace is a scaling factor. That is, if P1,P2,y,Pk are all theimportant patches and S1,S2,y,Sk are one set correspond-ing scaling factors, then (S1,S2,y,Sk) is a point in thescaling factor space. Each Si, i¼1.2,y,k, takes the valuesgiven by Eq. (2). (S1,S2,y,Sk) is called an effective combi-nation of factors if

Pki ¼ 1 Si9Pi9 is not more than the target

width, while 9Pi9 means the width of patch Pi. Only theeffective combination of scaling factors can be used toresize the important patch, and to compute the scalingfactors of unimportant patches.

2.2.2. How to compute the scaling factors of unimportant

patches

Based on above description, each column should beassigned only one scaling factor to reduce distortions sothe small patches on same column have the same scalingfactor in resizing. For the scaling factor S of an unimpor-tant patch P, if P is on the same column of an importantpatch Pi,j, its scaling factor should be the same as Pi,j. Ifthere is no important patch on the same column with P, itis reasonable to ask that all this kind of unimportantpatches have the same scaling factor because they havesame little importance and should equally sharing thedistortion in resizing. To give out the scaling factors of thiskind of unimportant patches, we denote by {C1,C2,y,CN}all the columns of the input image, and divide itinto important columns {Ci1,Ci2,y,Cim}, which include atleast one important patch and unimportant columns{Cj1,Cj2,y,Cjn} that do not include any important patch.At the same time, we describe W0 as the target width,Sik,Sjk as the scaling factors of Cik,Cjk, 9Cik9,9Cjk9 as the widthof Cik,Cjk. Then after an effective combination of scalingfactors is given by Eq. (2), i.e., the scaling factor Sik of Cik

are given and satisfy W 0imprW 0, with W0 imp¼

PSik9Cik9 is

the total width of all resized important columns, the

scaling factor S of an unimportant patch P can be com-puted as follows:

S¼Sik, if P 2 Cik and Cik 2 fCi1,Ci2,:::,Cimg,

1Wunimp

ðW 0�W 0

impÞ, if P 2 fCj1,Cj2,:::,Cjng,

(ð3Þ

where Wunimp ¼P

Cjk2fCj1 ,Cj2 ,:::,Cjng9Cjk9 is the total width of

unimportant columns.After each small patch is assigned a scaling factor by

Eqs. (2) or (3), we deform it to suit the target image by aninterpolating method such as the bilinear interpolation.Fig. 5 describes an image resizing with two importantcolumns, three unimportant columns, and five resizingresults by assigning five effective combinations of scalingfactors. Fig. 6 describes another image resizing withcomplicated patch division.

2.3. Obtain the optimal resizing result by image distance

According to the resizing scheme described in Section2.2, each effective combination of factors of importantpatches produces a resizing output and different effectivecombinations of factors produce different resizing out-puts. Each resizing output is considered as a possibleresult. After assigning all the effective combination offactors of important patches by Eq. (2) and computing thecorresponding factors of unimportant patches by Eq. (3),we obtain all the possible outputs for one image resizing.Then, we compute the optimal resizing result among allthe possible outputs based on the image distance definedby the following Eq. (4).

DImageðI,OÞ ¼ bDSimðI,OÞþð1�bÞDSmoothðI,OÞ ð4Þ

where I is the original image, O is a possible output, DSim

represents the bidirectional difference of color informa-tion between I and O, and DSmooth describes the differenceof smoothness of them, bA[0.1] is a parameter with


default value 0.7 in our experiments. DSim and DSmooth areto be defined later.

2.3.1. Image similarity based on bidirectional distance

We propose the bidirectional distance DSim by Eq. (6)to measure the similarity between the original image andits resizing output. Like the method in [29], we alsocalculate the similarity between images using colorsimilarity. An ideal resizing result should have twoproperties: one is that the output contains the informa-tion of the original image as much as possible; the other isthat the artifacts of the output are produced as few aspossible.

We still need to give some notations for Eq. (6). By a k-patch we mean a square portion of an image includingk� k pixels. Thus, an image of m�n pixels contains(m�kþ1)� (n�kþ1) k-patches. In our experiments, wetake k¼7. Let P¼(pi,j) and Q¼(qi,j) be two k-patches. Then,the distance D(P,Q) between P and Q is defined by

DðP,Q Þ ¼Xk�1

j ¼ 0

Xk�1

i ¼ 0

9pipþ i,jpþ j�qiqþ i,jqþ j92, ð5Þ

where (ip, jp) and (iq, jq) are the top-left sub-indexes of P

and Q, respectively. 9pi, j�qi, j9 is the difference of the colorinformation between pi, j and qi, j. In addition, we assumethe input image I is divided into m�n small patches, sayPi,j, i¼0.1,...,m�1, j¼0.1,...,n�1. O is a resizing output of I.At the same time, by PCPi,j we mean that P is a k-patch ofthe small patch Pi,j, and by QCQi,j we mean that Q is ak-patch of Qi,j. Finally, we define the similarity distanceDSim as follows:

DSimðI,OÞ ¼1

mn

Xm�1

i ¼ 0

Xn�1

j ¼ 0

l1

NPi,j

XP�Pi,j

minQ�Oi,j

DðP,Q Þ

zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{DSim_compðPi,j ,Oi,jÞ8>>>><>>>>:

þð1�lÞ1

NOi,j

XQ�Oi,j

minP�Pi,j

DðQ ,PÞ

zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{DSim_coheðOi,j ,Pi,jÞ9>>>>=>>>>;

, ð6Þ

where NPi,j,NOi,j are the numbers of k-patches in Pi,j andOi,j, respectively. (lA[0.1]) is a parameter with defaultvalue 0.5 in our experiments. Similar to [13], the first termDSim_comp of Eq. (6) is the deviation of the resized k-patchPi,j from the completeness and the second DSim_cohe is thedeviation of Pi,j from the coherence.

The completeness describes the preservation of inputimage. The coherence describes the distortions of outputimage. We evaluate them based on the comparisons oflocal k-patches of I and O. If each local k-patch P of I iscompletely included in O, the output is the most completeone. P is just a small k-patch of I, and describes its localinformation. Q describes a small k-patch of O, and is usedto compare with P, so the distance between P and Q has

no relation with the sizes of I and O. By minQ�Oi,jDðP,Q Þ,

we search the most similar k-patch of P in O, namely Q,and compute the completeness distance DSim_comp

between them to evaluate how much the P is preserved.If Q satisfies D(P,Q)¼0, it means P is wholly contained in

O. Smaller D(P,Q) means P is preserved better. By sum-ming the completeness distance DSim_comp of each P, weevaluate the completeness of I. The situation is similar for

minP�Pi,jDðQ ,PÞ.

Although similar to the method of [13], our similaritydistance is more powerful, because we compute thedistance based on the local information rather thanthe global information as in [13]. For example, using themethod of [13] to compute DSim_comp, the k-patch Q withthe least distance from k-patch P of Pi,j is globally

searched in the output O by minQ�ODðP,Q Þ, but by our

method it is searched only in Oi,j by minQ�Oi,jDðP,Q Þ. In

fact, it is more reasonable that Q from Oi,j than O. Thesituation is similar for DSim_cohe. In a word, our similaritydistance is more effective than the method in [13] and canbring better resizing result.

2.3.2. Smoothness distance

The smoothness distance between the original imageand its resizing output, defined by Eq. (7), is an importantterm in our image distance. Because our image resizing ispatchwise, it is inevitable to introduce discontinuitiesaround the common boundaries of adjacent smallpatches, such as the left side of the flower in Fig. 7(c). Inthis paper, an area around a common boundary is called amixed area which is composed of some pixel segmentsnear to the common boundary, such as the area of theblue rectangle in Fig. 7(b). Each pixel segment is the samesize with and parallel to the common boundary.

To horizontally resize image, each column has onescaling factor so the discontinuities exist in the mixedareas between adjacent columns. We describe this kind ofmixed area as horizontal mixed area, such as the area ofthe blue rectangles in Fig. 7(b,d,f). If only one directionalresizing applied, all the mixed areas are perpendicular tothis direction. In fact, there are six horizontal mixed areasin Fig. 7(d,f), although only two are marked by bluerectangles. If a horizontal mixed area of the original imageis formed by the pixel segments of patch Pi,j and Pi,jþ1, themixed area formed by the pixel segments of resizedPi,j,Pi,jþ1 is called its corresponding mixed area in theoutput. For example, the mixed area of the green rectan-gle in Fig. 7(d) is corresponding to that of the greenrectangle in Fig. 7(b). The difference of intensity andgradient information between the horizontal mixed areasand its corresponding mixed areas reflects the disconti-nuities of the output. Less difference means more con-tinuous. Unlike [12,13], based on the difference ofintensity and gradient information between mixed areas,we define smoothness term as Eq. (7) to measure thediscontinuity of the resizing outputs.

We denote by MAI the set of all the mixed areas of theinput image. By pAMAI we mean that p is one mixed areaof MAI. In addition, we denote by WP the width of p,HP theheight of p, and q the mixed area in an output corre-sponding to p. The width and height of q is the same as p.At the same time, pi,j,qi,j describes the pixel of mixed areap,q, rpi,j, rqi,j is the gradient of pi,j,qi,j. Then the smooth-ness distance between the original image and its resizing

Original image Mixed areas (output1) Output2 Mixed areas (input) Output1 Mixed areas (output2)

Fig. 7. Two mixed areas around the important patch covering flower. Each mixed area is composed of 24 pixel segments, 12 pixel segments to each side. The

smoothness distance between the mixed areas in the green rectangles of (d) and (b) is 381�103 and the smoothness distance between the mixed areas in the

green rectangles of (f) and (b) is 207�103. It is clear that there are more discontinuities in the mixed area of the green rectangle in (d) than that of the green

rectangle in (f). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Table 1Image distances of the different resizing outputs in Fig. 2.

Output image Scaling factor DSim_comp DSim_cohe DSim DSmooth DImage

Fig. 2(e) (0.5) 702.1�103 134.7�103 418.4�103 541�103 455.2�103

Fig. 2(f) (1.1) 183.4�103 298.4�103 240.9�103 256.4�103 245.6�103

Fig. 2(g) (1.4) 98.7�103 484.8�103 296.8�103 2406.3�103 929.6�103

Fig. 2(h) (1.7) 69.5�103 657.4�103 363.5�103 642.4�103 447.1�103

Fig. 2(i) (2.0) 25.6�103 807.6�103 416.7�103 458.5�103 429.2�103

Table 2Image distances of the different resizing outputs in Fig. 5.

Output image Scaling factor DSim_comp DSim_cohe DSim DSmooth DImage

Fig. 5(e) (0.9.0.7) 1501.1�103 318.9�103 910.1�103 928.6�103 915.6�103

Fig. 5(f) (1.0.1.0) 706.1�103 254.2�103 480.2�103 157.8�103 383.5�103

Fig. 5(g) (0.6.0.95) 1194.4�103 421.1�103 807.8�103 935.9�103 846.2�103

Fig. 5(h) (0.725.0.725) 481.4�103 342.6�103 412�103 734.4�103 508.7�103

Fig. 5(i) (1.2.1.2) 453�103 379.1�103 416.1�103 1492.7�103 738.9�103


output is defined by

DSmoothðI,OÞ ¼X

p2MAI

1

k

XWP

i ¼ 1

XHP

j ¼ 1

9pi,j�qi,j92þ

1

k

XWP

i ¼ 1

XHP

j ¼ 1

9rpi,j�rqi,j92

!, ð7Þ

where k¼2552.The first part of Eq. (7) describes the intensity differ-

ence between the mixed areas of the input image andtheir corresponding mixed areas of the output. The secondpart of Eq. (7) describes their gradient difference. Fordifferent outputs of an image, smaller DSmooth means lessdiscontinuity is produced and better resizing result isobtained.

The image distances of Fig. 2(e–i) are shown in Table 1.The scaling factors in Table 1 are assigned to the impor-tant patch including yellow flower, respectively. The out-put image given in Fig. 2(f) has the least distance and isthe best one among all the outputs (e–i). In fact, it is thebest one in the sense of common esthetics. The Table 2gives the distances of the outputs of Fig. 5(e–i). It is clearthat the best output with the least image distance of themis Fig. 5(f).

In summary, based on an image distance, we present apatchwise resizing method to obtain an optimal output asthe following steps:

(1)
Compute the important map of an image. (2) Divide the image into small patches based on its
important map, and classify the patches into importantpatches and unimportant patches.

(3)
To each important patch, independently assign differ-ent scaling factors by Eq. (2).
(4)
Select effective combinations of the scaling factors inthe scaling space, and for each effective combinationof the scaling factors compute the scaling factors ofunimportant patches by Eq. (3).
(5)
For each effective combination of the scaling factors,compute a resizing output by patchwise deforming.
(6)
For each resizing output calculate the image distance,and select the output image with minimum imagedistance as the final resizing result.
3. Experiments

We use PC with Intel Pentium(R) T2370 1.73 GHz, 2 GBRAM to implement our experiments and choose varykinds of images from landscapes to human body picturesas the inputs. It costs 3.21 min to compute an output(300�300) from an input (400�300). Some results arepresented in this section and more results are given out inthe appendix.

3.1. Image retargeting

Image retargeting is the process of shrinking image.Our method shrinks image patchwise, which is verydifferent from the current methods. We compare ourmethod with the multi-operator image resizing methods


in Figs. 8 and 9. Some other comparing results are shownin Figs. 10–12.

The multi-operator resizing method proposed in [11]performs well for some images. However, it loses someinformation (see the left stick in Fig. 8(b1)) for combiningcropping, and destroys the proportion of important parts forremoving too many seams across the same area (see thewings of the eagle in Fig. 8(b2)). The multi-operator resizingmethod of [12] brings discontinuities for using seam carving

Original image Multi-operator Ours

Fig. 8. Comparison of our result with that the result of multi-operator imag

(400�600) to (400�460), and resize (a2) (400�268) to (200�268).

Original image Dong’s

Fig. 9. Comparison of our result with the result of Dong’s [12] and Multi-op

(350�400).

Original image

Scaling

Important mapand area

Patch

Cropping ISC

Fig. 10. Some retargeting results for image bird. We shrink the image bird from

the retargeting result (d). (e–k) are some results from other retargeting metho

(see the right arm in Fig. 9(b)), and this shortage still existsin the result of [11] (see the right arm in Fig. 9(c)). Under theconstraint of similarity distance and smoothness distance,our method preserves the integrity (see Fig. 8(c1)) andsmoothness (see Fig. 9(d)). By patchwise scaling, ourmethod can preserve the proportion of important parts(see the wings of eagle in Fig. 8(c2)). Comparing with themulti-operator image resizing methods, our method canproduce more visually pleasing results.

Original image Multi-operator Ours

e resizing method proposed in [11]. We resize the original image (a1)

Multi-operator Ours

erator method in [11]. We resize the original image (a) (500�400) to

division Ours Dong’s

OSS SM BS

(430�250) to (250�250) by our method with patch division (c) to get

ds.

Original image

Scaling

Important mapand areas

Patch division Ours Dong’s

Cropping ISC OSS SM BS

Fig. 11. Some retargeting results for image man and girl. We shrink image (a) from (500�250) to (250�250) by our method with patch division (c) to

get the retargeting result (d). (e–k) are some results from other retargeting methods.

Original image

Scaling

Important mapand areas

Patch division Ours Dong’s

Cropping ISC OSS SM BS

Fig. 12. Some retargeting results for image heart. We shrink image (a) from (343�242) to (242�242) by our method with the patch division (c) to get

the retargeting result (d). (e–k) are some results from other retargeting methods.


As the simple retargeting methods, the scaling methodcannot emphasize important areas and squeezing them toomuch (see Figs. 10(f) and 11(f)). The cropping methoddestroys image integrity because of losing information outof the clipping window (see the branch in Fig. 10(g), theleft trees in Fig. 11(g) and the heart in Fig. 12(g)). The ISCmethod is very effective to preserve the areas of blurrybackground, but induces too many new artifacts (see thebird tail in Fig. 10(h), the girl in Fig. 11(h) and the boundaryof heart in Fig. 12(h)). As a mesh-based retargeting method,the OSS method is insensitive to the contours of importantobjects, and induces distortions on important objectsespecially on the boundaries of the important objects(see the body of bird (Fig. 10(i))). The SM method shrinksimage by removing unimportant pixels, but sometimespixels of important objects are removed for maintaining

the continuity or smoothness of the retargeting result(see the bird in Fig. 10(j), the people in Fig. 11(j) and thebuilding in Fig. 12(j)). The BS method brings unpredictableblurs and lost information for it did not consider imagestructure (see the bird tail in Fig. 10(k), the man’s left leg inFig. 11(k)). The multi-operator resizing method of Dong’sinduces discontinuities (see the bird tail in Fig. 10(e), theheart in Fig. 12(e)) for combining seam carving, andsqueezes the important objects (see the man and the girlin Fig. 11(e)) for using scaling.

Our results in Fig. 10–12 maintain image integritysince there is no main information lost. The bird of ourresult in the image bird enjoys better shape and muchsmoother outline, since the main part of bird is includedin the important patch and is not zoomed in too much.The man and girl in Fig. 11(d) are well preserved without


any distortion, because their regions are included in theimportant patches and undergo coherent resizing. Thebird in Fig. 10(d) and the heart in Fig. 12(d) are onlyweakly shrunk and well preserved by diffusing the strongshrinkage into unimportant patches. Comparing withexisting methods, our method preserves important areasmuch better while producing better visual effect, such assmoothness, coherence and integrity.

We employ the method of Simakov’s [13] to measurethe quality of retargeting results by their distances to theoriginal image, see Fig. 13. Smaller distance means betterresult. As shown in Fig. 13, it is clear that the values of ourresults are smaller than other retargeting results whichmean that our results are better than the others.

3.2. Magnify image non-homogeneously with shape

preservation

Non-homogeneously magnifying image is anotherimportant use of image resizing, which can display imageon bigger screens with different aspect ratios. The naıveand content agnostic method such as scaling cannotpreserve the shape of important object (see Fig. 14(e)).The ISC method cannot enlarge important objects (seeFig. 14(f)). The OSS method brings distortions on impor-tant areas (see the right foot in Fig. 14(g)) and inducesoverlaps (see the tower body and the hand in Fig. 14(g)).The multi-operator resizing method of Dong’s cannot

Fig. 13. The distances (defined in [13]) between the original images

(Figs. 10–12, 17) and their retargeting results by different methods. The

y-coordinates have been scaled out by 103.

Importantarea

Patchdivision

Originalimage

Importantmap

Scaling ISC

Fig. 14. Comparison of some results for image tower and boy.

emphasize the important objects enough (see the boyand the tower in Fig. 14(h)).

The shapes of the tower and the boy in Fig. 14(i) arewell preserved, because each of them is covered by someimportant patches and is patchwise resized. The resizedboy in Fig. 14(i) is well emphasized without destroyingthe integrity, unlike the result of Fig. 14(g), and alsowithout flattening the body, unlike the result ofFig. 14(e). Comparing to the others, our result looks morenatural.

We still use the method of Simakov’s [13] to measurethe quality of the results of Fig. 14, see Fig. 15. Again, it isclear that the values of our results are smaller than theother results which demonstrate that our results arebetter than the others.

3.3. Shrink image while magnifying important objects

When browsing images with high resolution onmobiles, PDA and other small display devices, we needimage retargeting. However, sometimes we want toenlarge important objects big enough to be recognized.In this section, we describe a novel technique to shrinkimage while magnifying its important objects by patch-wise resizing. To this end, we assign to each importantpatch a scaling factor of more than one, and then shrinkimage using the technique described in Section 2. Forexample, the important patches of covering the boat and

OSS Dong’s Ours

We magnify image (a) from (100�180) to (180�180).

Fig. 15. Distances (defined in [13]) of the original images (Fig. 14 (a)) to

the magnified results of different methods Fig. 14 (e–i). The y-coordi-

nates have been scaled out by 103.

Original image Patch divisionimportant mapand area

Scaling Ours

Fig. 16. Shrink image while magnifying its important objects. We shrink images boat and bud from (400�300) to (300�300) while magnifying the boat

and bud, respectively.

Original image

OSS

Cropping Scaling ISC

SM BS Dong’s Ours

Fig. 17. Retargeting results of image car. We shrink the image (a) from (400�300) to (300�300).


the bud (see Fig. 16(c)) are assigned a scaling factor ofmore than one. By this way, the important patch ismagnified while the image is shrunk. As shown inFig. 16, the boat and the bud are magnified while theimage is shrunk. Using this technique, the importantobjects can be preserved well or emphasized as the focuscontext in [30], although the method in [30] is proposedto deal with 3D objects.

4. Conclusion and discussion

This paper presents a patchwise image resizing method.That is, after the input image is divided into small patches,each small patch can be resized independently. The final

output is the optimal result obtained by an image distancewe present in this paper. The optimization based on thisimage distance makes it possible that the important objectsof the input can be well preserved while the output has alsobetter global visual effect.

A disadvantage of our method is that if the patchescovering a characteristic of the input are divided intodifferent columns, it could produce shearing around theboundary of important and unimportant objects (see thewhite lines in Fig. 17(i)). This shearing is produced fromthe different patch scaling for the same object. In fact, thisshearing also exists in the results of other methods,except cropping (see Fig. 17(c–h)). One of our futureworks is to propose some new constraints to reduce such


shearing, such as the straight line constraint. Anotherfuture work is to reduce the processing time, and to applyour method to real-time content aware video resizing.

Acknowledgments

We thank the anonymous reviewers for their valuablecomments. We are grateful to Mr. Yu-Shuen Wang forsharing his enlightening views and to Prof. Weiming Dongfor providing the multi-operator resizing comparisons.We also thank to Prof. Xiquan Shi for the insightfuldiscussions on algorithms, and thank to Mr. Sixuan Zhongfor polishing English. This work was supported by NSFC-Guangdong Joint Fund (nos. U0735001, U0835004 andU0935004), 973 Program of China (no. 2011CB302204),the National Science Fund of China (no. 61103162) andthe Fundamental Research Funds for the Central Univer-sities of China (no. 1109021170001137105).

Appendix 1

Comparison of the resizing results of different meth-ods. Scaling method cannot preserve the shape of theimportant object, such as the starfish in the second row.ISC method introduces discontinuity, such as the tow inthe first row. OSS methods brings distortions on

Original image Scaling ISC

Fig. A1. Non-homogeneous image magnifying with shape preservation

important objects, such as the tow in the first row andthe mushroom in the fourth row. Dong’s method can’temphasize the important objects, such as the tow in thefirst row and the water lily in the third row (Fig. A1).

Appendix 2

We shrink some images while magnifying its impor-tant objects. The tow in the first row, the buildings in thesecond and the third row, and the people in the fourthrow are magnified while the backgrounds of them areshrunk. Compared with the scaling method and croppingmethod, our method can produce more visually pleasingresults (Fig. A2).

Appendix 3

Comparison of some different retargeting methods.The scaling method squeezes the important objects, suchas the fish, the dear and the panda in the second column.The Cropping method loses information, such as theimage panda and the image fish in the third column.The ISC method brings discontinuity, such as the choco-late and the dear in the forth column. The OSS methodcannot emphasize the important objects such as thewindmill and the fish in the fifth column. The SM method

OSS Dong’s Ours

(change the aspect ratio of the image from 233:360 to 360:360).

Original image Scaling Cropping Ours

Fig. A2. Shrink image while magnifying its important objects (change the aspect ratio of the image from 500:373 to 373:373).

Original Image Dong’sScaling Cropping ISC OSS SM BS Ours

Fig. A3. Image retargeting (change the aspect ratio of image from 500:313 to 313:313).



removes some important objects, such as the fish, thechocolate and the dear in the sixth row. The BS methodbrings blurs such as the image windmill in the seventhcolumn. The Dong’s method introduces distortions anddiscontinuity, such as the chocolate and the panda in theeighth column. Our method can preserve the importantobjects without introducing more distortions. Comparedwith these retargeting methods, our method can producemuch better results (Fig. A3).

References

[1] F. Ar�andiga, R. Donat, P. Mulet, Adaptive interpolation of images,Signal Processing 83 (2003) 459–464.

[2] S. Avidan, A. Shamir, Seam carving for content-aware imageresizing, ACM Transactions on Graphics 26 (3) (2007) 267–276.

[3] M. Rubinstein, A. Shamir, S. Avidan, Improved seam carving forvideo retargeting, ACM Transactions on Graphics 27 (3) (2008) 1–9.

[4] Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-Map Image Editing, in: IEEEInternational Conference on Computer Vision, 2009, pp. 151–158.

[5] R. Achanta, S. Susstrunk, Saliency detection for content-awareimage resizing, in: IEEE International Conference on Image Proces-sing, 2009, pp. 1005–1008.

[6] Y.S. Wang, C.L. Tai, O. Sorkin, T.Y. Lee, Optimized scale-and-stretchfor image resizing, ACM Transactions on Graphics 27 (5) (2008).

[7] G.X. Zhang, M.M. Cheng, S.M. Hu, R.R. Martinx, A shape-preservingapproach to image resizing, Computer Graphics Forum 28 (7)(2009) 1897–1906.

[8] J. Shi, Y.W. Guo, Z.L. Du, F.Y. Zhang, Q.S. Peng, Mesh parameteriza-tion-based image retargeting method, Journal of Software Suppl 19(2009) 19–30.

[9] Y. Guo, F. Liu, J. Shi, Z.H. Zhou, M. Gleicher, Image retargeting usingmesh parameterization, IEEE Transactions on Multimedia 11 (5)(2009) 856–867.

[10] L. Wolf, M. Guttmann, D. Cohen-Or, Non-homogeneous content-driven video retargeting, in: IEEE International Conference onComputer vision, 2007, pp. 1–6.

[11] M. Rubinstein, A. Shamir, S. Avidan, Multi-Operator media retarget-ing, ACM Transactions on Graphics 28 (3) (2009) 1–11.

[12] W. Dong, N. Zhou, J.C. Paul, X. Zhang, Optimized image resizingusing seam carving and scaling, ACM Transactions on Graphics 28(5) (2009) 1–10.

[13] D. Simakov, Y. Caspi, E. Shechtman, M. Irani, Summarizing visualdata using bidirectional similarity, in: IEEE Computer SocietyConference on Computer Vision and Pattern Recognition, 2008,pp. 1–8.

[14] V. Setlur, T. Lechner, M. Nienhaus, B. Gooch, Retargeting imagesand video for preserving information saliency, IEEE ComputerGraphics and Applications 27 (5) (2007) 80–88.

[15] T.W. Ren, Y.W. Guo, G.S. Wang, F.Y. Zhang, Constrained samplingfor image retargeting, in: IEEE International Conference on Multi-media and Expo, 2008, pp. 1397–1400.

[16] A. Santella, M. Agrawala, D. Secarlo, D. Salesin, M. Cohen, Gaze-based interaction for semiautomatic photo cropping, in: the SIGCHIconference on Human Factors in Computing Systems, 2006,pp. 771–780.

[17] V. Setlur, S. Takagi, R. Raskare, M. Gleicher, B. Gooch, Automaticimage retargeting, in: the International Conference on Mobile andUbiquitous Multimedia, 2005, pp. 59–68.

[18] H. Liu, X. Xie, W.Y. Ma, H.J. Zhang, Automatic browsing of largepictures on mobile devices, in: ACM International Conference onMultimedia, 2003, pp. 148–155.

[19] B. Suh, H. Ling, B. Bederdon, D. Jacobs, Automatic thumbnailcropping and its effectiveness, in: ACM Symposium on User Inter-face Software and Technology, 2003, pp. 95–104.

[20] H.Y. Liu, S.Q. Jiang, Q.M. Huang, C.S. Xu, W. Gao, Region-basedvisual attention analysis with its application in image browsing onsmall displays, in: ACM Multimedia: 2007, pp. 305–308.

[21] L.Q. Chen, X. Xie, X. Fan, W.Y. Ma, H.J. Zhang, H.Q. Zhou, A visualattention model for adapting images on small displays, MultimediaSystems 9 (4) (2003) 353–364.

[22] T.S. Cho, M. Butman, S. Avidan, W.T. Freeman, The patch transform andits applications to image editing, in: IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition, 2008, pp. 1–8.

[23] C. Barnes, E. Shechtman, A. Finkelstein, D.B. Goldman,A. PatchMatch, Randomized correspondence algorithm for struc-tural image editing, ACM Transactions on Graphics 28 (3) (2009).

[24] F. Liu, M. Gleicher, Automatic image retargeting with fisheye-viewwarping, in: ACM Symposium on User Interface Software andTechnology, 2005, pp. 153–162.

[25] Y.S. Wang, H. Fu, O. Sorkine, T.Y. Lee, H.P. Seidel, Motion-awaretemporal coherence for video resizing, ACM Transactions onGraphics 28 (5) (2009) 1–10.

[26] Y.S. Wang, H.C. Lin, O. Sorkine, T.Y. Lee, Motion-based videoretargeting with optimized crop-and-warp, ACM SIGGRAPH 2010papers, 2010, pp. 1–9.

[27] J. Harel, C. Koch, P. Perona, Graph-based visual saliency, in:Proceedings of the NIPS. 2006.

[28] L. Itti, C. Koth, E. Niebur, A model of saliency-based visual attentionfor rapid scene analysis, IEEE Trans. on Pattern Analysis andMachine Intelligence 20 (11) (1998) 1254–1259.

[29] X. Zhang, J. Yang, The analysis of the color similarity problem inmoving object detection, Signal Processing 89 (2009) 685–691.

[30] Y.S. Wang, T.Y. Lee, C.L. Tai, Focusþcontext visualization withdistortion minimization, IEEE Transactions on Visualization andcomputer graphics 14 (6) (2008) 1731–1738.

Documents

Patchwise scaling method for content-aware image resizing