Scale and object aware image retargeting for thumbnail browsing

Scale and Object Aware Image Retargeting for Thumbnail Browsing

Jin Sun, Haibin Ling

Temple University, Philadelphia, PA, 19122

By perillaroc

ICCV 2011

Introduction

• Image browsing tasks

• Tiny thumbnails: a fixed small size

• Scaling

– bring difficulties in searching and recognition

• Automatic image retargeting methods

– target size is comparable to that of the original image

• Several important issues for thumbnail browsing

2012/6/19 2

Thumbnail Browsing

• Thumbnail scales

– Studies have shown that scales can have

significant effects on human visual perception

• Object completeness

– Low-level gradient-based information

– NOW! The object-level completeness

• Structure smoothness

– The contamination caused by pixel removal

methods, e.g. seam carving

2012/6/19 3

Scale and Object Aware Retargeting

• Thumbnail scales

– An image perceived by human vision system

– A new scale dependent saliency

• Object completeness

– integrating the objectness measurement recently

proposed by Alexe et al.

• Structure smoothness

– Use the thin-plate-spline(TPS) as warping function

– Use Cyclic seam Carving to guide the warping.

2012/6/19 4

Scale and Object Aware Retargeting

• A continuous retargeting algorithm but uses discrete retargeting to guide the warping transformation.

2012/6/19 5

Overview : Task Formulation

• Retargeting problem:

• Seam carving(SC)

• First get retargeted images to a target size that is comparable to the original image size.

• Then shrink the retargeted images to get the final thumbnails.

( , ( ))J F I S I

,

( , ), ( )

( , 1), ( )

sc

i j

I i j if j seam iF

I i j if j seam i

2012/6/19 6

Overview : Framework Overview

• Scale and object aware image retargeting(SOAP):

• Warping function: thin-plate-spline (TPS)

• Landmark points : cyclic seam carving (CSC)

• Scale dependent saliency

• Object awareness

• Scale and object aware saliency

• SOAP

soF

( )scaleS I

( )O I

( ) ( ) ( )so scaleS I S I O I

( , ( ))so soJ F I S I

2012/6/19 7

Flow chart of the proposed method

2012/6/19 8

Scale-dependent Saliency

• Original image size

• Display size

• Projection size

• DPI(Dots Per Inch)

,d rpo

d p

s Dss s

DPI D

2012/6/19 9


• How well such a system preserves the image information?

• We want to make the foreground object/ content/theme of the image as “clear” as possible

• According to the study, not all patterns in an image are recognizable by human

2012/6/19 10

Scale-dependent Saliency • the perceived image, denoted as , can be derived as

• is the mean value of N-connected neighbors of pixel I(i,j)

• N is determined by display device specifications.

• κ and ρ are the lower and upper bounds(in cycles per degree) respectively of human visual acuity.

• κ and ρ define the limits at which the visual stimuli frequency becomes too low or too high to be recognized by human.

• We use κ = 0.0175 and ρ = 0.83.

pI

( , ), / [ , ]( ( , ))

( , ),

d

p

N

I i j if s DI I i j

I i j otherwise

( , )NI i j

2012/6/19 11


• A pixel may become indistinguishable from its neighbors to a human observer.

• An image patch that was salient in the original image may not appear salient to a human observer.

• Scale dependent saliency

2012/6/19 12


• First, the original image is scaled in homogeneous into the final thumbnail size, i.e. the display size, 60x60 pixels in our experiment.

• Then, the minimum recognizable pattern, denoted by

is determined by Eqn.4.

• Finally, the scale dependent saliency is defined as

where S(.) is the standard saliency (Itti 1998)

( ) ( )scale

pS I S I

( )scaleS I

ds

2012/6/19 13

Scale-dependent Saliency • the minimum recognizable pattern

• In our monitor : 1680X1050@65Hz , 120 DPI

• is in average 0.009 inches.

• The size is approximately the distance between two pixel lines on the screen, i.e. N = 4 in Eqn.5.

• As a result, in the final thumbnail four adjacent neighbors of one image pixel patch with value differences in color space within certain threshold will be assigned their mean value, which means those pixels are unable to be distinguished by human.

• Differences : 50 in our experiment

ds

ds

2012/6/19 14


2012/6/19 15

Objectness Measure

• Preserve object completeness as much as possible

• Alexe et al. What is an object? (CVPR,2010).

– A novel objectness measure, which is trained to distinguish object windows from background ones.

– For a rectangular window w = (i,j,w,h) with a top-left corner at (i,j), width w and height h, its objectness is defined as the probability p(w) that w contains an object.

2012/6/19 16

Objectness Measure

• First sample windows for an input image

• Then calculate the objectness map O as the average objectness reponse at each pixel

Where

• = 10000 in our experiments

wn1{ }w

i iW w

( , )

1( , ) ( )obj

w W i j w

O i j p w

,max ( , )i j O i j

wn

2012/6/19 17

Scale and Object Aware Saliency

( , ) ( , )so scaleS S i j O i j

2012/6/19 18

Cyclic Seam Carving • In many cases a seam has no choice but to cross objects due

to the original definition of seam.

• Discontinuous seams

• Discontinuous seam-carving for video retargeting. CVPR, 2010

• Problem: object-structure damage, e.g. pixel shift problem

2012/6/19 19

Cyclic Seam Carving

• Cyclic Seams: warp the image

2012/6/19 20

Cyclic Seam Carving

• Energy function

where ρ is the weight and set to 0.3 .

• The improved energy is then combined with the CSC algorithm to provide landmark point pairs needed for estimating TPS warping.

(1 ) so

scale scE E S

2012/6/19 21

Cyclic Seam Carving

2012/6/19 22

Image Warping Function

• Many discrete retargeting methods generate excellent results in general but they sometimes create serious artifacts when the target has a size much smaller than the input.

• We combine a continuous warping model with a discrete retargeting guidance.

• Thin-plate-spline(TPS) as our warping function

2012/6/19 23

Thin-plate-spline(TPS)

• Landmark points and

, where p is mapped to q.

• the TPS transformation T is defined as the transformation from P to Q that minimizes the regularized bending energy E(f) defined as

• The TPS warping is defined as

2 , 1,2, ,i lP p i n

2 , 1,2, ,i lQ q i n

2

2 2 22 2 2

2 2

( ) ( )

( ) 2( ) ( )

i i

i

E f q f q

f f fdxdy

x x y y

argmin ( )fT E f

2012/6/19 24

TPS Warping Function

• Landmark point pairs P and Q is derived from the CSC retarget algorithm

• A two-way solution

• First, We sample randomly a landmark set b P (Fig. 6(a)) from original image and then trace their shifting in the CSC process.

2012/6/19 25

TPS Warping Function

• Then, point set Q is re-sampled uniformly on the target image, which is generated by CSC.

• Finally, a sample set P is generated by mapping Q to the original image using warping estimated by Q and P.

2012/6/19 26

Quantitative Experiments

• To find the thumbnail that matches the description

• From 10 x 10 image thumbnails

• Different methods

– Scaling(SL)

– Seam carving (SC)

– Improved seam carving (ISC)

– The proposed SOAR algorithm (SOAR)

2012/6/19 27


2012/6/19 28


2012/6/19 29

Qualitative Experiments

• In general our method performs the best regarding the (foreground) object size in the thumbnails.

2012/6/19 30

RetargetMe

2012/6/19 31

Limitations

• Saliency distribution is scattered

• Object is to big

2012/6/19 32

Conclusion

• A scale and object aware image retargeting method for thumbnail browsing.

• Several new techniques

– Scale dependent saliency

– Objectness

– Cyclic seam carving

– A TPS-based continuous warping model

2012/6/19 33

Technology

Scale and object aware image retargeting for thumbnail browsing