Shift-Map Image Shift-Map Image EditingEditing
Yael Pritch, Eitam Kav-Venaki, Shmuel Peleg
Computer Science and EngineeringThe Hebrew University of Jerusalem,
IsraelICCV 2009
Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph
LabelingShift-Map ApplicationConcluding Remarks
Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph
LabelingShift-Map ApplicationConcluding Remarks
IntroductionIntroductionGeometric image rearrangement
is becoming more popular◦Image resizing (a.k.a. retargeting)◦Object rearrangement and removal
Early methods manipulation mostly crop and scale◦For image resizing, examining image
content and removing “less important” regions
IntroductionIntroductionSeam carving [2, 13]Continuous image warping [19,
16]Shift-map editing
◦Avoids scaling and mostly remove or shift image regions
(a) Original image(b) Video-retargeting [19](c) Optimized scale-and-stretch [16](d) Improved Seam Carving[13](e) Our shift-map editing
(a) Original image(b) Our shift-map editing(c) Video-retargeting [19](d) Optimized scale-and-stretch [16](e) Improved Seam Carving[13]
Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph
LabelingShift-Map ApplicationConcluding Remarks
Image Editing as Graph Image Editing as Graph LabelingLabeling
Shift-map◦The relative shift of every pixel in the
output image from its source in an input image
◦Represents the selected label for each output pixel
Two terms are used in computing the optimal shift-map◦Data term◦Smoothness term
Image Editing as Graph Image Editing as Graph LabelingLabeling
Input image I(x, y)Output image R(u, v)The relationship between input
image and output image is defined by◦Shift-map M(u, v) = ( , )
R(u, v) = I(x + , y + )Each output pixel can be labeled
by a shift ( , )
xt yt
xt yt
xt yt
Image Editing as Graph Image Editing as Graph LabelingLabeling
The optimal shift-map M minimizes the cost function :
◦ : data term◦ : smoothness term◦N : neighboring pixels◦ = 1
dE
sE
Single pixel data termSingle pixel data termPixel rearrangement
Pixel saliency and removal
◦S : saliency map, very high for pixels to be removed, very low for pixels not to be removed
Smoothness term for Smoothness term for pixels pairpixels pair
The smoothness term represents discontinuities added to the output image by discontinuities in the shift-map
Two neighboring location and in the output image R if
The smoothness term account color difference and gradient difference
11,vu 22 ,vu ),(, 2211 vuMvuM
))(),(( qMpMEs
Smoothness term for Smoothness term for pixels pairpixels pair
◦ : four unit vectors - four spatial neighbors
◦Color differences are Euclidean distances in RGB
◦ and are the magnitude of the image gradients at these locstion
◦ = 2
I R
ie
Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph
LabelingShift-Map ApplicationConcluding Remarks
Hierarchical Solution for Hierarchical Solution for Graph LabelingGraph Labeling
Finding the optimal graph labeling, the number of possible labels is the number of pixels in the input image
Use heuristic hierarchical approach reduces the memory and computational ◦First solved in a coarse resolution◦Higher resolution level
Hierarchical Solution for Hierarchical Solution for Graph LabelingGraph Labeling
Example : 4th pyramid level◦The number of pixels and number of
labels are reduce by a factor of 64
Coarse level
Coarse level
Input image Output image
64x64
32x32
16x16
4x4
Coarse shift-map
Nearest neighbor interpolation
Hierarchical Solution for Hierarchical Solution for Graph LabelingGraph Labeling
Use three to five pyramid levelsThe coarsest level contains up to
100 x 100 pixels
Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph
LabelingShift-Map ApplicationConcluding Remarks
Shift-Map ApplicationShift-Map ApplicationImage retargetingImage rearrangementInpainting Image composition
Image retargetingImage retargetingLabel order constraint
◦The shift-map will retain the spatial order
◦In the case of reducing width and , ,
◦In the case of increasing width and , ,
),(),( yx ttvuM ),(),1( ''yx ttvuM xx tt ' 0xt
),(),( yx ttvuM ),(),1( ''yx ttvuM xx tt ' 0xt
Image retargetingImage retargetingControlling object removal
◦It is possible to control the size and number of removed objects by performing several steps of resizing
◦Also possible to control object removal by marking objects as salient
◦The number of steps becomes the number of removed columns
(a)Original image(b) Resizing in single
step(c) Six smaller resizing
steps(d) Ten smaller resizing
steps
Shift-map retargeting :(a) Original image(b)(c)(e) No saliency(d) Child was marked salient
Original image [13] [19] [16] shift-map
Image rearrangementImage rearrangementMoving an object to a new image
locationDeleting part of the imageSpecified in two parts using the
data term◦Force pixels to appear in a new
location using Eq. 2◦Marks these pixels for removal from
their original location using Eq. 3
Example – 1 : ◦move the person and a part of the
temple to the right, and keep the tourists at their original location
Example – 2 :◦Kid on the left should move to the
center, baby should move to the left, kid on the right should remain in place
InpaintingInpaintingUnwanted pixels are given an
infinitely high data term as described in Eq. 3
Maps pixels inside the hole to other locations in the input image
InpaintingInpaintingA good complition with no user
intervention
Image compositionImage compositionIn the shift-map framework the
input can consist of either a single image, or of a set of images◦ , is the index of
the input imageTolerate misalignments between
the input images
),,(),( indyx tttvuM indt
Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph
LabelingShift-Map ApplicationConcluding Remarks
Concluding RemarksConcluding RemarksShift-maps are proposed as a new
framework to describe various geometric rearrangement problems
Images generated by the shift map are natural looking
Minimal and intuitive user interaction
Distortions that may be introduced by stitching are minimized
Large regions can be synthesized
Thank you!!!Thank you!!!