Image RenaissanceUsing Discrete Optimization
Cédric Allène Nikos Paragios
ENPC – CERTIS
ESIEE – A²SIECP - MAS
France
Image Renaissance
Outline
1)Theory: positions and patches
2)Finding positions with good similarities
3)Determining the patch sizes
4)Results and discussions
Image Renaissance1
- T
heory
: p
osit
ion
s a
nd
patc
hes
Main idea: inpainting as a jigsaw
Inpainting: Reconstructing a destroyed part of an image (specified by a
mask) The process should be undetectable
Our method: Considering the reconstruction as a jigsaw which pieces are
patches taken from the “good” part of the image
Image Renaissance
Two steps1) Finding positions/offsets in the “good” image having strong
similarities with the content of the image at the boundary of the area to rebuild
2) For each offset found, finding which pixels should be copied to reconstruct the destroyed area
1 -
Th
eory
: p
osit
ion
s a
nd
patc
hes
Image Renaissance
Positions and offsets
Such a problem consists in finding, given an origin point at the boundary of the region to inpaint, a number of candidate positions in the image domain that have similar content with the one at the boundary of the area to rebuild
A few remarks: Missing content is not necessarily in the neighborhood of the
destroyed region… Testing each offset in the image for each position at the boundary of
the destroyed area would be too expensive in computation time…
Our solution: using a probabilistic method, the particle filters
2 –
Fin
din
g p
osit
ion
s w
ith
good
sim
ilari
ties
Image Renaissance
Particle filters theory Considering a fixed number of samples, we have to repeat this
steps: Weight the samples with a Probability Density Function (pdf) Generate a new set of samples (of equal number) from the existing
ones with the biggest weights applying small perturbations (importance of perturbation decreases with the iterations to obtain a convergence)
2 –
Fin
din
g p
osit
ion
s w
ith
good
sim
ilari
ties
Image Renaissance
Particle filters in positions research
A particle has an origin point (at the boundary of the area to inpaint, which is invariant), an offset and a size (width and height)
The perturbation is a displacement of the offset and a modification of the size
The weight is evaluated by a Sum of Square Differences (SSD) between the sized neighborhood around the origin point and the offset point balanced with the size
2 –
Fin
din
g p
osit
ion
s w
ith
good
sim
ilari
ties
Image Renaissance
How to find the optimal partition
For each pixel in the area to inpaint we have multiple offsets (and, as a consequence, multiple pixel values possible), so we have to find the best label partition to rebuild the image:
In that aim we use a combinatorial optimization: the α-expansion algorithm (using graph cuts)
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
Image Renaissance
Term 1 to minimize:constraints to data
: region of the image to be reconstructed
: region of the image to be reconstructed and its close neighborhood (which contains data)
: distance (similarity) between the label and the data of the image at position
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
Image Renaissance
Term 2 to minimize:constraints between patches
: region of the image to be reconstructed
: 4-neighborhood of
: distance (similarity) between the label L and the label at position
: Euclidian distance from to the boundary of the area to rebuild (to reconstruct firstly the borders of the region)
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
Image Renaissance
Term 3 to minimize:smoothing term
: region of the image to be reconstructed
: 4-neighborhood of
:
Potts model:penalty for neighbors with different
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
Image Renaissance
Global energy function on partition
to minimize
ω
Minimized through successive α-expansion (using graph cuts)
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
Image Renaissance
Alpha-expansion principle
With a given partition, the algorithm expands the label α over all the other labels minimizing the energy function
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
"green"-expansion
Image Renaissance
Alpha-expansion algorithm
For each offset selected (explaining the interest to reduce their number), we:
Construct a graph representing the energy on the area to inpaint Execute the graph-cut algorithm on the graph Modify the partition
This is repeated until stability (but in practice, one iteration on each offset gives a result near of the final result)
Each iteration will so enhance the result by minimizing the global energy
Y. Boykov, O. Veksler and R. Zabih.Fast Approximate Energy Minimization via Graph Cuts.
PAMI 2001.
3 –
Dete
rmin
ing
th
e p
atc
h s
izes
Image Renaissance
Conclusion
Graph-based combinatorial approach for inpainting
Advantages: Efficient for textured images Quite fast: in general a couple of minutes (but the final result is
nearly reached after the first series of α-expansion)
Drawbacks: Polynomial cost depending on the number of offsets selected (so
particular attention must be paid on this step)
4 –
Resu
lts a
nd
dis
cu
ssio
n