Upload
woonghee-lee
View
772
Download
1
Embed Size (px)
Citation preview
Region Filling and Object Removal by Exemplar-Based Image Inpainting
Criminisi, A., Perez, P., and Toyama, K. (2004)
Lee, WoongheeM.S. student at the Big Data Mining Lab.
Department of computer science and engineering at the Hanyang UniversityOctober 4th, 2015
Contents Prerequisite Key Observations Region Filling Algorithm Results and Comparisons
Prerequisite Texture synthesis Inpainting
PrerequisiteTo fill large image repetitively 2-D texture synthesis
inpainting
Exmaples from Ashikimin [1]
PrerequisiteProperties
• Cheap cost to generate image• Effectively generating image• Difficult to fill holes in photos• A complex product of mutual
influences between different boundaries
texture synthesis
inpainting
PrerequisiteTo fill holes in images by propagating linear structures (called isophote)
texture synthesis
inpainting
PrerequisiteTo fill holes in images by propagating linear structures (called isophote)
Depends on Gestalt Law of Continuation
texture synthesis
inpainting
PrerequisiteGestalt Law of Continuation texture synthesis
inpainting
PrerequisiteGestalt Law of Continuation texture synthesis
inpainting
PrerequisiteGestalt Law of Continuation
Human perceives a dotted line as a full line by implicit continuation.
texture synthesis
inpainting
PrerequisitePropagation direction texture synthesis
inpainting
propagate along isophotes
PrerequisiteProperties
• Effective to fill speckles, scratches, and overlaid text
• Causes noticeable blur to fill large regions
• Extremely slow (83’-158’ on a 384 X 256 image)
texture synthesis
inpainting
Main Idea
To combine the advantages of “texture synthesis” and “inpainting”
Key ObservationsA. Exemplar-Based Synthesis SufficesAlgorithm Core: Isophote-driven image-sampling process
Key ObservationsA. Exemplar-Based Synthesis Suffices
Key ObservationsB. Filling Order is Critical
artefacts
Key ObservationsB. Filling Order is Critical
Onion peel(concentric-layer odering) causes “over shooting” → To achieve balancing between the structured regions and texture regions.
Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ𝑝|
𝐷 𝑝 =|∇𝐼𝑝
⊥∙𝑛𝑝|
𝛼
Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω
Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ𝑝|
𝐷 𝑝 =|∇𝐼𝑝
⊥∙𝑛𝑝|
𝛼
Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω
higher priority
lower priority
Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)
|Ψ𝑝|
𝐷 𝑝 =|∇𝐼𝑝
⊥∙𝑛𝑝|
𝛼
Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω
similar priority
Region Filling Algorithm2) Propagating Texture and Structure InformationAfter computing priorities, setting the highest priority Ψ 𝑝
To avoid diffusion, propagating image texture from the source region
Ψ 𝑞 = arg𝑚𝑖𝑛Ψ𝑞∈Φ𝑑(Ψ 𝑝, Ψ𝑞)
Region Filling Algorithm3) Updating Confidence ValuesAfter filling the patch Ψ 𝑝, the confidence term is updated
𝐶 𝑝 = 𝐶 𝑝 , ∀𝑝∈ Ψ 𝑝 ∩ Ω
It does not require additional parameter to specify image.
Region Filling AlgorithmThe 𝑡 indicates the current iteration.
Region Filling AlgorithmProperties of the region filling algorithm
Recall 𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
The priority equation achieves balance of effects and an organic synthesis
Region Filling AlgorithmProperties of the region filling algorithm
𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝
• avoids an arbitrary fill order.• eliminates the risk of “broken-structure”
artefacts.• propagates strong edges.• reduces blocky and misalignment artefacts
without additional step.
Region Filling AlgorithmImplementation Details
The target 𝛿Ω is manually selected.The normal direction 𝑛𝑝 is computed as1) Contour’s “control” points are filtered via
2D Gaussian kernel2) estimated as the orthogonal unit vector of
𝛿Ω
Region Filling AlgorithmImplementation Details
The gradient ∇𝐼𝑝is computed as the MAX value in Ψ𝑝 ∩ 𝐼
Pixels are classified as belonging to• The target region Ω• The source region• The remainder
Results and Comparisons
Experimental environment was a 2.5-GHz Pentium IV with 1GB of RAM.
To compare with the results of earlier work.
Results and ComparisonsKanizsa Triangle and the Connectivity Principle
Results and ComparisonsComparing Different Filling Orders
original image target region
raster-scan concentric
Harrison’s2 m 45 s
Ours5 s
Results and ComparisonsComparing Different Filling Orders
original image target region
Results and ComparisonsComparing Different Filling Orders
raster-scan concentric
Results and ComparisonsComparing Different Filling Orders
Harrison’s45 m
Ours2 s
Results and ComparisonsComparing Different Filling Orders
Using only data term leads the “over shoot”
Results and ComparisonsComparisons With Diffusion-Based Inpainting
original image target region
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
Results and ComparisonsComparisons With Diffusion-Based Inpainting
onion peel ours
Results and ComparisonsComparisons With Diffusion-Based Inpainting
priority function for before image
Priority function is 0 for inside and 1 for outside Final priorities made the continuation of the pole
Results and ComparisonsComparisons With Diffusion-Based Inpainting
original image target region
Results and ComparisonsComparisons With Diffusion-Based Inpainting
Isophotes hits the thin boundary
Results and ComparisonsComparisons With Diffusion-Based Inpainting
ours traditional image inpainting (blurry)
Results and ComparisonsComparisons With Diffusion-Based Inpainting
target region
texture and structure inpainting (blurry) ours
Comparison With Drori et al.
Results and Comparisons
Drori et al. (blurry)
Examples on PhotographsResults and Comparisons
Drori et al. (blurry)
Examples on PhotographsResults and Comparisons
Drori et al. (blurry)
onion peel ours
Examples on PhotographsResults and Comparisons
ours
“bow-tie” effect
Thank you for listening