Equations of fluid dynamics for image inpainting of ﬂuid dynamics for image inpainting Evelyn M. Lunasin Department of Mathematics United States Naval Academy Joint work with E.S

Equations of fluid dynamics for imageinpainting

Evelyn M. LunasinDepartment of Mathematics

United States Naval Academy

Joint work with E.S. TitiWeizmann Institute of Science, Israel

.IMPA, Rio de Janeiro

March 21, 2014

Evelyn Lunasin · United States Naval Academy Equations of fluid dynamics for image inpainting


NSE, NotationNS-α model of turbulence [Foias, Holm, Titi 1998]

Generalized α-model [Holst, Lunasin, Tsogsogerel, 2010]

Grayscale image inpainting basics

[Bertalmio, Bertozzi, Sapiro ’01]

Connection between the image intensity function I and thestream function ψ in the 2D incompressible fluid.

[Y.Cao, Lunasin, Titi ’06], [Oskolkov ’73, ’80], [Kalantarov, Titi ’07]

Navier-Stoke-Voight equation as a subgrid scaleturbulence model.

[Ebrahimi, Lunasin, Holst ’11]

2D Navier-Stokes Voight for image inpainting[Lunasin, Titi, ’14]

Two-dimensional NSE with linear and nonlinear damping.


The Navier-Stokes Equations (θ = 1)

∂tu + (−∆)θu + (u · ∇)u +∇p = f , in

∇ · u = 0,

u(0) = u0

(1)

+ boundary conditions


The Navier-Stokes-α Equations (θ = 1, θ2 = 1)

∂tu + (−∆)θu + (Mu · ∇)u +∇(Mu)T · u +∇p = f ,

∇ · u = 0,

∇ · (Mu) = 0,

u(0) = u0,

(2)

where M = (I + (−α2∆)θ2)−1.


Consider the following system on a three-dimensional flat torusT3:

∂tu + (−∆)θu + (Mu · ∇)u +∇(Mu)T · u +∇p = f ,

∇ · u = 0,

∇ · (Mu) = 0,

u(0) = u0,

(3)

where M = (I + (−α2∆)θ2)−1.

Olson-Titi (2007): Viscosity vs. vorticity stretching: sufficientconditions on the relationship between θ and θ2 are establishedto guarantee global well-posedness and global regularity ofsolutions.

This family of NS-α-like model equations interpolates between incompressible hyper-dissipative equations and theNS-α models when varying the two nonnegative parameters θ and θ2.


A more general model: [Holst, Lunasin, Tsogsogerel, 2010]

∂tu + Au + (Mu · ∇)(Nu) + χ∇(Mu)T · (Nu) +∇p = f (x),∇ · u = 0,

u(0) = u0,

(4)

whereA, M, and N are bounded linear operators having certainmapping propertiesχ is either 1 or 0.θ to control the strength of the dissipation operator A.two parameters which control the degree of smoothing inthe operators M and N, namely θ1 and θ2, respectively,when χ = 0, and θ2 and θ1, respectively, when χ = 1.


Some examples of operators A, M, and N which satisfy themapping assumptions we will need in this paper are

A = (−∆)θ, M = (I − α2∆)−θ1 , N = (I − α2∆)−θ2 , (5)

for fixed positive real number α and for specific choices of thereal parameters θ, θ1, and θ2.


As a result, the system in (??) includes the Navier-Stokes equations and the various previously studied α

turbulence models as special cases, namely,

∂tu + Au + (Mu · ∇)(Nu) + χ∇(Mu)T · (Nu) +∇p = f (x),∇ · u = 0,u(0) = u0,

(6)

Table: Some special cases of the model (??) with α > 0, and withS = (I − α2∆)−1 and Sθ2 = [I + (−α2∆)θ2 ]−1.

Model NSE Leray-α ML-α SBM NSV NS-α NS-α-likeA −ν∆ −ν∆ −ν∆ −ν∆ −ν∆S −ν∆ ν(−∆)θ

M I S I S S S Sθ2

N I I S S S I Iχ 0 0 0 0 0 1 1


[Holst, Lunasin, Tsogsogerel, 2010]

We have established necessary and/or sufficient conditions on theranges of the three parameters for dissipation and smoothing in orderto obtain :

Existence, regularity, uniqueness, and stability.Existence and finite dimensionality of global attractors.Bounds on the number of determining degrees of freedom.



Direct Numerical Simulations


Settling for averages





The filtered velocity field u acts like the mean flow in RANS.











Overview and Recent Advances

IEEEJan ’14p.127


Overview and Recent Advances


What is image inpainting?

Image inpainting is the process of correcting a damaged imageby filling in the missing or altered data of an image with a bettersuited data for that region.


Applications

Recovery of missing areas in a decoded images of videotransmission suffering from packet losses.


Image inpainting basics

Idea: Fill damaged part using information from itssurrounding area.Goal: Automation of inpainting which mimics manualtechniques used by professional restorer.


Categories

Diffusion basedinpainting

1 PDE models topropagate localstructures fromexterior to interior ofthe hole

2 many variants: linear,nonlinear, isotropic,anisotropic, favoringpropagation inparticular directions

3 well-suited forinpainting smallregions


Categories:

Diffusion basedinpainting

1 PDE models topropagate localstructures fromexterior to interior ofthe hole

2 many variants: linear,nonlinear, isotropic,anisotropic, favoringpropagation inparticular directions

3 well-suited forinpainting smallregions

4 not well-suited forrecovering texture forlarge areas, as theytend to blur the image.Evelyn Lunasin · United States Naval Academy Equations of fluid dynamics for image inpainting

Categories - Second Family of Inpainting methods

Examplar-based1 Goal: produce an

image larger than theinput sample with asimilar visualappearance.

2 the texture to besynthesize is learnedfrom a similar regionby sampling and thenstitching togetherpatches calledexamplar.

3 better suited thandiffusion method forfilling large texturedareas.


Summary

Diffusion methods - good for small and sparsely distributedgaps, but unable to restore textureExamplar-based methods - work well in textured regionswith regular patterns, but not well-suited for perservingedges


Categories - Third Family of Inpainting methods

Hybrid Methods - Separates structure and texture then addthe result together.


PDE-based methods

We focus on PDE-based methods.Two main steps:

Retrieve local image geometry - by computing gray levellines (also called isophotes)Use PDEs to describe the evolution of the image structures



Image I:m× n matrix with grayscalevalue 0 to 255 at each pixel.

0.27 0.11 0.58 0.500.68 0.49 0.22 0.690.65 0.95 0.75 0.890.16 0.34 0.25 0.95



Isophotes - are lines of constant intensity within an image.Recall: The discretized gradient vector gives the largestspatial changes so one obtains the direction of isophotesby rotating the gradient vector 90.



Let Ω ⊂ D be the inpainting region, where D is the set of points(x, y) where I is defined.

The image inpainting problem

Find I∗ such that I∗ = I for fixels in D− Ω and I∗ appears tohave ’suitable’ values for the pixels in Ω.



Method: Moving image intensity along isophote lines.

The direction of isophotes (level lines of equal color gradient) andwhere smoothness should propagate.


Basic Mathematical Concepts

∇⊥I: Direction of isophotes (level lines of equal grayscaleintensity)∆I: Smoothness of the image.

We want

∇⊥I · ∇∆I ' 0.

The isophote lines, in the direction of ∇⊥I, should be almostparallel to the level curves of the smoothness ∆I of the imageintensity.


PDE for image inpainting

The proposed algorithm is then to get a discrete approximationof the steady state solution of the PDE

Bertalmio, Sapiro, Ballester, Casellas 2000

It = ∇⊥I · ∇∆I + ν∇ · (∇I)

The additional term ν∇ · (∇I) can be modified toν∇ · (g(|∇I|)∇I) to account for anisotropic diffusion (edgepreserving diffusion).Without the presence of viscosity in the method, we do nothave uniqueness of steady state solution.


Analogy of transport of vorticity in 2D incompressiblefluids

Recall the vorticity-stream formulation of the 2D NSE,

∂ω

∂t+ u · ∇ω = ν∆ω. (7)

Here u = ∇⊥Ψ is the velocity, where Ψ is the stream functionand ω = ∇× u is the vorticity. If the viscosity ν is zero, thesteady state solution for the stream-function Ψ in 2D satisfies

∇⊥Ψ · ∇∆Ψ = 0. (8)

Notice the similarity between

We want

∇⊥I · ∇∆I = 0.

The isophote lines, in the direction of ∇⊥I, must be parallel to thelevel curves of the smoothness ∆I of the image intensity.


This elegant analogy enables the automation of theinpainting procedure using techniques from fluid dynamics.However, the difficulties which arise in CFD are alsoinherited.Possible solution: sub-grid scale turbulence modeling.


The Navier-Stokes-Voight equations

We simulate the inviscid 2D NSV with diffusion

− α2 ∂

∂t∆ω +

∂ω

∂t+ u · ∇ω = ν∇ · (∇ω), (9)


NSE vs NSV


NSE vs NSV


NSE vs NSV


NSE vs NSV

Comparison between NSE and NSV.


Measuring the quality of an image using PSNR

Peak-signal-to-noise-ratio (PSNR)

DefinitionLet P(i, j) be the original image that contains N by M pixels andI(i, j) the reconstructed image. The pixel values range betweenblack (0) and white (255). The PSNR in decibels (dB) arecomputed as follows:

PSNR = 20 log10

(255

RMSE

)(10)

where

RMSE =

√∑i∑

j (P(i, j)− I(i, j))2

N ∗M.


PSNR of NSE vs NSV

Comparison between NSE and NSV.


PSNR of NSE vs NSV

Comparison between NSE and NSV with smaller timestep.


Uniqueness of steady state solution

Dependence of solution to the image at the boundary, size ofthe inpainting region and viscosity.

Idea:Given the physical data φ defined on ∂Ω, we can find anextension Φ of φ inside Ω. Then if Φ satisfy some conditions,we have at least one solution (u, p) to the non-homogeneoussteady problem. If

ν2 > 2Cλ−1/21 ‖f‖, (11)

where f = ν∆Φ− B(Φ,Φ), and λ1 ∼ 1/L2, then thenonhomogeneous 2D NSE and 2D NSV has a unique steadystate solution.


2D damped NSE

Instead of 2D NSV:

−α2 ∂

∂t∆ω +

∂ω

∂t+ u · ∇ω = ν∇ · (∇ω),

we use:

γω +∂ω

∂t+ u · ∇ω = ν∇ · (∇ω),


Uniqueness of steady state solution for 2D dampedNSE

Dependence of solution to the image at the boundary, size ofthe inpainting region and viscosity.

Idea:Given the physical data φ defined on ∂Ω, we can find anextension Φ of φ inside Ω. Then if Φ satisfy some conditions,we have at least one solution (u, p) to the non-homogeneoussteady problem. If

ν2(2γ +ν

L2 ) > 8‖f‖, (12)

where f = −γΦ + ν∆Φ− B(Φ,Φ), then the nonhomogeneous2D NSE and 2D NSV has a unique steady state solution.


2D nonlinearly-damped Euler equations with artificialviscosity

Instead of 2D NSV :

γ1ω +∂ω

∂t+ u · ∇ω = ν∇ · (∇ω),

γ2|u|2ω +∂ω

∂t+ u · ∇ω = ν∇ · (∇ω),




Observations

Non-linear damping gives better results vs linear damping(PSNR 48 vs 46 db).Both models satisfy Maximum Principle.


Challenges, Future work

Maximum principleAutomation for multi-region defects.Parameter selection.Hybrid methods. Joint work with E.Titi, I. Popovich, W.Withers


Thanks!

Thank you for your time.Many thanks again to the organizers for this opportunity.

contact info: [email protected]

Department of MathematicsUnited States Naval Academy

Annapolis, Maryland


Documents

Equations of fluid dynamics for image inpainting of ﬂuid dynamics for image inpainting Evelyn M. Lunasin Department of Mathematics United States Naval Academy Joint work with E.S