Monotonicity Enhancing Nonlinear Diffusion

Journal of Visual Communication and Image Representation 13, 313–323 (2002)

doi:10.1006/jvci.2001.0484, available online at http://www.idealibrary.com on

SHORT COMMUNICATION

Monotonicity Enhancing Nonlinear Diffusion1

Pavel Mrazek

Center for Machine Perception, Department of Cybernetics, Faculty of Electrical Engineering,Czech Technical University, Technicka 2, 166 27 Praha 6, Czech Republic

E-mail: [email protected]: http://cmp.felk.cvut.cz/˜mrazekp

Received February 7, 2000; accepted June 15, 2000

We consider the task of filtering the noise from images and other types of inputswhich are assumed to be piecewise continuous and piecewise monotone. We showthat nonlinear diffusion of the data, a powerful filtering method, is too restrictive forsuch a case, leading to piecewise constant functions. We claim that the piecewisemonotonicity can be enhanced by nonlinear diffusion of first partial derivatives ofthe input data. The method is developed in this paper; we introduce the algorithmsand present experimental results. C© 2002 Elsevier Science (USA)

Key Words: nonlinear diffusion; monotonicity; piecewise monotone approxima-tion; noise filtering; range data; 3D reconstruction.

1. INTRODUCTION

Consider the following situation: let f be a piecewise continuous real function defined ona rectangle A = [0, xmax] × [0, ymax] ⊂ R2: Moreover, let f be piecewise monotone; i.e.,the domain A can be partitioned into K connected subsets Ak, k = 1, . . . , K , such that fis continuous and monotone on Ak :

f (x1, y) ≤ f (x2, y) ∀(x1, y), (x2, y) ∈ Ak, x1 < x2(1)

or f (x1, y) ≥ f (x2, y) ∀(x1, y), (x2, y) ∈ Ak, x1 < x2

f (x, y1) ≤ f (x, y2) ∀(x, y1), (x, y2) ∈ Ak, y1 < y2(2)

or f (x, y1) ≥ f (x, y2) ∀(x, y1), (x, y2) ∈ Ak, y1 < y2.

1 This research was supported by the Czech Ministry of Education under the grant VS96049 and 4/11/AIP CR,by the Grant Agency of the Czech Republic under Grant GACR 102/00/1679, and by the Research ProgrammeJ04/98:212300013.

313

1047-3203/02 $35.00C© 2002 Elsevier Science (USA)

All rights reserved.

314 PAVEL MRAZEK

The function f is sampled and represented by a 2D array of values fi, j ,

fi, j = hi, j ∗ f + n, xi = i · �x, y j = j · �y, (3)

where hi, j is the sampling kernel for position xi , y j , and �x, �y the sampling intervals inthe directions of axes x and y, respectively. Some noise n is added to the samples duringthe discretization process.

In the discrete case the piecewise monotonicity assumption can be restated as follows:if K is the smallest number of connected sets Ak needed to partition the function domainso that the discrete function f is continuous2 and monotone on each Ak , we require thatK is much smaller then the number of pixels in the image. The discretization noise mayviolate this monotonicity assumption; the gradient of the noisy samples fi, j will change itsorientation much more often than that of the original function. Our task is to restore thedesired function properties, filter the noise, and smooth or simplify the sampled function fso as to enforce the piecewise monotonicity (reduce the number K described above) whilepreserving important discontinuities or edges.

To give a real world example of the situation where the piecewise monotonicity as-sumption could be appropriate, we may mention the range data for 3D reconstruction incomputer vision. The function f represents the distance of the objects in the scene from thecamera; the distance changes gradually on continuous surfaces, with abrupt changes, i.e.,discontinuities where a different object comes into view. These distance data are measuredat discrete positions xi , y j with some imprecision modeled by the noise n, thus forming a2D array (or image) of values fi, j .

In this paper we concentrate on the possibility to enhance piecewise monotonicity ofthe data by nonlinear (NL) diffusion. We first review the basics of nonlinear diffusion, apowerful filtering method, and present its simplest discrete algorithm. Then we discuss whyclassical NL diffusion is not suited for monotonicity enhancement and suggest to remedythe problems by moving to NL diffusion of first directional derivatives of the input data. Thismethod is developed further in the paper; we demonstrate its abilities in several experiments.

2. NONLINEAR DIFFUSION

Nonlinear diffusion has deservedly attracted much attention in the field of image process-ing for its ability to reduce noise while preserving (or even enhancing) important featuresof the image, such as edges or discontinuities; this can be opposed to linear diffusion (aliasGaussian filtering or linear scale-space representation; see [1]) which not only removesnoise but also blurs and dislocates edges. A good introduction to NL diffusion can be foundin [2, 3], Weickert in [4] gives a rich survey of the literature. The following brief presentationis adapted loosely from Weickert et al. [5].

We take the filter of Catte et al. [6], a regularization of the pioneering Perona–Malikmodel [7], as a typical representative of a well-founded nonlinear diffusion process. With

2 Obviously, the notion of function continuity does not exist in the discrete situation. The gradient (more exactlyits estimate from the discrete data) may serve as a replacement: the smaller the gradient at a given position, themore feasible it is to regard the function as continuous around that position.

MONOTONICITY ENHANCING NONLINEAR DIFFUSION 315

this scheme, the filtered image is found as a solution to the equation

∂u

∂t= ∇ · (g(|∇uσ |2)∇u) (4)

with the original image f as the initial state and the reflecting boundary conditions,

u(x, 0) = f (x),

(∂u

∂t

)· n = 0 on ∂�, (5)

where n denotes the normal to the image boundary ∂�. In words, Eq. (4) expresses thefact that the value of u(x, t) changes with time according to the flow to and from theneighborhood of x; this flow depends on the image gradient ∇u and its amount is controlledby the function g of the smoothed gradient ∇uσ (smoothing makes the filter insensitive tonoise at scales smaller then σ ); no flow passes through the image boundary.

The diffusivity function g(s) is typically constructed to be positive everywhere but rapidlyand monotonically decreasing for s > 0; to give an example, the diffusivity

g(s) ={

1 for s < 0

1 − exp(−3.315

(s/λ)4

)for s ≥ 0,

(6)

was used in [5]. The parameter λ in this formula can be understood as a threshold offunction continuity: if s � λ, g(s) is almost one and the position of such a small gradients = |∇uσ | is considered to belong to a continuous region where diffusion (or smoothing) isencouraged. On the other hand, g is close to zero for s � λ; almost no diffusion takes placeat positions of a larger gradient. This way the small-scale noise in otherwise homogeneousregions is removed, whereas important discontinuities between regions remain stable overlong periods of the diffusion process.

The positivity of g guarantees that the solution u(., t) converges to a constant for t → ∞.To obtain nontrivial results, a finite stopping time T has to be set.

3. DISCRETE NONLINEAR DIFFUSION

To be suitable for numerical computations with sampled data, the continuous equation (4)has to be discretized. Let us start in one dimension only, where (4) simplifies to

∂t u = ∂x (g(|∂x uσ |2)∂x u). (7)

For discrete data uki (approximating u at position xi = i · �x and time instant tk = k · τ ,

with τ the discretization time step and �x the spatial grid size), replacing the derivativesby finite differences, Eq. (7) becomes

uk+1i − uk

i

τ=

∑j∈N (i)

gki j

�x2

(uk

j − uki

), (8)

where N (i) is the set of the neighbors of pixel i and gki j is the diffusivity belonging to the

connection between pixels i and j at time tk .

316 PAVEL MRAZEK

Equation (8) is called the explicit discretization scheme of (7). It can be summarized bythe following iterative formula

uk+1 = (I + τ A(uk))uk, (9)

where τ is a discrete time step, I is the identity matrix and A(uk) contains the diffusivityinformation:

ai, j =

gki j

�x2 for j ∈ N (i),

−∑n∈N (i)

gkin

�x2 for j = i,

0 otherwise

(10)

(note that only the elements ai, j of A for which either j ∈ N (i) or i = j are nonzero; in1D A is tridiagonal). For two-dimensional data u another term appears,

uk+1 = (I + τ Ax (uk) + τ Ay(uk))uk, (11)

and Ax (uk) and Ay(uk) are matrices containing information about the diffusivities betweenindividual pixels in the directions of axes x and y, respectively.3

The explicit (or Euler) discretization scheme used in this section is the most straightfor-ward but requires a small time step τ (and thus more iterations) in order to be stable; moreefficient and more complicated, absolutely stable procedures like the semi-implicit schemeand the additive operator splitting have been introduced in [5].

4. MONOTONICITY ENHANCING NL DIFFUSION

Being smoothed more inside homogeneous regions, the function u solving Eq. (4) tendsto piecewise constant as the time t increases. This is illustrated on a simple function inFig. 1; the horizontalization of an increasing function can be observed first near the ends ofcontinuous function segments. While nonlinear diffusion yields impressive results on someimages and may be particularly useful for robust image segmentation, the model assumingpiecewise constancy is unsuitable for noise removal from most natural scenes.

A variety of other possible models for image filtering, such as piecewise linear, locallymonotone, or locally convex have been suggested in [8, 9]. The limitations of the piecewise-constant model for noise removal using minimization of the total variation of the image4

were observed by Chambolle and Lions in [10]. The authors suggest to alleviate the problemby introducing second order terms (such as the total variation of the image gradient) intothe functional to be minimized.

In this paper we try to exploit the following simple idea: if we differentiate the datafirst and run the diffusion on the arrays of partial derivatives instead of the original func-tion values, the piecewise smoothing of derivatives leads (after integration) to data whichare piecewise monotone, for higher t approaching a piecewise linear function. Whereclassical nonlinear diffusion simplifies an image into segments of similar gray levels,

3 The pixels of u must be arranged into a single column vector to allow the matrix multiplication.4 We should remark that image restoration by minimization of some functional is closely related to nonlinear

diffusion and scale-space theory, see e.g. [7, 4, 12, 13].


FIG. 1. Classical nonlinear diffusion approaches a (piecewise) constant function; this phenomenon can beobserved first near the ends of growing function segments.

the procedure using derivatives (the main topic of this paper) creates patches of similartrends, which can successfully approximate a large class of images and other types ofinputs and will be preferred if the piecewise monotonicity of the data is assumed anddesired.

5. FROM DATA TO PARTIAL DERIVATIVES

Consider the two-dimensional situation: a function f (x, y) is sampled and representedby values fi, j at positions (xi , y j ), xi = i · �x , y j = j · �y, i = 1, . . . , Ni , j = 1, . . . , N j ;the samples (pixels) fi, j form an image, our input data.

The partial derivatives of the original function in the direction of axes x, y, respectively,can be approximated from the discrete image by differences of the neighboring pixels,forming two arrays v and w:

∂ f (x, y)

∂x

∣∣∣∣xi ,y j

≈ vi, j ≡ fi+1, j − fi, j

�x, i = 1, . . . , Ni − 1, j = 1, . . . , N j , (12)

∂ f (x, y)

∂y

∣∣∣∣xi ,y j

≈ wi, j ≡ fi, j+1 − fi, j

�y, i = 1, . . . , Ni , j = 1, . . . , N j − 1. (13)

The arrays v, w must satisfy some requirements in order to represent the partial derivativesof a real function. In the continuous domain, the integral of a function gradient along anyclosed curve is zero, ∮

C∇ f (r) · dr = 0. (14)

In the discrete image any closed curve is composed of the elementary closed curves passingthrough four pixels5 as illustrated in Fig. 2, and the observation (14) transforms into

∀i, j : ei, j ≡ wi, j + vi, j+1 − wi+1, j − vi, j = 0. (15)

5 Assuming 4-neighbourhood, i.e., only vertical and horizontal connections are allowed.

318 PAVEL MRAZEK

FIG. 2. Geometry of the discrete image: the partial derivatives of the data u in the direction of axes x, y areapproximated by finite differences forming arrays v and w, respectively.

While this constraint is satisfied automatically by (12)–(13), we have to be more carefulabout it during the diffusion process.

6. THE DIFFUSION ALGORITHM

The continuous equations for nonlinear diffusion have been discretized above. Simplyrewriting (11) using v, w instead of u, we obtain the following set of equations:

vk+1 = (I + τ Ax (vk, wk) + τ Ay(vk, wk))vk (16)

wk+1 = (I + τ Bx (vk, wk) + τ By(vk, wk))wk . (17)

Here v, w are column vectors of generally different size, (Ni − 1) · N j and Ni · (N j − 1),respectively; the matrices A, B have also different dimensions. However, as the connectionbetween elements vi−1, j and vi, j , and between wi, j−1 and wi, j passes through a com-mon point (ui, j in Fig. 2), we find it reasonable to assign these two connections the samediffusivity, namely gi, j . This is why the matrices A and B depend on both vk and wk ;Equation (16) and (17) will be coupled through the common array of diffusivities andmany elements of Ax and By will be identical (similarly for the other pair of directionsand Ay , Bx , only the common point of the connections vi, j − vi, j+1 and wi, j − wi+1, j

does not coincide with any input datum u; we denote the diffusivity of that position bygi+1/2, j+1/2).

There are many possibilities on how to assemble the information from the smoothed,regularized versions v, w of the arrays v, w into the common diffusivities gi, j . Proceedingmost directly from (4) to the diffusion of derivatives, we obtain the following:

g(|∇2uσ |2) = g(|∇ · (∇uσ )|2) ≈ g(|∇ · (v, w)|2) = g

( ∣∣∣∣ v

∂x+ w

∂y

∣∣∣∣2 )

≈ gi, j ≡ g(|vi, j − vi−1, j + wi, j − wi, j−1|2). (18)

In our experiments, we used Eq. (6) to define the function g(·).


The complication with this approach is that derivatives amplify high frequency compo-nents of a signal (including noise), and the second order derivatives of the input data whichappear in formula (18) make the method more difficult to tune and unsuitable for highlycorrupted inputs. In some of our experiments we employed the following trick successfully:steer the diffusion not with a gradient of the partial derivatives, but with a gradient of theoriginal data, thus avoiding higher order derivatives. Using the arrays of partial derivativeswe can write

g(|∇uσ |2) ≈ gi, j ≡ g

( ∣∣∣∣ vi−1, j + vi, j

2

∣∣∣∣2

+∣∣∣∣ wi, j−1 + wi, j

2

∣∣∣∣2 )

,

(19)

gi+ 12 , j+ 1

2≡ g

( ∣∣∣∣ vi, j + vi, j+1

2

∣∣∣∣2

+∣∣∣∣ wi, j + wi+1, j

2

∣∣∣∣2 )

.

This alternative reveals a drawback, too: limited to first derivatives only, it may neglectdiscontinuities of the second derivatives and round corners of a continuous function. Seesome experiments below.

There is another problem with the simple formulation of nonlinear diffusion of partialderivatives: Equations (16) and (17) do not guarantee that the constraint (15) is satisfied. Inthe remaining part of this section we try to enforce the necessary properties of the arrays ofpartial derivatives.

Denote by z = [vk, wk]T the result of the diffusion process at time tk . We seek a solutionz as close as possible to z while obeying the constraint (15) which can be written in matrixform as

Cz = 0, (20)

where C is a [(Ni − 1) · (N j − 1)] × N sparse matrix with four nonzero entries in each row,and N is the total number of elements of the arrays v, w:

N = (Ni − 1) · N j + Ni · (N j − 1). (21)

The rigorous way to solve this problem would be to find z as a projection of z into the nullspace of matrix C ; such solution would minimize the norm ‖z − z‖2; see e.g. [11]. However,this mathematically correct solution involves the construction of an orthonormal basis of thenull space of C and full matrix multiplication (processes of complexities O(N 3) and O(N 2),respectively). Already for small images, these matrix computations become infeasible. As aviable alternative, we choose to restore the property (15) by the following iterative algorithm:

ALGORITHM 1 (RESTORATION OF THE DERIVATIVES).

1. Evaluate errors ei, j = wi, j + vi, j+1 − wi+1, j − vi, j , ∀i, j.2. For all i, j , update the values as follows,6

vi, j = vi, j + (ei, j − ei, j−1)/c, wi, j = wi, j − (ei, j − ei−1, j )/c,

with the obvious modifications at the image boundary.3. If max |ei, j | is smaller than a given threshold, finish; otherwise go to 1.

6 For simplicity, we use the same symbols for the original and the updated values, using a MATLAB-likenotation; new values are on the left.

320 PAVEL MRAZEK

The constant c in the algorithm divides the errors into the elements which contributed toit. The value c = 4 is a reasonable choice as four elements form each ei, j ; a slightly highernumber (c ≈ 4.3) damps down oscillations and leads usually to a faster convergence. Thecomplexity of one iteration of this algorithm is only O(N ). Although several iterations arenecessary, this still compares favorably with the complexity of the matrix method.

7. FROM DERIVATIVES BACK TO DATA

Let us now assume that the filtered arrays v, w are available and that they contain correctvalues in the sense of condition (15). These arrays contain (redundantly) all the informationneeded for the reconstruction of the image u up to a scalar u0 added to function values. Theintegration can be performed by the following algorithm:

ALGORITHM 2 (RECONSTRUCTION OF u FROM v, w).

1. Reconstruct the first row:

u1,1 = 0

for i = 2, . . . , Ni : ui,1 = ui−1,1 + vi−1,1.

2. Reconstruct the columns:

for i = 1, . . . , Ni

for j = 2, . . . , N j : ui, j = ui, j−1 + wi, j−1.

3. Fix the shift of function values:

for all i, j : ui, j = ui, j + u0.

The choice of the scalar u0 influences significantly the behavior of the filtering procedureas a whole. One possibility is to select it so that the average gray values of the original, f ,and the filtered image, u, remain equal:

u0 = 1

Ni · N j

[Ni∑

i=1

N j∑j=1

fi, j −Ni∑

i=1

N j∑j=1

ui, j

]. (22)

8. EXPERIMENTS

The results of all the methods of nonlinear diffusion mentioned in this paper applied to asimple 1D function are shown for comparison in Fig. 3. In the center column the preferenceof the classical nonlinear diffusion for (piecewise) constant functions can be seen: dependingon the parameters, it either approximates the input by several steps or bends the functionnear the extrema. On the right, two results of the nonlinear diffusion of first derivatives areshown for comparison: the upper one, where the diffusion was controlled by first derivatives,shares some properties of the ordinary diffusion (rounding of the corner) but behaves betternear the ends of continuous segments. The lower one, where the diffusion was dominated bya function of second derivatives, is also able (for carefully chosen parameters) to preciselylocate the corner in the function values, i.e., the discontinuity of the second derivative.


FIG. 3. Gaussian noise was added to a triangle function (a) to obtain the noisy data (b), used to test thediffusion filtering methods. The filtering results (continuous lines) are shown together with the noisy input (dots).(c) Ordinary nonlinear diffusion, σ = 1, λ = 1, T = 10. (d) Ordinary nonlinear diffusion, σ = 1, λ = 3, T = 10.(e) Filtering by nonlinear diffusion of first derivatives, σ = 1, λ = 1, T = 10. The diffusion is controlled by firstderivatives according to (19). (f ) Filtering by nonlinear diffusion of first derivatives, σ = 3, λ = 0.05, T = 10.The diffusion is controlled by the second order derivatives as described by (18).

An experiment with synthetic noisy 2D data is presented in Fig. 4. The image gradient islarger on the sloped surface; to remove the noise from that area with the ordinary nonlineardiffusion, a larger diffusivity parameter λ has to be chosen, which leads to a higher riskof blurring the discontinuities. You can observe this phenomenon, as well as the bending

FIG. 4. Experiment with two-dimensional data. (a) original data; (b) noisy input; (c) classical nonlineardiffusion of (b), σ = 1, T = 6, p = 0.8; (d) the noisy input filtered with the nonlinear diffusion of first derivatives,controlled by second derivatives, σ = 1, T = 20, p = 0.8.

322 PAVEL MRAZEK

of increasing segments, in Fig. 4c. In contrast, our method controlled by second deriva-tives according to Eq. (18) behaves in the same way regardless of the surface slope, thediscontinuity is well preserved for a large interval of parameters, and the function bendingis avoided. These properties are demonstrated in Fig. 4d. You may notice that some noisehas remained near the areas of higher second derivatives, i.e., near the discontinuities andcorners, where the diffusion is inhibited. This misbehavior could be avoided by using ananisotropic filter where the diffusion is controlled by a diffusivity tensor; see Weickert [4].

In this experiment, the parameter λ was computed in each diffusion iteration as a p-tile ofthe histogram of the data gradients as suggested by Perona and Malik in [7]. This way, insteadof setting λ manually, we have to supply a (more intuitive) parameter p corresponding tothe percentage of the smallest image gradients which should be subjected to diffusion.The diffusivity threshold λ then typically decreases as the diffusion iterations advance, sothe diffusion slows down after the first few steps and the solutions become more stablecompared to the results obtained with a fixed λ.

Although not designed originally for that type of data, the monotonicity-enhancing fil-ter may be applied successfully to photographic images. Compared to the classical NLdiffusion, it performs better at preserving gradual transitions between different light inten-sities. Some examples are shown on our Web page http://cmp.felk.cvut.cz/∼

mrazekp/Diffusion/diffusion.html.

9. CONCLUSION

We have presented a method for piecewise monotone approximation which allows usto remove noise and enhance the trends while preserving important discontinuities presentin the input data. The task is accomplished by nonlinear vector-valued diffusion of partialderivatives of image data. The method is applicable to filtering or smoothing of sampledfunctions expected to be piecewise continuous, piecewise monotone, or piecewise linear.Many images fulfill these properties; range data for 3D reconstruction represent a particularexample.

In the future we would like to extend the method into an anisotropic filter to improve theedge and corner enhancement, analyze the method in more detail, and develop a generalmechanism which would allow us to adapt the parameters of the procedure autonomouslyto obtain the best results for any given input data.

ACKNOWLEDGMENT

The author expresses his gratefulness to Mirko Navara for his constant support, assistance, and valuable com-ments on the manuscript.

REFERENCES

1. T. Lindeberg, Scale-Space Theory in Computer Vision, Kluwer Academic, Dordrecht/Norwell, MA, 1994.

2. B. M. ter Haar Romeny (Ed.), Geometry-Driven Diffusion in Computer Vision, Kluwer Academic,Dordrecht/Norwell, MA, 1994.

3. J. Weickert, A review of nonlinear diffusion filtering, in Scale-Space Theory in Computer Vision (B. M.ter Haar Romeny, L. Florack, J. Koenderink, and M. Viergever, Eds.), Lecture Notes in Computer Science,Vol. 1252, pp. 3–28, Springer-Verlag, Berlin/New York, 1997.


4. J. Weickert, Anisotropic Diffusion in Image Processing, European Consortium for Mathematics in Industry,Teubner, Stuttgart, 1998.

5. J. Weickert, B. M. ter Haar Romeny, and M. A. Viergever, Efficient and reliable schemes for nonlineardiffusion filtering, IEEE Trans. Image Process. 7, 1998, 398–410.

6. F. Catte, P.-L. Lions, J.-M. Morel, and T. Coll, Image selective smoothing and edge-detection by nonlineardiffusion, SIAM J. Numer. Anal. 29, 1992, 182–193.

7. P. Perona and J. Malik, Scale-space and edge-detection using anisotropic diffusion, IEEE Trans. Pattern Anal.Mach. Intell. 12, 1990, 629–639.

8. S. T. Acton and A. C. Bovik, Piecewise and local image models for regularized image restoration usingcross-validation, IEEE Trans. Image Process. 8, 1999, 652–665.

9. S. T. Acton and A. C. Bovik, Nonlinear image estimation using piecewise and local image models, IEEETrans. Image Process. 7, 1998, 979–991.

10. A. Chambolle and P.-L. Lions, Image recovery via total variation minimization and related problems, Numer.Math. 76, 1997, 167–188.

11. D. G. Luenberger, Optimization by Vector Space Methods, Wiley, New York, 1969.

12. E. Radmoser, O. Scherzer, and J. Weickert, Scale-space properties of regularization methods, in Scale-SpaceTheories in Computer Vision (M. Nielsen, P. Johansen, O. E. Olsen, and J. Weickert, Eds.), Lecture Notes inComputer Science, Vol. 1682, Springer-Verlag, Berlin/New York, 1999.

13. O. Scherzer and J. Weickert, Relations Between Regularization and Diffusion Filtering, Technical ReportDIKU-98/23, Dept. of Computer Science, University of Copenhagen, Denmark, Oct 1998.

PAVEL MRAZEK graduated in computer science from the Czech Technical University, Prague, in 1996. Hehas been working toward his PhD at the Center for Machine Perception, Czech Technical University, since 1997;his research concentrates on interpolation, approximation, and noise reduction in image processing and computervision, with special emphasis on monotonicity of the data.

Documents

Monotonicity Enhancing Nonlinear Diffusion