Relaxation labelling — the principle of ‘least disturbance’

Pattern Recognition Letters 1 (1983) 385-391 July 1983

North-Hol land

Relaxation labelling- the principle of "least disturbance'*

A n d r e w B L A K E Machine Intelligence Research Unit, Edinburgh University, Edinburgh, Scotland

Received 20 January 1983

Revised 14 February 1983

Abstract: A new approach to constrained labelling is described and applied to labelling edges in images. An initial labelling is adjusted iteratively, to reach the nearest labelling that satisfies given constraints. The iterative method is provably convergent.

Key words: Constrained labelling, relaxation, optimisation, parallel array processor, computer vision.

1. Introduction

This paper describes a new approach to constrained labelling by relaxation, and its application to refining the labelling of edges in an image. A labelling task is cast as a constrained optimisation problem and solved by a linear iterative method, whose convergence is guaranteed.

Relaxation labelling may be an important method of programming parallel array processors. Work on image recognition systems - for instance the system by Perkins (1978) for recognising car parts by 2-D analysis of grey-tone images - makes it clear that extraction of primitive descriptions of objects from image data is a processing bottleneck. Badly needed processing power could be obtained from special hardware but the parallel array processor is a flexible source of power because it is programmable, and should become cheaper as it continues to be developed (e.g. Duff (1978)). Parallel array processors - large arrays of simple processing cells, each connected to its near neighbours - perform local operations like con-

* This research was conducted with the aid of a grant f rom the

SERC to Professor Donald Michie for work on a versatile programmable industrial image processor. The author is also in- debted to the University of Edinburgh for the provision of

facilities.

volution and thresholding very efficiently. However, extraction of lines and curves, as primitives for object description, involve operations which are not local, because the edges are long and must be examined over their entire length. Relaxation labelling achieves a global effect by repeated application of a local operation. The way of deriving that local operation from a problem specification, and the relationship to constrained optimisation, is the subject of this paper.

2. The principle of 'least disturbance'

Numerous applications of constrained labelling by iterative methods have been reported, for instance: analysing line drawings (Waltz (1972)), character recognition (Ullmann (1974)), edge labelling (Hanson and Riseman (1978)), stereopsis (Marr (1978)) and object recognition (Hinton (1978)). A commonly used formalism is that of labelling the nodes of a graph; a pair of nodes is joined by an arc to indicate that the labelling of one node is constrained by the labelling of the other. An initial labelling, or a set of potential labellings, is adjusted successively, propagating the effect of the constraints through the graph, to find consistent labels. This paper looks at labelling

0167-8655/83/$3.00 © 1983, Elsevier Science Publishers B.V. (North-Holland) 385

Volume 1, Numbers 5,6 PATTERN RECOGNITION LETTERS July 1983

edges in an image, in which the nodes of the graph are all the possible sites for an edge - the boundaries between each pixel and its neighbours. The labels are numerical measures, 'strengths' of the edge at each site and constraints between neighbouring edges demand that they should form unbroken chains.

The principle of 'least disturbance' is used to derive a labelling algorithm in the following way: an initial estimate of the labelling is assumed to be available from a previous process (a differentiator, in the case of edge labelling). The initial estimate is then coaxed into satisfying the constraints, by the most direct available route. The labelling achieved is then that consistent set of labels which is closest to the initial set of labels. A minimal disturbance is applied to the initial labelling, altering it only insofar as is necessary to achieve con- sistency. In this way, information in the initial labelling is preserved, as far as possible, for subsequent processes.

Generating a labelling like this is a constrained optimisation: the objective function is the difference (Euclidian distance) between the initial and final labelling, to be minimised subject to the constraints on the final labelling. The method of solution of this optimisation problem is described below and applied to the labelling of edges.

3. Imposing constraints on edges

3.1. The constraints

There is a duality between edges and regions. An image can be partitioned into a number of regions - sets of connected pixels - in which case the boundaries between adjacent regions are chains of edges. Conversely edges extracted from an image may form boundaries which delineate a number of regions. This duality is used here to derive constraints for edges. A set of edges is regarded as consistent if it corresponds to a set of regions, in which each region is of uniform grey level, so that the common boundary between two regions is of uniform strength (Fig. la). This requires edges to be continuous; they may not have gaps, or other discontinuities in strength. We refer to these constraints, in this paper, as 'continuity constraints'.

8 region reglon B [

A

b

Fig. 1. The duality between edges and regions. (a) The strength along the common boundary between two regions is constant. (b) Moreover, following any closed path, the edge strengths on all the region boundaries crossed must sum to zero.

Representing regions with uniform grey level is likely to be reasonable under diffuse lighting, with Lambertian surfaces, and when reflectivity is roughly constant within each region.

An initial set of edges, obtained from an image by differentiation, already satisfies the continuity constraints (see Sections 3.3, 3.4). However, if it is necessary to impose additional constraints then we must ensure that the edges continue to satisfy the continuity constraints. For instance, an image often contains many spurious, weak edges due to noise from the camera and from quantisation, and as a result of small variations in surface reflec- tance. We wish to retain only those edges caused by physically significant events such as object boundaries, which are likely to appear as substantial changes in image intensity. This can be achieved by a 'minimum strength constraint ' , as used in this paper: no edges may exist where the magnitude of the grey-level difference between adjacent pixels in the original image falls below a certain threshold. The threshold itself might be determin- ed from sensor parameters. Alternatively it could be derived from a histogram of initial edge strengths, set to supress a certain fixed proportion of the edges. Horn (1974) uses a similar constraint to eliminate the effect of slow illumination gra- dients from image intensity data.

There may be other ways of picking out important edges than applying the minimum strength constraint. One alternative that has been tried by the author involves applying a nonlinear transfor-

386


mation to the initial edge strengths, to enhance the strong edges and diminish the weak ones. In

general the result no longer satisfies the continuity constraints but may be adjusted to do so, by relaxation. However, the minimum strength constraint is very much more effective, because it eliminates edges from substantial areas of the image, rather than merely diminishing the strength of those edges while still allowing the possibility of restor- ing them during subsequent relaxation.

The constraints of continuity and minimum strength, enforced simultaneously, will produce a set of edge strengths, with a corresponding set of regions in which the grey levels of any pair of adjacent regions differ by at least the amount of the minimum strength threshold.

3.2. Hanson and Riseman's algorithm

Hanson and Riseman (1978) have described an iterative algorithm to derive edge strengths, based on the idea of 'good line continuation' - roughly that good boundaries (lines) are unbroken ones. They extend this idea to cover edges labelled with strengths. An edge's six neighbours (Fig. 2b) are inspected: its strength is enhanced if it forms part

d b

Fig. 2a. Local constraints on edges. The diagram shows four

pixels with grey levels A, B, C, D. Edge strengths are: a = B - A,

b = C - B , c = D - C and d = A - D . These edge strengths are

related by the constraint: a + b + c + d = 0. This is simply a local

version of the constraint illustrated in Fig. lb.

of a continuous chain of strong edges, and diminished if it is isolated. Their algorithm has drawbacks in its design, however:

(1) Constraints on edges are not explicitly stated. The spirit of the algorithm is to perform some sort of edge cleaning operation, in an iterative fashion, but there is no clear definition of the problem that is being solved.

(2) At each iteration, new edge strengths are computed from current edge strengths, by a nonlinear formula, following Rosenfeld et al. (1976). Analysis of the effect of the repeated application of a nonlinear formula is difficult, and was not attempted by Hanson and Riseman. They show two examples in which edge strengths appear to converge, but there is no guarantee of convergence, and oscillation has been observed in a similar algorithm (Blake (198 lb)).

The algorithm presented here overcomes these drawbacks. Constraints have already been described, the continuity constraint is described in detail below, and convergence can be proved. First, some terms must be defined.

3.3. Definitions

An image is an array of positive numbers

~,j, i=1 . . . . . N, j = l . . . . . N, (1)

in which each element represents the intensity of light falling on one picture element (pixel) and is called the 'grey-level' of that pixel. An edge is the dividing line between two adjacent pixels. A boundary is a chain of edges, connected end to end. Horizontal and vertical edge strengths are arrays

H, j a n d V/j, i , j = l . . . . . N. (2)

Strengths are signed, to distinguish a light-to-dark transition from a dark-to-light transition, and their magnitude indicates the absolute difference of intensity across the edge. Finally, edge strengths (H, V) are said to represent an image I when

Ii, j - - Ii_ l , j = t-Ii, j and Ii, j - [i,j + l = Vi, j

for i,j=O . . . . . N + 1. (3)

Fig. 2b. The edge-region duality constraints, in their local form

(Fig. 2a), relates each edge to its six nearest neighbours.

This means that (H, V) are simply differences of intensity between adjacent pixels of I where, on the

387


array boundary ( i = 0 or i = N + I or j = 0 or j = N + 1)//j is taken to be zero.

3.4. The continuity constraint

The duality between regions and edges - that a set of edge strengths (H, V) must correspond to a set of regions of uniform intensity - requires that, at the very least, (H, V) must represent some image I , because a set of regions, each with a grey-level, is itself an image. The strengths (H, V) have 2N 2 degrees of freedom but an image I , has only N 2 degrees of freedom. We expect, therefore, that edge-region duality imposes N 2 constraints on (H, V). A set o f N 2 constraints can be derived

directly f rom the definition of representation (3):

for all i,j, Sij = 0

where S i j m n i j - V i j -n i , j+ 1 q- Vi_l, j. (4)

Each edge is linked, by one or other of these constraints, to its six immediate neighbours (Fig. 2b). The meaning of the constraints is that over each of N 2 small closed paths (Fig. 2a) the strengghs of

edges crossed sum to zero. In fact this is true for any closed path (Fig. lb). This result is directly analogous to Stokes' theorem in vector calculus, as Table 1 shows.

Table 1

Discrete array Vector field

Image I Edge strengths (H, V) Constraint on (H, V) implies

strengths sum to 0 over a closed path

Scalar field (o Vector field A = V ~p curl(A) = curl(V ~o) = 0 implies (Stokes theorem)

JA. dl= 0 over a closed path

3.5. The relaxation formula

Given an i m a g e / w e take, as the natural estimate of edge strengths, the strengths ( H (°), IA °)) that represent I (3). This means that initially, edge strengths are simply the differences between adjacent pixels. They are modified according to the principle of least disturbance, to conform to the two constraints defined earlier. This generates an optimisation problem, to find the edge strengths (H, V) which minimise

T= IIH- H(°)ll 2 + U V - V(°)ll 2 (5)

subject to the constraints of continuity and minimum strength, where for some matrix M

[IM][ 2= E (Mu) 2,

the euclidian matrix norm. The objective function T is a convex function of

the edge strengths and the constraints are linear, so there is a unique local minimum of T, which is also its global minimum. The constrained optimisation can be t ransformed to an equivalent unconstrained one using the method of Lagrange multipliers, and solved by a variant of Jacobi relaxation (Varga (1962)). The solution is expressed as the limit of an iterative formula:

Whenever H/~ °) is inactive (below minimum strength) then

/-/i} n)= 0, and similarly for V; (6)

elsewhere, for active edges,

/ S !~) S,~)1,; ~ Hi(jn + 1)= H(n) (1 - k) ( "q , (7)

\ rij r i+l , j /

where S (defined in (4)) is a measure of the degree to which the continuity constraints are violated, rij is the number of active edges contributing to Sij, in (4), so that 1 <rij<4; also 0 < k < 1 but k should be close to 0 for rapid convergence. The final labelling is the limit

(H, V)= l ira (/-/(n), V(n)). (8)

It can be proved that the formula (7), converges for 0 < k < 1. Note that when k = 0, (7) is the Jacobi relaxation formula (Varga (1962)). The derivation of the formula and the proof of convergence extend easily to deal with two or more arrays of labels, under linear homogeneous constraints of a certain type. The derivation and p roof cannot be included here, for lack of space.

Relaxation methods of solving matrix equations have previously been used in computer vision for deriving shape f rom shading (Ikeuchi (1980)) and optical flow (Horn and Schunk (1980)), al though they use the Gauss-Seidel method (Varga (1962)) which is good for serial computers, whereas Jacobi relaxation is a parallel method. Ullman (1979)

388

Volume I, Numbers 5,6 PATTERN RECOGNITION LETTERS July 1983

e

Fig. 3. (a) A spanner. (b) Initial edge strengths, obtained by differentiating the intensities in the spanner picture. (The absolute value

of the strength is displayed.) (c) Applying the minimum strength constraint removes weak edges. (d) The continuity and minimum

strength constraints, applied together by relaxation, adjust edge strengths and remove boundaries with unattached ends (any apparent

unattached ends here are artefacts of the printing process). (e) Edges have a dual representation as regions. These are the regions cor-

responding to Fig. 3d. (f) The relaxation process is convergent as theoretically predicted.

389


describes a general method for optimisation under concave constraints, by relaxation, which could be put to work here. However the method used here is simpler, if less general.

3.6. Parallel processor implementation

The next section briefly describes results obtained from executing the edge relaxation algorithm on a parallel array processor emulator. Parallel array processors like CLIP4 (Duff (1978)) pose a problem, because of limited storage space for arrays. Floating point or high precision integer arrays may be unacceptably extravagant; however, care must be taken to retain the convergence of the algorithm and, as far as possible given limited precision arithmetic, the correctness of the solution.

The usual termination condition for iteration in an array is that some measure of the change in the array, in one iteration, falls below some value. Oscillatory behaviour can be eliminated provided a bound on its magnitude is known. With arrays of limited precision (8 or 16 bit) it is unlikely that such an upper bound will be sufficiently small to be useful. An alternative strategy in this case, is to partition the edges into four sets, so that each Sij (4) depends on exactly one edge from each set. The iteration scheme (7) is also partitioned to adjust edges in one of the sets, recompute S, adjust edges in another of the sets, and so on. In this way convergence can be guaranteed, at the expense of ac- curacy: the final edge strengths will obey the constraints (within the limitations of rounding error) and will approximate to the desired 'least disturbance' solution.

4. Results and discussion

Figure 3 shows a picture of a spanner (Fig. 3a), taken under diffuse lighting with a CID camera, and the effect of the edge relaxation algorithm applied to it. The initial edge strengths are shown in Figure 3b. The minimum strength constraint requires that all edges below some low threshold be removed (Fig. 3c); the threshold was chosen at a turning point in the edge strength histogram, and removes the bulk of low strength edges. The ap-

plication of the continuity constraint, by relaxation, results in a clean edge strength map (Fig. 3d). The effect of the relaxation is both to erode any boundaries in Figure 3c with unattached ends, and to redistribute edge strengths within the remaining boundaries. The partitioned algorithm for limited integer precision (described above) was used. It was executed on an emulator of a parallel array processor which is programmed in the PARAPIC parallel image processing language (Blake (1981a)). The edges of Figure 3d have a dual representation as regions (Fig. 3e), obtained from the edge strengths by a process of integration. The grey level within each region is constant, as expected, except for some striping which is due to the effect of rounding error in relaxation, accumulated and smeared by the integration.

Convergence of the relaxation process can be traced by plotting the maximum deviation, in the array, from the continuity constraint, at successive iterations (Fig. 3f). A theoretical prediction of what to expect in the case of a closed boundary of M edges, can be obtained using eigenvalue analysis and circulant matrix theory (Davies 1979). The rate of convergence is found to be strongly dependent on the initial pattern of edge strengths. Some patterns converge within a few iterations, while in the worst case, the deviation from continuity decays to 1/e only after about 2M2/~z 2 iterations. The convergence graph (Fig. 3f) appears to reflect this, converging rapidly initially, followed by a slow

decay. Note also that in the partitioned scheme for limited integer precision, described earlier, the graph is guaranteed to be monotonic decreasing.

How fast can the edge relaxation algorithm be expected to run? Using the figures published for the CLIP4 design (Duff (1978)) the parallel processor time needed for a 96 × 96 pixel image is

2.9 ms +4.1 ms per iteration: about

210 ms for 50 iterations (as used in Fig. 3).

This is a reasonable time for robotic applications, with perhaps one second available to recognise an object.

5. Conclusions and further work

The principle of 'least disturbance' has been

390


proposed as a strategy for altering a labelling just sufficiently to comply with some linear constraints, but preserving the initial labelling as far as possible. A relaxation algorithm can be defined that achieves this, and is guaranteed to converge.

Such a relaxation scheme for edge labelling has been derived and implemented on an emulator of a parallel array processor. The duality between edges and regions imposes a continuity constraint on edge strengths. A further constraint (minimum strength) supresses low edge strengths to remove weak edges that may have been generated by noise.

The edge relaxation scheme seems to exemplify a natural way of using parallel array processors. However it acts at the lowest level of scene analysis. The next step would be to organise the edges into larger patterns. For instance arranging 'strokes' (oriented line segments) into curves and straight lines is an important task in some 2-D vision systems (Perkins (1978), Mero (1980)). The idea of using relaxation for this has been suggested by Zucker et al. (1977). However, when treated as an optimisation problem, according to the principle of least disturbance, an obstacle appears: the constraints are not linear. The problem may have many local optima, rather than just one as under linear constraints. Only one of those local optima is the required global optimum. It may be possible to transform the problem into an equivalent one with the same global optimum, but no additional local optima; otherwise the global optimum can be found by searching amongst the local optima, which is feasible only if the search can be suitably guided. Alternatively a good but not necessarily optimal labelling may be acceptable, if one can be found. Further work is required to find a good solution for such problems.

References

Blake, A. (1981a). PARAPIC language definition. Unpublished report, MIRU, Edinburgh University, Edinburgh.

Blake, A. (1981b). Edge growing and relaxation, in parallel. Research memorandum MIP-R-134, MIRU, Edinburgh University, Edinburgh.

Davies, P.J. (1979). Circulant Matrices. John Wiley and Sons, New York.

Duff, M.J.B. (1978). Review of the CLIP image processing system. National Comput. Conference, 1011-1060.

Hanson, A.R. and E.M. Riseman (1978). Segmentation of natural scenes. In: A.R. Hanson and E.M. Riseman, eds., Computer Vision Systems. Academic Press, New York.

Hinton, G.H. (1978). Relaxation and its Role in Vision. Ph.D. thesis, University of Edinburgh, Edinburgh.

Horn, B.K.P. (1974). Determining lightness from an image. Comput. Graphics and Image Processing 3, 277-299.

Horn, B.K.P. and B.G. Schunk (1980). Determining optical flow. AI memo 572, AI Lab, MIT, Cambridge, USA.

Ikeuchi, K. (1980). Numerical shape from shading and oc- cluding contours in a single view. AI Memo 566, AI lab, MIT, Cambridge, USA.

Marr, D. (1978). Representing visual information. In: A.R. Hanson and E.M. Riseman, eds., Computer Vision Systems. Academic Press, New York.

Mero, L. (1980). An algorithm for scale- and rotation-invariant recognition of two dimensional objects. Comput. Graphics and Image Processing 15, 279-287.

Perkins, W.A. (1978). A model-based system for industrial parts. IEEE Trans. Computers 27, 126-143.

Rosenfeld, A., R.A. Hummel and S.W. Zucker (1976). Scene labelling by relaxation operations. IEEE Trans. Systems Man Cybernet. 6, 420-433.

Ullman, S. (1979). Relaxed and constrained optimisation by local processes. Comput. Graphics and Image Processing 10, 115-125.

Ullmann, J.R. (1974). A use of continuity in character recognition. IEEE Trans. Systems Man and Cybernetics 3,294-300.

Varga, R. (1962). Matrix Berative Analysis. Prentice-Hall, Englewood Cliffs, N J, USA.

Waltz, D.L. (1972). Generating Semantic Descriptions from Drawings of Scenes with Shadows. Ph.D. thesis, MIT, Cam- bridge, USA.

Zucker, S.W., R.A. Hummel and A. Rosenfeld (1977). An application of relaxation labelling to line and curve enhance- ment. IEEE Trans. Computers 26, 394-403.

391

Documents

Relaxation labelling — the principle of ‘least disturbance’