Segmentation of brush strokes by saliency preservingdual graph contraction

P. Kammerer, R. Glantz *

Institute for Computer Aided Automation, Pattern Recognition and Image, Processing Group 183/2,

Vienna University of Technology, Austria

Abstract

Brush strokes are segmented from works of art by a combination of filtering and grouping. Filtering yields local

evidence for crossings and lines. Grouping is done on two levels of scale and abstraction. The first level is a dual pair

ðGu;GuÞ of attributed plane graphs, the vertex and edge attributes of which are derived from the filtering. The result of

the grouping on this level is given by a topological minor Gtop of Gu. The derivation of Gtop from Gu is done by dual

graph contraction, i.e. by parallel steps, each of which involves only local operations on Gu and Gu. This step is shown

to preserve connections via most salient paths. On the second level consecutive edges of Gtop are grouped to strokes

which are consistent with our model of strokes from superimposed brush moves. Experimental results are presented for

portrait miniatures.

� 2002 Elsevier Science B.V. All rights reserved.

Keywords: Segmentation; Dual graph contraction; Preservation of saliency; Brush strokes; Portrait miniatures

1. Introduction

The segmentation of brush strokes in portraitminiatures (Fig. 1a) is a crucial part in a project on

the application of pattern recognition and image

processing methods to works of arts (Sablatnig

et al., 1998). The saliency of the brush strokes

suffers from the painting method, i.e. the strokes of

a previous move are intersected and disturbed by

the strokes of subsequent moves. Especially in the

shaded regions of a portrait miniature the brushstrokes form a cross-hatch as in Fig. 1b. However,

the strokes of more than two moves may be pre-

sent in the image (see the filtered image in Fig. 7a).

Thus, the goal of this paper is to segment a max-

imal set of brush strokes from a hypotheticalsuperposition of at least two moves.

Proper detection of line segments requires to

adapt the filters to the orientation and the thick-

ness of the line segments (Perona, 1995). In this

paper non-linear combinations of such line filters

are used to collect local evidence for crossings and

lines. For grouping local evidence Amir and Lin-

denbaum (1998) suggest

(1) to construct a graph Gu representing the topo-

logical and geometric relations in the data set

(pixels), and

(2) to find ‘‘the best’’ partition of Gu.

Pattern Recognition Letters 24 (2003) 1043–1050

www.elsevier.com/locate/patrec

*Corresponding author.

E-mail address: glantz@dsi.unive.it (R. Glantz).

0167-8655/03/$ - see front matter � 2002 Elsevier Science B.V. All rights reserved.

PII: S0167-8655 (02 )00250-7

In this paper the vertices and edges of Gu rep-

resent the pixels and the 4-neighborhood of the

pixels, respectively. Each vertex is equipped with

an attribute that indicates local evidence for

crossings and lines. The grouping is done in two

successive steps and thus on two levels of scale and

abstraction.

(1) A topological minor Gtop of Gu is formed by

dual graph contraction (Kropatsch, 1995).

Thus, the set of vertices in Gu is grouped by

the faces and the edges of Gtop. A vertex of

Gu belongs to more than one group only if it

is an end vertex of more than one edge from

Gtop.(2) After contracting edges that do not fit to our

model of strokes from superimposed brush

moves, consecutive edges are grouped to

strokes.

In accordance with Lades et al. (1993) we deal

with graphs, the nodes of which carry information

about the orientation in the neighborhood of the

nodes. In contrast to Malsburg (1988) and Lades

et al. (1993) there is no model graph that Gtop

could be matched to. The accordance of the re-

sulting groups with the model goes back to the

criteria for contraction and grouping only.The plan of the paper is as follows. Section 2 is

devoted to collecting local evidence for lines and

crossings. In Section 3 the plane graph Gtop is de-

rived from the plane graph Gu by dual graph

contraction. Section 4 deals with grouping the

edges of Gtop and with experiments on portrait

miniatures.

2. Detection of lines and crossings

In the following we will present a filter that

detects a pair of lines crossing at angles within the

range ½0; p�. An angle close to 0 or p indicates asingle line.A line detector L ¼ Lðu; rx;ry1; ry2Þ which ex-

tracts even symmetric lines can be defined as an

oriented difference of two Gaussian (DoG) filters,

Grx;ry1;u and Grx;ry2;u (J€aahne, 1999). The parametersof the filter are given by the orientation u and

standard deviations of the two Gaussians, rx, ry1,

and ry2. Similar to Chen et al. (2000) we com-

bine two line detectors L1 and L2 to a crossingdetector

C ¼ sðL1 � L2ÞL1 þ sðL2 � L1ÞL2; ð1Þ

where

Fig. 1. (a) Portrait miniature, (b) zoom into cheek.

Fig. 2. Crossing detector: (a) filter kernel to detect a cross with principal orientations u1 ¼ p=4 and u2 ¼ 3p=4, (b) surface plot of (a).

1044 P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050

sðxÞ ¼ 1 if x < 0;0:5 if x ¼ 0:

�ð2Þ

Fig. 2 depicts a filter kernel for detecting a

crossing with principal orientations u1 ¼ p=4 andu2 ¼ 3p=4.Given an angular resolution of d, i.e. n ¼ p=d

orientations, the filter set will consist of n2 þ n=2

filters. Using the PCA method, the number

of filters can be reduced by an approximation

with linearly combinable basis filters (Perona,

1995; Moghaddam and Pentland, 1997). Thus, to

approximate the set of 136 filters needed to de-

tect 16 orientations, 17 basis filters suffice forthe reconstruction error to be below 0.82% (Fig.

3).

Fig. 4 depicts the responses of the crossing de-

tectors with 16 orientations. The test image con-

sists of two perpendicular lines with a Gaussian

profile. The magnitudes of the detectors at two

positions (marked by the white circles) are de-

picted in a 16� 16 matrix.The segmentation of the brush strokes will be

based on the maximal responses of all crossing

detectors, i.e. each pixel p is associated with the

maximal response of all crossing detectors at p

(Fig. 7a).

3. Saliency preserving dual graph contraction

In the following we collect the definitions nee-

ded to explain our concept for grouping by means

of topological minors.

Definition 3.1 (Graph, subgraph). A graph G ¼ðV ;E; iÞ is given by a finite set V of elements calledvertices, a finite set E of elements called edges withE \ V ¼ ; and an incidence relation i which asso-ciates with each edge e 2 E a subset of V with oneor two elements. The vertices in iðeÞ are called the

Fig. 3. Mean and first 17 basis filters of crossing detector (error

below 0.82%).

Fig. 4. Crossing detector applied to test image: (a) test image, (b) response at central point on the crossing, (c) response at point on

upper left branch.

P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050 1045

end vertices of e. The graph G0 ¼ ðV 0;E0; i0Þ is asubgraph of G, if V 0 V , E0 E and i0ðe0Þ ¼iðe0Þ 8e0 2 E0.

Note that the definition includes graphs withself-loops (i.e. edges with only one end vertex) and

multiple edges (i.e. several edges with identical sets

of end vertices). From now on we assume that G isplane, i.e. G is embedded onto the real plane suchthat the edges meet only at their end vertices. Next

we specify a parallel method to derive topological

minors of G, i.e. graphs that have a subdivision(Diestel, 1997) which is a subgraph of G. The mainidea is to involve the abstract duals (Definition

3.2). The method is known as dual graph contrac-tion (Kropatsch, 1995).

Definition 3.2 ([Minimal] cut, abstract dual). Let Gbe a graph with edge set E. A subset ECut of E iscalled [minimal] cut, if ECut is a [minimal] set suchthat the graph induced by E n ECut is not con-nected. A graph G is called abstract dual of G, ifthe edge set of G is E and if the minimal cuts in Gare precisely the edge sets of circuits in G.

In (Diestel, 1997) it is shown that a graph has

an abstract dual, if and only if it can be drawn as a

plane graph. The abstract dual G of a plane graphG may always be drawn such that the vertices of Gare placed in the regions of G, i.e. in the connectedcomponents of G n E (Diestel, 1997) (Fig. 5).

Dual graph contraction makes use of the fact

that the contraction of an edge e in a plane graphcorresponds to the deletion of e in the abstractdual of the graph (see again Fig. 5). Generally,

removing a set F of edges from G [G] correspondsto contracting the edges of F in G [G] (Theorem3.5). The proof can be found in (Thulasiraman and

Swamy, 1992). Formally, the contraction of all

edges from F may be expressed by means of anequivalence relation.

Definition 3.3 (Equivalence relation �F ). Let G bea graph with vertex set V and edge set E. ForF E the equivalence relation �F on V is definedas follows. v �F w : () there exists a path P in Gwith start vertex v and end vertex w such that alledges of P are contained in F.

Definition 3.4 (Subgraph G n F , contracted graphG=F ). Let G ¼ ðV ;E; iÞ be a graph and let F E.The graph G n F is the subgraph of G with vertex

set V and edge set E n F . If ½v�F denotes the classof v 2 V with respect to the equivalence relation

in Definition 3.3, the contracted graph G n F ¼ðVF ;EF ; iF Þ is defined by VF ¼ f½v�F j v 2 V g, EF ¼E n F , and iF ðeÞ :¼ f½v�F g [ f½w�F g for e 2 E n Fwith iðeÞ ¼ fvg [ fwg.

Theorem 3.5 (G n F , duality of contraction and

deletion). Let G be the abstract dual of a graph Gwith edge set E and let F E. Then G n F and G=Fform a pair of abstract duals. The graph G n F isconnected, if and only if no circuits in G can beformed by edges from F.

In the following we will describe the maximal

responses of the crossing detectors in geographic

terms, i.e. the maximal responses are interpreted asaltitudes of a topographic surface. Among many

others this intuitive approach to segmentation has

been chosen by Vincent and Soille (1992);

Koenderink and Doorn (1994); Meyer (1999) and

Meijsler and Roerdink (1998). In (Glantz and

Kropatsch, 2000) a neighborhood graph (Fig. 6a)

and its abstract dual are used to describe the

neighborhood relations of hills via crest lines andof basins via river lines, respectively. In this section

Fig. 5. Duality of deletion and contraction. The contraction of

e in G corresponds to the deletion of e from G.

1046 P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050

the method is adapted to the segmentation of

brush strokes.

The topographic surface is represented by an

underlying neighborhood graph Gu ¼ ðVN ;EN ; iN Þ.The vertices of Gu represent the pixels and the

edges of Gu represent the 4-neighborhood of the

pixels. The following initialization of attributes

attð�Þ in Gu and its abstract dual Gu ¼ ðV N ;EN ;�iiN Þwill allow to remove edges from Gu which do not

correspond to parts of crest lines on the topo-

graphic surface. Since each vertex of Gu stands for

a pixel, we may consider the maximal response ofall crossing detectors at vertex v. Let this value bedenoted by MaxResðvÞ. We set

• attðvÞ :¼MaxResðvÞ 8v 2 VN ,• attðeÞ :¼ minfattðvÞjv 2 iN ðeÞg 8e 2 EN ,

• attðvÞ :¼ minfattðeÞjv 2 �iiN ðeÞg 8v 2 V N .

Note that the removal of an edge e in Gu isequivalent to the fusion of the regions on both

sides of e. This fusion, in turn, is equivalent tocontracting e in Gu (Fig. 5). In terms of watersheds

it is intuitive to fuse the regions of Gu until each of

the resulting regions corresponds to exactly one

basin of the topographic surface. Two neighboring

regions may be fused, if there is no separating

ridge between the regions. Due to the initializationof the attribute values in Gu, we may formulate a

criterion for the fusion of two regions as follows:

Let r1 and r2 denote two regions of the neighbor-hood graph and let �rr1 and �rr2 denote the corre-sponding vertices in Gu. The regions r1 and r2(r1 6¼ r2) may be fused, if there exists an edge e with�iiNðeÞ ¼ f�rr1;�rr2g such that attðeÞ ¼ attð�rr1Þ or

attðeÞ ¼ attð�rr2Þ. Assume that attð�rr2Þ6 attð�rr1Þ.

Then the fusion of r1 and r2 is achieved by thecontraction of �rr1 into �rr2. Thus, during the wholecontraction process the attribute values of a vertex�vv in Gu indicates the altitude of the deepest point inthe region represented by v. Multiple fusions canbe done by iterating the following parallel step.

(1) Form edge disjoint trees in Gu from the di-

rected edges (directed toward an end vertex

with a minimal attð�Þ-value) that fulfill theabove condition for contraction. The trees

are required to be rooted trees of depth one,the root being the target of all edges within

the tree.

(2) Contract all edges of all rooted trees in a single

parallel step.

The iteration stops, when none of the edges

meets the above criterion in Gu. The remaining

edges of Gu induce a connected subgraph G0u of Gu

that still contains all vertices of Gu. (Fig. 6b). Due

to the above conditions on the contraction of

edges from Gu the following condition holds. For

each path P between two vertices v and w in Gu

with

minfattðvÞjv is on PgP L for some number L

ð3Þthere exists a path on G0

u with the same property.

Since the attð�Þ-values of the vertices from Gu in-dicate the saliency of the strokes and the crossings,

the mapping Gu ! G0u is saliency preserving. Our

model of strokes from superimposed brush moves

excludes strokes that do not belong to some cross-

hatch. Therefore we form a graph G00u by excluding

all edges from G0u that do not belong to any circuit.

Fig. 6. (a) Plane neighborhood graph Gu, (b) saliency preserving subgraph of neighborhood graph, (c) topological minor Gtop.

P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050 1047

This can be done in a single parallel step, since the

excluded edges are exactly the self-loops in Gu. The

topological minor Gtop is derived from G00u by

contracting all edges, at least one end vertex of

which has degree two.

4. Grouping the edges of the topological minor

Our model, i.e. strokes from superimposed

brush moves, allows us to remove all self-loops

from Gtop and to represent parallel edges in Gtop by

single edges. Furthermore, all edges of Gtop arereplaced by straight line segments between their

end vertices (Fig. 7b).

Intuitively, strokes are represented by paths of

the topological minor Gtop which have minimal

tortuosity. In the following we will make use of the

fact that the vertex degrees in Gtop can only be 3 or

4. In case of a vertex� with degree 4, there is butone way to pursue a stroke across �, since for each

edge e with end vertex � there is a unique opposite

edge at �.Let Y be a vertex with degree 3. For an edge

with end vertex Y there is no unique opposite edge.Moreover, the continuation of two strokes across

Y yields overlapping strokes, i.e. the two strokesshare an edge. The idea for solving such conflicts is

to

(1) Group the edges of Gtop in a greedy way: Ini-

tially, the groups consist of the single edges.

Groups G1 and G2 (edge sets of paths) may

be united, if G1 [ G2 is the edge set of another

path and if the bend (angle) of the joint bet-ween G1 and G2 is minimal with respect to al-

ternative joints. The group of an edge e isdenoted by groupðeÞ.

(2) Contract edges e of Gtop that fulfill the follow-

ing conditions (Fig. 7c).

• Both end vertices v and w of e have degree 3.Thus, the contraction of e yields a new ver-tex � with degree 4.

Fig. 7. Strokes from Fig. 1b: (a) maximal responses, (b) simple Gtop with straight line segments, (c) contracting edge e, (d) final graph,

(e, f) two problematic examples. The final graphs are superimposed on the original images.

1048 P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050

• There exist edges ev ¼ fv; v0g, ew ¼ fw;w0gsuch that ev; ew 6¼ e, groupðevÞ, groupðewÞ ¼groupðeÞ and the sign of hv� v0; ðw� vÞ?iequals the sign of hw0 � w; ðw� vÞ?i (Fig.7c). Here ðw� vÞ? denotes a vector perpen-dicular to the vector w� v and h�; �i denotesthe scalar product. The latter requirement

ensures that ev and ew are opposite edgesof the new vertex � after the contraction

of e. The edge contractions are done in agreedy way: The harshness of the contraction

of e is expressed by the ratio (linear distancebetween v0 and w0)/ðlengthðevÞ þ lengthðeÞþlengthðewÞÞ. Low-harshness contractions areperformed first.

(3) Group the two opposite edges at� not con-tained in the group of e.

Before the edge contractions the more promi-

nent vertical strokes dissected the horizontal

strokes in the example of Fig. 7. The horizontalstrokes, however, could be restored by the edge

contractions (Fig. 7d).

Our method is not robust with respect to per-

turbations of the graph attributes. Thus it fails

wherever the maximal filter responses do not

emerge from the background along even short

parts of the strokes. Furthermore, our model of

superimposed brush strokes often cannot be ap-plied to areas close to the border of the image.

Cross-hatches cut by the border are either not

represented correctly (Fig. 7e) or they are not

represented at all (Fig. 7f).

5. Conclusions

In this paper a graph-based method to segment

strokes from superimposed brush moves was

proposed. Local evidence for crossings and lines,

coming from a set of line filters, is grouped byforming a topological minor of the underlying

neighborhood graph. We used dual graph con-

traction to do the grouping in parallel steps, each

of which consists of local operations on the

neighborhood graph and its abstract dual. In a

second grouping step the strokes are formed by

partitioning the edges of the topological minor.

Here we utilized the special properties of the topo-

logical minor, i.e. the restriction on the vertex

degrees, and the embedding. In particular, the

orientation of the edges around the vertices and

the angles between the edges allowed us to for-

mulate grouping criteria in accordance with ourmodel.

Acknowledgements

Supported by the Austrian Science Fonds

(FWF) under grants P12028-MAT and P14445-

MAT. The authors gratefully acknowledge fruitful

discussions with Walter G. Kropatsch and Horst

Bischof.

References

Amir, A., Lindenbaum, M., 1998. A generic grouping algo-

rithm and its quantitative analysis. IEEE Trans. Pattern

Anal. Machine Intell. 20 (2), 168–184.

Chen, J., Sato, Y., Tamura, S., 2000. Orientation space filtering

for multiple orientation line segmentation. PAMI 22 (5),

417–429.

Diestel, Reinhard, 1997. Graph Theory. Springer, New York.

Glantz, R., Kropatsch, W.G., 2000. Plane embedding of dually

contracted graphs. Discrete Geometry for Computer Imag-

ery, DGCF2000, December 2000, vol. 1953. Lecture Notes

in Computer Science, Uppsala, pp. 348–357.

J€aahne, B., 1999. Local structure. In: J€aahne, B., Haussecker, H.,

Geissler, P. (Eds.), Handbook of Computer Vision and

Applications, vol. 2. Academic Press, San Diego.

Koenderink, J., van Doorn, A., 1994. Image structure. In:

Paulus, E., Wahl, F. (Eds.), Mustererkennung 1997. Springer.

Kropatsch, W.G., 1995. Building irregular pyramids by dual

graph contraction. IEEE Proc. Vision, Image and Signal

Process 142 (6), 366–374.

Lades, M., Vorbr€uuggen, J.C., Buhmann, J., Lange, J., von der

Malsburg, C., W€uurtz, R.P., Konen, W., 1993. Distortioninvariant object recognition in the dynamic link architec-

ture. IEEE Trans. Comput. 42, 300–311.

van der Malsburg, C., 1988. Pattern recognition by labeled

graph matching. Neural Networks 1, 141–148.

Meijsler, A., Roerdink, J., 1998. A disjoint set algorithm for the

watershed transform. In: Proceedings of the EUSIPCO’98,

IX European Signal Processing Conference, Rhodes, Greece,

1998, pp. 1665–1668.

Meyer, F., 1999. Graph based morphological segmentation. In:

Kropatsch, W.G., Jolion, J.-M. (Eds.), 2nd IAPR-TC-15

Workshop on Graph-based Representation, OCG-Schrif-

tenreihe, Band 126, €OOsterreichische Computer Gesellschaft,

pp. 51–60.

P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050 1049

Moghaddam, B., Pentland, A., 1997. Probabilistic visual

learning for object representation. PAMI 19 (7), 696–

710.

Perona, P., 1995. Deformable kernels for early vision. IEEE

Trans. Pattern Anal. Machine Intell. 17 (5).

Sablatnig, R., Kammerer, P., Zolda, E., 1998. Structural

analysis of paintings based on brush strokes. In: Anti-

Counterfeiting in Art, IS& T/SPIE’s 10th Annual Sympo-

sium on Electronic Imaging, 1998.

Thulasiraman, K., Swamy, M.N.S., 1992. Graphs: Theory and

Algorithms. J. Wiley & Sons, New York, USA.

Vincent, L., Soille, P., 1992. Watersheds in digital spaces: an

efficient algorithm based on immersion simulations. IEEE

Trans. Pattern Anal. Machine Intell. 13 (6), 583–598.

1050 P. Kammerer, R. Glantz / Pattern Recognition Letters 24 (2003) 1043–1050