Employing structural representation for symbol detection, symbol spotting and indexation in line drawing document images Muhammad Muzzamil Luqman [email protected]

Employing structural representation for symbol detection, symbol spotting and indexation in line drawing document images

Muhammad Muzzamil [email protected] student (II year)

Jean-Yves Ramel Université François Rabelais de Tours, France

Thierry Brouard Université François Rabelais de Tours, France

Josep Lladós Universitat Autònoma de Barcelona, Spain

Thesis supervisors

Wednesday, 02 June 2010

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 2Plan

Part 1

Representation and recognition of graphics content in line drawing

document images

Part 2

Unsupervised indexation and content based (focused) retrieval for

line drawing document image repositories


Part 1

Representation and recognition of graphics content in line drawing document images

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 4Representation phase

Representation phase

Representation of structure of graphics content by an Attributed Relational Graph.

Description phase

Learning and Classification phase

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 5Description phase



Description phase

Extraction of signature from ARG.

Num

ber

of n

ode

s

Number of connections

L-Junctions

T-Junctions

Successive connections (S)

Parallel connections (P)

Intersections (X)

Number of nodes with

Medium density of connections

Number of nodes with High

density of connections

Number of Small-Length

primitives

Number of Medum-Length

primitives

Number of Full-Length

primitives

Number of Small-Angle connections

Number of Medum-Angle connections

Number of Full-Angle

connections

Arrangement of connections (between primitives)

Distribution of relative length of primitives

Distribution of relative angle of connections

Density of Connections at nodes

Number of primitives in symbol

Number of nodes with Low



Num

ber

of n

ode

sNumber of

connections

L-Junctions

T-Junctions



Intersections (X)






primitives


primitives


primitives




connections









Num

ber o

f nod

es


L-Junctions

T-Junctions



Intersections (X)






primitives


primitives


primitives




connections








A value laying here fully

contributes (i.e. membership

weight 1) to the interval “Small”

A value laying here contributes

in part to the interval “Medium” and in part to the

interval “Full”


Num

ber

of n

odes


L-Junctions

T-Junctions



Intersections (X)






primitives


primitives


primitives




connections








Two iterations over set of ARGs:

First iteration1. Compute ‘connection density counts’ for all

ARGs2. Distribute these ‘connection density counts’ in

an optimal number of bins3. Arrange the bins in a fuzzy fashion to form

overlapping intervals for ‘Low’, ‘Medium’ & ‘High’ connection densities.

Second iterationCompute signature for graphic symbols (ARGs)

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 9Learning phase (Structure & Parameters of BN)


Description phase


Encoding of Joint Probability Distribution of signatures by a Bayesian Network.

P(Nodes)

P(Class|Nodes)

P(DenH|DenM)

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 10Classification phase (Graphics Recognition)


Description phase


Encoding of Joint Probability Distribution of signatures by a Bayesian Network.

Bayesian probabilistic inference for recognition.

Bayes rule:

ikelihoodMarginal l

robability * Prior pLikelihood yprobabilitPosterior

)(

)()|(

)(

),( e) |(

eP

cPceP

eP

cePcP iii

i

k

1i

)()|(),(e)(

21,...,2,1

iii cPcePcePP

fffe

where

Query is recognized as class which gets highest posterior probability!

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 11Example images

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 12Noise and deformations

2D linear model symbols from GREC databases

Learning on clean symbols and testing against noisy and deformed symbols

Results presented in CIFED2010 – With Fuzzy Intervals

Results presented in ICDAR2009 – Without Fuzzy intervals

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 13Noise and deformations

2D linear model symbols from GREC databases

Learning on clean symbols and testing against noisy and deformed symbols

Comparing results with (Qureshi et al., 2007) and (Luqman et al., 2009)

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 14Context noise

2D linear model symbols from GREC databases (SESYD dataset)

Learning on clean symbols and testing against context-noise

Results presented in CIFED2010 – With Fuzzy Intervals

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 15Some remarks

Based on vectorization and hence is sensitive to noise and deformation (which

produce irregularities in signature). The proposed signature is more vulnerable to

symbols that are composed of circles/arcs.

However, lightweight signature and use of an efficient classifier makes it suitable to be

used as a pre-processing step to reduce search space or as a quick discrimination

method for sufficiently large number of graphic symbols … an application to

symbol spotting!

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 16Generalizing fuzzy signature - Explicit Graph Embedding

Vector for explicit embedding of attributed graphs

Fuzzy zones for “features for node degrees” (for example)

A value laying here contributes

in part to the interval “Fi2” and

in part to the interval “Fi3”

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 17ICPR2010 contest on Explicit Graph Embedding (GEPR)

ICPR2010 contest Graph Embedding for Pattern Recognition (GEPR)

Results on sample contest data

ALOI (Performance Index: 0.379)

COIL (Performance Index: 0.376)

ODBK (Performance Index: 0.353)

ALOI - Amsterdam Library of Object ImagesCOIL - Columbia Object Image LibraryODBK - Object Databank Performance Index measures the

quality of clustering (that could be obtained for the embedded vectors).

The closer it gets to zero the better the embedding results are!


Part 2

Unsupervised indexation and content based (focused) retrieval for

line drawing document image repositories

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 19A Symbol Spotting & Focused Retrieval System

Document base

Spotting system

QBE

Localization results

Utilisateur


Unsupervised indexation of line drawing document images

Represent document images by attributed relational graphs

Spot Regions Of Interest (ROIs) in the ARG of document image

Learn parameters for fuzzy structural signature from the set of ROIs

Describe each ROI by a fuzzy structural signature

Cluster signatures of ROIs

Prepare an index (clusterID vs ROIs vs documentImage) and

Learn a BN


Content based focused retrieval for line drawing document images

Represent query ROI by attributed relational graph

Spot Regions Of Interest (ROIs)

Describe each query ROI by a fuzzy structural signature

Classify query ROIs using BN and

Retrieve documents using repository index


Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 23Experimentation

Dataset

SESYD (Systems Evaluation SYnthetic Documents)

During learning phase our system detected a total of 10285 ROIs in electronic diagrams and 4586 ROIs in floorplans, which approximately corresponds to 108% of the symbols in each of the datasets.

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 24Experimentation

Document Retrieval Results

Results presented in ICPR2010

Each point in the graph represents the precision

and recall values for a query image.

Part1: Recognition of graphics contentExperimentationSome remarksPart2: Content based (focused) retrievalExperimentationConclusion - 25Conclusion and Future work

The Overall framework allows to prepare an index for the document repository in an

unsupervised fashion, which is a very important contribution.

However the underlying method for ROI localization is based on a set of heuristics and

does not return a single symbol in most of the cases and needs to be improved.

Future lines of work include the designing of a method to replace the manually

selected heuristics by automatic learned heuristics for spotting a ROI.

- 26References

Delalandre et al., “Building synthetic graphical documents for performance evaluation,” in GREC, vol. 5046 of LNCS, pp. 288–298, Springer, 2007.

Delaplace et al., Two evolutionary methods for learning bayesian network structures, in LNAI 2007.

Luqman et al., A Content Spotting System For Line Drawing Graphic Document Images, International Conference on Pattern Recognition, 2010, to appear.

Luqman et al., Vers une approche floue d’encapsulation de graphes: application à la reconnaissance de symboles, Colloque International Francophone sur l'Ecrit et le Document, 2010, 169-184.

Luqman et al., Graphic Symbol Recognition using Graph Based Signature and Bayesian Network Classifier, Tenth International Conference on Document Analysis and Recognition (ICDAR), IEEE Computer Society, 2009, volume 10, 1325-1329.

Luqman et al., Employing fuzzy intervals and loop-based methodology for designing structural signature: an application to symbol recognition, Eighth IAPR International Workshop on Graphics RECognition (GREC), 2009, volume 8, 22-31.

Qureshi et al., Combination of symbolic and statistical features for symbols recognition, in IEEE ICSCN’2007.

Qureshi et al., “Spotting symbols in line drawing images using graph representations,” in GREC, pp. 91–103, 2007.