Toward an Automatic Annotation Approach Based …Semantic annotation process relaying on: semantic rules definition, ontology enrichment and new images annotation based on the enriched

International Journal of Engineering & Technology IJET-IJENS Vol:13 No:02 80

138102-9797-IJET-IJENS © April 2013 IJENS I J E N S

Abstract— In this paper, we present a new approach for image

semantic annotation to facilitate the retrieval of the image. It

discusses works in progress of our approach, which comprises

the extension of ontology by the semantic rules to facilitate and

optimize research. This comprises the development of the domain ontology used for annotation, the functionality for annotating

image with an underlying ontology and search features based on

these annotations.

Our approach is based on the definition and the generation of semantic rules presented via the logic of predicates. Then, we

proposed the classification of these rules by a method of

clustering Fuzzy C-means. For this, we used the method of co-citation to calculate the similarity measure between images. This

classification has grouped thematically these images with the aim

to facilitate research and annotation. We also describe a method

for automatic enrichment query SPARQL language using new

properties that we defined.

Index Term— Automatic annotation, Classification,

Enrichment ontology, Enrichment query, Hierarchical

organization, Image, Semantic rules, SPARQL,

I. INTRODUCTION

Recent years have witnessed rapid evolution of technology

acquisition, analysis and multimedia data representation [1].

However, the exploitation of this data becomes very difficult

and imprecise to search relevant information due to the huge

volume of these data. In front of this mass data, the users don’t

find the most relevant documents with regard to their

requirements. So, to facilitate the search for multimedia

documents by their contents, we need to annotate them.

In the literature and according to Azouaou and al. [2], there

This paragraph of the first footnote will contain the date on which you

submitted your paper for review. It will also contain support information, including sponsor and financial support acknowledgment. For example, ―This

work was supported in part by the U.S. Department of Commerce under Grant BS123456‖.

The next few paragraphs should contain the authors’ current affiliations, including current address and e-mail. For example, F. A. Author is with the

National Institute of Standards and Technology, Boulder, CO 80305 USA (e-mail: author@ boulder.nist.gov).

S. B. Author, Jr., was with Rice University, Houston, TX 77005 USA. He is now with the Department of Physics, Colorado State University, Fort

Collins, CO 80523 USA (e-mail: [email protected]). T . C. Author is with the Electrical Engineering Department, University of

Colorado, Boulder, CO 80309 USA, on leave from the National Research

Institute for Metals, Tsukuba, Japan (e-mail: [email protected]).

is not a consensual definition of the annotation, but rather

several definitions (from diverse dictionaries) according to the

search domains. For example Gillaume [3] define annotation

as ―a particular mark attached to a collection of documents, a

document, a segment of document, the annotation it be

corresponding to contents which are a mental track

representation that the annotator is made for the document‖.

Multimedia annotation is an important open problem in

computer vision specify for image. It is necessary to define, in

the images documents for example, which vocabulary, to use,

at which levels annotate the image, according to which facet,

and how to organize them. Shatford [11] shows that there are

several manners to describe image:

In objective, but also in subjective way. For example, the

image of somebody who cries (objective) can also represent

the sadness (subjective).

At various levels between generic and specificity.

By various facets: temporal, spatial, according to the

activity or the event, and according to objects.

Often, annotations are manually, and represent a very long

process in comparison with an automatic. Furthermore, the

manual annotations are very subjective because various people

can adopt different viewpoints for the same image. It is more

difficult to master the coherence of various annotations by

various people as the immense database. These immense

databases motivate, in fact, the development of automatic

annotation and querying methods. The purpose of the

automatic images annotation is to develop methods which can

predict, for a new image, the relevant keywords among

annotation vocabulary.

However, several problems persist during the automation

images annotation, such as: how can we extract the semantic

contents in an automatic way to assure and facilitate the

research? How can we associate the high-level knowledge

with the low-level characteristics?

The aim of our work is to define an automatic images

annotation approach to facilitate and optimize research, for

advanced query. We intend, at first, to extract the low-level

information. Then, we intend to define semantic rules set

allowing defining semantic characteristics. These rules are

presented in the logic expression using ontology. Various

Toward an Automatic Annotation

Approach Based on Ontological

Enrichment for Advanced Research

Yassine Ayadi, Ikram Amous, and Faiez Gargouri MIRACL, Institut Supérieur d’Informatique et de multimédia, cité Ons, Sfax, Tunisia

[email protected], [email protected], [email protected]



semantic levels are automatically generated based on inference

rules and upgraded in the ontology. Finally, we propose a

querying process based on the enriched ontology.

The paper is organized as follows way. We present some

related works in section 2. Section 3 describes our proposed

annotation approach. Then, section 4 sketchers the querying

process. Experimental results are discussed in section5.

Finally, we conclude and outline avenues of future work.

II. RELATED WORKS

With the digitization, the document became more and more

rich and complex by integrating heterogeneous contents (text,

video, audio, graphic, images). The digital images for example

are now used in any circumstances, in a professional or private

frame, in an artistic or functional way.

To facilitate the search for the multimedia documents by

their contents, we need a solution to analyze and visualize the

images. The best solution to describe the images contents is

the annotation. Several methods are proposed, which can be

based on: neighborhood, words bag, hierarchy… We detail

some of them.

The method based on Neighborhood. Its objective is to be

able to locate the neighbors of an image (the images having

the similar characteristics) [5]. For example, the authors [5]

proposed a process to define a model allowing finding from an

image I, the model capable of allocating to this image labels

which can describe at best its semantic.

The method based on Words Bag. It consists in extracting

and grouping the image characteristics compared with clusters

constituting a visual vocabulary. Among the works using this

method, we cite Griffin and Perona [4], which build

hierarchies to accelerate the classification. Other works can be

quoted, as those of Lazebnik and al. [10], which consists in

cutting the image in a pyramidal way, by concatenating and

balancing histograms obtained at every level.

The method based on Hierarchy. Its objective is to classify

and annotate images based on hierarchical organization. This

later can be based on visual characteristics , or based on the

Wordnet hierarchy (Srikanth and al. [9]).

Despite the important research made in images annotation,

the problem of the images semantic analysis is still remaining

as well as finding a solution to the semantic gap.

Recent works are made which uses structured vocabulary to

annotate images. These methods are based on the knowledge

exploitation and keywords organization. These methods are

based on Ontology allowing the knowledge representation; it

is defined by concepts and relationships. During the

annotation process, an image is annotated by associating rules

with the ontology. Other works concerning ontologies as those

of Gao and Fan [18], and Liu and al. [19], combines the

ontologies at WordNet terms to find annotation.

Note that the images semantic annotation improves the

result of the querying process.

There are two directions for the existing images research

systems: those based on the symbolic characteristics and those

based on the semantic. The first compares the similarity of the

global image characteristics. This kind of search is very weak

because there is difficult to link semantics with the low levels

characteristics such as the histogram, the textures...

In the research systems based on the semantic

characteristics, the image is analyzed by concepts, which is

defined as semantic terms used to describe the image. For the

representation of these concepts, they use OWL ontologies,

which does not allow the semantic definition of the complex

relations, such as, a parking is formed by cars the one in front

to the other.

Several interrogation system based on the ontologies were

proposed. Among, we quote Aqualog [12], which is based on

a waterfall model, in which the query in natural language is

translated into an intermediate representation with ontology.

An extension of this system, named PowerAqua, allowing

interrogation of heterogeneous and distributed ontologies on

Web.

Other system called ONLI [8] translates a query user in a

natural language to the nRQL language.

We can summarize several observations, the problem of the

semantic gap, which is not solved yet, and the images

annotation problem which remains still manual. We can again

notice that the hierarchical approaches are the object of an

increasing interest. They appear as a solution to reduce the

system complexity. We notice as well as the semantic

hierarchies allow improving the performances in images

research. As regards the ontologies, they are not well used for

the images annotation. The classification is used only for low-

level characteristics. Often, the images annotations are

manually. It is thus convenient to find an automating way to

annotate semantics to facilitate the search for the multimedia

documents.

The next section presents an overview on our annotation

approach. We present afterward the ontology enrichment used

to help our annotation process. Finally, we present the

inference process to generate automatically new semantic

annotation.

III. OVERVIEW ON OUR APPROACH

We propose an automatic approach of image annotation for

an advanced research guided by domain ontology. [14]

This approach is composed of four steps:

Training: propose a local structure of image. Training

process, extract the preliminary characteristics of image. In

our study, the idea is to exploit the visual descriptors and

topological relationships in image to determine their

semantics. Actually, neither tool presenting concepts

annotates exactly the images. The existing tools do not

combine the object detection and the relation one.

Semantic annotation process relaying on: semantic rules

definition, ontology enrichment and new images annotation

based on the enriched ontology. The objective of this

process is to know the rules structure; these rules represent

and reflect the human knowledge.

Inference process: generate automatically new semantic.



These rules are under declarative shape, so that the

interpretation procedures can use them easing.

Querying process: consist on the automatic

transformation of query reformulation through the various

ontology properties to the SPARQL code. This process is

based on the enrichment steps, which aims at searching the

relevant terms to enrich query. This enrichment leans on the

neighborhood methods, the semantic rules hierarchies and the

semantic rules classification.

In the following we detail the Semantic annotation process,

then querying process finally the experimental evaluation.

IV. SEMANTICS ANNOTATION PROCESS

The goal of semantics annotation process is to use existing

annotation to annotate new images. The proposed process is

composed of three parts: automatic annotation, ontology

enrichment and inference process.

A. Automatic annotation

The automation images annotation is recommended to

facilitate the research of images by contents. The Semantic

automatic Annotation, consist on the semantics rules

construction which express the deducted information and not

explicitly specified.

Semantic rules can be formulated using concepts and

relationship extracted from ontology. Each semantic rule can

be expressed with the formula: Oi . [R . Oj] Semantic

(e.g. car in front of car Parking).

In the following we presents semantics rules which are

expressed by users.

RG1: Taxi Car

RG2: Car In front of Car Parking

RG3: Car Transport Means

Once the semantic rules are defined, we can use them in the

images annotation. The automatic annotation of a new image

is definite choice criteria which determine the best annotation

(semantic rules) to use for the new image. For the more

objective is to import and to order the semantics annotations,

and to annotate automatically new images. To import and

order the semantic rules, we define two factors:

Fact.Aff (Appearance Factor): which represents the

time’s number of semantic rule appearance for the new

image annotation.

Fact.Salt (Selection Factor): which represents the time’s

number of semantic rule selection for the new image

annotation.

Based on these factors we define two rates. The first one:

the selection rates (Sel_Rate) of a semantic rule for new image

annotation. It improves to imported rules, which represents the

use percentage of a rule posted as proposal annotation. It is

calculated by the following formula:

affFactSelFactRateSel ./._ (1)

The use rate (Use_Rate) of an imported rule, which

represents the use percentage of a rule compared with all rules

rates. It is calculated by the following formula:

n

i

i

j

j

RuleRateSel

RuleRateSelRuleRateUse

_

__ (2)

The algorithm 1 presents outlines the steps of our automatic

new images annotation. It allows extracting from objects

composing the image, all the semantic rules from the

ontology. Then, it orders all the rules according to the

selection rate. Finally, it calculates for every rule the use rate

and re-orders them.

Algorithm 1: Automatic Semantic Annotation

Data

C {C1, C2,…,Cn] Concepts Set of New Image

ReG {RG1, RG2, …, RGn} Rules Set

Begin

//Extract all the Rules for each concepts couples Extract_Rules (Ci, Cj);

//order rules selection rate Ordonner(ReG);

//Calculate use rate

Use_Rate(RGi) ;

//re-order rules use rate Reordonner(ReG);

End

B. Ontology Enrichment

Ontologies are used to formalize the concepts of a given

domain. It is a formal explicit description of the knowledge

domain by concepts (classes), properties of every concept

describing characteristics and attributes (facets) and

relationship between concepts. The classes constitute the main

concepts of several ontologies. A class can have subclasses

which represent more specific concepts than the super-class

(or superior class). The attributes describe the classes and the

instance properties.

Our work proposes semantic image annotation guided by

domain ontology. So to annotate images we used

Fig. 1. Proposed Annotation Approach



TOWNTOLOGY [6], ontology which represents the concepts

used in the urban image.

Formally, that is O (Co, Ro, Xo) where: Co is a concept

set; Ro is a relationships between a concepts instances; Xo is

the axioms set defined on the concepts and the relations.

Figure 2.a depicts the meta-model of the ―TOWNTOLOGY‖.

To take account the semantic rules proposed in previous

step we proposed to enrich ontology by a new component

―RULES‖. So the formal definition the enriched ontology is:

Or (Co, Ro, Xo, Rgo). Where ―Rgo‖ is a rule set between the

concepts and the relations providing semantics.

In fact, we define some ontology properties, such as HasSet,

DefinedBy, ComposedBy, HasSemantic. We present in next

the definition of them.

HasSet Property: For each concept c, relation r and axiom a

of ontology Or (Co, Ro, Xo, Rgo), Such as c Co, r Ro,

aXo, the HasSet property are generated.

For each HasSet property generated, c Co, r Ro, a

Xo / c is a subset of concept SetOfConcept, r is a subset of

relation SetOfRelation and a is a subset of axiom

SetOfAxiom.

For the rule concepts, we can define the HasSet property

such as: for each rule rg of ontology Or = (Co, Ro, Xo, Rgo)

and rg Rgo, one HasSet property is generated. Thus, if the

HasSet property is generated, rg Rgo/ rg is a subset of

semantic rules SetOfRules.

DefinedBy Property: is generated for each ontology

Or = (Co, Ro, Xo, Rgo), such as Co is a set of concept

SetOfConcept, Ro is a set of relation SetOfRelation and Xo is

a set of axiom SetOfAxiom. Each concept c Co is defined by

property set SetOfProperty.

The DefinedBy property is generated for Ontology Or =

(Co, Ro, Xo, Rgo), such as Rgo is a set of semantic rules

SetOfRulest.

ComposedBy Property: Or ontology, such as Or =

(Co,Ro,Xo, Rgo). ai Xo, c1 Co et c2 Co, such as ai

= c1 ri c2. ci Co, cj Co, such as ci is composed of cj.

O ontology, such as Or = (Co, Ro, Xo, Rgo).

rgi Rgo, c1 Co, c2 Co, ri Ro, such as

rgi = c1 ri c2 s et s Co. s is semantic of rule and s is a

concept.

HasSemantic Property: For each semantic rules of

ontology O the HasSemantic property is generated. For the

semantic concept, we can define the HasSemantic property

generated, such as c is a sub-concept of SetOfConcept, r is a

sub-relation of SetOfRelation. ci Co, cj Co, rRo, such

as Rg = (Ci r Cj) generate a semantic by HasSemantic

property.

C. Inference Process

Remind that the inference is an action which allows a

machine to increase its knowledge. This person or this

machine deducts a result from a set of data.

The inference process consists to generate a new semantics.

The inference rules we have proposed is based on three

components which are the elements base (BE), the relations

base (BR) and the rules base (BGR). These inference rules are

specified as follows:

Rule 1:

∀{OB1, OB2, S1, S2} ∈BE

∀R ∈BR

∃ {RG1, RG2} ∈BGR /

RG1 = OB1, R, OB2 S1

RG2 = OB1 S2

∃RG3 a new Rule / RG3 = S2, R, OB2 S1

Rule 2:

∀{OB1, OB2, S1, S2} ∈BE

∃ {RG1, RG2} ∈BGR /

RG1 = OB1 S1

RG2 = S1 S2

∃RG3 ∈BGR / RG3 = OB1S2

The inference process generates new semantic rules

following the sequence of four steps which are:

1) Put all the semantic rules in a list (Begin_List)

2) For every rule, browse all the list (Begin_List), then apply

the two inference rules,

3) Register every new semantic rules, put it in the end of the

list (Begin_Listt)

4) To repeat the steps 2 and 3 till the end of the semantic

rules (Begin_List =)

Applying the inference in our example produces new

semantic rules presented in following:

RG4: Taxi in front of Taxi Parking

RG5: Transport Means In front of Transport Means

Parking

RG6: Taxi Transport Means

The semantic rules upgraded in the ontology improve the

query process. The following section details how semantic

rules are used in this process.

Fig. 2. Ontology Meta-model: (a) initial ontology. (b) Enrichement ontology



V. QUERYING PROCESS

The result of the annotation process is a set of semantics

rule [17], enriched in the ontology to help the querying

process. The objective of our querying process is to define a

query reformulation independent of any language. The

figure 3 presents the querying process architecture. It’s

composed of three main steps.

Acquisition Querying which makes possible to the user

to propose his query. This query is described by a set of

ontology terms.

Enrichment whose aim resides in proposing relevant

terms to enrich query? This step is based on the neighbor

concepts calculating, the semantic rule hierarchies and the

semantic rules classification.

The first enrichment called 'Enrichment by neighborhood’,

aim in bring neighboring concepts of the query terms from

ontology. The second enrichment called 'Enrichment by

semantics hierarchy', aim in bring semantics the closest to the

initial query terms. The third enrichment called 'Enrichment

by class', aim in bring semantics belonging to same class as

initial query terms.

The query is specified in SPARQL language.

Visualization of the query result.

The classification and the hierarchical semantic

organization are used to enrich the query, which represents the

objective of the following sub-sections.

A. Classification

The semantic rules classification process is based on the

construction of the distance matrix, which is a common point

of various classification methods. This distance matrix is

calculated by the Co-citation method, whose application in the

image context is designed to create from the semantic rules

(RS) in the same context, semantic relations between these

images. Given an image I containing several semantic rules,

each RS rules may be cited at least once by an image I [17].

So the semantic rules can be represented in a Citation graph or

co-citation matrix. In this work, the distance functions

representing the similarity between the rules:

2,

1,jiC

jiS (3)

Where C (i, j) represents the Co-citation index, which is

defined as the number of times in which the rules i and j are

cited sets. S(i, j) is in the [0, 1] interval. In fact, more two

semantic rules are cited together, more distance S(i, j) will be

close to zero. Since the semantic rules are cited, we can build

the MC distance matrix.

The classification results by Fuzzy C-means algorithm are a

matrix dimension where any elements present the membership

degree of rule i in class j.

The classification process can be presented in figure 4.

B. Semantic hierarchical organization

In order to represent association (Rules/Images), we use the

Galois lattice [16]. The latter is a mathematical structure

which represents the undisjoint classes subjoined with set

objects.

For example the following table II represents the

correspondence between six images with five rules {RG1,

RG2, RG3, RG4, RG5}.

The example of Galois correspondence is:

O1 = {I3, I4} f(O1) = {RG1, RG5}

A1 = {RG1, RG5} g(A1) = {I1, I3, I4}

In this example, we have the couple ({RG5}, {I1, I3, I4})

which means that the result of the query RG5 gives the images

{I1, I3, I4}.

The Galois lattice can be constructed following the Bordat

algorithm which is illustrated as follow.

Fig. 4. Classification Process

Fig. 3. Querying Process Architecture

TABLE I IMAGES/ RULES ASSOCIATED

RG1 RG2 RG3 RG4 RG5

I1 X X X I2 X X X

I3 X X X

I4 X X

I5 X X

I6 X X



C = (Ø, {I1, I2, I3, I4, I5})

δC = max {fC(I1), fC(I2), fC(I3), fC(I4), fC(I5), fC(I6)}

= max {{RG1, RG2, RG5}, {RG3, RG4, RG5}, {RG1,

RG4, RG5}, {RG1, RG5}, {RG4, RG5}, {RG3,

RG5}}

= max {{RG1, RG2, RG5}, {RG3, RG4, RG5}, {RG1,

RG4, RG5}}

In this case the direct successors of C are:

C1 = ({I1}, {RG1, RG2, RG5}); C2 = ({I2}, {RG3,

RG4, RG5}); C3 = ({I3}, {RG1, RG4, RG5}}

In the same way, the direct successors of C1 are calculated:

δC1 = max {fC1(I1), fC1(I2), fC1(I3), fC1(I4), fC1(I5),

fC1(I6)}

= max {{RG5}, {RG1, RG5}, {RG1, RG5}, {RG5},

{RG5}}

The C1 Successors are: C4 = ({I1, I3, I4}, {RG1, RG5})

The lattice structure is used to extract the hierarchical

relations between the semantic rules .

C. Query Enrichment

The objective of our process is to define an intermediate

query reformulation expressed with the ontology properties

which will be independent of every language.

Recall that the user query REQ = {t1, t2… t3} is described

by a set of terms from the ontology, where is rewired

following the proposed proprieties (especially HasSet).

REQ is defined flowing formula 4.

ONOO CtHasSetCtHasSetCtHasSet ,...,, 21 (4)

The enrichment process aims at proposing to the user

relevant terms to enrich its query and to guide it in its search.

Three types of reformulation are proposed: Neighbor

Reformulation, semantics Reformulation and class

Reformulation. We detail all of them.

1) Neighbor Reformulation

The goal of reformulation by Neighbor is to reach the nodes

(Concepts) which could satisfy the user.

Principe: ci Co, such as HasSet(Co, ci). cj Co, such

as HasSet(Co, cj) and SetOf(ci, cj).

Any property SetOf(Ci, cj) generated, such as cj is a direct

sub-concept of Ci. The SetOf property represents in fact the

relation is-a (cj has Ci). The reformulation by Neighbor can

then generate new results if we add new concepts at the initial

query.

Example: Initial query REQ1 = {c1, c2} is presented by

formula 5.

OO CcHasSetCcHasSet ,, 21 (5)

If c3 Co and SetOf (C1, c3) generated, then a new

reformulation of query is obtained as follows:

OOO CcHasSetCtHasSetCtHasSet ,,, 321 (6)

It is clear that the number of reformulations, obtained from

Neighbor reformulation grows in an exponential way with the

concepts number of the initial query. This enrichment phase

consists of research of the candidates concepts from Neighbor

concepts list. The various stages of this enrichment are as

follows:

Calculate the concept popularity measurement

―ConceptRank‖.

Calculate the concept importance based on popularity.

Search candidate’s concepts.

2) Semantic Reformulation

The goal of semantics reformulation is to reach the nodes

(semantics rules) which could satisfy the user.

Principe: ci Co, such as HasSet(Co, ci); rgi RGo, if

HasSemantic(rgi, Ci) then Ci is the semantics of the rule rgi.

Any property Fils (rgi, rgj) generated, such as rgi is a

semantic rule more abstract than the rule rgj

Example: Initial query REQ1 = {c1, c2}. If rg1 RGo and

HasSemantic (rg1, c1) a new reformulation of query as

follows:

1121 ,.,, CrgcHasSemantiCtHasSetCtHasSet OO (7)

If rg2 RGo, c3 Co, such as HasSemantic(rg2, c3),

and if fils (rg1, rg2) then a new reformulation of query is

obtained as follows:

32

1121

,

,,,

CrgcHasSemanti

CrgcHasSemantiCtHasSetCtHasSet OO

(8)

Note that the stop condition of reformulation is the end of

the semantic rules hierarchy.

3) Class Reformulation

The goal of class reformulation is to reach the nodes

belonging to the same class which could satisfy the user.

Principe: ci Co, such as HasSet(Co, ci); cj Co,

such as HasSet(Co, cj); rgi RGo, if HasSemantic (rgi, Ci)

then Ci is the semantics of the rule rgi. rgj RGo, if

HasSemantic(rgj, cj) then cj is the semantics of the rule rgj.

Any property CLASS (rgi, rgj) generated, such as rgi and

rgj semantic rules of the same class .

Example: Initial query REQ1 = {c1, c2}. If rg1 RGo and

HasSemantic (rg1, c1).

If rg2 RGo and HasSemantic (rg2, c3), such as CLASS

(rg1, rg2) a new reformulation of the query is as follows:

32

1121

,

,,,

CrgcHasSemanti

CrgcHasSemantiCtHasSetCtHasSet OO

(9)

D. Automatic SPARQL Query Construction

The query is translated in an interrogation language as for

example the SPARQL language recommended by the W3C

since 2008 [13]. The latter is a language which defines as well

as the syntax and semantics necessary for the query expression

in RDF schema.



Figure 7 shows an SPARQL query allowing to search the

images of parking in the RDF file (see Annex). The query

includes three clauses: the SELECT clause which identifies

variables that must appear in the result, the WHERE clause

which defines the elementary graph patterns to put in

correspondence with the RDF graph and the UNION clause

which realizes the union of alternative graphs. The elementary

graph patterns in this example consist of a single patterns of

triplet with a single variable (?Image). PREFIX ONTOLOGY: <http://ONTOLOGY.com/>

SELECT ?image

WHERE

{

{

?x ONTOLOGY:CONCEPT_NAME "Parking".

?x ONTOLOGY:MULTIMEDIA ?image.

}

UNION

{

?t ONTOLOGY:SEMANTIQUE "Parking".

?t ONTOLOGY:IMAGES ?image.

}

}

The translated in SPARQL is done in an automatic way

based on the correspondences between query properties and

the equivalent notation in SPARQL language.

SPARQL Query construction by follow:

HasSet(Co, ci) : ?x ONTOLOGY:CONCEPT_NAME "ci".

ComposedBy(ci, cj) : ?x ONTOLOGY:CONCEPT_NAME "ci".

?x ONTOLOGY:CID ?y.

?t ONTOLOGY:CONCEPT_NAME "cj".

?t ONTOLOGY:CID ?z.

?r ONTOLOGY:CONCEPT_ORIG ?y.

?r ONTOLOGY:CONCEPT_DEST ?z.

DefinedBy(ci, pi) : ?x ONTOLOGY:CONCEPT_NAME "ci".

?x ONTOLOGY:CONCEPT_DEF "pi".

SetOf(ci, cj) : ?x ONTOLOGY:CONCEPT_NAME "ci".

?x ONTOLOGY:CID ?y.

?t ONTOLOGY:CONCEPT_NAME "cj".

?t ONTOLOGY:CID ?z.

?r ONTOLOGY:CONCEPT_ORIG ?y.

?r ONTOLOGY:CONCEPT_ORIG ?z.

?r ONTOLOGY:TYPE "IS A".

VI. EXPERIMENTAL EVALUATION

To illustrate the feasibility of our proposed approach, we

developed a prototype called "MIRACLAnnotizer". Indeed, it

allows the automatic annotation and the advanced research.

Our prototype is experienced on the images bunch-mark

"Iaptrc ".

MIRACLAnnotizer assures the acquisition of annotations

using domain ontology in RDF file. It assists the user to

annotate a new image. This stage is based on the enriched

ontology. Once annotations are validated by annotator, the

tool assures the visualization of the ontology contents

(concepts and semantic rules) and the associated images.

MIRACLAnnotizer allows the research based on enrichment by

three methods: Neighbor Concepts, Rules Hierarchies and

Rule’s classifications.

MIRACLAnnotizer offer different functionality. The Figure

5 shows the global architecture.

It is composed of five modules: Corpus construction,

Annotation, Classification and hierarchical organization,

Querying and Visualization.

A. Annotation

The tool regroups many steps for the new image annotation.

These steps are illustrated by:

1. New image Selection;

2. Display semantic rules set which can be used to a new

annotation;

3. Manual creation of a new semantic rule when the user is

not satisfied with the shown rules.

4. Annotation is saved once semantics is chosen.

B. Annotation Querying

Remained that the querying process is based on: the query

description according to terms, and the automatic construction

of the SPARQL code after enrichment.

For example if the query composed of: traffic and parking.

Once the user proposed his query, our system proceeds to the

generation of the intermediate form, as well as the query

enrichment.

The query is enriched; the SPARQL code is generated, as

shown in Figure 7.

Fig. 5. MIRACLAnnotizer Architecture

Fig. 6. New Image Selection and Semantics Rules Display



The result of this query is displayed, where the user can

navigate through all the images.

C. Classification Evaluation

In order to check the classification result with the K-means

algorithm, we used the Cluster 3.0 tool and JMinHEP 1.0 tool.

The following table shows the result of these tests with a

database of 1000 images and 250 semantic rules and 3

clusters.

According to the above table, it is worthy to note that both

of these tools with a database of 1000 images give us almost

the same result.

As for the ascending hierarchical classification, we notice

that with "the k-means algorithm" a rule belongs to only one

class, which is not relevant in this case. This leads us to

choose a solution for a multi-classification, a rule that may

belong to several classes. For this reason, we have chosen to

use the extended form which is the fuzzy k-means algorithm

―fuzzy c-means‖ [7] which use the fuzzy set theory.

The result of our example gives the following matrix by

setting the number of clusters 3 and = 0.03, = 0.04 and = 0.05:

We have fixed the rate to value 0.05; the number of rules

in overlapping becomes stable.

D. Querying Process Evaluation

The querying systems based on the ontology are frequently

used. They improve the keywords search in ontology.

Enrichment step aims at offering terms to enrich the users

query. In this evaluation we have experiment 100 query.

For example, for the query REQ1 {Taxi, Traffic},

enrichment consists of three stages: by neighborhood, by

semantics and by class.

The enrichment system analyses every concept extracted

from query and calculates the importance of other concepts of

ontology comparing with initial query. At the end, the

enrichment system gives a list of concepts candidates for the

query enrichment.

In our example, two concepts Taxi and Traffic jam are kept

and offered by the user. The result of query is introduced by

table III.

The curve of recall strives towards 1, and the curve of

silence decries towards 0. Both curves of recall and of

precision show very good results at the level of the

performance of our research system.

For a sample of 100 queries got results are the following,

the medium of RECALL = 0,85; PRECISION equates in

0,834; the medium SILENCE equates in 0,147 and NOISE

equates in 0,165

The Results got so from our prototype show a very good

satisfaction rate.

VII. CONCLUSION

The semantic annotation allows associating images with

elements ontology. These latter allows to represent image

semantics, by the knowledge representing the studies domain.

We have proposed an approach which objective is to solve

the problem of extraction automatically of images semantics

to facilitate the research. Our approach also allows enriching

ontology, which consists of the semantic rules addition.

Our proposal can be summarized by the points:

The first contribution of our approach, concerns the

images components extraction and the construction of the

first semantic rules level. We defined afterward, an inference

system determining rules to deduct automatically more

abstract semantics.

The second contribution concerns, the ontology

enrichment by semantic rules extracted. This enrichment

allows at the same time to develop the ontology, but also to

TABLE II OVERLAPPING RULES

= 0.03 = 0.04 = 0.05

1000 Images

1000 Images

1000 Images

Overlapping Rules 35/120 63/120 67/120

Fig. 7. Automatic SPARQL code generated

TABLE III EVALUATING QUERY

Q UERY : Traffic Parking

RESULT enrichment

Recall Precision Silence Noise

Without 0,416 0,666 0,583 0,333

Neighborhood

0,625 0,789 0,375 0,210

Semantics 0,583 0,777 0,416 0,222 Class

0,625 0,882 0,375 0,117

With 0,833 0,869 0,166 0,130



help the querying process. We have presented also semantic

rules classification to allow grouping the images by class.

The third contribution concerns the definition of

automatic images annotation, which is guided by domain

ontology.

The fourth contribution is to define an advanced querying

model. In this model, the query is enriched by concepts from

the domain ontology. The objective of our querying process

is to define an intermediate reformulation with the ontology

properties independently to any query language. It is based

on an enrichment algorithm, which objective propos ing

relevant terms to enrich the query. The enrichment is

realized before its execution to allow more complete results.

Then, we construct automatically a SPARQL query.

Several perspectives are envisaged, such as:

Experiment our approach on other domains ontology.

Improve the shape of semantic rules. In fact we have

used a simple shape of rules which can’t take into account

complex information.

Define user profile which deserves to be taken into

account to improve the querying process .

REFERENCES [1] A. Jedidi, I. Amous, and F. Sèdes. ―A contribution to multimedia

document modeling and querying,‖ Multimedia Tools and Applications (MTAP), Kluwer academic publishers, 2005, Vol. 25 N. 3, pp. 391-404.

[2] F. Azouaou, C. Desmoulins, and D. Mille Dominique, ―Formalismes pour une mémoire de formation à base d’annotations : articulé

sémantique implicite et explicite,‖ in Actes de la conférence EIAH, France, 2003, pp. 43–54.

[3] G. Cabanac, ―Fédération et amélioration des activités documentaires par

la pratique d’annotation collective,‖ M.S. thesis. Toulouse university, 2008.

[4] G. Griffin, and P. Perona, ―Learning and using taxonomies for fast visual categorization,‖ in CVPR’08 : IEEE Conference on Computer

Vision and Pattern Recognition , 2008, pp. 1–8. [5] H. Hacid, ―Annotation semi automatique de grandes BD images:

Approches par graphes de voisinage,‖ Conférence francophone en recherche d’information et applications CORIA. Lyon, France, 2006, pp.

205-210. [6] J. Teller, A. F. CuttingDecelle, and R. Billen, ―Future of Urban

Ontologies,‖ Proceedings of the final conference Presse, Université de Liège, 2009.

[7] J.C. Dunn, ―A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters‖. Journal of Cybernetics, vol 3, 1973, pp. 32-57.

[8] L. Kosseim, R. Siblini, C. J. Baker, and S. Bergler, ―Using Selectional

Restrictions to Query an OWL Ontology ,‖ International Conference on Formal Ontology in Information Systems (FOIS 2006) , Baltimore, Maryland, USA, November, 2006.

[9] M. Srikanth, J. Varner, M. Bowden, and D. Moldovan, ―Exploiting ontologies for automatic image annotation ,‖ in SIGIR’05, New York, NY, USA, 2005, pp. 552–558.

[10] S. Lazebnik, C. Schmid, and J. Ponce, ―Spatial Pyramid Matching,‖

Object Categorization: Computer and Human Vision Perspectives, Cambridge University Press, 2009.

[11] S. S. Layne, ―Some issues in the indexing of images,‖ in J. Am. Soc. Inf. Sci, 1994, pp. 583–588.

[12] V. Lopez, M. Pasin, and E. Motta, ―Aqualog: An ontology portable question answering system for the semantic web,‖ in Proceedings European Semantic Web Conference (ESWC), Crete, 2005, pp. 546–562.

[13] W3C. http://www.w3c.org

[14] Y. Ayadi, I. Amous, A. Jedidi, and F. Gargouri, ―Towards a Semi Automatic Description of the Multimedia Image Document ,‖ in The International Business Information Management Conference (8th

IBIMA), Dublin, Ireland, 2008, pp. 203-209.

[15] Y. Ayadi, I. Amous, and F. Gargouri, ―Advanced images research based on an ontological extension,‖ MoMM, Ho Chi Minh City, Vietnam,

2011, pp. 252-255. [16] Y. Ayadi, I. Amous, and F. Gargouri, ―Hierarchical Organization of the

Semantic Rules for the Images Annotation By Co-Quotation Method,‖

IARIA SEMAPRO, Florence, Italy, 2010, pp. 138-144. [17] Y. AYADI, M. GARGOURI, I. AMOUS, and F. GARGOURI.

―Semantic Rules Classification for Images Annotation,‖ Eleventh International Conference on Hybrid Intelligent Systems (HIS 11),

Malysia. December 2011, pp. 72-77. [18] Y. Gao, J. Fan, X. Xue, and R. Jain, ―Automatic Image Annotation by

Incorporating Feature Hierarchy and Boosting to Scale up SVM Classifiers,‖ in ACM Multimedia, Santa Barbara, CA, 2006.

[19] Y. Liu, A. Coulet, P. LePendu, and N. H. Shah, “Using ontology based annotation to profile disease research,” Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA. 2012.

Yassine Ayadi received her Bachelor degree in business

data processing of Sfax University, Tunisia, in 2000. He received his Master Degree in informatcs from the higher institute of computer Science University of Mannouba Tunsia. He is currently a member at MIRACL laboratory of Sfax University, Tunisia. His research interests include multimedia document

annotation and querying.

Ikram Amous received her Bachelor degree in business

data processing of Sfax University, Tunisia, in 1998. She received her Master Degree in data processing and telecommunication from the Paul Sabatier University of Toulouse III, French in June 1999. She received her doctorate in Informatics from the Paul Sabatier University of Toulouse III in December

2003. She is currently a member at LARIM laboratory of Sfax University, Tunisia. Her research interests include semi-structured document modeling, multimedia document personalization, multimedia document annotation and

querying.

Faïez Gargouri is Professor at the Institut Supérieur

d’Informatique et de Multimédia de Sfax (Tunisia), where he is the head. He obtained his Master’s degree in Computer Science from the Paris 6 University

(1990) and a PhD from the Paris 5 University (1995). In 2002, he obtained an Habilitation Universitaire en Informatique from the Faculté des Sciences de Tunis (Tunisia). His research interest focus of information systems field: Design, Quality measurement, Verification, Data Warehousing, Multimedia,

Knowledge management, Ontology. He published more than 100 papers in journals, international conferences and he is a PC member of multiple international conferences.

http://www.w3c.org/



Annex

Documents

Toward an Automatic Annotation Approach Based …Semantic annotation process relaying on: semantic rules definition, ontology enrichment and new images annotation based on the enriched