Benchmarking the interoperability of ontology development tools

Benchmarking the interoperability of ODTs. April 7th 2005 1 © Raúl García-Castro, Asunción Gómez-Pérez

Benchmarking the interoperability of ontology development tools

Raúl García-Castro, Asunción Gómez-Pérez

<rgarcia,[email protected]>

April 7th 2005


Table of Contents

• The interoperability problem

• Benchmarking framework

• Experiment to perform

• Participating in the benchmarking


Ontology development tools interoperability problem

• It appears due to ontology reuse.

Tool 1

Tool 2

Tool 3

Tool 4

Tool 5

Ontology development tools

Potential functionalities

Real functionalities


Ontology development tools interoperability problem

• Why is it difficult?– Different KR formalisms

frames description logic conceptual graphs

first order logic semantic networks

– Different modelling components inside the same KR formalism

• Some results:– It is difficult to preserve the semantics and the intended

meaning of the ontology– Interoperability decisions…

• At many different levels• Usually hidden in the programming code of ontology

exporters/importersO. Corcho. A Layered Declarative Approach to Ontology Translation with Knowledge PreservationFrontiers in Artificial Intelligence and Applications, Volume 116, January 2005


Knowledge Model comparison

Classes

Template Slots/properties/instance attributes

InstancesData types

RDF(S)

Protégé-2000

Subclass-of

Subproperty-of

Literals

Containers

Collections

Statements

Own slots/Class attributes

PAL constraints/WAB axioms

Concept groups

Disjoint decompositions

Exhaustive decompositions

Partitions

Constants

Relation properties

Synonyms

Abbreviations

Bibliographic references

Metaclasses


Elements outside RDF(S) K.M.

Thesis

MSc Thesis PhD Thesis

Disjoint-subclass

Thesis


subclass

Thesis


subclass

Partial loss

Total lossDon’t export

Insert ad-hoc RDF(S)

<rdfsClass rdf:about=“#Thesis”> <a:disjoint rdf:Resource=“#MsC Thesis”> <a:disjoint rdf:Resource=“#PhD Thesis”></rdfs:Class>

Doesn’t import

Thesis

RDFSEXPORT IMPORT


Example: WebODE and Protégé

RDFS <rdf:Description rdf:about='#distanceToSkiResort'> <rdf:type rdf:resource=‘&rdf;Property'/> <rdfs:comment>The distance from the hotel to a ski resort</rdfs:comment> <rdfs:domain rdf:resource='#accommodation'/> <rdfs:range rdf:resource=‘&rdfs;Literal'/> </rdf:Description>

WebODE

RDF(S)

Protégé-2000

Protégé-2000


Example: WebODE and Protégé

WebODE

RDF(S)

Protégé-2000

Protégé-2000

RDFS<rdf:Property rdf:about=“#distanceToSkiResort“

a:maxCardinality="1“ a:minValue="0.0“ a:range="float“ rdfs:comment="The distance from the hotel to a ski resort“ rdfs:label="distanceToSkiResort">

<rdfs:domain rdf:resource=“#accommodation"/> <rdfs:range rdf:resource="&rdfs;Literal"/></rdf:Property>

Protégé-2000 generatesad hoc RDF(S) code


Table of Contents






Benchmark and benchmarking

Benchmarking• Systematic evaluation• Comparison with the best tools• Extraction of best practices

Benchmark Benchmarking

IS A Test Continuous process

PURPOSE

• Measure• Evaluate

• Search for best practices– Measure– Evaluate

• Improve

TARGET

• Method• System

• Process• Product• Service


General framework for benchmarkingBENCHMARKING ITERATION

Recalibration task

PLAN PHASE

1. B. goals identification 2. B. subject identification 3. Participant identification 4. B. proposal writing 5. Management involvement 6. B. partner selection 7. B. planning and resource allocation

IMPROVE PHASE

11. B. report writing 12. B. findings communication 13. Improvement planning 14. Improvement 15. Monitor

EXPERIMENT PHASE

8. Experiment definition 9. Experiment execution 10. Experiment results analysis

General evaluation criteria:• Interoperability• Scalability• Robustness

Benchmark suites for:• Interoperability• Scalability• Robustness

Benchmarking supporting tools:• Testing frameworks• Workload generators• Monitoring tools• Statistical packages

García-Castro, Maynard, Wache, Foxvog and González-Cabero. Knowledge Web Deliverable 2.1.4 Specification of a methodology, general criteria, and benchmark suites for benchmarking ontology tools. December 2004.


Plan phase

Benchmarkinggoals

identification

Benchmarkingsubject

identification

Participantidentification

Need for benchmarking

Organisation goals and strategies Benchmarking

proposal writing

Benchmarking goals, benefits, costs

Benchmarking subject, tool functionalities, evaluation criteria

List of involved members, benchmarking team

Management involvement

Benchmarking partner selection

Benchmarking planning and

resource allocation

Benchmarking proposal

Management support

Benchmarking partners, updated benchmarking proposal

Benchmarking planning

Organisation's tools

Tools from outside the organisation

Organisation planning

Improve the interoperability of ontol. development tools

RDF(S) import and export capabilities

Identify ontology components

exported/imported

B.P.


Experimentdefinition

Experimentexecution

Experimentanalysis

Benchmarking planning

Benchmarking proposal Experiment

definition, experimentation planning

Experiment results

Experiment report

Experiment phase

RDF(S) Import benchmark suitesRDF(S) Export benchmark suites

• test 1• test 2• test 3• ...


NOOKOK


OKNOOK

OKOKNO ...



NOOKOK


OKNOOK

OKOKNO

E.R.


Benchmarkingreport writing

Benchmarking findings

communication

Improvementplanning

Updated benchmarking proposal

Experiment report

Improvement

Benchmarking report

Updated benchmarking report

Monitor

Necessary changes, improvement planning, improvement forecast

Organisation support

Improved tool

Monitorisation report

Improve phase

• Comparative analysis• Compliance with standards• Weaknesses• Recommendations on tools• Recommendations on practices


Table of Contents






Benchmarking goalsGoal 1:• To assess and improve the interoperability of ontology development tools using

RDF(S) for ontology exchange.

Goal 2:• To identify the subset of RDF(S) elements that ontology development

tools can use to correctly interoperate.

Goal 3:• Next step: OWL.


Experiment to perform1. Export to RDF(S)• Check if ontology development

tools can export the core elements of RDF(S).

• Check if ontology development tools can export other elements of their knowledge models.

2. Import from RDF(S)• Check if ontology development

tools can import the core elements of RDF(S).

• Check if ontology development tools can import the non-core elements of RDF(S).

• Check if ontology development tools can import the other elements of the knowledge model the tools exported to RDF(S).

RDF(S)

RDF(S)


Extended Benchmarks

Sample export benchmarks

Core Benchmarks Common to every tool

Particular to each tool

Benchmark 3 Export concept

NL description Export an ontology containing 3 concepts and no further property or relation between them.

Graphical representation

Expected result <rdf:RDF

xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"

xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">

<rdfs:Class rdf:about="#concept1“/>



</rdf:RDF>

Concept 1 Concept 2 Concept 3



Benchmark 6 Export linear concept taxonomy

NL description Export an ontology containing 3 concepts with a subclass-of relation between each pair of them forming a linear taxonomy.


Expected result <rdf:RDF



<rdfs:Class rdf:about="#Concept 1“>

</rdfs:Class>


<rdfs:subClassof rdf:resource=“Concept 1”>

</rdfs:Class>

<rdfs:Class rdf:about="#Concept 3”>


</rdfs:Class>

</rdf:RDF>

Concept 1

Concept 2

Concept 3



Concept 1


Concept 1


Concept 1

Concept 2

Benchmark 7

Export a concept with multiple children

Export an ontology containing four concepts, being three of them subclass of the fourth one.

Benchmark 8

Export a concept with multiple parents

Export an ontology containing four concepts, one of them subclass of the other three.

Benchmark 9

Export two concepts subclass of each other

Export an ontology containing two concepts, each one subclass of the other.


Export process

Load ontology

Concept 1

Concept 2

Export ontology

<rdf:RDF

xmlns:rdf="http://www.w3.org/2-rdf-syntax-ns#"


</rdfs:Class>


<rdfs:subClassOf rdf:resource=“Concept 1”>

</rdfs:Class>

</rdf:RDF>

Compare result with expected

<rdf:RDF

xmlns:rdfs="http://www.w3.org/f-schema#">


</rdfs:Class>



</rdfs:Class>

</rdf:RDF>

<rdf:RDF



</rdfs:Class>



</rdfs:Class>

</rdf:RDF>

= ?YES

NO

Steps can be manual or automatic

Export strategy: minimal

knowledge loss in exports


Export resultsWebODE Protégé Tool A Tool B Tool C …

Benchmark 1 YES YES YES YES YES





Benchmark 6 YES YES YES NO YES

Benchmark 7 YES YES YES YES NO

Benchmark 8 NO YES NO YES YES

Benchmark 9 NO NO YES YES YES

Benchmark 10 YES YES NO NO YES

Benchmark 11 YES YES YES NO NO

Benchmark 12 NO NO YES NO NO

…


Extended Benchmarks

Sample import benchmarks

RDF(S) Core Benchmarks Common to every tool

Common to every tool. From export extended benchmarks

Benchmark 2 Import class

NL description Import a graph containing 3 classes and no further property between them.


RDF(S) source <rdf:RDF






</rdf:RDF>

Expected result

Class 1 Class 2 Class 3

RDF(S) Extended Benchmarks

Class 1 Class 2 Class 3


Sample import benchmarks

Benchmark 5 Import classes with a property

NL description Import a graph containing 2 classes and a property between them.


RDF(S) source <rdf:RDF



<rdfs:Class rdf:about="#Person“>

<ns:Is_author rdf:Resource=“#Book”>

</rdfs:Class>

<rdfs:Class rdf:about="#Book“/>

</rdf:RDF>

Expected result

Person BookIs_author

Person BookIs_author


Import process

Load graph

<rdf:RDF



</rdfs:Class>



</rdfs:Class>

</rdf:RDF>

Import graph

Class 1

Class 2

Steps can be manual or automatic

Compare result with expected = ?

YES

NO

Class 1

Class 2

Class 1

Class 2


Import resultsWebODE Protégé Tool A Tool B Tool C …






Benchmark 6 YES YES YES NO YES

Benchmark 7 YES YES YES YES NO

Benchmark 8 NO YES NO YES YES

Benchmark 9 NO NO NO YES YES

Benchmark 10 YES YES NO NO YES

Benchmark 11 YES YES YES NO NO

Benchmark 12 NO NO YES NO NO

…


Table of Contents






Benchmarking benefits

• For the participants:– To know in detail the interoperability of their ODTs.– To know the set of terms in which interoperability between

their ODTs can be achieved.– To show the rest of the world that their ODTs are able to

interoperate and are among the best ODTs.• For the Semantic Web community:

– To obtain a significant improvement in the interoperability of ODTs.

– To know the best practices that are performed when developing the interoperability of ontology development tools.

– To obtain instruments to assess the interoperability of ODTs.– To know the best-in-class ODTs regarding interoperability.


Participating in the benchmarking

• Every organisation is invited to participate:– If you are developers, with your own tool.– If you are users, with your preferred tool.

• Supported by the Knowledge Web NoE.

• The results will be presented in the EON 2005 workshop.


Timeline

April 22nd 2005 First definition of export and import benchmarks.

May 6th 2005 Agreement on the benchmark definitions and the experimentation.

July 1st 2005 Results from Export experiments.

September 16th 2005

Results from Import experiments.

If you want to participate in the benchmarking or have some further question/comment about it, please contact:

Raúl García-Castro <[email protected]>


KW Deliverable 1.2.2Deliverable 1.2.2: “Semantic Web Framework Requirements Analysis” • Analyse applications and tools for identifying the set of requirements for interoperation and

exchange of ontologies. • Identify the main components that an unified semantic web framework should have.

Should contain:• The main systems developed on the field.• For each application or system, include its architecture, design criteria, main components,

how they interoperate with other systems or exchange their ontologies.

These studies should allow you to identify: • The main and additional requirements for each type of tool/system.• The main and additional functionalities.• Results on evaluation of the requirements of interoperation and exchange. • Results on evaluation of other criteria like scalability, etc.• Summarize in a table such criteria in order to have a clear picture of the field.

Needs contributions from tool developers in order to obtain accurate descriptions of their tools.


Benchmarking the interoperability of ontology development tools

Raúl García-Castro, Asunción Gómez-Pérez <rgarcia,[email protected]>

April 7th 2005

Documents

Benchmarking the interoperability of ontology development tools