CHAPTER – 7shodhganga.inflibnet.ac.in/bitstream/10603/36602/17/17_chapter7.pdf · Chapter-7: Prototype Implementation and Evaluation Page | 155 An Integrated Approach to Improve

Chapter-7: Prototype Implementation and Evaluation Page | 150

An Integrated Approach to Improve Ontology Mapping Process in Semantic Web

CHAPTER – 7

Prototype Implementation and

Evaluation

7.1 Prototype Implementation of Proposed

System

7.2 Evaluation of Proposed System

7.3 Summary



CHAPTER – 7

PROTOTYPE IMPLEMENTATION ANDEVALUATION

This chapter gives conceptual system design used for prototype

implementation and data sets used to evaluate the developed

prototype. It also describes in brief Precision, Recall, and F-Measure,

widely used metrics for evaluating the correctness of such algorithm.

Finally, it provides the analysis of result for the sample data sets while

proposed approach is applied on them in prototype.

7.1 Prototype Implementation of ProposedSystem

A GUI based system, called AI-ATOM, is developed to implement the

proposed algorithm in prototype. As the system is developed in

prototype using Rapid Application Development, the efficiency of the

system is not given a well thought. The main objective is to

demonstrate effectiveness of the proposed algorithm. The following

section describes abstract system requirement specification, system

module chart, database design, class diagram, and sample screens of

user interface of the prototype system.

7.1.1 System Requirement Specification

I. User Requirement (User should be able to):

a. Ontology Management

i. Create New Ontology

ii. View Ontology

iii. Update Ontology

iv. Delete Ontology



v. Import Ontology from File

vi. Export Ontology to RDF File

b. Ontology Mapping Project Management

i. Create New Ontology Mapping Project

ii. View Ontology Mapping

iii. Update Ontology Mapping Project

iv. Delete Ontology Mapping Project

v. Assign users to Ontology Mapping Project

c. Mapping Element Management

i. Generate mapping elements

ii. Store mapping elements

iii. View all mapping elements

iv. View suggested mapping elements

v. View un-processed mapping elements

vi. Accept selected mapping elements

vii. Reject selected mapping elements

viii. Add new mapping elements

d. System Settings

i. Auxiliary Resource Setting

Set Context Directory

Set Stop Word List

Set Domain Specific Abbreviation List

Set Domain Specific Synonym List

ii. Threshold Management

Set threshold values for matchers

iii. Algorithm Configuration and Parameter Setting

Enable/Disable matchers

Set execution order of matchers

iv. Matcher Rules Management

Enable/Disable specific matcher rule

Set parameters for matcher rule



e. User Management

i. Add user

ii. Delete user

iii. Update user

II. System Requirement (Develop Code Library for):

a. Generating all possible mapping elements by cross joining

classes of both the ontologies

b. Filtering Mapping Elements based on incompatible data

types and constraints.

c. Filtering Mapping Elements based on previously rejected

mappings by user.

d. Reusing previously accepted mappings by user and

eliminate all Mapping Elements involving either entity of

accepted mapping.

e. Jaccard Similarity coefficient

f. Vector Space Model Engine

i. Create feature vector

ii. Calculate VSM similarity score

iii. Select potential mapping elements for further

processing

g. Label Matcher

i. Edit Distance

ii. N-Gram

iii. Prefix

iv. Suffix

v. Soundex

vi. Heuristic Rules

Equality

Vowel less

Right most digit less

Digit less



Non Alpha Numeric less

Same Character Order

h. Linguistic Matchers

i. Domain Specific Synonym

ii. WordNet Synonym

iii. WordNet Gloss

i. Structure Matchers

i. Path Label Matching

ii. Upward Cotopic Distance

iii. Anchor-Prompt Path Propagation

iv. Heuristic Rules

All children to all children

All children to class label

Parent and some children to parent and some

children

j. Language Processing Activities

i. Spell Corrector

ii. Expand Abbreviation

iii. Remove Stop words

iv. Stemming

v. Get Domain Synonym

vi. Get WordNet Synonym

vii. Get WordNet Gloss

7.1.2 System Module Chart

The Figure 33 shows module hierarchy chart for the majorcomponents of the system and major modules of these components.



Figure 33: System Module Chart

7.1.3 Data Structure

The Data Structure used by Algorithm is listed below:

User (User ID, User Name, Password)

AI-ATOM

UserManagement







Ontology

New Ontology

View and Maintain Ontology

Import From File

OntologyMappingProject

New Ontology Mapping Project

View and Maintain OntologyMapping Project

System Settings

AuxiliaryResource Setting


Set Stop Words

Set Abbreviation List

Set Synonym ListThreshold

Management

AlgorithmConfiguration

Matcher RulesManagement

OntologyMappingEngine

Label Matcher

Edit Distance

Label Huerstic RulesVSM

LanguageProcessing

LinguisticMatcher

get Domain Synonym

get WordNet Synonym

get WordNet Gloss

StructureMatcher

Upward Cotopic Distance

Anchor-Prompt

Structure Heuristic RulesUserManagement Add User

Update User

Delete User








Set Stop Words

Set Abbreviation List

Set Synonym List

Edit Distance

N-Gram

Prefix

Suffix

Equality

Label Huerstic Rules

get Domain Synonym

get WordNet Synonym

get WordNet Gloss

Path Label

Upward Cotopic Distance

Anchor-Prompt

Structure Heuristic Rules



Ontology (Ontology ID, Ontology Name, Description, Owner User

ID, Date of Creation)

Class (Class ID, Class Name, Description, Ontology ID)

Class Has Sub Class (CHSC ID, Class ID, Sub Class ID)

Ontology Mapping Project (OMP ID, OMP Name, Description,

From Ontology ID, To Ontology ID, Created By User ID, Start

Date, Due Date, Completed Date)

Ontology Mapping Project User (OMPU ID, OMP ID, User ID)

Domain Specific Label (DSL ID, Label, Meaning, Frequency of

Use)

Domain Specific Abbreviation (DSA ID, Abbreviation, Expanded

Text)

Domain Specific Synonym (DSS ID, Synonym, Synonym Group

ID)

Stop word (Stop word ID, Stop word)

Algorithm Configuration (AC ID, AC Name, Description, For

OMP ID, Precision, Recall, F Measure)

Parameter (Parameter ID, Parameter Name, Description, Default

Value Text, Default Value Number)

Parameter Possible Value (PPV ID, Parameter ID, Possible Value

Text, Possible Value Number)

Algorithm Parameter Value (APV ID, For AC ID, Parameter ID,

User PPV ID)

Mapping Element (ME ID, OMP ID, From Class ID, From Class

Name, To Class ID, To Class Name, Status, Relation,

Confidence, Explanation, Processed By System YN, Processed

By User YN, User ID)

o Status= Candidate/Potential/System

Generated/Accepted/Rejected

Accepted Rejected Mapping Group (ARMG ID, Class Name,

Relation, Group ID, Accepted or Rejected)



The Figure 34 shows the Database Relationship Diagram for the

prototype system.

Figure 34: Database Relationship Diagram

7.1.4 Class Diagram

The Figure 35 and Figure 36 shows sample class diagrams for fewmatchers and language processing activities, and Figure 37 showssample abstract class diagram for the GUI component of the system.




prototype system.


7.1.4 Class Diagram





prototype system.


7.1.4 Class Diagram




Figure 35: Sample Class Diagram of the System – I



Figure 36: Sample Class Diagram of the System - II



Figure 37: Abstract Class Diagram for GUI of the System

7.1.5 User Interface

7.1.5.1. Sample Screens for Menu Design

Opening Screen

The Figure 38 shows opening screen of the system, which allows the

user to navigate around the system modules with the help of

horizontal main menu.

Figure 38: Opening Screen of the System






Opening Screen










Opening Screen







Ontology Management

The Ontology Menu shown in Figure 39 allows the user to perform

tasks specific to ontology management. It allows the user to create

new ontology, to open existing ontology, and to close the currently

opened ontology. It also allows importing ontology from text file.

Figure 39: Ontology Management Screen

Ontology Mapping Project ManagementThe Ontology Mapping Menu shown in Figure 40 allows the user to

perform task specific to ontology mapping project management. It

allows the user to create new ontology mapping project, to open

existing ontology mapping project, and to close the currently opened

ontology mapping project. It also allows assigning users to ontology

mapping project.

Figure 40: Ontology Mapping Project Management Screen



Ontology Management











mapping project.




Ontology Management











mapping project.




Application Setting and Algorithm Configuration Management

The Setting Menu shown in Figure 41 allows the user to set auxiliary

resources such as context dictionary, acronym dictionary, domain

specific synonym list, and stop word list. It also allows configuring

algorithm, setting threshold values for different matchers, and setting

parameters for matcher rules.

Figure 41: System Setting Screen

7.1.5.2. Sample Screen for Forms Design

Use of Context Dictionary to suggest domain specific labels to user

The Figure 42 shows the benefit of the context dictionary. When user

creates new ontology using system, it suggests the domain specific

labels according to its usage frequency for partially entered label by

user. This reduces the problem of word sense disambiguation.

Depending on his intended meaning, he can directly select word from

suggested list. This helps the ontology mapping process greatly.

Importing Ontology from Text File

The Figure 43 shows the option for importing the ontology from the

text file. It is assumed that ontology tree is stored in this text file

using the parenthesized representation for general tree as shown in

Figure 32. When user selects the file and clicks Open button, ontology















































in imported from the text file on this screen. By clicking on Save

button, user can save this ontology in native form of the system.

Figure 42: Using Context Dictionary to suggest Domain Specific Labels

Figure 43: Importing Ontology from Text File















Ontology Mapping Project Management

The Figure 44 shows screen for ontology mapping project

management. It shows both the ontologies considered for mapping

tasks. When user clicks on Generate button, it fills the grid displayed

in the middle of both the ontologies with system generated mappings

as shown in Figure 45. When user clicks on any of this mapping

element, it shows the explanation for this mapping element in text

boxes shown above the Generate button. The user can select mapping

elements using Checkbox against the mapping element in the grid,

and can accept or reject such checked mapping elements. The user

can also add new mapping element by selecting nodes from both the

ontologies and clicking on Add button.

Figure 44: Ontology Mapping Project Management

































System Generated Mapping Elements

Figure 45: System Generated Mapping Elements

System Setting and Configuration

The Figure 46 shows the System Setting screen. It allows the user to

do following type of system settings

Enable or disable specific feature of the algorithm such as

whether to reuse previously accepted mappings or not

Enable or disable specific matcher and sub matcher

Enable or disable the use of threshold value for a specific

matcher

To specify threshold value for a specific matcher

To set execution order of the matchers












matcher














matcher






7.2 Evaluation of proposed system

7.2.1 Evaluation Measure

The effectiveness of the system in processing the mapping elements ismeasured by looking at Precision and Recall [67] [24]. The ontologymapping systems are evaluated with respect to the notion ofcorrectness perception – a judgment by a human that a mappingelement found by ontology mapping algorithm is correct or not.

A system’s ability to retrieve correct mapping elements is assessedwith a Recall measure that is defined as below:

Recall = | Relevant and Retrieved | / | Relevant |

A system can achieve 100% recall by simply returning all the possiblemapping elements between two ontologies.

A system’s accuracy is based on how many of the mapping elementsgenerated by system are actually correct as per user’s decision, whichcan be assessed by a Precision metric and is defined below.

Precision = | Relevant and Retrieved | / | Retrieved |

























Ideally both should be 100%. But, most of the system scarifies one forthe other. Hence, to measure the optimum balance between these twomeasures, F-Measure is used which is defined as below:

F-Measure = 2 * (Precision * Recall) / (Precision + Recall)

The Precision and Recall can be understood from the Figure 47, Where:

A = False Positives B = True Positives C = False Negatives D = True Negatives.

Figure 47: Precision, Recall, and F-Measure

Using these notions; Precision, Recall, and F-Measure can be definedas following.


= B / (B+C)


= B / (A+B)

F-Measure = 2 * (Precision * Recall) / (Precision + Recall)

= 2 * B/ ((A+B) + (B+C))

System

Matches

User

Matches

D

A B C



7.2.2 Data Sets used for evaluation

The system is evaluated with two data sets, which are shown in Figure

48 and Figure 49. The first data set represents sample ontologies

represented by two different academic institutes, whereas second data

set represents snapshot of database schema from two academic

institutes.

Figure 48: Data Set-1: Sample Ontologies from two different Academic

Institutes

Both the data sets are selected from the academic domain as one of

the objectives of study is to analyze and to present the significance of

domain knowledge in the automated ontology mapping process.



Figure 49: Data Set-2: Sample Database Schema from two different

Academic Institutes

In experimental setup, these data sets are given to few users having

varying knowledge regarding the ontology mapping. The information

about manual mappings identified by different users for above two

data sets are aggregated and summarized in the Table 6.

Table 6: User Performance for example Data sets

DataSet

UserPerformance

FP TP FN P R FM

DS-1 Actual 0 23 0 100 100 100

Maximum 12 23 6 88.46 100 93.88

Minimum 3 17 0 58.62 73.91 65.38

Average 6 19.33 3.67 77.60 84.06 80.36

DS-2 Actual 0 13 0 100 100 100

Maximum 5 12 4 100 92.31 88.89

Minimum 0 9 1 64.29 69.23 72

Average 2.30 10 2.30 83.33 81.12 80.90FP=False Positive, TP=True Positive, FN=False Negative, P=Precision, R=Recall, FM=F-Measure



7.2.3 Analysis of Result

The basic objective of work is to improve automation of ontology

mapping process which yields quality mappings. Thus, system is

tested with different algorithm configuration and parameter settings

for its effectiveness only. The efficiency of the system with respect to

memory and/or CPU is not given due importance for this prototype

system. The following section describes experimental study performed

for the prototype system.

7.2.3.1 Overall Performance of the System

The Figure 50 and Figure 51 represent overall performance of the

system with actually expected number of mappings by the creator of

the ontology. It is observed that system offers reasonably good

performance without any fine tuning of the algorithm. It provides

100% precision for one of the data set. It proves the system’s potential

ability to be used as a completely automated system.

Figure 50: Overall System Performance for Data set - 1

Precision Recall F-Measure

Actual 100.00 100.00 100.00

System 100.00 43.48 60.61

0.00

20.00

40.00

60.00

80.00

100.00

120.00

Perf

orm

ance

in %

Graph-1.1: Overall SystemPerformance

Actual

System



Figure 51: Overall System Performance for Data set - 2

7.2.3.2 System Performance against User

The comparative study between system and user for the selected data

sets are depicted in the Figure 52 and Figure 53. The study shows

that system can be rated as better when it is compared with user

driven manual activity for the establishment of ontology mappings for

a given data sets. That is, system gives F-Measure value of 65.54%

against actual result. But, when system’s average performance (not

best) is compared to user’s average performance; it gives F-Measure

value of 71.49%, which may be considered as good result.

7.2.3.3 System Performance for different Threshold Values

The system is run for five threshold values 0.6, 0.7, 0.8, 0.9, and 1.0.

The performance of system for these threshold values for data set-1

and data set-2 is shown in the Figure 54 and Figure 55 respectively. It

is observed that precision of the system increases when threshold

value is set to high value for both the data sets.

Precision Recall F-Measure

Actual 100.00 100.00 100.00

System 55.56 76.92 64.52

0.0020.0040.0060.0080.00

100.00120.00

Perf

orm

ance

in %

Measures

Graph-2.1: Overall SystemPerformance

Actual

System



Figure 52: System Performance against User for Data set - 1

Figure 53: System Performance against User for Data set - 2

The Recall and F-Measure slightly decreases for data set-1, and

decreases moderate for data set-2. It also reveals that as the demand

for precision increases, the recall of the system decreases, and vice

versa. Another interesting effect of setting high threshold is that it

Precision RecallF-

Measure

User Average 77.60 84.06 80.36

System Average 90.00 42.61 57.45

System against User 115.98 50.69 71.49

0.0020.0040.0060.0080.00

100.00120.00140.00

Perf

orm

ance

in %

Graph-1.2: System Performance AgainstUser

User Average

System Average

System against User

Precision RecallF-

Measure

User Average 83.33 81.12 80.90

System Average 65.02 52.31 56.14

System against User 78.02 64.48 69.39

0.0010.0020.0030.0040.0050.0060.0070.0080.0090.00

Perf

orm

ance

in %

Measures

Graph-2.2: System Performance AgainstUser

User Average

System Average

System against User



yields 100% precision which helps for total automation of the ontology

mapping process.

Figure 54: System Performance for different Threshold Values (DS - 1)

Figure 55: System Performance for different Threshold Values (DS - 2)

0.60 0.70 0.80 0.90 1.00

Threshold

Precision 83.33 66.67 100.00 100.00 100.00

Recall 43.48 43.48 43.48 43.48 39.13

F-Measure 57.14 52.63 60.61 60.61 56.25

0.0020.0040.0060.0080.00

100.00120.00

Perf

orm

ance

in %

Graph-1.3: The effect of differentThreshold Values on Performance

Precision

Recall

F-Measure

0.60 0.70 0.80 0.90 1.00

Threshold

Precision 55.56 66.67 60.00 71.43 71.43

Recall 76.92 61.54 46.15 38.46 38.46

F-Measure 64.52 64.00 52.17 50.00 50.00

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

Perf

orm

ance

in %

Graph-2.3: The effect of differentThreshold Values on Performance

Precision

Recall

F-Measure



7.2.3.4 The effect of Language Processing Activities on

System Performance

The prototype system developed implements several language

processing activities. The effect of using/performing selected

language-processing activities, viz., removal of stop words, stemming

the words, and spelling correction; on system performance for data

set-1 and data set-2 is shown in Figure 56 and Figure 57 respectively.

It is observed that all language processing activities improves the

precision of the system. At the same time there is no adverse effect on

the F-Measure, except that of stemming for the data set-1.

Figure 56: The effect of Language Processing Activities on System

Performance for Data set - 1

With AllLanguageProcessing

WithoutStop words

WithoutStemming

WithoutSpell

Corrector

Precision 100.00 100.00 100.00 100.00

Recall 43.48 39.13 52.17 39.13

F-Measure 60.61 56.25 68.57 56.25

0.00

20.00

40.00

60.00

80.00

100.00

120.00

Perf

orm

ance

in %

Graph-1.4: The effect of LanguageProcessing Activities on Performance

Precision

Recall

F-Measure



Figure 57: The effect of Language Processing Activities on System

Performance for Data set - 2

7.2.3.5 The effect of Auxiliary Resources on System

Performance

The system used the concept of semantic similarity with the help of

domain specific knowledge and WordNet knowledge. The fig- and fig-

depicts the effect of excluding the specific type of external resources

for the data set-1 and data set-2 respectively. It shows that F-measure

is highest when all auxiliary resources are used. For data set-2, the F-

Measure reduces to almost 50% when none of the auxiliary resources

is used. This signifies the importance of Linguistic Matcher used by

the system.

With AllLanguageProcessing

WithoutStop words

WithoutStemming

WithoutSpell

Corrector

Precision 75.00 55.56 62.5 62.5

Recall 69.23 38.46 76.92307692 38.46153846

F-Measure 72.00 45.45 68.96551724 47.61904762

0.0010.0020.0030.0040.0050.0060.0070.0080.0090.00

Perf

orm

ance

in %

Graph-2.4: The effect of LanguageProcessing Activities on Performance

Precision

Recall

F-Measure



Figure 58: The effect of Auxiliary Resources on System Performance for

Data set - 1

Figure 59: The effect of Auxiliary Resources on System Performance for

Data set - 2

WithAuxiliary

Resources

WithoutDomain

Knowledge

WithoutWordNet

WithoutAuxiliary

Resources

Precision 100.00 100.00 100.00 100.00

Recall 43.48 39.13 34.78 30.43

F-Measure 60.61 56.25 51.61 46.67

0.00

20.00

40.00

60.00

80.00

100.00

120.00Pe

rfor

man

ce in

%

Graph-1.5: The effect of AuxiliaryResources on Performance

Precision

Recall

F-Measure

WithAuxiliary

Resources

WithoutDomain

Knowledge

WithoutWordNet

WithoutAuxiliary

Resources

Precision 75.00 100.00 100.00 100.00

Recall 69.23 23.08 30.77 23.08

F-Measure 72.00 37.50 47.06 37.50

0.00

20.00

40.00

60.00

80.00

100.00

120.00

Perf

orm

ance

in %

Graph-2.5: The effect of AuxiliaryResources on Performance

Precision

Recall

F-Measure



The values shown in all the graphs shown here is obtained by keeping

all other parameter constant just to see the effect of specific

parameter. Moreover, a particular parameter setting is applied in

totality here. For example, if stemming is disabled, it is disabled for all

matchers and sub matchers. During trial run of the system it is

observed that this does not give the optimum performance. For

example, the N-Gram similarity improves the result if stemming is

used on class label before they are passed to it. Similarly, the effect of

different parameters is measured with constant threshold values, 0.7

for data set-1 and 0.8 for data set-2. Moreover, this threshold value is

used uniformly for all matcher and sub matchers. Using different

threshold values for different matchers and sub matcher may reveal

other interesting behavior of the system.

7.3 Summary

This chapter presented comparative evaluation of the proposed

integrated approach against user driven manual mapping activity. The

effectiveness of the proposed system is analyzed with two data sets

from an academic domain. The proposed algorithm is highly

configurable with many possibilities of combining different parameter

values. Some selected combination is used to test the performance of

the system. In best case, algorithm yields extremely encouraging

result. Though, the algorithm is yet to be tested for their optimum

configuration setting which is a gigantic task. The engineering

contribution is made by developing a prototype system, which can be

further used or extended to add scientific contribution.

Documents

CHAPTER – 7shodhganga.inflibnet.ac.in/bitstream/10603/36602/17/17_chapter7.pdf · Chapter-7: Prototype Implementation and Evaluation Page | 155 An Integrated Approach to Improve