The Concept of Connecting Ontology and itsExploitation for Knowledge and InformationPresentation for People with Special Needs
Muhammad Shuaib Karim1,2
1Institute of Software Technology and Interactive SystemsVienna University of Technology (http://www.ifs.tuwien.ac.at/)
2Computer Science DepartmentQuaid-i-Azam University, Islamabad (http://www.qau.edu.pk/)
October 09 / 2007October 09 / 2007
Technisches Universität Wien
2
Motivation
Providing a generic accessibility solution for SemanticLIFE using the Semantic Web TechnologyProviding a generic accessibility solution for SemanticLIFE using the Semantic Web Technology
3
Semantic Web – Promises
● Ability to integrate heterogeneous data sources
● Ability to formally describe the information Formal data description makes it understandable,
sharable and thus processable by software agents Automatic reasoning becomes possible
● Abundance of OS tools from modeling, storage, annotation, reasoning, & query to user interfaces
4
Semantic Web - Architecture
5
Semantic Web – Resource Description (1/5)
● Every Thing can be thought of as an Entity
● The Entity is described as a Resource
● The Resource may range from literal to concept map, and has formal Description
● The Resource is identified by a URI
● The Description has the consensus within CoP
6
Semantic Web – Resource Description (2/5)
● The Description is asserted in terms of (Subject,
Predicate, Object) triples where each of them is a Resource
● For example, <Person workingOnTask Task> <:John workingOnTask :Ontology_Template>
7
Semantic Web – Resource Description (3/5)
In principle, every “piece of information”● can be conceptualized in terms of inter-relation of entities
– Ontological (Philosophical) Level, AND● has schema (ontology) described in terms of chains of
triples as well as its instances (individuals) – Developer or Content Author Level, AND
● has implementation for those chains of triples based upon strong theoretical formalism of Description Logic – Implementation Level Allows Reasoners to make inference Deduce relationships such as containment, symmetrical,
transitive, inverse relations
8
Semantic Web – Resource Description (4/5)
● So, Semantic Web provides a platform from Conceptual Modeling to Implementation with added inference capabilities
● Not necessary to implement all of it for getting started. “Even a little semantics go a long way – James Hendler”
9
Semantic Web – Resource Description Example (5/5)
John is working on tasks “Ontology_Templates´´ and “Message_Bus´´. The task “Ontology_Templates´´ is for project “DynamOnt´´ and the task “Message_Bus´´ is for project “SemanticLIFE´´
Person
John
Shuaib
Task
Ontology_Templates
Message_Bus
Project
DynamOnt
SemanticLIFE
workingOnTask
workingOnTask
workingOnTask
partOf
partOf
workingOnProject
10
SemanticLIFE - Architecture
Message Bus Plug-in
Data Feed Processing
Repository Plug-in
PersonalRepository
Ontologies
Pipeline Plug-in
Stylesheets
User ProfilePlug-in
AnnotationPlug-in
Web ServicePlug-in
AnalysisPlug-in
Visualization Plug-in
Accessibility Plug-in
Pipelines
(Ahmed et al., 2004). SemanticLIFE - A Framework for Managing Information of A Human Lifetime. In Austrian Computer Society Book Series: iiWAS'04 Proc., (vol.183), pages 687-696.
11
Concept of SemanticLIFE UI
SemanticLIFE - x
Scratch Bags
Misc.
Control Panel
TimeLine
AbstractControls Query
Items Panel Text TableXML
AssociationPanel
RDF
4 associated mails, 5 web pages, 1 file, 12 process state data
Status Bar
Annotation
User Profile Settings HelpData Feeds
Query Trash
Information Items
User Abstract Information
System Information
PIM
EmailWeb PageProcess StateFile SystemCalendarNoteContactFile
Email Web Page FileAll Contact Note
EmailFrom ToSubject
Re: Reification withindividuals
Farewell Party Shehzad ATScolars@yahoogr...
FilePath SizeName
Semantic Desktop Proposal
C:\prj 20 Kbs
Text GraphRDFNativeReification withindividuals
A
AnnotationA
Trash
ProjectResourceTaskDeliverableLocationCost
Slife Core Ontology
Imapirments OntologyUI Ontology
Project Ontology
Khalid,modeling of n-ary relationships and reified relationships is very well described in a working draft of the Semantic Web Best Practices working group:http://www.w3.org/TR/swbp-n-aryRelations/The paper describes several patterns and examples that can help you in modeling.TimLatif, Khalid wrote:> >Hello,>I have made ontology of vehicle parts with many properties. I need to
12
Accessibility – Key Issues
● Not only an issue for people with popular disabilities and specific age group, rather it is Acc4All
● Not only related with user interface issues, rather applicable to whole system
13
Accessibility – Web Accessibility Components
User
Content
Evaluation Tools(WCAG)
Authoring Tools(ATAG, WCAG)
User Agents(UAAG)
Assistive Technology(UAAG)
Developer(WCAG)
14
Research Questions
● Can Semantic Web Technology be used for providing a Generic Accessibility solution?
● How far is it true that the investment on UI alone could provide maximum possible Accessibility?
● Is the sought-after approach exploitable towards diversity in general, and integration of Information Management Systems in particular?
15
Sample Components Requiring Connections
User Profile ↔ Assistive Technology
Assistive Technology ↔ User Agents
User Agents ↔ Contents
….
….
User Interfaces ↔ User Profile
Info. Semantics of Domain1 ↔ Info. Semantics of Domain2
Encoded in software or in data → based upon some rules → ?
Connection of heterogeneous domains => Rules
User
Content
Evaluation Tools(WCAG)
Authoring Tools(ATAG, WCAG)
User Agents(UAAG)
Assistive Technology(UAAG)
Developer(WCAG)
16
Significance of Connections (Ex. Food & Wine)
Suggests different foods and wine combinations
→ Domain expert´s knowledge encoded as OWL DL, such as;
• taste• appetite or stimulation to eat• digestion• availability
17
Ex. Food & Wine
Another Approach (“Connecting Ontology´´) Ontology of Food – Taxonomy with concepts such as;
• the calories with respect to the quantity• the tendency to get digested on ist own (Quick, Medium
duration, Long duration)• …
Ontology of Wine - Taxonomy with concepts such as;• the appetizing effects (Nil, Low, Medium, High)• the digestive effects (Nil, Low, Medium, High)• …
Above knowledge normally accessible from the available literature
18
Food & Wine (new CQ)
New CQ possible to fulfill: Food requiring no drinks Food requiring drinks of certain quality & brand Food & wine combination requiring least digestion time
19
Connecting Ontology - Characteristic Features of OC between O1 and O2
● O1 , O2 not related with same domains of discourse
● O1 , O2 developed using their own CODeP
● Similarity b / w CODeP indicate inter-connection
● Creation of new knowledge
20
Connecting Ontology - Benefits
● Useful for top-down evolution of ontologies / applications
● Incompatibilities between the two ontologies are solved at the ontological level
● Helpful in code automation● Possibility to determine the Cause - Effect
relationship between the two ontologies● Reliance on domain experts reduced due to
encoding of domain knowledge
The key is “How to capture and represent the domain knowledge ??´´
21
Connecting Ontology - Domain Knowledge
Available as;
Structured documents
Unstructured documents
Tacit knowledge with domain experts
22
Connecting Ontology – Capturing Domain Knowledge
● Text processing of knowledge about domains of CO and the two participating ontologies
● Refinement of the above by aligning participating ontologies with global standard ontologies
● Exploiting valid queries on the two ontologies
● Exploiting queries result set and data mining
● Using Semantic Web Rules
23
Connecting Ontology - Workflow
Load Ontology A
Load Ontology B
1. createModel
Ontology Model
2. addModel
Load Rules
ConnectingOntology
4. storeOntology
3. applyRules
24
Accessibility Plug-in (A scalable framework using Semantic Web Technology)
Info-VizBridge Service
Accessibility Framework
AccessibilityService
ConnectingOntology
DeviceProfiles
DomainOntology
User Profile(Impairments)
map
s maps
maps
ConnectingOntology
map
s
maps
Uses
Use
s
RepresentationOntology
Karim, S., Latif, K., and Tjoa, A. M. (2007). Providing Universal Accessibility using Connecting Ontologies: A Holistic Approach. Universal Access to Applications and Services, In Proc. of HCII’07., LNCS 4556 (vol. 7).
25
CODeP for Accessibility - Generic Pattern with Spatio-Temporal Dimensions
InterfaceElement
GenericAccessibilityPattern
Representation Capability
InformationObject
composedOf
User
CausalObject
describes has
exposedBy
ImpairmentProfile
belongsTo
dependsOn
Space-Region
affects
spatialLocation
Time-Interval
temporalLocation
26
CODeP for Accessibility – Simplified Generic Pattern
InterfaceElement
GenericAccessibilityPattern (Simplified)
Representation Capability
InformationObject
composedOf
effects
User
CausalObject
describes has
exposedBy
ImpairmentProfile
belongsTo
dependsOn
27
CODeP for Accessibility - Memory Recall Pattern
28
CODeP for Accessibility - Perception Effect Pattern
29
CODeP for Accessibility – Mobility Enhancement Pattern
30
Persistence of Patterns
Formal description of semantics for each component
=> Ontology Oi for each component
Formal description of consequences and effects of potentially interacting component on each other
=> Connecting Ontology for O1 and O2
31
Test case
Connecting Ontology for
User’s Impairments & UI Characteristics
Accessibility Guidelines
User Interface C
haracteristics Use
r Im
pairm
ents
Pro
file
Accessible User Interface
32
Role of User Impairments
Assistance to be provided = User's Needs - User's Capabilities ...(1)
Assistance to be provided = User's Needs + User's Impairments needs - User's Capabilities of (1)
Quality of Life Technology (QoLT)Ref: Kanade, T. (2007). Digital Human Modeling and Quality of Life Technology. In Keynote address in 12th International Conference on Human-Computer Interaction, Beijing, China.
33
Motivation - Ontology of Visualization Techniques
informationEntity Tree
VisualizationTechnique
visualizationAbstraction
suitableFor
Schema
hasSchema
Data
hasData
breadthFirst traversal
visualizationTransformation analyticalAbstraction
breadthFirst traversal
Layout
visualMappingTransformation
Ref: Chi, E. (2000). A Taxonomy of Visualization Techniques using the Data State Reference Model. In INFOVIS, pages 69-75.
34
Overview of Ontology Interaction for On-Demand Visualization
Information Semantics
Device Needs
Visualization Techniques
User Needs / Impairments
Connecting Ontologies
*Application Domain Ontologies
**
*
35
Human Disease Ontology
HumanDisease
Type
Treatment
Cause
Environmental
Symptom
Genetic
Physiotherapy
Chemotherapy
Surgery
DrugTherapyPsychotherapy
Ref: Hadzic, M. and Chang, E. (2005). Ontology-Based Support for Human Disease Study. In Proc. of HICSS'05, IEEE Computer Society.
36
Extension of Human Disease Ontology
HumanDisease
Type
Treatment
Cause
Environmental
Symptom
Genetic
Physiotherapy
Chemotherapy
Surgery
DrugTherapyPsychotherapy
InterfaceAdaptation
Impairment
Ref: Hadzic, M. and Chang, E. (2005). Ontology-Based Support for Human Disease Study. In Proc. of HICSS'05, IEEE Computer Society.
37
Impairment-User interface Connecting Ontology - Sample user scenarios
● Avoiding the confusing colors on an interface for a user with particular type of color blindness
● Font adjustments according to user‘s visual acuity
● Information presentation on the better part of the screen for a user suffering from Hemianopsia (absence of vision in half of visual field)
38
Impairment-User Interface Ontology
Karim, S. and Tjoa, A. M. (2006). Towards the Use of Ontologies for Improving User Interaction for People with Special Needs. In Proc. of ICCHP’06, LNCS (vol. 4061), pages 77-84.
39
Exploring Relationship between Motor Control Impairments and UI Components
Impairment Related With UI ComponentNormalMotorControl suggests ComboBox
NormalMotorControl suggests RadioButton
NormalMotorControl suggests CheckBox
NormalMotorControl suggests ScrollBar
NormalMotorControl suggests Spinner
NormalMotorControl suggests ToggleButton
VeryLowMotorControl suggests AudioFeedback
VeryLowMotorControl suggests ToggleButton
VeryLowMotorControl prohibits Spinner
VeryLowMotorControl prohibits ScrollBar
40
Representing Relationship between Motor Control Impairments and UI Components
http://www.ifs.tuwien.ac.at/ontologies/imp/Impairment
isA
.../imp/SeverityMeasure
hasSeverityMeasure
OneOf:#Normal, #High, #VeryHigh
http://www.ifs.tuwien.ac.at/ontologies/ui/UiComponent
isA
.../ui/ConvenienceMeasure
OneOf:#Normal, #High, #VeryHigh
.../ui/ComboBox#
.../ui/Spinner#
.../ui/ToggleButton#
.../ui/ScrollBar# .../ui/RadioButton# .../ui/CheckBox#
isA
isA
isA isA isA
.../imp/MotorControlImapirment#
hasMotoricConvenience
41
Representing Relationship in DL
● MotorControlImpairment with SeverityMeasure “VeryHigh” suggests UiComponent with MotoricConvenienceMeasure “VeryHigh”
VeryHighasureseverityMeyMeasurehasSeveritorControlVeryLowMotntolImpairmeMotorControrControlVeryLowMot
.
● MotorControlImpairment with SeverityMeasure “Normal” suggests UiComponent with MotoricConvenienceMeasure “Normal”
NormalasureseverityMeyMeasurehasSeveritrControlNormalMotontolImpairmeMotorContrrControlNormalMoto
.
and also;
VeryHigheMeasureconvenienceConvenienchasMotoricnentnientCompotoricConveVeryHighMo
tUiComponennentnientCompotoricConveVeryHighMo
.
NormaleMeasureconvenienceConvenienchasMotoricntentComponericConveniNormalMoto
tUiComponenntentComponericConveniNormalMoto
.
42
Impairment ontology - competency questions
For a given impairment name;● What is / are the related body parts ?
● What is the impaired side (right, left,...) ?
● What is its severity (on a predefined scale) ?
● What are the perception cues which are affected, and up to what degree (on a predefined scale) ?
● What is the effect of one impairment on another w.r.t. affected perception ?
43
Impairment Ontology
44
Some derived concepts (1 / 2)
● LeftSidedImpairment
LeftnhasPositioyPartrelatedBodmpairmentLeftSidedIImpairmentmpairmentLeftSidedI
.
● RightSidedImpairment
RightnhasPositioyPartrelatedBodImpairmentRightSidedImpairmentImpairmentRightSided
.
● BothSidedImpairment
RightnhasPositioLeftnhasPositioyPartrelatedBodmpairmentBothSidedI
ImpairmentmpairmentBothSidedI
.
.
45
Derived concepts (2 / 2)
● AnySidedImpairment
RightnhasPositioLeftnhasPositioyPartrelatedBodpairmentAnySidedIm
ImpairmentpairmentAnySidedIm
.
.
● AnySidedImpairment
ImpairmentRightSided ImpairmentLeftSidedpairmentAnySidedIm
46
User interface Ontology - competency questions
● Find the part-whole relationship of UI components
● Find the attributes of a component and their values (according to predefined usability scale for a normal user in normal conditions)
● For a given attribute, find the related UI components
47
User Interface Ontology
48
Some derived concepts (1 / 2)
● GoodUsabilityComponent
GoodtyhasUsabilitUiComponenntityComponeGoodUsabiltUiComponenntityComponeGoodUsabil
.
● FairUsabilityComponent
FairtyhasUsabilitUiComponenntityComponeFairUsabiltUiComponenntityComponeFairUsabil
.
● FairUserControlComponent
FairluserControtUiComponennentntrolCompoFairUserCotUiComponennentntrolCompoFairUserCo
.
49
Rules to Connect Impairments and UI
Low perception implies suggesting high usability components (VisualAcuityLow UI LegibilityGood)
Karim, S. and Tjoa, A. M. (2007). Connecting User Interfaces and User Impairments for Semantically Optimized Information Flow in Hospital Information Systems. Journal of Universal Computer Science: In Proc. of I-MEDIA'07 and I-SEMANTICS'07, pages 372-379.
(?x rdf:type imp:VisualAcuity) (?x imp:perceptionMeasure imp:Low)(?y rdf:type ui:UiComponent) (?y ui:hasLegibility ui:Good) (?x eg:suggests ?y).
50
Rules to Connect Impairments and UI
High perception implies suggesting fair usability components (VisualAcuityHigh UI LegibilityFair)
(?x rdf:type imp:VisualAcuity) (?x imp:perceptionMeasure imp:High)(?y rdf:type ui:UiComponent) (?y ui:hasLegibility ui:Fair) (?x eg:suggests ?y).
51
Rules to Connect Impairments and UI
High rheumatism implies suggesting easily operatable components (RheumatismHigh UI UserControlGood)
(?x rdf:type imp:Rheumatism) (?x imp:impairmentMeasure imp:High)(?y rdf:type ui:UiComponent) (?y ui:userControl ui:Good) (?x eg:suggests ?y).
CO Workflow
52
Generated RDF triples
<rdf:RDF xmlns:co="http://www.ifs.tuwien.ac.at/ontologies/co#" ...> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#VisualAcuity_High"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_09"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_26"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextStyle_Italic"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextStyle_BoldItalic"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_08"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_24"/> </rdf:Description> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#Rheumatism_High"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#LabelledButton"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#ComboBox"/> </rdf:Description> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#VisualAcuity_Low"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_18"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_10"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextFont_TimesNewRoman"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextStyle_Bold"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_22"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_11"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#TextSize_20"/> </rdf:Description><rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#ColorBlindness_RG"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Grey"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Orange"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Cyan"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Blue"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Yellow"/> </rdf:Description> <rdf:Description rdf:about="http://www.ifs.tuwien.ac.at/ontologies/imp#ColorBlindness_YB"> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Grey"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Orange"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Cyan"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Green"/> <co:suggests rdf:resource="http://www.ifs.tuwien.ac.at/ontologies/ui#Color_Red"/> </rdf:Description></rdf:RDF>
53
54
55
56
Visualization and Accessibility Plug-ins
Triple store
Ontology
Storage
Data Preparation for UI
UI Mapping Transformation
Data Transformation
Grouping Aggregation Rendering
Transformation for Image
Formatted Data
User InterfaceQuery
XMLTreeRDF
Graph
Filtering
Visualization Plug-in
Accessibility Plug-in
Info-VizBridge Service
AccessibilityService
57
Grouping and Aggregation of Items Arranged on a Timeline
58
Consequences of Impairments-user Interface Connecting Ontology
• Helpful in automatically adapting UI for the user• Helpful in deducing the best match of UI characteristics
for a user with multiple impairments• Possibility to use the ontology for diversity• Historical data for studying the cause-effect relationship
b/w the impairments and the computer interfaces• Useful for rehabilitation purposes• Possibility to extract impairment related semantics from
user´s information stored in SemanticLIFE repository, and modify the impairments ontology accordingly
59
Concluding Remarks – Contributions (1/5)
● Ontology for Impairments and Usability introduced● A number of Accessibility CODeP introduced● The concept of Connecting Ontology introduced● Demonstration of Semantic Web Rule Layer for
developing higher level ontologies ● The Connecting Ontology concept using the ontological
rule-based approach paves the way for a generic solution● Introduced the integration of heterogeneous domains by
persisting the tacit knowledge using Connecting Ontology● Application of the approach in the motivating scenarios
60
Concluding Remarks - Future directions (2/5)
● User testing● Semantic Web Service for the Accessibility and
the Info-Viz Bridge modules● Instantiating the Impairments and UI ontologies● Integrating capability measuring tools (MMSE)● Realization of the remaining CODeP for
Accessibility● Integration for ontology of visualization techniques● Elevation / Lifting of Connecting Ontology
61
Concluding Remarks – Goals Evaluation (3/5)
● Can Semantic Web Technology be used for providing a Generic Accessibility solution?Generic foundation, not the complete generic solutionUI adaptation possible Visualization toolkits still not ready
62
Concluding Remarks – Goals Evaluation (4/5)
● How far is it true that the investment on UI alone could provide maximum possible Accessibility?Annotation of resources, ontologies of life events, tasks
UI & Applications
Accessib
ility
63
Concluding Remarks – Goals Evaluation (5/5)
● Is the sought-after approach exploitable towards diversity1 in general, and integration of Information Management Systems2 in particular? 1Modeling of user specific attributes (impairments) 2Domain Knowledge, Business rules 2Motivating scenarios
64
Thanks!
http://www.ifs.tuwien.ac.at/~skarimSemanticLIFE Project http://storm.ifs.tuwien.ac.at/
ASEA-UNINET(Asean European University Network) http://www.uibk.ac.at/asea-uninet/
HEC (Higher Education Commission of Pakistan)http://www.hec.gov.pk/
Quaid-i-Azam University, Islamabadhttp://www.qau.edu.pk/
Technisches Universität Wien