Vocabularies for Descriptionof Accessibility Issues in MMUI

Željko Obrenović, Raphaël Troncy, Lynda HardmanSemantic Media Interfaces, CWI, Amsterdam

zeljko.obrenovic@cwi.nl http://www.cwi.nl/~obrenovi/

Introduction

• What (output) modalities aremost suitable in which situation?

• How should different (output) modalitiesbe combined?

• Using rich accessibility descriptionof MM UIs to answer these two questions

Motivation

• Expressing human functionality andanatomical structures required by modalities– Capturing existing knowledge– Reasoning over modalities

• Limited scope, small non-standard vocabularies – Card, Mackinlay, Robertson:

• Morphological analysis of input device;

– Modality theory, Niels Ole Bernsen:• Based on taxonomy of output modalities;

• Linguistic, analog, arbitrary, static, media…

Describing Accessibility Issuesin Multimodal User Interfaces

• MM user interfaces - systems that communicate a message, an effect,– Stimulating a particular human

functionality or anatomical structures

• Interaction constraints– Influence of various factors on human

anatomical structures and functionalities.

Describing Accessibility Issuesin Multimodal User Interfaces

Modality

Modality

Modality

User constraints

Social constraints

Environment constraints

Device constraints

Accessibility issues(interaction constraints)

Mu

ltim

od

al is

su

es

Computer Human

Sensing

Perception

Cognition

Motor skills

Linguistic skills

Interaction Modalities

Interaction Constraints

Interaction Context

described in terms of

ReasoningFramework

define

query result

Applications

described in terms ofVocabularies

Vocabularies

WHO Resources

Bioinformatics

Additional concepts

ReasoningFramework

query result

Applications

Interaction Modalities

Interaction Constraints

Interaction Context

described in terms of

define

described in terms of

Vocabularies: ICF1/2

• WHO International Classification ofFunctioning, Disability and Health (ICF):– 2001, 9 years revision, widely used in health

community

• Describing “the person in his/her world“– Applicable to all people, whatever their health

condition– Carefully designed to be relevant across cultures

as well as age groups and genders– Uses neutral terms (function vs. disease)– Allows for an assessment of the degree of disability

Vocabularies: ICF2/2

• Around 1500 concepts:– Body functions and structure– Activities (tasks and actions by individual)

and participation (involvement in a life situation)– Environmental factors

• Problems with formalization– Not defined by knowledge engineering experts

• Formalized ICF Checklist: 180 core concepts

• Possibility to reuse millions of profilesand statistical data expressed by ICF!

Vocabularies: Anatomy

• Foundational Model of Anatomy (FMA)– Detailed description of human anatomy– Open source and available for general use– Used in bioinformatic community– ~ 100 000 concepts– Available in OWL format– Problems: size

Vocabularies: Interaction Effects• Some examples:

– Gestalt visual grouping• By similarity, motion, texture, symmetry, proximity,

parallelism, closure, good continuation

– Gestalt visual highlighting• By color, polarity, brightness, orientation, size, motion,

flicker, depth, shape

– 3D cues• Visual: stereo vision, motion parallax, linear

perspective, rel. size, shadow, familiar size, interposition, horizon

• Audio: inter-aural time/intensity difference, HRTF, head movement, echo, attenuation of high frequencies

Vocabularies

WHO Resources

Bioinformatics

Additional concepts

ReasoningFramework

query result

Applicationsdefine

described in terms of

Interaction Context

described in terms of

Interaction Modalities

Interaction Constraints

Example Modality: Speech

Example Constraint: Noise

Interaction Constraints

Interaction Context

•Device profile•User profile•Environment profile

Userinterface

described in terms of

described in terms of

Vocabularies

WHO Resources

Bioinformatics

Additional concepts

ReasoningFramework

query result

Interaction Modalities

define

described in terms of

Applications

Reasoning: Using Descriptions

• Possible with rich and explicit descriptionsof (implicit) accessibility requirements

• What modalities are suitable in which situation?– Combining descriptions of constraints

and modalities (speech in noisy environment?)

• How should different modalities be combined? – Combining modality descriptions to identify conflicting

requirements (speech & short term memory);

Implementation

• In early stage• Using Semantic Web technologies:

– OWL, RDF standards– Tools and database to support reasoning

• Sesame, Jena…;

– Reusing some of our and existing toolsfor data visualization and exploration• Facet browsing

Conclusions

• Expressing human functionality andanatomical structures required by modalities

• Relations with other projects:– W3C Multimodal Interaction Framework - content– W3C Accessibility Initiative (WAI) - guidelines

• Future work:– Full implementation of reasoning framework– Solving problems with vocabularies:

• No relations among concepts, overlapping – New vocabularies are coming:

• The Human Brain Project (with FMA)• DARPA Digital Human – unifying medical ontologies