A Transformational Approach for Multimodal Web User Interfaces based on UsiXMLAdrian Stanciulescu, Quentin Limbourg, Jean Vanderdonckt, Benjamin Michotte and Francisco Montero

Université catholique de Louvain, School of Management (IAG){stanciulescu, limbourg, vanderdonckt, michotte, montero}@isys.ucl.ac.be

Objectives:Developing multimodal web UI based on UsiXMLApproach based on model-to-model graph transformationsProgresivelly moves from Task&Domain to FUIConsists on four steps divided into sub-steps

Hypotheses:Four modalities that involve four devices: keyboard, mouse,microphone, loud speakers No fusion/fission needed: independent interpretation/rendering for each modality is performedModality type is exclusive: sequential and independent interactionSupported CARE properties: assignement and equivalence

Step 1: The Task & Domain ModelsGraphical description of the models and the mappings between them IdealXML tool: automatic generation of UsiXML specification

Transformation engine (TransformiXML)

Transformation engine (TransformiXML)

Concrete userInterface #1(graphical)

Concrete userInterface editor

(GrafiXML)

XHTML code

XHTML browser

XHTML Generator

Abstract user interface #1

Domain modelTask model

…

…

Abstract user interface #2 Abstract user interface #n

Concrete userInterface #1(multimodal)

X+V code

IBM X+V browser

XSL Transformation

…

Step 1

Step 2

Step 3

Step 4

General development scenario of UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts:interactive tasks carried out by the end user & domain objects

AUI: UI definition independent of any modality of interaction

CUI: concretizes an AUI into CIOs (widget sets found in popular graphical and vocal toolkits)

Co

nte

xt

of

us

e:

use

r, c

om

pu

ting

pla

tform

, en

viro

me

nt

FUI: operational UI that runs on a particular platform either by interpretation or by execution

Step 2: From Task & Domain Models to AUI Model

Defining transformation systems that realize the transition from Task & Domain Models to AUI Model

LHS

RHS

NAC Resultant AUI fragment

www.similar.ccBCHI

www.isys.ucl.ac.be/bchi www.usixml.org

LHS

RHSNAC

Step 3: From AUI Model to CUI ModelDefining transformation systems applied to realize the transition from AUI Model to CUI ModelFour variants of CUI are considered: total graphic, predominant graphic, predominant vocal, total vocal

Resultant CUI fragment

Step 4: From CUI Model to FUI Model Transforming each variant of CUI into their respective FUI:

•Total graphic•Predominant graphic•Predominant vocal•Total vocal

XHTML genarator of GrafiXML tool

XSLT to produce

VoiceXML code

XSLT to produce X+V codeXSLT to produce X+V code

Transformation rule mechanism: G: initial UsiXML specification

LHS matches into GNAC doesn’t match into GRHS replaces LHS

G is transformed into G’ (resultant UsiXML specification

Specification of transformation:

TransformiXML tool:Choose input file (models to transform)Select development path(starting and destination point)Edit rules in an XML editorSave the result in a UsiXML file