Work on ModelWork on Model--based UI at HIIS / ISTIbased UI at HIIS / ISTI--CNRCNR

Fabio Paternò, Carmen Santoro, Davide Spano

http://giove.isti.cnr.it/ISTI-C.N.R.

HIIS Laboratory Pisa, Italy

Model-Based UI at HIIS / ISTI-CNR 2

ModelModel--basedbasedUserUser Interface DesignInterface Design

MotivationsFirst generation

UIDE, Humanoid

Second generationMastermind, Adept, Mobi-D

Third GenerationUIML, TERESA, PUC

Model-Based UI at HIIS / ISTI-CNR 3

AbstractionAbstraction LevelsLevelsin in InteractiveInteractive SystemsSystems

Task and object– I want to select a work of art

Abstract Interface – Single selection object with high cardinality

Concrete Interface – List Interaction object with X elements

Implementation– List object in Java or XHTML or ....

Model-Based UI at HIIS / ISTI-CNR 4

PossiblePossible TransformationsTransformations

Concrete Interface

AbstractInterface

Task Model

ForwardReverse

Model-Based UI at HIIS / ISTI-CNR 5

Multiple Levels of AbstractionMultiple Levels of AbstractionAdvantagesAdvantages

Focus on the main design choicesLinking semantic information and implemementation elementsSemantic Web vs Interaction SemanticInteroperability through many possible implementation languages

Model-Based UI at HIIS / ISTI-CNR 6

The The ConcurTaskTreesConcurTaskTreesNotationNotation forfor Task Task ModelsModels

Hierarchical structure

Task Allocation

Temporal relations

Model-Based UI at HIIS / ISTI-CNR 7

SpecifyingSpecifying PlatformPlatform--dependentdependent TasksTasks

Model-Based UI at HIIS / ISTI-CNR 8

The user interface of the The user interface of the CTTE toolCTTE tool -- DEMODEMO

Model-Based UI at HIIS / ISTI-CNR 9

Task model Task model -- based designbased design

1. 1. AnalysisAnalysis of of temporaltemporal operatorsoperatorsamongstamongst taskstasks

IdentifyIdentify structurestructureof of dialoguesdialogues

2. 2. AnalysisAnalysis of of eacheachtask (task (objectsobjects, , attributesattributes, ...), ...)

ChooseChoose suitablesuitableinteraction interaction objectsobjects

Model-Based UI at HIIS / ISTI-CNR 10

Design Design PracticePractice

Model-Based UI at HIIS / ISTI-CNR 11

CommunicationCommunication--orientedorientedCompositionComposition operatorsoperators

Grouping: a set of elements logicallyrelated to each other

Ordering: existing of an order amonginteractors (i.e. temporal)

Hierarchy: a logical hierarchy among a set of interactors

Relation: 0ne interactor related to a group of other interactors (i.e. disablingthem)

Model-Based UI at HIIS / ISTI-CNR 12

StructuringStructuring the the UserUserInterfaceInterface

Grouping – Example: Task decomposition -> grouping of correspoding interaction techniquesOrdering – Sequential communicating tasks -> adjacent interaction techniquesRelation – Control tasks (one to manyrelations)Hierarchy – Frequent tasks –> More space or larger attributes

Model-Based UI at HIIS / ISTI-CNR 13

TERESA XMLTERESA XMLTwo platform-independent languages : task (CTT) and abstract interfaceOne level (concrete interface) representedthrough a number of platform dependentlanguagesDesigners aware of the potential platforms (not devices) early on in the design processMethod allows developers to avoid dealing with a plethora of low-level details (transformation from concrete description to implementation is automatic)Easy to add support for new implementation languages

Model-Based UI at HIIS / ISTI-CNR 14

Presentation2HierarchyDescription

Edit

Multiple-selection…….

The Structure of the Abstract User Interface

User Interface

Presentation1GroupingSelection

Edit

Navigator

Connection

Model-Based UI at HIIS / ISTI-CNR 15

Concrete User Interface

Defines some concrete aspects of the user interface depending on the current platform

Provides indications for the implementation of abstractinteractors

- <interactor id="Go_to_section3">-<interaction>- <control>-

- <navigator>- <button label="Go to selected section"/>- </navigator>- </control>- </interaction>- </interactor>

- <interactor id="Go_to_section3">-<interaction category="interaction">-

- <control type="control">- <navigator object="navigator"/>- </control>- </interaction>

Abstract levelI Concrete level

EXAMPLE

Model-Based UI at HIIS / ISTI-CNR 16

Example of platform-dependentconcrete interactor choice

EXAMPLE:Single choice abstract interactor

Mobile Phones

Radio Button

Drop Down List

Drop Down List

Desktop Computers

Radio Button

List Box

List with scrollbars

Cardinality

Low cardinality

Medium cardinality

High cardinality

Model-Based UI at HIIS / ISTI-CNR 17

Example of platform-dependentcomposition operator implementation

EXAMPLE: Grouping Operator Desktop Computers

FieldsetBulletBackground Colour/Image

Mobile PhonesUnordered List in ColumnFieldset

Column-orientedorganizationRow-orientedorganization

Model-Based UI at HIIS / ISTI-CNR 18

The The AuthoringAuthoring EnvironmentEnvironment

PresentationsPresentationsListList

ConnectionsConnectionsListList

PresentationPresentationAbstractAbstractDescriptionDescription

ElementElementConcreteConcreteDescriptionDescription

Model-Based UI at HIIS / ISTI-CNR 19

ExampleExample of of TERESATERESA--generatedgenerated UserUser InterfaceInterface--

DEMODEMO

Model-Based UI at HIIS / ISTI-CNR 20

VocalVocal InteractionInteraction

Characteristics: linear, not persistent, fasterand more natural for some operationsProvide feedback to check the status of applicationBrief prompts and short lists of options toreduce memory capabilityManagement of events (no-input , no-match, help)

Model-Based UI at HIIS / ISTI-CNR 21

SpeechSpeech implementationimplementation ofofcompositioncomposition operatorsoperators

Grouping:Insert a soundInsert a pauseUse some keywordsUse a specific volume of synthesizer voice

OrderingAlphabetical orderUse some keywords

RelationChange context(change type of menu)

HierarchyIncrease or decrease the volume of synthesizer voice

Model-Based UI at HIIS / ISTI-CNR 22

SpecifyingSpecifying generalgeneralparametersparameters forfor allall

presentationspresentations

Model-Based UI at HIIS / ISTI-CNR 23

VUI VUI vsvs GUIGUI

Welcome Welcome messagemessage

System:

Welcome to the Marble Museum Voice Response System. This service recognises your speech to provide you with the information you request.

System:

(grouping sound ) If you would like some general information, say information; if you would like information about specific artworks, say artworks; if you would like to book a ticket, say ticket. (grouping sound )

Caller: Artworks

First presentation

System:Ok, loading information about artworks. (The system goes to presentation about Artworks Section)

System:

(grouping sound ) The artworks contained in the section are the following: Boat, Totem, Hole. If you would like information on one of these please say its name (grouping sound ) Remember that if you would like to return to the main menu, say home.

System: (Time out) Please say your choice. Caller Boat

Fourth presentation

System: (feedback) ok, you have chosen Boat. (The system goes to presentation about Boat)

System:

The Boat has been achieved through the subtle divisions of the planes enveloping its central part, which is only rough-hewn; the material is white marble. (Five second pause) Remember that if you would like to return to the main menu, say home or if you would like to go back to the previous menu, say back.

Fifth presentation

Caller: Home

Management of no input Management of no input eventevent

ProvideProvide feedbackfeedback

DescriptionDescription ObjectObjectCompositionComposition operatorsoperators

Model-Based UI at HIIS / ISTI-CNR 24

AddingAdding supportsupport forfor a new a new multimulti--modalmodal platformplatform

Define new concrete description languages(as refinement of the common abstract one)Identify target implementation language(s)Identify how to support multi-modality

Model-Based UI at HIIS / ISTI-CNR 25

Generation of Generation of MultiMulti--ModalModalInterfacesInterfaces

X+V – W3C standardSupported by OPERA and NetFront Browser, also for PDAsEMMA not supported by any public toolSMIL not interaction orientedX+V application structured into three parts(document definition, head and body)

Model-Based UI at HIIS / ISTI-CNR 26

Design of Design of MultiModalMultiModal SupportSupport

CARE properties (Complementarities, Assignment, Redundancy, Equivalence)Application to composition operators, interaction and output-only elementsInteraction structured into prompt, input, feedback Identify meaningful solutions, providesuggestions

Model-Based UI at HIIS / ISTI-CNR 27

Design of Design of MultimodalMultimodalSupportSupport ((graphical+voicegraphical+voice))

Identification of new platforms (multimodaldesktop, multimodal PDA, …)Design how to support composition operatorsand interactorsmultimodal desktop:

compositon operators -> graphically supportedinteractors -> graphical prompt, input either graphicalor vocal, feedback in both modalities

multimodal pda:compositon operators -> supported both graphicallyand vocallyinteractors -> redundant/complementary prompt, input either graphical or vocal, feedback in bothmodalities

Model-Based UI at HIIS / ISTI-CNR 28

The The AuthoringAuthoring EnvironmentEnvironment

Multi-Modal attributesdefinitions

Model-Based UI at HIIS / ISTI-CNR 29

The The AuthoringAuthoring EnvironmentEnvironment

30

MultiModalMultiModal TERESATERESAhttp://http://giove.isti.cnr.itgiove.isti.cnr.it//teresa.htmlteresa.html

Direct ManipulationXHTML/SVG

Graphical+GestureC#

Digital TV

Form-basedXHTML

Vocalinterfaces(VoiceXML)

Graphical+VocalX+V

Mobile XHTML MP

Model-Based UI at HIIS / ISTI-CNR 31

MARIA XML MARIA XML RequirementsRequirements((MARIA MARIA –– ModelModel--basedbased AuthoringAuthoring enviRonmentenviRonment

forfor InteractiveInteractive ApplicationsApplications))

XML-based Languages with SchemasSupport for Abstract Data TypesMore engineered and powerful language(e.g. Pacman) Able to generate user interfaces including complex Javascripts and Ajax scripts

Model-Based UI at HIIS / ISTI-CNR 32

MARIA MARIA ToolTool RequirementsRequirements((MARIA MARIA –– ModelModel--basedbased AuthoringAuthoring enviRonmentenviRonment

forfor InteractiveInteractive ApplicationsApplications))

New Authoring EnvironmentIntegrated Support for Web Services

Mappings WSDL/LUIGeneration/Refinement

Not only traditional top-down approachesTransformations not hard-coded but definedexternally and performed with XSLT Integration of BPMN/BPEL with Model-based UIs.

Model-Based UI at HIIS / ISTI-CNR 33

AgendaAgenda

How to Manage JavaScritpsIntegration of wider set of interaction modalities in multi-device environmentsMulti users applicationsIntegration of model-based user interfaceswith Semantic WebServFace and OPEN