Bruno Fontes Barroca Building Adaptive Three-dimensional Interfaces for Critical Complex Control Systems Rapid UI Prototyping Infrastructures Applied to

Bruno Fontes Barroca

Building Adaptive Three-dimensional Interfaces for Critical Complex Control Systems

Rapid UI Prototyping InfrastructuresRapid UI Prototyping InfrastructuresApplied to Control SystemsApplied to Control Systems

A survey

BATIC3S 2006 2/24

In this presentation: Introduction:

Building User Interfaces for Control Systems

Major Topics: Adaptability Three dimensional user interfaces Development Methodologies Modelling: A three layer model Adaptation and Navigation

Conclusions

BATIC3S 2006 3/24

Building User Interfaces for Control systemsBuilding User Interfaces for Control systems

Too many blinkingAnd flashing lights!

BATIC3S 2006 4/24

Building User Interfaces for Control systemsBuilding User Interfaces for Control systems

The problem: Past (Legacy control systems)

Hardware interfaces to hundreds of controls Requires an expert operator Difficult to interpret the actual state of the

system Difficult to manage the interface to solve a

problem Poor user experience

Present (Technologic evolution) Thousands of controls User interfaces manage several control

systems Requires developers/programmers

Code a specific UI to a specific control system

Non-specific development framework No much code reuse, time is wasted

Future Adaptability and Navigation Three-dimensional interfaces Flexibility and Fiability

<< Framework >>RUID Designer

Operator

<< Software >>InterfaceLayer

Operator

Programmer

<< Hardware >>Control Systems

HardwareArchitect

Operator

BATIC3S 2006 5/24

AdaptabilityAdaptability

Motivation: Different kinds of operators may need different user

interfaces Different geometric metaphors Different process of diagnosing/solving a problem or

situation Different system states may require special user

interfaces The programmer would have to code [nProfiles X

nStates] different UI’s Issues: The designer has also to specify [nProfiles X nStates]

different UI’s? Can the designer reuse UI specifications for each user

profile?So, how hard is it to specify a UI for a control system?

Please, state the nature of those User Interfaces.

BATIC3S 2006 6/24

Three dimensional interfacesThree dimensional interfaces

Motivation: The 3D metaphor is more “human readable”

Humans think in 3D Contact with the 3D geometric model of the

actual control system Enhances user experience – increase of

productivity Numerous 3D CAD systems supports geometric

modellingIssues: Too much control given on 3D navigation

It could be hard to find a certain control Restrictions in navigation aids increases usability

Hard to associate UI logic to geometric models

See - “A Specification Language and System for the Three-Dimensional Visualisation of Knowledge Bases” - El Mustapha El Ati and Gilles Falquet – DSVIS’05

OK, so when do we start?

Wait! How do the designer starts?!

BATIC3S 2006 7/24

Development MethodologiesDevelopment Methodologies

The development process of user interfaces starts: On a top-down approach

Developer identifies and subdivides the user interface into several subUI’s in an hierarchical way.

On a bottom-up approach Developer defines basic components of the user interface and

reuses them to build more complex components. On a vertical approach

Each developer is assigned to specify a set of independent controls.

On an horizontal approach Each developer is assigned to specify a certain

property/behaviour over a large number of UI controls.

Has anyone though in a solution?

BATIC3S 2006 8/24

A three layer modelA three layer model

System Layer: System interaction logics for each system control Handles commands and events from control

system controls Formalization language: CO-OPNUser interaction logic Layer: User interaction logics for each UI control Binding between system objects and user objects Formalization language: CO-OPN(?)Visual Layer: Three-dimensional geometric models of the

controls Configure several options given about navigation

issues Other user interface resources

BATIC3S 2006 9/24


System Design

User Interaction Logic

Visual(geometric modeling)

Re-generation and merge with previous

customizations

Re-generation and merge with previous

customizations

Core Model refinement

(to model New requisites)

Automatic layer generation Starting from any layer formalization, deduce the others

BATIC3S 2006 10/24


Formalize the main requisites/contributions of each layer to the model environment

Build specific meta-model bridges to use any other languages.

BATIC3S 2006 11/24


Related Work (ERGO-CONCEPTOR*): ERGO-CONCEPTOR uses a model based approach on

UI development Module 1 presents three stages of design

Physical model (controls, variables, etc.) Structural model (subsystems and

interconnecting dataflows) Functional model (relations between variables)

Module 2 generates an abstract User Interface specification (with design alternatives)

Module 3 permits the designer to take his options on the final UI and validate them on a ergonomic KB.

This ergonomic validation can be usefull to automate the generation of three-dimensional interfaces?

Very similar approach to the three layer model

* “A model based approach to semi-automated user interface generation for process control interactive applications” F. Moussaa, C. Kolskib, M. Riahia, Elsevier

BATIC3S 2006 12/24

A three layer modelA three layer modelRelated Work (ERGO-CONCEPTOR*):


BATIC3S 2006 13/24


Tic-Tac-Toe

Inteligent WeldingGun

Related Work (DWARF*)

* “A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality” - Christian Sandor Æ Gudrun Klinker Springer-Verlag’05

BATIC3S 2006 14/24


SHEEP



BATIC3S 2006 15/24


ARCHIE



BATIC3S 2006 16/24


Related Work (DWARF*): DWARF is not oriented to control systems DWARF presents an interesting functional decomposition

of generic UI’s


BATIC3S 2006 17/24


Related Work (DWARF*): Some modules uses PetriNets on XML to specify

user interaction logic on each control Use third-party opensource Viewers (OpenInventor)

on a XML specification.


BATIC3S 2006 18/24

Adaptation and NavigationAdaptation and Navigation

System Layer: System states raises specific properties that are

embed in each user interface Ex: on an emergency state

on a “novice” profile UI, the affected controls do not transmit events to the upper layers, so it won’t appear to the user.

User Interaction Logic Layer: More or less automated controls are given to the

user interface, depending on operator’s profile. Some controls may dissapear or behave differently

according to operator’s profile.Visual Layer: Different operator’s may have different worlds or

geometrical metaphors of some controls may be different.

BATIC3S 2006 19/24


Related Work (Multi-modal Operator Interfaces*): Several levels of autonomy:

Traded control – human and machine completes subtasks exclusively and sequentially

Shared: automation and human tele-manipulation Interactive: automation and human highly skilled functions

(vision-based, speech-based controls) Supervised: Full automation, operator intervenes if required.

Based on Rassmussen multi-modal interaction model

* “Integration and design of multi-modal interfaces for supervisory control systems” - A. Zaatri, M. Oussalah, Elsevier

BATIC3S 2006 20/24


Related Work (Multi-modal Operator Interface*):

Anticipatory system detects abnormal situations before they happen on the real system.

CERS – recovery system Operator tries several highy

automated UI’s in order to solve a problem

The anticipatory system can detect the impact of the commands issued by the operator on that UI

While the problem is not solved the operator tries less and less automated UI’s in order to solve it.

* “Integration and design of multi-modal interfaces for supervisory control systems” - A. Zaatri, M. Oussalah, Elsevier

BATIC3S 2006 21/24


Related Work (ERGO-CONCEPTOR*): Means/goals abstraction hierarchy Operators navigate between several levels of resolution in

order to solve a problem


BATIC3S 2006 22/24

Conclusions:

Three dimensional User Interfaces

Use of standard formats for the geometric description of user controls. Opensource support libraries to deal with the 3D world viewer (DWARF).

Adaptation (designer) The answer to the problem of rapidly specifying adaptability in a UI specification was not tackled yet.

Adaptation (operator) Modern control systems UI’s presents several degrees of automation to the user (MOI).

The three layer model

The three layer concept is supported by the model-driven design of ERGO-CONCEPTOR. Formalization of the user interaction logics for each control is best done with PetriNets (DWARF). It should be extensible to other languages. Automatic layer generation accelerates the design process.

BATIC3S 2006 23/24

References: Stanislav Chachkov, Didier Buchs, From an

Abstract Object-Oriented Model to a Ready-to-Use Embedded System Controller, Rapid System Prototyping’05

“A Specification Language and System for the Three-Dimensional Visualisation of Knowledge Bases” - El Mustapha El Ati and Gilles Falquet – DSVIS’05

F. Moussaa, C. Kolskib, M. Riahia, A model based approach to semi-automated user interface generation for process control interactive applications. Interacting with Computers 12 (2000) 245–279, Elsevier.

A. Zaatri, M. Oussalah, Integration and design of multi-modal interfaces for supervisory control systems. Information Fusion 4 (2003) 135–150, Elsevier.

Christian Sandor, Gudrun Klinker, A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality, Springer-Verlag London Limited 2005.

BATIC3S 2006 24/24

Q&A

Documents

Bruno Fontes Barroca Building Adaptive Three-dimensional Interfaces for Critical Complex Control Systems Rapid UI Prototyping Infrastructures Applied to