Alessandro Costa Pereira, Falk Hartmann, Kay Kadner

SAP Research/TU Dresden

A Distributed Staged Architecture for Multimodal Applications

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

2/13Agenda

The Project SNOW

Prerequisites

Enabling Multimodal Access (MMI-F)

The SNOW Architecture

The Implementation

Lessons Learned/Research Questions

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

3/13Project SNOW

SServices for Nomadic Workers

EU FP6 Project, 2004-2006, 6 Partners

www.snow-project.org

A large-scale industrial diffusion of multimodal mobile documentation

for maintenance operations.

How to author multimodal mobile maintenance documentation?

How to exploit this documentation through robust interaction modalities?

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

4/13Project SNOW

Job CardFrom paper …

…to wearable electronic documentation

Maintenance procedures and related information

Airplane maintenance worker uses paper-based documentation

Unanticipated problems cause high expenses

Multimodality essential

Increased productivity for future maintenance

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

5/13Prerequisites

Multimodality is the use of multiple types of interaction between a user and a computer.

W3C Activity "Multimodal Interaction"

"Multimodal Interaction Framework" (MMI-F)

"Extensible MultiModal Annotation markup language" (EMMA)

A role model is "the description of a (possibly) infinite set of object collaborations using role types".

Riehle/Gross "Role model based framework design and integration"

A staged architecture is "a sequence of n subarchitectures, in which stage produces the (data for the) next stage"

Aßmann "Invasive Software Composition"

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

6/13

Input ComponentsOutput Components

Enabling Multimodal Access

W3C Multimodal Interaction Framework (MMI-F)

The multimodal interaction framework is not an architecture .

The multimodal interaction framework is a level of abstraction above an architecture. […]

High-Level View

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

7/13Enabling Multimodal Access

Output

InputInteraction Manager

Application Functions

Session Component

System & Environment

Generation

1..*Integration

Styling

InterpretationRecognition

Rendering1..*

1

1

1

1

1

Short illustration of the core of the MMI-F

Applicable to the software architecture of your choice

MMI-F is a role model

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

8/13The SNOW Architecture

Architectural Requirements

Client-/Server architecture

Domain independence

Device independence ( multimodality)

(Smallest) End-user device: PDA No speech-/gesture-recognition, no speech synthesis

Available modalities depend on network connection

Reuse standard software on PDA (Browser)

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

9/13The SNOW Architecture

Server

Documentation Application

Dialog Manager

Adaptation Manager

ClientBrowser

Domain-independent

Procedures

Domain-specific

Device-specific

Device-independent,Domain-independent

Staged Architecture

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

10/13The SNOW Architecture

Server

Documentation Application

Dialog Manager

Adaptation Manager

Client

Keyboard Control

Integration Manager

Browser

Gesture Recognition

Speech Recognition

Text to Speech

Speaker Control

Procedures

Microphone Control

Camera Control

Integration

RecognitionRecognitionRecognition RenderingRecognition RenderingRendering

InterpretationInterpretationInterpretationInterpretation

Recognition

Integration

Interaction ManagerInteraction Manager

StylingStylingStyling

GenerationGeneration

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

11/13The Implementation

Staged Architecture XML transformation pipeline

Problem 1: Implementation effort More components

Use of a common base technology in the components

Special template engine with bypassing (XTL Engine)

Problem 2: Performance penalty Several parse/serialize processes

Mitigated using StAX for XML processing

Compensated by deployment options (esp. within development)

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

12/13Lessons Learned/Research Questions

Lessons Learned

Mapping the MMI-F to a concrete architecture can be difficult Not very suitable for client/server architectures

Improvement of the MMI-F (fission role)

Staged Architecture can enable domain independence Evaluated the architecture in a medical scenario (WearIT@Work)

Only the Documentation Application needs to be substituted

But: there is no best-suited architecture for multimodality

Research Questions

How to generically apply the MMI-F role model?

SAP AG 2007, A Distributed Staged Architecture for Multimodal Applications

13/13Questions

…Tablet PC based demonstrator now also available

Mobile Client (PDA w/

camera and headset)

WLAN Access Point

“Server” (Text-to-speech,

Speech Recognition, Gesture recognition)