ARCHITECTURES AND STANDARDS FOR IVAS AT THE SOCIAL

COGNITIVE SYSTEMS GROUP

H. van Welbergen, K. Bergmann, H. Buschmeier, S. Kahl,

I. de Kok, A. Sadeghipour, R. Yaghoubzadeh, S. Kopp

MAIN RESEARCH THEMES

How can cognitive systems be designed as intelligent, socially adept interaction partners that allow a fluent and coordinated interaction with humans

Developing methods to model the behavioral, preceptual motor and cognitive mechanisms of embodied human-like communication and cooperation

Application and evaluation in IVAs

Scenarios: Embedding IVAs in mouse-keyboard interaction Virtual coaches Virtual assistents for the elderly and cognitively impaired Cognitive models that investigate semantic speech/gesture

coordination Computational models of dialog coordination based on linguistic

feedback

CURRENT ARCHITECTURE AND STANDARDS

Our components: A multimodal memory component realized as a spreading

activation model of semantic coordination for speech-gesture production (Bergmann et al., 2013)

Incremental NLG based on SPUD (Buschmeier et al., 2012) A Behavior Planner for iconic gestures (Bergmann and Kopp,

2009) A BML 1.0 Realizer for incremental and adaptive behavior

(van Welbergen et al. 2014) An IS-based incremental dialog manager, capable of handling

uncertain input (yet unpublished)

Several external components Often multiple alternatives available, allowing trade-offs

between recognition/synthesis quality, reactivity and control

CURRENT ARCHITECTURE AND STANDARDS

Combining components We follow the SAIBA reference architecture

BML 1.0 for realization, extensions for incrementality and adaptivity

No standardized communication (via FML) between the Intent and Behavior Planner

Components Run on various programming languages, OSs, computers Allow incremental processing IPAACA

Implements the Incremental Unit architecture (Schlangen and Skantze, 2011)

Embeds it in a message oriented middleware (RSB; Wienke and Wrede 2011)

FUTURE ARCHITECTURES AND STANDARDS: LONG TERM Base our architecture on universal (less problem

specific) and cognitively motivated principles

Ongoing work: Fully incremental production an recognition and

production, e.g. in attentive speaking (Buschmeier et al 2012)

Representation and decision making under uncertainty Using uncertainty as valuable information rather than as a

mere nuisance Priming and imitation learning of gesture (Sadeghipour

and Kopp, 2011)

FUTURE ARCHITECTURES AND STANDARDS: SHORT TERM

SAIBA provides us: A common terminology for large scale IVA components A standardized interface for behavior realization

We would like to enhance it to a full reference architecture for IVAs

Our requirements Handle both input processing and output generation Coordination between these at multiple levels Incremental input and output processing Representing memory Representing uncertainty

FUTURE ARCHITECTURES AND STANDARDS

Asap(Kopp et al., 2014)

SUGGESTIONS FOR DISCUSSION: A NEW REFERENCE ARCHITECTURE FOR IVAS Our requirements

Handle both input processing and output generation Coordination between these at multiple levels Incremental input and output processing Representing memory Representing uncertainty

Requirements of others Representing and interacting with ‘the world’ Generalization to multi-agent/multi-human

SUGGESTIONS FOR DISCUSSION: A NEW REFERENCE ARCHITECTURE FOR IVAS Desired outcome:

Prioritized list of requirements, plan to incrementally embed them

Definition of shared terminology Architecture sketch, identification of shared interfaces Agenda for shared interfaces between modules

Workshops/plans of progress for BML, FML, PML, …

SUGGESTIONS FOR DISCUSSION: SHARING AND COMBINING SMALLER COMPONENTS Many interesting ‘small’ IVA components are

presented every year In isolated projects/experiments

How can these be embedded in the larger effort of developing a full IVA?

We are especially interested in doing this for the Behavior Planner The challenge is not only in implementing individual

parts that are potentially useful but also in combining the components

SUGGESTIONS FOR DISCUSSION: SHARING AND COMBINING SMALLER COMPONENTS Desired outcome:

(Start of) design pattern catalogue for the design of (parts of) a behavior planner E.g. MAS, blackboard, MOM, IU-architecture, subsumption

architecture Ideas to share smaller components

List of good candidate components for sharing Inspired by best practices from related fields (e.g. ROS:

Quigley et al., 2009) Can we develop standardized components using challenges

for each of them?

SUGGESTIONS FOR DISCUSSION: CHALLENGES

Goal: Foster the development and comparison of reusable IVA

components

Different from the goals of IVA’s GALA Provide a forum to present the state-of-the art in

technology of virtual humans and innovative applications

Aimed to stimulate (under)graduate student participation

Disseminate demos in an ever evolving gallery for study and reuse In practice the “dissemination” was limited to a video

SUGGESTIONS FOR DISCUSSION: CHALLENGES

Desired outcome: GALA post-mortem Best practices of successful challenges in related fields

RoboCup@Home, Blizzard TTS, Give NLG, INTERSPEECH, … Requirements/guidelines for a challenge Plan to embed challenge in a workshop/conference

(IVA?) Example design (sketch) of a first challenge

THANKS FOR YOUR ATTENTION

Questions?

SUGGESTIONS FOR DISCUSSION: CHALLENGES

Challenges in related fields: RoboCup@Home, Blizzard TTS, Give NLG, INTERSPEECH, …

Some interesting ideas from these: Shared set of ‘resources’, fosters comparable system building

E.g. recorded voices to build off (Blizzard), shared environment to navigate through (Give, RoboCup@Home), shared corpus (INTERSPEECH)

Given by the organizers of the challenge Predefined set of automatically obtained evaluation criteria (GIVE,

INTERSPEECH) Baselines (INTERSPEECH)

‘Sharing’ subjects (Blizzard, RoboCup@Home) Both the challenge and the submissions are full-fledged, peer-

reviewed papers (INTERSPEECH challenges, Blizzard challenges) Mandatory (but delayed) publishing of source code and technical

documentation (RoboCup@Home)

SUGGESTIONS FOR DISCUSSION: SHARING AND COMBINING SMALLER COMPONENTS Inspiration from related fields: ROS (Quigley et al.,

2009) Provides over 3000 robotics components Provides the ‘plumbing’ of robotic systems

Connecting components: Publish subscribe, rpc, actions, distributed parameter setting,

IDL’s for messages, starting components, central time, … Debugging connections: visualization, recording/playback

Building and distributing components: Tools for building and distributing ROS-packages

We have reinvented the wheel on several of these… Some adhering to conventions required