Implementing Diverse Robotic Interactive Systems Using VOnDA

Bernd Kiefer1 , Christian Willms21paragon semvox GmbH 2Deutsches Forschungsinstitut fur Kunstliche Intelligenz (DFKI GmbH)

kiefer@semvox.de, christian.willms@dfki.de

AbstractThis paper describes the implementation of differ-ent interactive robotic systems using the dialogueframework VOnDA, an interactive robotic arm, aguidance and a transport robot developed in the IN-TUITIV project, and the social robotic companionand tutor of the PAL system. The applications dif-fer in terms of complexity in multiple dimensions,e.g., concerning dialogue structure, frequency ofsensory input, architecture (embedded vs. cloud),etc. We highlight the different requirements, andaim to demonstrate the flexibility of the frameworkand its feasibility in multiple contexts and on dif-ferent levels of control, from mere interaction mod-elling to being the central agent control component.

1 IntroductionInteractive robotic applications come in many flavours, fromspeech or multi-modal command and control interfaces forindustrial robots [SynDiQuAss Team, 2020; Mocan et al.,2016; Ramgire and Jagdale, 2016] to social companion robots[ASAROB Team, 2020; Foster et al., 2016] or interactivelearning companions for different areas [Belpaeme et al.,2013; PAL, 2018; Belpaeme et al., 2015]. The different ap-plications have quite diverse requirements in terms of reactiv-ity, sensor input and processing, interaction capabilities, useradaptivity or long-term interaction abilities.

In this paper, we present different applications of in-teractive robotic agents implemented with the VOnDAframework[Kiefer et al., 2019], which was initially designedfor dialogue management, but turned out to be able to serveas a basic management layer for many kinds of interactiverobotic (and other) systems.

1.1 Related WorkIn this section, for reasons of space, we will concentrate oninteractive robotic applications similar to the ones we are go-ing to describe later. There are many competitive frameworksavailable, e.g. OpenDial [Lison and Kennington, 2016],which takes a similar hybrid approach (more are discussedin [Kiefer et al., 2019]), with less focus on long-term mem-ory. Some are already embedded into a robot ecosystem, such

as rrSDS [Kennington et al., 2020], a feature that VOnDA iscurrently lacking.ASARob The aim of the research project ”Aufmerk-samkeitssensitiver Assistenzroboter” (ASARob)1, performedby Fraunhofer IAB, is to develop new skills for interactiveassistance robots that allow to detect attention state of theircounterpart and, if necessary, influence and direct it throughtheir behaviour. Using different robot sensors and computinglocation and motions of the user, including gaze, head direc-tion and posture, it recognises where the user’s attention isdirected and what his/her possible intention might be. Ver-bal interactions are another source of information and playan important role in regaining the user’s attention on sharedjourneys. In addition, stylised facial expressions and move-ments of the robot are used for this purpose. The multi-modal interaction component in ASARob is implemented bythe geni:OS2 platform of paragon semvox, which shares as-pects such as ontological modelling of interactions and reac-tive rules with VOnDA, but is commercially available only.MuMMER The goal of this project [Foster et al., 2019] isto build a Multimodal Mall Entertainment Robot, that pro-vides help and guidance to customers, as well as marketingactivities. The challenge is that the robot has to be socially ap-propriate and entertaining to gain broader acceptance. It usesverbal (speech-based) and non-verbal communication fusedwith appropriate motions to achieve this. Multiple sensorstrack people in the vicinity, their location, gaze and socialsignals and share their results with the interaction modules togenerate modulated motions for expressing non-verbal socialsignals. The conversational system is built upon several ma-chine learning based modules which are nevertheless basedon dialogue acts and frame semantics to achieve functionali-ties for social talk that go beyond usual slot-filling dialoguesemployed in voice agents. To synchronise verbal utterancesand motions, which may be quite different in duration, theirexecution system uses recipes that tie dialogue and physicalactions together, and the execution system has been extendedto manage multi-threaded parallel executions and their arbi-tration, including the possibility that users interrupt such aprocess and switch to a completely different action. The robotis fully autonomous, albeit using heavy computing machinery

1https://asarob.de/2https://www.semvox.de/technologien/genios-3/

to accomplish the different complex computations for sen-sory and other modules, and was tested in a three-month userstudy. The dialogue and agent control system of this project isvery different from a VOnDA implemented agent and there-fore hard to compare, but the agents described in sections 3.2and 3.3 share a lot of the demanding aspects of this system.

2 Background: The VOnDA frameworkVOnDA uses a hybrid rule-based and statistical approach tobe able to trade control against flexibility depending on thedemands of the application. This covers the range from fullydeterministic systems to socially interactive robots, wherecentral requirements are extensive user adaptivity and vari-ation in then interaction, including some surprise or unpre-dictability.

Reactive Rules forDialogue

Management

ActionSelection

RDF Store

General Reasoning RulesExtended Reasoning Service

SpeechActs

EntitiesIndividualsUser Model

Frames(Actions)

Speech Recognition

Application Data

Perceptions(Sensor Data)

Speech ActsApplication Actions

NL Generation

Text To Speech

Speech Acts &Actions

(Alternatives)

Updates

Cha

nges

Generation

Parameters

Figure 1: Architecture of a VOnDA Agent

In figure 1, which shows the schematic of a VOnDA agent,the statistical component is in the box labelled action selec-tion. It can be implemented by probabilistic models of anycomplexity, from Bayes Nets to deep neural network solu-tions, and has access to all the data available in the informa-tion state, if needed.

The information state is kept in an extended RDF Store andreasoner (HFC, [Krieger, 2013; Krieger, 2014]) which allowsto efficiently attach time information for all facts in the database and to reason about them. In this way, the informationstate contains a full history of past interactions, even beyonda single session, and all relevant system and contextual datafor dealing with a user in a socially appropriate and pleasantform. Data can be analysed and aggregated over this historyto implement interaction patterns that foster long-term use ofsuch a system, e.g. by showing recollection or familiaritywith the current user. On the other hand, the semantic longterm history also allows to model gradual forgetting on anyscale, e.g., by specifying reasoning rules that influence thevalidity of data chunks depending on their class and lifetime.For dealing efficiently with, e.g., high-frequency sensor data,the RDF store has support for streaming reasoning, whichcan also be used for aggregating modules like an episodicmemory.

The use of an RDF store as a central database has other ad-vantages: Firstly, the specification of application data struc-tures, such as sensor data, natural language semantics orother internal data and the data objects itself are representedin the same format. We will describe in the next section

how VOnDA uses for example the type and property systemof OWL to support a compact rule specification language,and the time information to obtain modifiable object in thedatabase. Secondly, reasoning rules and queries can be usedto perform complex tasks in a more declarative way. HFCallows to add custom reasoning rules, which gives a lot offlexibility, but has to be handled with care since it may intro-duce performance non-termination problems.

Relevant changes in the information state, which are deter-mined by efficient filters similar to those used for streamingreasoning, trigger the rule system of a VOnDA agent. In com-bination with the efficient processing of sensory and otherinput data, highly reactive systems with very low responsetimes are possible.

The VOnDA framework uses shallow semantic structurescommonly used in dialogue systems for natural language un-derstanding (NLU) as well as generation (NLG). The advan-tage of this symbolic representation for generation is that theoutput (verbal and non-verbal, e.g., motions) can be flexiblymodulated for different situations, depending on informationstate data, such as the emotional state of the robot or the user,etc. This will play a role in the use cases we describe insection 3.2 and 3.3 . The semantic structures consist of adialogue act (intent) that adheres to the DIT++ schema andISO 24617-2 standards [Bunt et al., 2017], followed by themain frame/topic, taken from FrameNet where possible, anda number of (optional) key-value parameters. An examplewith slot fillers from the information state is shown here:#YNQuestion(IsCloseBy, title={user.title},

lastName={user.surname}, emotion=neutral)

The system outputs these dialog acts and sends them to amultimodal generation module, as well as to software mod-ules of other partners, which may generate additional multi-modal outputs, e.g., for an avatar.

Rule LanguageThe rule language of VOnDA was designed to be as com-pact and declarative as possible, but also allowing seamlessintegration of Java code and classes. One of the most impor-tant features is that RDF objects and classes can be treatedsimilarly to Java objects, but additionally including multipleinheritance and type inference, which are built into the RD-F/OWL semantics. The ontology that is part of every VOnDAproject provides the necessary information for the compila-tion of the compact rule specifications into native Java code.3

To obtain modifiable objects in the RDF store, VOnDAdoes not delete or modify existing data, but only adds the up-dated information that carries the current time stamp. Internalbackground reasoning in the run-time system makes sure thatwhen the rules are evaluated, the objects and “fields” (prop-erties) are referenced, the current value is available, retainingthe history of previous values.

3 Application ScenariosBy describing the current application scenarios we want tohighlight the qualities of VOnDA that make it well-suited forimplementing interactive systems, which are, among others:

3For a more in-depth discussion, see [Kiefer et al., 2019]

• reliability and predictability of the resulting system• real-time reactivity• flexible dialogue modelling for complex dialogues with

compact rule sets and reasonable implementation effort• integration and use of heterogeneous data types, from

sensor to semantic 2D map data• an information state that is usable by all layers of pro-

cessing: NLU, dialogue, action selection, NLG andmulti-modal generation

• control and orchestration of major ensembles of com-plex modules and applications

• co-operation of modules by synchronised belief states ordirect communication with exchangeable middlewares

• implementation of complex socially adequate agents ex-ploiting the integrated long-term history

3.1 Command and Control for a Robotic ArmOne of the maybe least complex speech-based robotic appli-cations is a verbal or multi-modal command and control in-terface for a robotic arm and its various capabilities.

The main challenge when implementing such an applica-tion is the high reactivity and reliability that is required to beable to control the arm safely. The latency between utteringa command and its execution has to be very small for somecommands, e.g., for stopping a running command execution.Reliability in this case mostly depends on the accuracy of theautomatic speech recognition, which can be tuned quite welltowards accurateness despite the short utterances because ofthe small number of relevant possibilities. From the view ofthe dialogue and control management, latency and reliabilityare not an issue for a VOnDA implementation, since the rulescan implement fully deterministic behaviour, and thanks to asynchronised information state between the dialogue systemand the robot, the dialogue system is fully aware of the robot’sinternal state.

In the concrete case, we use a VOnDA dialogue system tocontrol a robotic Franka Emika arm. The work is carried outas part of the INTUITIV project ([Kruijff-Korbayova et al.,2020]), acronym for ”Intuitive-nonverbal and informative-verbal robot-human communication“, which aims at devel-oping non-verbal and verbal human robot interactions thatcan be immediately understood by humans. The use-casesto study these interactions are applications in a rehabilitationclinic.

Here, the robotic arm is used to hand out required medi-cal supplies to hospital staff during an examination session.The staff member moves it using speech commands in phaseswhere both hands are already bound by other activities. Moreprecisely, it directs the arm’s end effector holding a tabletwith medical consumables to (predefined) positions in spaceby means of voice instructions. The available guidance com-mands are shown in Figure 2.

The dialogue part of the system is responsible for translat-ing the verbal commands into ROS4 messages for the robotand generating speech output in case of errors or problems.

4http://wiki.ros.org/Documentation

Figure 2: INTUITIV arm scenario: Support movement directionsand target positions.

The robot performs the calculation of the specific actions nec-essary to reach the target position, while the VOnDA subsys-tem focuses on the speech interactions.

The information state contains concepts for the dialog his-tory, dialog structure, history of commands, execution state ofthe robot as well as a list of predefined target poses/positions.For future applications, a facility to add poses and positionswith the help of speech input is planned.

It is updated using two different input sources, firstly theintents determined by the language processing: AutomaticSpeech Recognition (ASR) and Natural Language Under-standing (NLU) provided by Cerence Studio (former Mix)5

platform, and secondly the execution status updates of therobotic system, e.g. ExecutionFinished Success.

As described above, the rules for controlling the dialogtranslate the results of the language processing into ROS com-mands or generate language output if the execution status ofthe robot indicates an error. There are 4 main classes of rules:

• Directions, e.g. right, left, up, down etc.• target positions, e.g. go to target pose 2• Repeat/Cancel previous tasks, e.g. continue, stop• Error messages, as reactions to robot execution status

All commands have instantaneous effects, which is espe-cially important for the Stop command, which halts all ac-tivity of the arm, suspending it in the current position. Al-though an emergency button to halt the robot is still required,the voice command can probably be given faster than the me-chanical signal.

3.2 Synchronised Multi-Modal InteractionModule for a Robotic Guide

The INTUITIV project also investigates how intentions of arobot can be made transparent to humans through anticipatorypath selection in combination with iconic and verbal commu-nication in order to minimise the discomfort due to uncer-tainty regarding the actions of a robot. The robotic platforms

5https://www.cerence.com/cerence-products/cerence-studio

used in this scenario are a guidance robot and a luggage trans-porter. Both robotic platforms are equipped with speech inputand output as well as an stylised avatar face on a display.

This scenario poses quite different challenges to the under-lying conversational system than the command-and-controlscenario. Besides more complex interactions, which alsohave to be intuitive and compliant to the social interplay be-tween robot and user, there are more real-time signals arrivingfrom the robot in a higher frequency. Here, VOnDA makesit possible to react to these signals in real time, while at thesame time providing a consistent and socially adequate inter-active robot persona.

The robots picks up patients at some location within theclinic, such as the reception desk, and accompanies them toa destination such as the patient’s room or a treatment/train-ing facility. After receiving a pick-up order from the recep-tion desk, the robot drives autonomously to the location ofthe patient and starts the guidance. During the guiding andautonomous phases of the robot, the main purpose of theVOnDA agent is the planning and execution of the situateddialogues, which consist of verbal as well as multi-modaloutput. The following main situations are covered by the sys-tem, with varied demands and requirements on the interactionmanagement subsystem:

• Greeting and Farewell: The robot greets the client, in-troduces himself and states its current task. On a firstmeeting, the robot also describes how it works and howthe client can interact with it and signals acquaintanceotherwise. When reaching the destination, the robotpolitely says goodbye and returns autonomously to itsstarting position. Important here is that the interactionsare adaptive, taking different forms depending on thelevel of familiarity between user and robot.

• Navigation and guidance: One of the main tasks of therobots is to guide patients through the clinic by givingthem verbal (and multi-modal) route instructions, serv-ing as an indoor navigation device. At first an (opti-mal) route based on the robot’s position and destinationon a semantic map of the building is computed, anno-tated with important landmarks and the functions of therooms. During navigation, the robot’s position data iscompared continuously with the calculated path and out-puts information about upcoming changes in directionetc. In addition to the guidance, the robot gives infor-mation about potentially interesting places on the way,and engages in other forms of social dialogue to makethe journey more pleasant and less boring. This is facil-itated in particular by VOnDA’s information state con-taining the map information. Here, the flexibility of theRDF store to represent all kinds of data without the needof additional programming code is a big advantage. TheRDF data for the map is mostly generated automaticallyusing annotated OpenStreetMap data from the buildingsin the Indoor Mapping format.

• Episodic Encounters during autonomous drive: An in-tuitive and user friendly system also implies that robots,even if they are autonomous, greet people in the corridor,make room for them, or otherwise interact with them,

for example by announcing an overtaking manoeuvre orwhen the robot is addressed. The dialogue system usesa shared information state with the belief state of therobotic system to carry out such interactions, for exam-ple to find out if and how many people are around. Theexchange with the robotic system is implemented via aROS interface.

• Using an Elevator: During autonomous drive and whenguiding the patient/client, it may become necessary forthe robot to use an elevator. The use of an elevator posesa particular challenge to the robotic and the dialog sys-tem. The dialog required for this is very complex, sincethere are so many different situations and solutions pos-sible, resulting in a wide spectrum of possible robot anduser dialogue turns. This starts with asking for helpwhen arriving at the elevator, since the robot can nei-ther open the door by itself, nor can it press the button tobring it to the right level. In the following, appropriatereactions are needed if this help is granted or denied. Inthe following, the robot has to find a way to determinethat it has arrived at the correct floor. This, again, canbe done by asking people who are potentially present. Ifthat is not the case, the robot might take up its plannedjourney until it can get evidence about which floor it isin, either with camera system support, map comparison,or asking people it encounters, if neither of the previousmethods gives conclusive results (or is not available atall). Also, the situation with and without guided personmight be quite different, and be even complicated by thefact that an elevator might be too small to accommodatethe robot and the person. In situations like this, VOnDAallows very flexible dialogue flows that take a lot of pa-rameters from the sensors and the information state intoaccount.

• Supporting the robot’s recovery behaviour: Exceptionalsituations, which occur both in the context of the au-tonomous journey and the journey with the patient, aresolved in a collective effort between the robotic systemand the dialogue system. For this purpose, there is aclose cooperation between the components, which takesplace via the ROS interface. The robotic system informsthe dialog system about the nature of the problem, forexample, the robot may not be able to continue its routebecause the corridor is blocked by a group of people.This information is used by VOnDA to launch a corre-sponding dialogue sequence. During the dialogue thesystem will send repeated requests and status updatesto the robotic system. This way, the robotic system hasknowledge of the dialog process and can initiate own ac-tions if necessary, e.g. calculating an alternative route ifthe persons do not cooperate.

Although maybe not as advanced as the autonomous socialrobot from the MuMMER project, this is already a quite ad-vanced use case, which highlights the abilities and advantagesof a VOnDA based implementation.

In contrast to the robotic arm described above, the informa-tion state to be managed by the VOnDA system comprises aconsiderably higher number of information types from differ-

ent sources. These include, among others, the semantic mapsof the hospital, basic patient information, dialog history, rep-resentation of the different tasks and status and sensory in-formation from the robots. The VOnDA agent maintains adistributed belief state, by synchronising the robotic knowl-edge state, e.g., robot status (idle, stuck, etc.), as well as theoverall mission knowledge, with its internal information staterelevant for dialogue, which is especially relevant for routeguidance and recovery behaviour.

3.3 Central Control of a Robotic eHealth TeachingCompanion

A fundamental goal of the PAL project6 is to provide a plat-form for child health education (Type 1 Diabetes Mellitus),supported by an embodied robotic companion. After an as-

Secure

Proxy

AgentIntelligence

Database

Figure 3: Architecture of the PAL system. External components arethe robot, the tablet app and the patient control panel.

sessment session to determine the child’s physical situationand education needs, where a browser-based control panel isused to create an initial user profile, the children interact witha physical robot. First, it introduces itself and its purpose,and asks the child questions about him/her. Then, they playsome rounds of educational games, which are selected by thechild. Finally, a tablet with the PAL app is given to the chil-dren to accompany their learning process. A virtual avatarexactly simulating the robot (and the same software pilotingit), is the central interactive component of the eHealth app onthe tablet.

The Hybrid BrainFor every user login, a complete instance of the “PAL brain”is spawned, which consists of the modules shown in fig-ure 4: dialogue manager with natural language understand-ing and generation (NLU/NLG), action selection/statisticaluser model, multimodal robot behaviour manager, targetedfeedback, episodic memory, off-activity-talk, Explainable AIcomponent. Many components are tightly linked do thedialogue manager that is predominantly implemented withVOnDA functionality, and is in fact the central agent be-haviour control component, piloting the whole “brain” withthe help of the rest of the modules and tying their functional-ity together.

The PAL platform is a cloud-based hybrid AI system[Kaptein et al., to appear 2021], a complex mix of machinelearning and symbolic AI components, implemented usingdifferent paradigms and programming languages. In the be-ginning, the hybrid approach allowed the first prototype of

6https://www.pal4u.eu

the system to be quickly available using more hand-craftedmodules and settings. Then, the system gradually evolved tousing more and more learned and sophisticated componentsonce the data became available.

Dialogue Manager

NLU

NLG

Quiz Logic

Off-Activity-Talk

Feedback Episodic Memory

Task Suggestions

BehaviourManager

Action SelectionExplainable AI

Goal Calculator

Sentiment MinerNexus Event MessagesDatabase access

Figure 4: The “Hybrid Brain” of the PAL system

A single RDF database (a HFC storage/reasoner) instanceprovides the central cloud data hub for all subsystems. Mostof the communication between modules and components, in-cluding that of the (partly virtual) robotic agents go throughthis hub. In addition, modules of one “brain” instance canexchange direct messages using a communication layer im-plemented by Apache Thrift (https://thrift.apache.org/).

The RDF storage contains an extensive ontology of healthand learning related information, as well as sub-ontologies forhuman/machine roles or tasks, events, processes, natural lan-guage semantics, episodic memory structures to capture spe-cific events, an off-activity database for social talk, etc. Theheterogeneous knowledge and data in the database is sharedamong all components, and used in such diverse tasks as ac-tion selection, the robot behaviour management, which in-cludes motions and utterances, interaction management, suchas task suggestions, social talk, relating to past episodes, andall higher-level applications. Almost all components use thedatabase in both ways, for gathering as well as providing data.

Activities in the eHealth AppThe child can interact with her/his robotic pal in several ac-tivities which are components of the app, and make quite dif-ferent demands on the central control module:Educational GamesQuiz, Memory Games, Break and Sort, Diabetes NotebookFun activitiesRobotic Dance Creator, Add-On Shop (gamification)Conversational activitiesTask suggestions and explanations, Achievement tracking,Off-activity talk, immediate progress feedback and mid-termaggregated feedback through the episodic memory moduleThere are many demanding aspects in these activities that theimplementation with VOnDA is able to support:

The already large number of components and apps, manyinfluencing each other in their behaviour by generating newdata about the user, needs to be arbitrated. This is compli-cated by the fact that the user can spontaneously stop anyactivity at almost every point in time or switch between them,and the agent has to consistently react to such spontaneoususer behaviour. The pseudo-parallel nature of the VOnDA

rule system (all rules may be triggered more or less simulta-neously) is a much bigger advantage than the potential prob-lems which may arise by such an architecture. The size ofthe rule system implementing the PAL system could be keptquite small, using rules with general preconditions that areequally applicable in many situations, and since all rules areexecuted on information state changes unless special mea-sures are taken, rules are in principle “always on”. On theother hand, it is also easy to fence certain rule subsystemsfrom others by applying “bail-out” rules based on the currentsystem state. In fact, the core control component with all nat-ural language resources was build in about half a person-year,parallel to the VOnDA framework.

To allow for variability and spontaneity in the reactions, al-ternative action paths are added at all places where it is sensi-ble in the interaction flow. The choice is left to the data-drivenstatistical action selection module, which takes the currentinformation state into account to take the most appropriatechoice at the given situation. This choice is highly adaptivealso to the user model and perceived user state, e.g, the cur-rent game or achievement progress.

While the real-time data frequency (game scores, user an-swers, data entry) is on the medium to low level, the reac-tivity of VOnDA allowed fine grained synchronisation withthe all tablet components and the robot, e.g., to guarantee thatrobot movements and utterances don’t get into each other’sway, game results and tablet input are immediately passed be-tween tablet and the central component for proper reactions,etc. Additionally, the multimodal generation modulates therobot gestures and utterances using knowledge stored in theuser profile of the information state.

Having all interaction history available allows differentforms of long and short-term interaction patterns: immedi-ate game and achievement progress, mid-term reactions usingaggregated data in the episodic memory, machine-learningbased user modelling for adaptivity of tasks suggestions, andbonding effects using interactions with the gamification in-centives and the off-activity talk.

4 Evaluation of System UseWe present some numbers from the PAL system’s last exper-imentation cycle. The other projects are still in progress andhave unfortunately not produced enough data for decisive re-sults.

In this experiment, children interacted with the continu-ously running system over an extended period of time (48children from Italy and the Netherlands over 2.5 to 3 months),in 571 resp. 471 sessions. Table1 lists figures indicating us-age patterns per child for both countries: Number of sessions,activities during one session, and average session and activ-ity duration in seconds. While children seem to use the sys-

# sessions duration # activities act. durationIT 17.3(14.5) 819.8(1018.1) 14.0(30.0) 58.4(159.8)NL 16.8(13.2) 721.5(939.2) 12.6(17.7) 57.1(154.7)

Table 1: Average usage parameters per child: standard deviationsare in parentheses

tem quite differently, which is visible in the high variation inusage duration, usage across countries looks rather similar.Unfortunately, we have no in-depth evaluation about usagepatterns in different child populations. The robot addressedthe child about 38 times in every session, from a repertoire ofaround 4800 unique utterances per language. Repetitive in-game feedback is not taken into account here. The childrenaddressed the robot using free-text input around ten times persession. Most of the interaction could be handled using Appbuttons or other GUI elements.

5 Future WorkWhile there are already tools for editing, browsing and debug-ging the rule system present in the current framework, havingadditional and more sophisticated tools almost always helpsfor implementing system, and makes it more pleasant to workwith it. We are currently adding a graphical editor for hier-archical state charts, which is an easy to grasp format for be-ginners, and a compiler that transfers these state charts intoVOnDA rules. The goal is to give new developers an easyhead start and at the same time be able to tweak the result(the compiled automaton) with the much more powerful andbetter generalising reactive rule framework.

There are also plans to make the syntax easier to read forthe developer and the compiler, since some language con-structs of Java that are currently supported are unnecessarilyhard to parse.

6 ConclusionWe believe to have demonstrated that the VOnDA frameworkis feasible for implementing a significant range of roboticconversational systems. Its compact specification language,seamless integration with a programming language, and con-nection to the extended RDF reasoner HFC open many waysand opportunities for exploitation.

The PAL system gives proof that stable and reliable sys-tems can be build in this way. Although the server hardwarewas very moderate, no performance problems were reported,even when 40 users were logged in at the same time. Theserver also did not have to be restarted at all in the final trialperiod, which lasted 3 months.

A bigger developer base would provide the best hints asto in which direction to develop the framework, and we stillbelieve that hybrid AI has many benefits over pure data drivenapproaches, at least for a significant range of applications, sowe hope to have drawn in more adopters with this paper.

Acknowledgements We want to thank all the collaboratorsin the projects that helped us implementing these systems.Special thanks go to Ivana Kruijff-Korbayova for her valuablecomments and suggestions, and to Bert Bierman for constantmental support.

The research described in this paper has been funded bythe Horizon 2020 Framework Programme of the EuropeanUnion within the project PAL (Personal Assistant for healthyLifestyle), Grant agreement no. 643783, and the BMBFproject INTUITIV, Grant No. 16SV7979.

References[ASAROB Team, 2020] ASAROB Team. ASAROB project

website. https://asarob.de/, 2020.[Belpaeme et al., 2013] T. Belpaeme, P. Baxter, R. Read,

R. Wood, H. Cuayahuitl, B. Kiefer, S. Racioppa, I. Kruijff-Korbayova, G. Athanasopoulos, V. Enescu, R. Looije,M. Neerincx, Y. Demiris, R. Ros-Espinoza, A. Beck,L. Canamero, A. Hiolle, M. Lewis, I. Baroni, M. Nalin,P. Cosi, G. Paci, F. Tesser, G. Sommavilla, and R. Hum-bert. Multimodal Child-Robot Interaction: Building SocialBonds. Journal of Human-Robot Interaction, 1(2):33–53,2013.

[Belpaeme et al., 2015] Tony Belpaeme, James Kennedy,Paul Baxter, Paul Vogt, Emiel EJ Krahmer, Stefan Kopp,Kirsten Bergmann, Paul Leseman, Aylin C Kuntay, TilbeGoksun, et al. L2TOR-second language tutoring using so-cial robots. In Proceedings of the ICSR 2015 WONDERWorkshop, 2015.

[Bunt et al., 2017] Harry Bunt, Volha Petukhova, DavidTraum, and Jan Alexandersson. Dialogue Act Annotationwith the ISO 24617-2 Standard, pages 109–135. 11 2017.

[Foster et al., 2016] Mary Ellen Foster, Rachid Alami, OlliGestranius, Oliver Lemon, Marketta Niemela, Jean-MarcOdobez, and Amit Kumar Pandey. The MuMMER project:Engaging human-robot interaction in real-world publicspaces. In International Conference on Social Robotics,pages 753–763. Springer, 2016.

[Foster et al., 2019] Mary Ellen Foster, Bart Craenen, AmolDeshmukh, Oliver Lemon, Emanuele Bastianelli, Chris-tian Dondrup, Ioannis Papaioannou, Andrea Vanzo, Jean-Marc Odobez, Olivier Canevet, et al. MuMMER: Sociallyintelligent human-robot interaction in public spaces. arXivpreprint arXiv:1909.06749, 2019.

[Kaptein et al., to appear 2021] Frank Kaptein, BerndKiefer, Antoine Cully, Oya Celiktutan, Bert Bierman,Rifca Peters, Joost Broekens, Willeke van Vught, Michaelvan Beckum, Yiannis Demiris, and Mark A. Neerincx.A cloud-based robot system for long-term interaction:Principles, implementation, lessons learned. Transactionson Human-Robot Interaction, page 17, to appear (2021).

[Kennington et al., 2020] Casey Kennington, Daniele Moro,Lucas Marchand, Jake Carns, and David McNeill. rrSDS:Towards a robot-ready spoken dialogue system. In Pro-ceedings of the 21th annual meeting of the special interestgroup on discourse and dialogue, page 132–135, 2020. Ci-tation Key: kennington2020rrsds.

[Kiefer et al., 2019] Bernd Kiefer, Anna Welker, andChristophe Biwer. VOnDA: A Framework for Ontology-Based Dialogue Management. In International Workshopon Spoken Dialogue Systems Technology (IWSDS),page 12. Springer, 4 2019.

[Krieger, 2013] Hans-Ulrich Krieger. An efficient imple-mentation of equivalence relations in OWL via rule andquery rewriting. In Semantic Computing (ICSC), 2013IEEE Seventh International Conference on, pages 260–263. IEEE, 2013.

[Krieger, 2014] Hans-Ulrich Krieger. A detailed comparisonof seven approaches for the annotation of time-dependentfactual knowledge in rdf and owl. In Proceedings 10thJoint ISO-ACL SIGSEM Workshop on Interoperable Se-mantic Annotation, 2014.

[Kruijff-Korbayova et al., 2020] Ivana Kruijff-Korbayova,Johannes Hackbarth, Caspar Jacob, Bernd Kiefer,Matthias Schmitt, Tanja Schneeberger, Tim Schwartz,Hanns-Peter Horn, and Karsten Bohlmann. TowardsIntuitive Verbal and Non-Verbal Communication forIncidental Robot-Human Encounters in Clinic Hallways.In International Conference on Human-Robot Interaction2020, 2020.

[Lison and Kennington, 2016] Pierre Lison and Casey Ken-nington. OpenDial: A toolkit for developing spoken di-alogue systems with probabilistic rules. Proceedings ofACL-2016 System Demonstrations, page 67–72, 2016.

[Mocan et al., 2016] Bogdan Mocan, Mircea Fulea, andStelian Brad. Designing a multimodal human-robot in-teraction interface for an industrial robot. In Advancesin Robot Design and Intelligent Control, pages 255–263.Springer, 2016.

[PAL, 2018] PAL. The PAL Project Website.http://www.pal4u.eu/, 2018.

[Ramgire and Jagdale, 2016] J. B. Ramgire and S. M. Jag-dale. Speech control pick and place robotic arm with flex-iforce sensor. In 2016 International Conference on Inven-tive Computation Technologies (ICICT), volume 2, pages1–5, 2016.

[SynDiQuAss Team, 2020] SynDiQuAss Team. SynDi-QuAss project website. https://syndiquass.de/, 2020.