DenK and iCatDenK and iCatTwo Projects on Cooperative Two Projects on Cooperative Electronic Assistants (CEA’s)Electronic Assistants (CEA’s)

Robbert-Jan Beun, Rogier van Eijk Robbert-Jan Beun, Rogier van Eijk & Huub Pr& Huub Prüüstst

Department of Information and Computing Department of Information and Computing SciencesSciences

Utrecht University Utrecht University

rj,rogier,huub@cs.uu.nlrj,rogier,huub@cs.uu.nl

DenKDenK(Dialoogvoering en Kennisopbouw)(Dialoogvoering en Kennisopbouw)

- Collaborative project between University Tilburg, Collaborative project between University Tilburg, Eindhoven University and IPO Eindhoven University and IPO

- 1990 – 19981990 – 1998- Fundamental principles of communicationFundamental principles of communication

Generic architectureGeneric architecture- Natural language interfaceNatural language interface- Knowledge representation Knowledge representation - Object oriented animationObject oriented animation

- Result: implementation, 8 dissertations, Result: implementation, 8 dissertations, conference proceedings, papers, reports, …conference proceedings, papers, reports, …

Triangle-paradigmTriangle-paradigm

A A human-computer interface should human-computer interface should correspond to our natural interaction in two correspond to our natural interaction in two ways:ways:

- Direct:Direct:- physical worldphysical world- perception and actionperception and action

- IndirectIndirect- with humans by languagewith humans by language- speech actsspeech acts

UserUser

direct observationdirect observationand manipulationand manipulation

TaskTaskdomaindomain

CEACEA

language/symboliclanguage/symbolic

softwaresoftwareinterfaceinterface

DenK propertiesDenK properties

- Mental representationMental representation- ContextContext- Reasoning Reasoning

- CommunicationCommunication- Typed English textTyped English text

- InteractionInteraction- Evaluation (software interface)Evaluation (software interface)

- Task domainTask domain- Electron microscope (EM)Electron microscope (EM)

UserUser

observation & observation & actionaction

TaskTaskdomaindomain

CEACEA

communicationcommunication

interactioninteraction

C2-lens

mini-condensor

specimen

screen

Spotsize

Magnification

On/Stand by

Beam Shift Y

Beam Shift X

I/D

Intensity

Focus

μP/nP

Filament

Emission

GUN

Example DenK InterfaceExample DenK Interface

A: Why is the contrast of the visible A: Why is the contrast of the visible image regulated by this aperture?image regulated by this aperture?

B: Because the diffraction image is in B: Because the diffraction image is in the OA-plane when the microscope the OA-plane when the microscope is in HM-mode.is in HM-mode.

Mental RepresentationMental RepresentationContexts in Type TheoryContexts in Type Theory

Contains Contains - ontological information - ontological information - private beliefs about application domain (EM)- private beliefs about application domain (EM)- shared beliefs about application domain (EM)- shared beliefs about application domain (EM)- dialogue history (discourse)- dialogue history (discourse)

Beliefs, Shared Beliefs & DialogueBeliefs, Shared Beliefs & DialogueDialogue ≤ Dialogue ≤ Shared beliefs Shared beliefs ≤ Beliefs≤ Beliefs

Extra context: Pending stack Extra context: Pending stack

Type TheoryType Theory

- Ontological information Ontological information - Contexts and sequentialityContexts and sequentiality

[recipe : *s][recipe : *s][sobanito : recipe][sobanito : recipe][vegetarian : recipe [vegetarian : recipe *p] *p][macrobiotic : recipe [macrobiotic : recipe *p] *p]

- Reasoning Reasoning - Legal extension (well-formedness)Legal extension (well-formedness)

[macrobiotic(sobanito) : *p] [macrobiotic(sobanito) : *p]

- Propositions-as-types (beliefs)Propositions-as-types (beliefs)[p17 : macrobiotic(sobanito)][p17 : macrobiotic(sobanito)]

From Natural Language From Natural Language to Type Theory to Behaviorto Type Theory to Behavior

structuralanalysis

pragmatic processing

semantic interpretation

Natural language

ULF

Type Theory

Behavior

grammar

lexicon

conceptuallexicon context

information

error/help

DenK-architectureDenK-architecture

EvaluationNL-parser

NL-production

Contextualinterpreter

Responsegenerator

Pending stack

Sharedbeliefs

Private beliefs

User

Taskdomain

CEA

CC-interface

HC-interface

Context Types in DenKContext Types in DenK

- Physical ContextPhysical Context- Private BeliefPrivate Belief- Shared BeliefShared Belief- DialogueDialogue- Pending StackPending Stack- SentenceSentence

User

Taskdomain

CEA

Context 1Context 1The dialogueThe dialogue

U: U: Turn on the microscope!Turn on the microscope!

C: C: OKOK

U: U: Which button is on?Which button is on?

C: C: The blue oneThe blue one..

U: U: Turn Turn itit off! off!

- ‘‘itit’ refers to the last introduced ’ refers to the last introduced buttonbutton

U: U: Is Is the button next to the the button next to the start-buttonstart-button on? on?

C: C: YesYes

U: U: Turn Turn itit off! off!

- Two objects: button_1 and button_2Two objects: button_1 and button_2- ‘‘itit’ refers not to the last introduced ’ refers not to the last introduced

button!button!- ‘‘itit’ refers to the last resolved button.’ refers to the last resolved button.

Context 2Context 2The dialogueThe dialogue

Context 3Context 3The dialogueThe dialogue

U: U: Is Is the red buttonthe red button on? on?

C: C: YesYes

U1: U1: Turn Turn itit/that/this off!/that/this off!

U2: U2: Turn Turn thatthat/this one off!/this one off!

U3: U3: Turn Turn the/that/this buttonthe/that/this button off! off!

- The object can be in different contextsThe object can be in different contexts- NL provides information about context NL provides information about context

searchsearch

Context 4 Context 4 Beliefs of system Beliefs of system

Ontological Ontological

U1:U1: Can you restart the internet? Can you restart the internet?S1:S1: It is impossible to restart the internet. It is impossible to restart the internet.

U2:U2: Is this Is this the enginethe engine of that car? of that car?S2:S2: Yes. Yes.U2:U2: Is this the key that starts it? Is this the key that starts it?S2:S2: Yes. Yes.U2:U2: Does Does itit runrun smoothly? smoothly?

Context 5Context 5 Goals Goals

Remove Remove the black blockthe black block!!

Context 6Context 6 Physical domain Physical domain

Remove Remove this onethis one!!

ConclusionsConclusions Use Use linguisticlinguistic component (not only icons and component (not only icons and

menus) menus) This component should have knowledge about:This component should have knowledge about:

– the task domain (ontology, private beliefs)the task domain (ontology, private beliefs)– the user (shared beliefs, dialogue)the user (shared beliefs, dialogue)– general communication principles general communication principles

Use Use contextcontext for interpretation and production of for interpretation and production of messagesmessages

The knowledge to build this component partly The knowledge to build this component partly exists (but is distributed):exists (but is distributed):– parsers (from language to mental states)parsers (from language to mental states)– mental states in Type Theory (to model belief states)mental states in Type Theory (to model belief states)– Gricean rules for communication (in progress)Gricean rules for communication (in progress)

In iCat many results from DenK can be reusedIn iCat many results from DenK can be reused