DenK and iCatDenK and iCatTwo Projects on Cooperative Two Projects on Cooperative Electronic Assistants (CEA’s)Electronic Assistants (CEA’s)
Robbert-Jan Beun, Rogier van Eijk Robbert-Jan Beun, Rogier van Eijk & Huub Pr& Huub Prüüstst
Department of Information and Computing Department of Information and Computing SciencesSciences
Utrecht University Utrecht University
rj,rogier,[email protected],rogier,[email protected]
DenKDenK(Dialoogvoering en Kennisopbouw)(Dialoogvoering en Kennisopbouw)
- Collaborative project between University Tilburg, Collaborative project between University Tilburg, Eindhoven University and IPO Eindhoven University and IPO
- 1990 – 19981990 – 1998- Fundamental principles of communicationFundamental principles of communication
Generic architectureGeneric architecture- Natural language interfaceNatural language interface- Knowledge representation Knowledge representation - Object oriented animationObject oriented animation
- Result: implementation, 8 dissertations, Result: implementation, 8 dissertations, conference proceedings, papers, reports, …conference proceedings, papers, reports, …
Triangle-paradigmTriangle-paradigm
A A human-computer interface should human-computer interface should correspond to our natural interaction in two correspond to our natural interaction in two ways:ways:
- Direct:Direct:- physical worldphysical world- perception and actionperception and action
- IndirectIndirect- with humans by languagewith humans by language- speech actsspeech acts
UserUser
direct observationdirect observationand manipulationand manipulation
TaskTaskdomaindomain
CEACEA
language/symboliclanguage/symbolic
softwaresoftwareinterfaceinterface
DenK propertiesDenK properties
- Mental representationMental representation- ContextContext- Reasoning Reasoning
- CommunicationCommunication- Typed English textTyped English text
- InteractionInteraction- Evaluation (software interface)Evaluation (software interface)
- Task domainTask domain- Electron microscope (EM)Electron microscope (EM)
UserUser
observation & observation & actionaction
TaskTaskdomaindomain
CEACEA
communicationcommunication
interactioninteraction
C2-lens
mini-condensor
specimen
screen
Spotsize
Magnification
On/Stand by
Beam Shift Y
Beam Shift X
I/D
Intensity
Focus
μP/nP
Filament
Emission
GUN
Example DenK InterfaceExample DenK Interface
A: Why is the contrast of the visible A: Why is the contrast of the visible image regulated by this aperture?image regulated by this aperture?
B: Because the diffraction image is in B: Because the diffraction image is in the OA-plane when the microscope the OA-plane when the microscope is in HM-mode.is in HM-mode.
Mental RepresentationMental RepresentationContexts in Type TheoryContexts in Type Theory
Contains Contains - ontological information - ontological information - private beliefs about application domain (EM)- private beliefs about application domain (EM)- shared beliefs about application domain (EM)- shared beliefs about application domain (EM)- dialogue history (discourse)- dialogue history (discourse)
Beliefs, Shared Beliefs & DialogueBeliefs, Shared Beliefs & DialogueDialogue ≤ Dialogue ≤ Shared beliefs Shared beliefs ≤ Beliefs≤ Beliefs
Extra context: Pending stack Extra context: Pending stack
Type TheoryType Theory
- Ontological information Ontological information - Contexts and sequentialityContexts and sequentiality
[recipe : *s][recipe : *s][sobanito : recipe][sobanito : recipe][vegetarian : recipe [vegetarian : recipe *p] *p][macrobiotic : recipe [macrobiotic : recipe *p] *p]
- Reasoning Reasoning - Legal extension (well-formedness)Legal extension (well-formedness)
[macrobiotic(sobanito) : *p] [macrobiotic(sobanito) : *p]
- Propositions-as-types (beliefs)Propositions-as-types (beliefs)[p17 : macrobiotic(sobanito)][p17 : macrobiotic(sobanito)]
From Natural Language From Natural Language to Type Theory to Behaviorto Type Theory to Behavior
structuralanalysis
pragmatic processing
semantic interpretation
Natural language
ULF
Type Theory
Behavior
grammar
lexicon
conceptuallexicon context
information
error/help
DenK-architectureDenK-architecture
EvaluationNL-parser
NL-production
Contextualinterpreter
Responsegenerator
Pending stack
Sharedbeliefs
Private beliefs
User
Taskdomain
CEA
CC-interface
HC-interface
Context Types in DenKContext Types in DenK
- Physical ContextPhysical Context- Private BeliefPrivate Belief- Shared BeliefShared Belief- DialogueDialogue- Pending StackPending Stack- SentenceSentence
User
Taskdomain
CEA
Context 1Context 1The dialogueThe dialogue
U: U: Turn on the microscope!Turn on the microscope!
C: C: OKOK
U: U: Which button is on?Which button is on?
C: C: The blue oneThe blue one..
U: U: Turn Turn itit off! off!
- ‘‘itit’ refers to the last introduced ’ refers to the last introduced buttonbutton
U: U: Is Is the button next to the the button next to the start-buttonstart-button on? on?
C: C: YesYes
U: U: Turn Turn itit off! off!
- Two objects: button_1 and button_2Two objects: button_1 and button_2- ‘‘itit’ refers not to the last introduced ’ refers not to the last introduced
button!button!- ‘‘itit’ refers to the last resolved button.’ refers to the last resolved button.
Context 2Context 2The dialogueThe dialogue
Context 3Context 3The dialogueThe dialogue
U: U: Is Is the red buttonthe red button on? on?
C: C: YesYes
U1: U1: Turn Turn itit/that/this off!/that/this off!
U2: U2: Turn Turn thatthat/this one off!/this one off!
U3: U3: Turn Turn the/that/this buttonthe/that/this button off! off!
- The object can be in different contextsThe object can be in different contexts- NL provides information about context NL provides information about context
searchsearch
Context 4 Context 4 Beliefs of system Beliefs of system
Ontological Ontological
U1:U1: Can you restart the internet? Can you restart the internet?S1:S1: It is impossible to restart the internet. It is impossible to restart the internet.
U2:U2: Is this Is this the enginethe engine of that car? of that car?S2:S2: Yes. Yes.U2:U2: Is this the key that starts it? Is this the key that starts it?S2:S2: Yes. Yes.U2:U2: Does Does itit runrun smoothly? smoothly?
Context 5Context 5 Goals Goals
Remove Remove the black blockthe black block!!
Context 6Context 6 Physical domain Physical domain
Remove Remove this onethis one!!
ConclusionsConclusions Use Use linguisticlinguistic component (not only icons and component (not only icons and
menus) menus) This component should have knowledge about:This component should have knowledge about:
– the task domain (ontology, private beliefs)the task domain (ontology, private beliefs)– the user (shared beliefs, dialogue)the user (shared beliefs, dialogue)– general communication principles general communication principles
Use Use contextcontext for interpretation and production of for interpretation and production of messagesmessages
The knowledge to build this component partly The knowledge to build this component partly exists (but is distributed):exists (but is distributed):– parsers (from language to mental states)parsers (from language to mental states)– mental states in Type Theory (to model belief states)mental states in Type Theory (to model belief states)– Gricean rules for communication (in progress)Gricean rules for communication (in progress)
In iCat many results from DenK can be reusedIn iCat many results from DenK can be reused