View
218
Download
0
Category
Preview:
Citation preview
Daniel Sonntag | 1
LREC 2010Speech Grammars for Textual Entailment Patterns in Multimodal QuestionAnswering
Daniel Sonntag, Bogdan Sacaleanu, DFKI21/05/2010
Daniel Sonntag | 2 | 2
Outline
» Semantic Dialogue Shell
» Textual Entailment
» Processing Example
» Conclusions
Daniel Sonntag | 3
Acknowledgements
»Thanks go out to Robert Nesselrath, Yajing Zang, Günter Neumann, Matthieu Deru, Simon Bergweiler, Gerhard Sonnenberg, Norbert Reithinger, Gerd Herzog, Alassane Ndiaye, Tilman Becker, Norbert Pfleger, Alexander Pfalzgraf, Jan Schehl, Jochen Steigner, and Colette Weihrauch for the implementation and evaluation of the dialogue infrastructure.
RobustQuestion
Understanding
Ease the interface to external third-party
components.
SPARQL
Daniel Sonntag | 4 | 4
Semantic Dialogue Shell
Daniel Sonntag | 5
Dialogue Shell Workflow
SpeechInterpretation
TextInterpretation
GestureInterpretation
GraphicGeneration
TextGeneration
SpeechInterpretation
ModalityFusion
PresentationPlanning
Pe
rso
na
lis
ati
on Dialogue
and
Interaction
Management
InteractiveSemanticMediator
InteractiveServiceCompo-sition
eTFS/SPARQL
SPARQL
SPARQL
SPARQL
OWL-API
Visualisation VisualisationService
ExternalInformation
Sources
Semantiic (Meta)Services
RDFKOIOS(Yago
Ontology)
RemoteLinked Data
Services
OWLAOIDE(Music
Ontology)
TextSummarisation
- Domain Model- Context Model- User Model
Daniel Sonntag | 6
THESEUS’s Semantic Dialogue Shell: Goals and Requirements
»Multimodal interaction with the Semantic Web and the Internet of Services
»Components customisable to different use case scenarios
»Flexible adaptation to» Input and output modalities» Interaction devices» Knowledge bases
»To understand a greater number ofqueries:
» Robust question understanding (NLU) when using both speech and written text input
» Semantic (i.e., a RDF or OWL based) query interpretation » The combination of robust question understanding and
ontology-based answer retrieval
Daniel Sonntag | 7
SPARQL Query Editor
SPARQL
Daniel Sonntag | 8
Daniel Sonntag | 9
Speech Grammar
<utterance name="SHOW_CV_OF_PERSON"> <phrases> <phrase>zeige ?mir den [werdegang lebenslauf] [von zu] PERSON</phrase> <phrase>sage ?mir mehr über den [werdegang lebenslauf] von PERSON</phrase> <phrase>wie ist der [werdegang lebenslauf] von PERSON</phrase> </phrases> <semantic-interpretation> <object type="odp#TaskRequest"> <slot name="odp#fusion-confidence"> <value type="Float">1.0</value> </slot> <slot name="odp#hasContent"> <object type="dialogshell#BackendRetrievalTask"> <slot name="dialogmanager#backendComponent"> <value type="String">mediator:summarizer</value> </slot> <slot name="odp#hasContent"> <variable name="PERSON"/> </slot> </object> </slot> </object> </semantic-interpretation> </utterance>
Daniel Sonntag | 10 | 10
Textual Entailment
Our idea is that an NLU grammar for speech input can be reused to build more robust multimodal text-
based question understanding by automatically generating textual entailment patterns.
Daniel Sonntag | 11
Textual Entailment & Information Access
Request
Method1
Method2
Information (RDF)
Implicit Mapping
Method1: Speech / Semantic Grammars
• RDF/OWL reasoning
Method2: RTE• textual reasoning
Ontology (RDF/OWL)
Conceptual
Textual
Reasoning
Daniel Sonntag | 12
Textual Entailment through Alignments
» For textual entailment to hold we need:» text AND background knowledge hypothesis» but background knowledge should not entail hypothesis alone
»Background Knowledge» Unsupervised acquisition of linguistic and world knowledge from
general corpora and web» Acquiring larger entailment corpora» Manual resources and knowledge engineering
» Alignment-based TE and Background Knowledge» Preprocessing: POS, morphology, cognates» Representation: bag-of-words» Knowledge Sources: WordNet, Roget‘s Thesaurus, Wehrle
Thesaurus
Daniel Sonntag | 13
Argumentation
» Input modalities are usually interpreted according to separate models and aligned to a shared model (often coarse-grained).
»Present a method of interpretation based on a common model (propagated changes to multiple modalities).
»Built on the grammar for speech inputs = Leveraging Existing Speech GrammarKnowledge
Daniel Sonntag | 14 | 14
Processing Example
Daniel Sonntag | 15
Entailment Patterns and Possible Hypotheses
Daniel Sonntag | 16
Entailment Patterns and Alignment Engine
»Association-based word alignment. Three steps:
» lexical segmentation, when boundaries of lexical items are identified;
» correspondence, when possible similarities are suggested in line with some correspondence measures;
» alignment, when the most likely semantically similar word is chosen.
Daniel Sonntag | 17
Entailment Patterns and Alignment Techniques
»Question: What is the birthplace of Angela Merkel?
»Pattern: Where is Angela Merckel born?
»Filters on a full alignment.
Daniel Sonntag | 18
Entailment Patterns and Alignment Techniques
POS Filter: Exclude unlikely alignments based on POS. Allow for the additional mappings: verb to noun (i.e., born vs. birthplace)Lexical Semantic Resource Filter: WordNet (synonyms); Roget Thesaurus (conceptually related words)String Similarity Filter: Dice coefficient, Longest common subsequence ratio; submatches, misspellings
System of weights: nouns, verbs, and adjectives are better scored than function words.
Daniel Sonntag | 19
Dialogue Example
» (1) U: “Open my personal address book. What do you know about Claudia?”
» (2) S: “There’s an entry: Claudia Schwartz. The personal details are shown below. She lives in Berlin.” + Google Map Display of street coordinates.
» (3) U: “Which is Claudia’s favorite kind of music? Do you know the bands she likes most?”
» (4) S: “Nelly Furtado” + Displays videos obtained from YouTube. (Rest API)
» (5) U: “How did experts rate her last album?”» (6) S: Shows an expert review according to the BBC Linked Data Set.» (7) U: “Show me other news.”» (8) S: Opens a browser + Text field and a new agency Internet page
(featuring Angela Merkel)» (9) U writes: “Where was Angela Merkel born? / In which town was
Angela Merkel born?” etc.» (10) S: “She was born in Hamburg.”» (11) U speaks again: “And Barack Obama?”» (12) S: “He was born in Honolulu.”» (13) U: “Show me Angela Merkel’s career.”
Daniel Sonntag | 20
Touchscreen Installation
Daniel Sonntag | 21
Image Analysis in Biomedicine MEDICO
Retrieval and examination of 2D picture series
Daniel Sonntag | 22 | 22
Conclusions
Daniel Sonntag | 23 | 23
Conclusions
»We described a multimodal dialogue shell for QA and focussed on the robust multimodal question understanding task.
»The textual interpretation is based on automatically generated textual entailment patterns.
»As a result, we can deal with written text input and different surface forms more flexibly according to the derived entailment patterns.
Daniel Sonntag | 24
Method Comparison
» Method 1: Speech Grammars» Speech grammars are verbose» Requires full coverage of expected input» Hard-coded reasoning in rules» Example:
» Show me all pictures of X.» What pictures does X have?» Show me all images of X.
» Method 2: NLU Grammars» Use of Textual Entailment» NLU grammars are compact» Requires partial coverage of possible input» Example:
» Show me all pictures of X.» Entailed utterances:
» What pictures does X have?» Show me all images of X.
2
1
Recommended