Upload
stella-norris
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Intelligent Information Retrieval and Presentation
with Multimedia Databases
Floris Wiesman (IKAT/UM)
Stefano Bocconi (CWI)
Boban Arsenijevic (ULCL/UL)
Yulia Bachvarova (CWI)
Nico Roos (IKAT/UM)Lambert Schomaker (AI/RUG)
2
Outline
• From search to presentation
• I2RP architecture
• Query processing & presentation generation
• Natural languange generation
• Ontology mappings
• Conclusions
3
From search to presentation
• Standard (multimedia) IR: result of query is ordered list
• Question-answering system: result of query is answer
• Our approach: result of query is multimedia presentation containing the answer
4
I2RP architecture
Presentationgenerator
Ontology agent
Semantic network
MMDB 1
MMDB n
DB 1ontology
DB nontology
Query processorqueryanswergraph Playermultimedia
presentation
Natural languagegenerator
facts text
5
6
Query processing and presentation generation
• Currently simple ‘closed’ queries
• Answers are determined from semantic network
• Rules determine which information to present
• Rules determine which modalities to use
7
Semantic network:Rembrandt’s worldChiaroscuro
Caravaggio
Caravaggist
Italy
PieterLastman
Bible Mythology
Rembrandt Saskia
Rubens
teacher teacher
Bol
is_founded_by
paints paints
inspired_by compared_to
belongs
studies
paintspaints
studies
uses
Bent Birds
Jan Lievens teacher
works_with
operated_in
Portraits
HistoryPaintings
paints
paints
paints
Night Watch
has_genre
Hendrickje
is_married
has_relation
Maria Trip
has_genre ProphetessAnna
has_genre
8
Metadata
• Search in databases by metadata, no content-based retrieval
• Metadata has to be such that:– The system can find what it looks for– The system can assemble the retrieved
information in a meaningful way– The presentation really means what was
intended: a new context is created
9
The information context
• Information items do not exist standalone, they have a context
• A presentation needs to combine the retrieved information items in a new context
• Information sources, from most structured to less structured:– Multimedia Databases (e.g., ARIA)– Digital Library (e.g, Open Archives Initiative)– The Internet
10
Narrative as a context• Example of narrative structure (Greimas):
– 6 Actants: Subject, Object, Helper, Opponent, Destinateur, Receiver
– 4 Narrative Units: Contract, Competence, Performance, Sanction
• Every character in the story plays a role in the narrative (identified by rules)– e.g. Artist biography: roles are main character, family
members, teachers, collaborators, students
• Structuring according to role– e.g. family members are grouped in private life section
11
Multimedia presentation
12
Natural-language generation
• Starts from semantic level• Semantic representation may contain:
– Participating concepts– Event structure– Temporal organization– Quantification– Relevant discourse functions
• Transforms selected meaning to natural language sentence
13
Learning ontology mappings
• The task:– Establish a mapping of concepts in ontology 2
to concepts in ontology 1
• Two steps:– Establish joint attention: which are common
instances of the ontologies?– Establish mapping: which operations map the
concepts best? (wrt joint attention)
14
Two example ontologiesOntology 1
Object
Title = Self portrait as St. Paul
Artist = Rembrandt Harmensz. van Rijn
Materials = oil on canvas
Date = 1661-1662
Ontology 2
Artefact
Title = Self portrait as the apostle St. Paul
Creator
family name = Rijn
given name = Rembrandt
Harmensz.
other name = van
Material = oil on canvas
Period
start = 1661
end = 1662
Mapping can be made by copying, splitting, and merging leaf conceptsMapping can be made by copying, splitting, and merging leaf concepts
15
Establishing joint attention
• do: – agent 1: sends instance of concept to agent 2
– agent 2: returns instance with highest proportion of words in common
• until enough common instances found above threshold
result is joint attention set: concepts of agent 1 with instances known to agent 1 & 2
16
Mapping of concepts• Mapping of concepts consists of operators:
– field n: copy leaf concept– merge s: merge leaf concepts using separator s– split s: split leaf concept at separator s– first: copy first part of split– last: copy last part of split
• Separator: none, space, colon, semicolon (,TC)• Example:artefact.period.end
field object.date, split(-), last
17
Establishing a mapping
• Search space consists of all possible mappings• Value of a mapping: number of correct mappings
on joint attention set• Mapping with highest value wins• Search is guided by proportion of words that
instances have in common• Search space is reduced by ignoring mappings
between leaf concepts with a low proportion of common words
18
Conclusions
• We have shown approach for IR from multimedia databases using:– Knowledge-based query augmentation
– Combination of IR results in a single multimedia presentation
– Natural language generation
– Automatic ontology mapping
• Parts are realized as prototypes;
to be combined in one system