A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

Nederlandse Organisatie voor Wetenschappelijk Onderzoek

A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

LREC 2008, Marrakech, Morocco

Hennie BrugmanMPI for Psycholinguistics, Nijmegen, Netherlands Véronique MalaiséFree University, Amsterdam, Netherlands

Laura HollinkFree University, Amsterdam, Netherlands


Overview

– The CATCH programme and its annotation requirements– Existing models– Annotation Meta Model (AMM) and its application to CATCH cases– Software and infrastructure– Conclusions


CATCH – Continuous Access To Cultural Heritage

– Dutch research program funded by NWO– Apply state of the art methods to the construction and

exploitation of digital collections of large Cultural Heritage institutions

– Currently 10 projects, hosted by Cultural Heritage institutions– Rijksmuseum Amsterdam, Dutch National Archive, Dutch National Library, Netherlands

Institute for Sound and Vision, etc

– Results and software applicable across institutions and collections


Annotations in CATCH

• Objectives:

- Cross media, cross collection, cross institution annotation of digital objects and segments of objects

- Add new layers of annotation to existing annotations

- Centralize storage and exploitation of annotations generated by CATCH projects

- Apply and showcase annotation recommendation modules/services from several CATCH projects



“some text description”Resource

Primitive value (string, date, number,…)

http://www.beeldengeluid.nl/GTAA#Subject_kunst_

Semantic value

property

property



“Abraham van Beijeren”

Resource

• complete resource

• catalog, metadata

• resource types

• images

• text

• html, xml

• audio

• video

artist



“very much”Resource segment – sound-video

choral:transcription



“roemer”

Resource segment - image

racm-glass:Shape



“boven eener verloting te Amsterdam”

Resource segment – scanned handwriting

scratch:transcription




Resource segment – text

scratch:transcription

http://geonames.org/NL/Amsterdam

choice:location






“bommenwerpers boven de hoofdstad”






“bommenwerpers boven de hoofdstad”

http://TGN/Amsterdamhttp://TGN/NL



– Further requirements for the annotation formalism– Project and media specific extensions– Predefined annotation schemes– Generic and specific queries possible– Expressive and simple– Reuse or include existing annotation models or vocabularies


Existing annotation models

– Semantic web community– Focus on (semantic) annotation values– Anchoring mainly to complete resources or web pages

– Linguistic annotation community– Anchoring to text or time series– Usually no semantic values

– Media industry (e.g. MPEG-7)

– Objections– Not all media types covered– Too complex or specialized– Hardly ever annotation of annotations, and of segments of annotation

values


Annotation Meta Model (AMM) – why RDF?

– RDF, RDFS and OWL seem good modeling languages for the domain of annotation - graphs versus hierarchies

– Some of our requirements automatically met:– Class and property inheritance– Constraints (e.g. domains and ranges for properties)– Integration of semantic values– Classes and instances for annotation schemes and annotation

resp.– General and specific queries


AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit

AnnotatableObject rdfs:type


Unit rdfs:type

Coordinates rdfs:type


AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit



Unit rdfs:type



AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit



Unit rdfs:type



AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit



Unit rdfs:type



AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit



Unit rdfs:type



AMM – Core model

AnnotatableObject

Coordinates

TimeSeriesObjectTextObject ImageObject

TimeSegmentRegion2DTextSpan


AMM – Scanned handwriting case

ann

“in Amsterdam is het alle dagen feest”

image1 amm:addressesRegion

amm:anchorsTo

(454,107,110,204)

“handwriting.jpg”

text1

text2

amm:anchorsTo

(beginNode, endNode)

http://www.geonames.org/places#Amsterdam

amm:addressesTextSpan

hasText

image2 dc:title

hw:transcription

hw:location “Amsterdam”

hasText

“pixels” amm:hasUnit

AnnotatableObject

Coordinates



ann



amm:anchorsTo

(454,107,110,204)


text1

text2

amm:anchorsTo




hasText

image2 dc:title

hw:transcription


hasText


AnnotatableObject

Coordinates



ann



amm:anchorsTo

(454,107,110,204)


text1

text2

amm:anchorsTo




hasText

image2 dc:title

hw:transcription


hasText


AnnotatableObject

Coordinates



ann



amm:anchorsTo

(454,107,110,204)


text1

text2

amm:anchorsTo




hasText

image2 dc:title

hw:transcription


hasText


AnnotatableObject

Coordinates



ann



amm:anchorsTo

(454,107,110,204)


text1

text2

amm:anchorsTo




hasText

image2 dc:title

hw:transcription


hasText


AnnotatableObject

Coordinates


AMM – Other cases

– Semantic annotations of segments of text documents– Manually annotated image regions– Complex linguistic annotation of co-occurring speech and gesture– Syntactic annotation of text


Software and infrastructure

– RDF repository, wrapped with AMM web service– Stores AMM model, project-specific annotation schemes and annotation data

– Java API, defining and implementing this web service– Clients:

– [email protected] documentalist support system– Integrated multimedia and web based “Annotation and

Recommendation” demonstrator for CATCH


CHOICE Documentalist Support System


Annotation and Recommendation demonstrator


Conclusions

– All requirements seem to be met– Applicable to a wide range of very different cases– Repository works efficiently, however not yet tested with a large

number of AnnotatableObjects (so far, approx. 50.000)– Highlights:

– Layered annotation– All media types are or can be supported– Annotation with multimedia objects or object segments possible


Thank you


Instances: CHOICE text annotation

“http://www.beeldengeluid.nl/Thesaurus/GTAASkosv7.owl#”

“Subject”

a1addressesTextSpan

partOf

(n1, n2)

“AndereTijdenGemmeker.txt”

apoldaSubject

apoldaOntology

Subject_bevelhebbers_

apoldaIdentifier

r1dc:title


Instances: video/audio annotation

“dit is gesproken tekst”

a1

addressesTimeInterval

partOf

t1

“gesturing_people.mpg”

transcription

textObjecta2partOf

partOfSpeech(beginNode, endNode)

http://www.isodatcats.org/part-of-speech#Noun

t2

hasTime

hasTime

“3521”

“4692”

addressesTextSpan

“some gesture description”

a3addressesTimeInterval

partOf

t3

“gesturing_people.mpg”

t4

hasTime

hasTime

“3854”

“5290”

handshape

http://www.mpi.nl/myShapes#fist

next

dc:description

dc:title

dc:title


Instances: syntax tree on text

a1

addressesTextSpan

partOf

(n0, n3)

“german sentence 1”r1dc:title

“Der Mann geht schnell”hasText

a2

a3

a4

partOf

partOfpartOf

“Der”hasText

“Mann”

hasText

(n4, n8)

(n9, n13)

(n14, n21)

addressesTS

addressesTS

addressesTS

“geht”

“schnell”

syntaxD

syntax

N syntax

V

syntax

Adv

NP1

VP1

S1

anchorsToanchorsToNP

SVP

anchorsTo

anchorsTo anchorsToanchorsTo

syntax

syntax

syntax

A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

annotation of annotations

annotation formalismproject

domain of annotation

cross media

catch projectsannotations

catch objectives

catch projectsapply

cross collection

Text of A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

Connecting the Dots: Linking Digitized Collections Across Metadata Silos

Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks

Semantic annotation and search of large virtual heritage collections

I Annotation - Webdamwebdam.inria.fr/wordpress/wp-content/uploads/2013/10/peter.pdf · databases, various forms of provenance ... "Annotation is the linking of a new commentary node

Review Free EGASP: the human ENCODE Genome Annotation ...repository.cshl.edu/25307/1/...Genome-Annotation...Automatic genome annotation methods To date, accurate automatic annotation

Linking Born-Digital News and Social Media Collections via Automated Entity Detection and Authority Matching

Automated annotation of Orthoptera songs: first results ... · Krusch 2000). Taxonomic and behavioral studies were successfully combined, and resulted in vast collections of sound

Open Annotation: Social Bookmarking and Annotation of eBooks

LinkOut Kernel: keys to linking your collections to PubMed

Collections for Automatic Image Annotation and Photo Tag Recommendation Philip McParlane, Yashar Moshfeghi and Joemon M. Jose University of Glasgow, UK

Design of Embedded DSP Processors - Linköping · PDF fileParse tree optimization ... bb cost table Annotation Static analysis Annotated parse tree Code generation Linking Probe lib

English PropBank Annotation Guidelinesverbs.colorado.edu/~mpalmer/projects/ace/EPB-annotation...Chapter 1 Verb Annotation Instructions 1.1 PropBank Annotation Goals PropBank is a corpus

Linking the International Wheat Genome Sequencing ... · Eduard Akhunov IWGSC Survey sequence v3 assembly and annotation Scaffolds, gene models Download, BLAST, and browser Andy Sharpe,

1 Reference Linking in Project Euclid …with some thoughts on the preservation of digital collections. A presentation at the Workshop on Linking and searching

Linking Structured Data - microsoft.com Linking Structured Data Michael Hausenblas ... 2.3 Enabling commercial use of BL collections ... object tuples that represent the individual

NeoVision 2 Annotation Guidelines - Welcome to iLab!ilab.usc.edu/neo2/dataset/neovision2-annotation-guidelines.pdf11. Annotation Quality Assurance Quality is assured throughout annotation

Linking collections to related resources: Multi-scale, multi-dimensional, multi- disciplinary collaborative research in biodiversity. Is this a “Big Data”

Graph based semantic annotation for enriching documents with … · 2014-09-30 · "Semantic Linking of Learning Object Repositories to Dbpedia”. Manuel Lama, Juan Carlos Vidal,

Annotation consistency using annotation intersections

CEVO: Comprehensive EVent Ontology Enhancing Cognitive Annotation … · 2018-10-05 · Event Classiﬁcation LODE: An ontology for Linking Open Descriptions of Events3 deﬁnes a

Medical and Health Sciences Collections & User Servicesmed.fau.edu/faculty/Preceptor_Brochure_2019-2020.pdf · Med’s routine linking out to available “free articles” from PubMed

English PropBank Annotation Guidelinesverbs.colorado.edu/propbank/EPB-Annotation-Guidelines.pdfChapter 1 Verb Annotation Instructions 1.1 PropBank Annotation Goals PropBank is a corpus

Semantic Annotation of Image Collectionslaurah/papers/Hollink_saic.pdf · 2003-12-02 · Semantic Annotation of Image Collections Laura Hollink 1Guus Schreiber Jan Wielemaker2 Bob

Linking biodiversity data with the Biological Collections Ontology Ramona Walls (iPlant Collaborative, University of Arizona) John Deck (University of

Session 3: Audio editing, Hardware, & time-linking · Session 3: Audio editing, Hardware, ... (taught here), Eclipse, EXMARALDA . June 24-27, 2008 & June 30 ... one annotation tier

D15.3 Report on Semantic Annotation and Linking - ARIADNE PLUSlegacy.ariadne-infrastructure.eu/wp-content/uploads/2019/07/ARIAD… · ARIADNE D15.1 gives an overview of vocabulary

Pathema Burkholderia Annotation Jamboree: Introduction to Annotation Jamboree

Semantic Annotation of Image Collections

Linking Collections Through Linked Open Data

Classifications Semantic annotation and linking of scientific · SKOS Simple Knowledge Organization System ... there are some overlaps between classifications of relative scientific

Documents

A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

Text of A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections