A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

Nederlandse Organisatie voor Wetenschappelijk Onderzoek

LREC 2008, Marrakech, Morocco

Hennie BrugmanMPI for Psycholinguistics, Nijmegen, Netherlands Véronique MalaiséFree University, Amsterdam, Netherlands

Laura HollinkFree University, Amsterdam, Netherlands

Overview

– The CATCH programme and its annotation requirements– Existing models– Annotation Meta Model (AMM) and its application to CATCH cases– Software and infrastructure– Conclusions

CATCH – Continuous Access To Cultural Heritage

– Dutch research program funded by NWO– Apply state of the art methods to the construction and

exploitation of digital collections of large Cultural Heritage institutions

– Currently 10 projects, hosted by Cultural Heritage institutions– Rijksmuseum Amsterdam, Dutch National Archive, Dutch National Library, Netherlands

Institute for Sound and Vision, etc

– Results and software applicable across institutions and collections

Annotations in CATCH

• Objectives:

- Cross media, cross collection, cross institution annotation of digital objects and segments of objects

- Add new layers of annotation to existing annotations

- Centralize storage and exploitation of annotations generated by CATCH projects

- Apply and showcase annotation recommendation modules/services from several CATCH projects

“some text description”Resource

Primitive value (string, date, number,…)

http://www.beeldengeluid.nl/GTAA#Subject_kunst_

Semantic value

property

“Abraham van Beijeren”

Resource

• complete resource

• catalog, metadata

• resource types

• images

• text

• html, xml

• audio

• video

artist

“very much”Resource segment – sound-video

choral:transcription

“roemer”

Resource segment - image

racm-glass:Shape

“boven eener verloting te Amsterdam”

Resource segment – scanned handwriting

scratch:transcription

Resource segment – text

scratch:transcription

http://geonames.org/NL/Amsterdam

choice:location

“bommenwerpers boven de hoofdstad”

http://TGN/Amsterdamhttp://TGN/NL

– Further requirements for the annotation formalism– Project and media specific extensions– Predefined annotation schemes– Generic and specific queries possible– Expressive and simple– Reuse or include existing annotation models or vocabularies

Existing annotation models

– Semantic web community– Focus on (semantic) annotation values– Anchoring mainly to complete resources or web pages

– Linguistic annotation community– Anchoring to text or time series– Usually no semantic values

– Media industry (e.g. MPEG-7)

– Objections– Not all media types covered– Too complex or specialized– Hardly ever annotation of annotations, and of segments of annotation

values

Annotation Meta Model (AMM) – why RDF?

– RDF, RDFS and OWL seem good modeling languages for the domain of annotation - graphs versus hierarchies

– Some of our requirements automatically met:– Class and property inheritance– Constraints (e.g. domains and ranges for properties)– Integration of semantic values– Classes and instances for annotation schemes and annotation

resp.– General and specific queries

AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit

AnnotatableObject rdfs:type

Unit rdfs:type

Coordinates rdfs:type

AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit

Unit rdfs:type

AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit

Unit rdfs:type

AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit

Unit rdfs:type

AMM – Core model

hasCoordinates

anchorsTo

feature

hasUnit

Unit rdfs:type

AMM – Core model

AnnotatableObject

Coordinates

TimeSeriesObjectTextObject ImageObject

TimeSegmentRegion2DTextSpan

AMM – Scanned handwriting case

“in Amsterdam is het alle dagen feest”

image1 amm:addressesRegion

amm:anchorsTo

(454,107,110,204)

“handwriting.jpg”

amm:anchorsTo

(beginNode, endNode)

http://www.geonames.org/places#Amsterdam

amm:addressesTextSpan

hasText

image2 dc:title

hw:transcription

hw:location “Amsterdam”

hasText

“pixels” amm:hasUnit

AnnotatableObject

Coordinates

amm:anchorsTo

(454,107,110,204)

amm:anchorsTo

hasText

image2 dc:title

hw:transcription

hasText

AnnotatableObject

Coordinates

amm:anchorsTo

(454,107,110,204)

amm:anchorsTo

hasText

image2 dc:title

hw:transcription

hasText

AnnotatableObject

Coordinates

amm:anchorsTo

(454,107,110,204)

amm:anchorsTo

hasText

image2 dc:title

hw:transcription

hasText

AnnotatableObject

Coordinates

amm:anchorsTo

(454,107,110,204)

amm:anchorsTo

hasText

image2 dc:title

hw:transcription

hasText

AnnotatableObject

Coordinates

AMM – Other cases

– Semantic annotations of segments of text documents– Manually annotated image regions– Complex linguistic annotation of co-occurring speech and gesture– Syntactic annotation of text

Software and infrastructure

– RDF repository, wrapped with AMM web service– Stores AMM model, project-specific annotation schemes and annotation data

– Java API, defining and implementing this web service– Clients:

– CHOICE@CATCH documentalist support system– Integrated multimedia and web based “Annotation and

Recommendation” demonstrator for CATCH

CHOICE Documentalist Support System

Annotation and Recommendation demonstrator

Conclusions

– All requirements seem to be met– Applicable to a wide range of very different cases– Repository works efficiently, however not yet tested with a large

number of AnnotatableObjects (so far, approx. 50.000)– Highlights:

– Layered annotation– All media types are or can be supported– Annotation with multimedia objects or object segments possible

Thank you

Instances: CHOICE text annotation

“http://www.beeldengeluid.nl/Thesaurus/GTAASkosv7.owl#”

“Subject”

a1addressesTextSpan

partOf

(n1, n2)

“AndereTijdenGemmeker.txt”

apoldaSubject

apoldaOntology

Subject_bevelhebbers_

apoldaIdentifier

r1dc:title

Instances: video/audio annotation

“dit is gesproken tekst”

addressesTimeInterval

partOf

“gesturing_people.mpg”

transcription

textObjecta2partOf

partOfSpeech(beginNode, endNode)

http://www.isodatcats.org/part-of-speech#Noun

hasTime

“3521”

“4692”

addressesTextSpan

“some gesture description”

a3addressesTimeInterval

partOf

“gesturing_people.mpg”

hasTime

“3854”

“5290”

handshape

http://www.mpi.nl/myShapes#fist

dc:description

dc:title

Instances: syntax tree on text

addressesTextSpan

partOf

(n0, n3)

“german sentence 1”r1dc:title

“Der Mann geht schnell”hasText

partOf

partOfpartOf

“Der”hasText

“Mann”

hasText

(n4, n8)

(n9, n13)

(n14, n21)

addressesTS

“geht”

“schnell”

syntaxD

syntax

N syntax

syntax

anchorsToanchorsToNP

anchorsTo

anchorsTo anchorsToanchorsTo

syntax

A Common Multimedia Annotation Framework for Cross Linking Cultural Heritage Digital Collections

Documents

Semantic annotation and search of large virtual heritage collections

Annotation consistency using annotation intersections

Linking Structured Data - microsoft.com Linking Structured Data Michael Hausenblas ... 2.3 Enabling commercial use of BL collections ... object tuples that represent the individual

Linking biodiversity data with the Biological Collections Ontology Ramona Walls (iPlant Collaborative, University of Arizona) John Deck (University of

English PropBank Annotation Guidelinesverbs.colorado.edu/~mpalmer/projects/ace/EPB-annotation...Chapter 1 Verb Annotation Instructions 1.1 PropBank Annotation Goals PropBank is a corpus

Linking collections to related resources: Multi-scale, multi-dimensional, multi- disciplinary collaborative research in biodiversity. Is this a “Big Data”

Ariadne: Semantic Annotation and Linking

Automated annotation of Orthoptera songs: first results ... · Krusch 2000). Taxonomic and behavioral studies were successfully combined, and resulted in vast collections of sound

PhD Hub Semantic annotation and linking of scientific ...€¦ · (SKOS). SKOS is an RDF vocabulary, W3C recommended, designed for representation of thesauri, classification schemes,

Graph based semantic annotation for enriching documents with … · 2014-09-30 · "Semantic Linking of Learning Object Repositories to Dbpedia”. Manuel Lama, Juan Carlos Vidal,

LinkOut Kernel: keys to linking your collections to PubMed Medical Library Association 2007 Philadelphia Lidia Hutcherson, NCBI/NLM

A Scalable Active Framework for Region Annotation in 3D ...ericyi/papers/part... · A Scalable Active Framework for Region Annotation in 3D Shape Collections Supplemental material

I Annotation - Webdamwebdam.inria.fr/wordpress/wp-content/uploads/2013/10/peter.pdf · databases, various forms of provenance ... "Annotation is the linking of a new commentary node

NCBI’s Genome Annotation: Overview Incremental processing Re-annotation ( batch ) Post-annotation review Case studies NOTE: limiting discussion to annotation

D15.3 Report on Semantic Annotation and Linking - ARIADNE PLUSlegacy.ariadne-infrastructure.eu/wp-content/uploads/2019/07/ARIAD… · ARIADNE D15.1 gives an overview of vocabulary

Review Free EGASP: the human ENCODE Genome Annotation ...repository.cshl.edu/25307/1/...Genome-Annotation...Automatic genome annotation methods To date, accurate automatic annotation

Annotation - 碁峰資訊epaper.gotop.com.tw/pdf/A155.pdf · Retention meta-annotation Java annotation annotation class class Java virtual machine annotation class class annotation

Annotation. Traditional genome annotation BLAST Similarities

A Scalable Active Framework for Region Annotation in 3D ...ericyi/papers/part_annotation_16_small.pdfA Scalable Active Framework for Region Annotation in 3D Shape Collections Li Yi1

PhD Hub Semantic annotation and linking of scientific ... Semantic... · Semantic annotation and linking of scientific Classifications Authors Charalampos Bratsas, Sotirios Karampatakis,