Ariadne Harvesting

ARIADNE K.U.Leuvenwww.ariadne-eu.org www.cs.kuleuven.be/~hmdb

HARVESTING METADATA IN REPOSITORY

NETWORKS

Bram VandeputteK.U.LeuvenARIADNE

11

http://www.ariadne-eu.org


http://www.cs.kuleuven.be/~hmdb



• Working since 2006 with Erik Duval

• ARIADNE harvester, OAI-PMH target support, validation service, etc.

• International projects : MACE, MELT, Globe, Ariadne, Aspect, iCoper,...

• Other interests : Information visualisation, Interoperability, TEL

2

WHO AM I

2






• OAI-PMH

• Validation Service

• Harvester Infrastructure

• Projects

• MACE

• MELT

• GLOBE

• OAI-PMH Target3

Overview

3






• Server - Client model

• Pull mechanism

• options :

• selective harvesting

• incremental harvesting

4

OAI-PMH

4






• Verbs : Identify, ListRecords, GetRecord

• Parameters :

• baseUrl

• from & until date

• metadataPrefix

• sets

5

OAI-PMH

5






• Interoperability : Application Profile (AP)

• Manual check : very time consuming

• Need a tool for enforcing an AP => validation scheme

• A set of validation rules

• Reusable

6

Validation Service

6






7

Validation ServiceLOM loose

MELTv1.0/minimal

MELTv1.0/full

lomloose.xsd

conditional fields check

LRE taxon field numerical

vcard validator

empty attribute fields

MACEv1.0

classification vocab service

melt.xsd mace.xsd

Legend

uses

extends

MACEv1.0

vcard validator

validationScheme

validation component

7






88






The ARIADNE Harvester Infrastructure

88






ARIADNE Harvester

harvester log

88






ARIADNE Harvester

validation serviceharvester log

ARIADNE Repository

8

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

8

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

8

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

8

LOMLOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

8

LOM

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

Validation Msg

8

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

Validation Msg

8

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

Validation Msg

8

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

Validation Msg

8

LOM

SQISPI

8






OAI

External Repository

LOMLOM

OAI-PMH

LOM

LOMLOM

OAI-PMH

ARIADNE Harvester


ARIADNE Repository

Validation Msg

9

LOM

LOM

SQISPI

LOMLOM

OAI-PMH

LOM

Validation Msg

LOM

LOM

Validation Msg

1 2

34

5

6

9






• MACE

• MELT

• GLOBE

10

Harvesting in networks

10






• Metadata for Architectural Contents in Europe

• www.mace-project.eu

• Harvesting, Searching, Enriching, (geo-) Tagging, attention metadata

11

MACE

11





http://www.mace-project.eu

http://www.mace-project.eu


Harvestedmetadata

WINDS

IRB

DYNAMO

ARIADNE Harvester

OAI-PMH

OAI-PMH

OAI-PMH

SPI

Enrichedmetadata

MACEmetadata

Uses Lucene for index search

CuminCAD

OAI-PMH

Validation service

SPI

SPI

12

MACE

12






13

MACE

13






• MELT : a Metadata Ecology for Learning and Teaching

• Ministries of education across europe

• enriching, translating, better quality metadata

• practical test for scalability of harvester framework

14

MELT

14






15

MELT

MEC

MESC

MSU

NCTE

ARIADNE Harvester

Validation service

BMBWK

C-H

CFL

EDUCATIO

EUN

FWU

INDIRE

KUL

RVP

TLF

UL-FRI

XTECO

AI-

PM

H

SPI

ARIADNEMETADATASTORE

OAI

SQIARIADNE Harvester

(EUN)

EUN MDS

Enrichment Toolkit

Multilingual Thesaurus

Vocabulary Registry

EUN AMG

SQI

15






1616






• Global Learning Objects Brokered Exchange

• Worldwide network of networks of repositories

• => Both SQI and OAI-PMH are used on various levels in the network

17

GLOBE

17






18

GLOBEMEC

MESC

MSU

NCTE

ARIADNE Harvester

BMBWK

C-H

CFL

EDUCATIO

EUN

FWU

INDIRE

KUL

RVP

TLF

UL-FRI

XTEC

OA

I-P

MH

WINDS

IRB

DYNAMOARIADNE Harvester

CuminCAD

MELT

OAI

SQI

OA

I-P

MH

Federated

Search

EdNA CGIAR Pubelo Merlot

III COSL LORNET Ariadne

LACLO NIME EUN

ISKME KERIS

SQI UDDI

Registry

MACE

OAI

SQI

SQI SQI SQI SQI

SQI SQI SQI SQI

SQI Client

SQI

OAI

SQI

OAI

SQI

OAI

SQI

OAI

SQI

OAI

SQI

ARIADNE Harvester

GLOBE

OAI

SQI

SPI

18






1919






20

OAI-PMH target

20






21

OAI-PMH result

DYNAMO LOM metadataDYNAMO LOM

metadataDYNAMO LOM metadataRepository LOM

metadata

ARIADNEHarvester

1. Get metadata

out of database

Repository metadata

- identifier- title- url- project description- ...

Repository LOM metadata

- lom.general.identifier- lom.general.title- lom.general.description- lom.technical.location- ...

2. Mapping Process

4. Serve results

3. Copy results in

OAI-PMH service

OAI-PMH Target Software

Repository

21






• Questions ?

22

どうも　ありがとうございます！

22






• SPI : http://ariadne.cs.kuleuven.be/lomi/index.php/SimplePublishingInterface

• SQI : http://ariadne.cs.kuleuven.be/lomi/index.php/LorInteroperability

• IEEE LOM : http://ltsc.ieee.org/wg12/

• OAI-PMH : http://www.openarchives.org/

• OAI-PMH target wiki :http://ariadne.cs.kuleuven.be/lomi/index.php/Setting_Up_OAI-PMH

23

References

23





Education

Ariadne Harvesting