Repositories, Workspaces, Web Services- some ideas -

Peter WittenburgThe Language Archive - Max Planck Institute

CLARIN Research InfrastructureNijmegen, The Netherlands

scope of workshop

• clear focus on technology and architecture issues for preservation and access

• many other issues not in focus although relevant • IPR, license issues only partially • quality of data & metadata • certification (RAC, DSA, etc)• AAI• cost aspects • etc.

• let's have interactive presentations • should be able to extract essentials

Definitions?

so simple

repository

- orange- 2010- plum

- 2010- pear- 2010- apple

- 2010

+ Metadata

repository

metadata registry

?

?dangerous since physical paths may change etc

- orange- 2010- plum

- 2010- pear- 2010- apple

- 2010

+ replication due to preservation

repository

metadata registry

repository

?

?dangerous since metadata records can be re-used metadata should be stable

transfer at physical level

- orange- 2010- plum

- 2010- pear- 2010- appel

- 2010

+ replication and PIDs

repository

metadata registry

repository

- PID4- 2010- PID3

- 2010- PID2- 2010- PID1

- URL1- URL 2

PID registry

?

dangerous: another indirection layer

transfer at physical level

access possiblewhich rights?

same access rights

- orange- 2010- plum

- 2010- pear- 2010

what about collections

repository

metadata registry

repository

- PID4- 2010- PID3

- 2010- PID2- 2010- PID1

- URL1- URL 2

PID registry

transfer at physical level

- collection- 2010

- appel- 2010

- PIDx- URL

PS:collections are dynamic

topic of high relevance

• ESFRI Task Force on Repositories (report)

• e-IRG/ESFRI Task Force on Data Management (report)

• Blue Ribbon Task Force on Sustainable Digital Preservation and Access (report)

• EC High Level Expert Group on Scientific Data (report)

• ASIS&T Summit Phoenix on Research Data and Access (slides & summary)

• T. Hey et al. The Fourth Paradigm: Data-Intensive Scientific Discovery (book)

summarizing the challenges

• how to • manage the data Tsunami • maintain data visibility • preserve the data (just seen one solution)

• protect the data integrity • ensure that we get the object we wanted• guarantee data authenticity (how to present)• maintain context and provenance information

• protect privacy and rights in complex data world • maintain trust in data

• federate repositories to (virtually) integrate data• achieve (partial) interoperability • exploit distributed data without copying

speaking about metadata harvesting

Access DataWith Extraction and Analysis, Through CatalogDirect to Partner Sites

View Information on DataThrough CatalogLink to Data at Partner Site

Search Shared Catalog

Data Mirror

MetadataCatalog

Harvester

Online Catalog

Online Analysis

speaking about architectures

speaking about federations

speaking about federations

speaking about federations

general configuration

repository A- architecture- rights domain- access paths- etc.

mirror repository X- architecture- rights domain- access paths- etc.

adapter(s)

adapter(s)

repository B- architecture- rights domain- access paths- etc.

adapters

repository C- architecture- rights domain- access paths- etc.

adapters

mirror repository Y- architecture- rights domain- access paths- etc.

adapters

mirror repository Z- architecture- rights domain- access paths- etc.

adapters

can be special does not scale

general configuration

repository A- architecture- rights domain- access paths- etc.

mirror repository X- architecture- rights domain- access paths- etc.

API

API

repository B- architecture- rights domain- access paths- etc.

API

repository C- architecture- rights domain- access paths- etc.

API

mirror repository Y- architecture- rights domain- access paths- etc.

API

mirror repository Z- architecture- rights domain- access paths- etc.

API

replication layer

generic HLEG figure

Data generators Users

Common Data Services

Community Support Services

Dat

a Cu

ratio

nUser functionalitiesData capture & transferVirtual Research Environments

Data discovery & navigationWorkflow generationAnnotation, Interpretability

Safe & persistent storageIdentifiers, Authenticity, Workflow execution, Mining

Trus

t

requirements for intermediate layer

• needs to cope with large diversity of solutions and architectures• may only minimally interfere with local repository solutions

(too much has been invested along community traditions)• needs to respect rights domains and preserve access rights

• needs to be transparent to proven utilization mechanisms • needs to operate at logical level (canonical collections)• needs to scale with number of (community) data centers

• only one way to go:• separate functionality into independent components

(data, metadata, PIDs, etc)• specify proper interfaces (of course)

requirements for layer

• how to manage procedures/workflows in complex landscape • how to assess quality and correctness of all workflows • how to maintain provenance information

• only one way to go• make use of an easy-to-interpret declarative language • establish proper "policy rules on all levels" • map these rules to robust and proven activities • separate declarative language from interpretation engine

• iRODS is an attempt in this direction respect to Reagan Moore and his team

• at MPI since some years such a declarative language to manageaccess rights for the million objects which need to be treated individually and which are part of collections

Reagan's data environments

• moving not bytes but collections • need to maintain integrity of collections (incl. relations) • collections are assembled for a certain purpose• collections have properties to ensure their purpose• policies ensure maintenance of properties• procedures implement policies • procedures result in state information• assessment step to validate state

• purpose, properties, policies, procedures, state info

program - 1st part

• Larry Lannom (CNRI): about a digital object architecture • Alex Wade (MS): approach from MS• Malte Dreyer: thoughts about generic API• John Kennedy: heterogeneity of repositories in DEISA• Ken Galluppi: federating several repositories • Willem Elbers: federation tests with iRODS• Jean-Yves Nief: iRODS in professional use • Peter and Johannes: summary + discussion

utilization challenge

• utilization software may not be affected by replication • utilization software should also make use of copies • any replication solution needs to demonstrate this !!!!

existingutilizationsoftware

work spaces and profiles

• users want to • store data • protect data• share data• enrich data• change data• etc.

• data is somewhere in this complex domain

• users want transparentaccess

• how to get this done?

profilesattributes

quotasetc

processing chains - specification

datametadataregistries

toolmetadataregistries

data operation data* operation

workflow specification framework

this is very discipline specific - various possibilitiescuration/annotation/enrichment/visualization pipelines, etc

processing chains - execution

workflow execution framework

the challenges

• large amounts of data is at mirroring repositories • let's execute operations on the mirroring sites • how to easily deploy operators • how to inform execution environment about invocation way• how to let them act on the user's behalf

• etc

program - 2nd part

• SARA colleagues: workspace in NL • Morris Riedel (FZJ): workspace ideas• Johannes & John (RZG): operational aspects • Thomas & Erhard (U Tübingen): WebLicht example• Mike Papazoglou (U Tilburg): generic SOA aspects

• Peter: wrap up and discussion

thanks for the attention