UC3 M e r r i t t

Curation Micro-Services“It’s a Series of Tubes”

Curation Micro-Services“It’s a Series of Tubes”

UC3 M e r r i t t

The Unix philosophy

“Make each program do one thing well”

“To do a new job, build afresh rather than complicate old programs by adding new features”

“Expect the output of every program to become the input to another, as yet unknown, program”

“Design and build software … to be tried early”

“Don't hesitate to throw away the clumsy parts and rebuild them”

— D. L. McIlroy et al., “Unix time-sharing system forward,” Bell System Technical Journal 57:6, part 2 (1978): 1902

UC3 M e r r i t t

The micro-services “philosophy”

http://www.flickr.com/photos/elsie/8229790/CC http://www.flickr.com/photos/oskay/265899811/CC

UC3 M e r r i t t

Curation micro-servicesMetaphors Assumptions Principles Preferences Practices

Pipeline Safety through redundancy Modularity The small and simple over

the large and complexFocus on outcomes, not means

Lego bricks Meaning through context Granularity The minimally sufficient

over the feature ladenComplexity through composition, not addition

Utility through service Orthogonality The configurable over the

prescribedPolicy neutral, platform and protocol independent

Value through use (and reuse) Emergence The proven over the

(merely) novelApproach sufficiency through incrementally necessary steps

Stewardship is a relay Evolution Early prototyping, frequent

refactoring

Parsimony Code to interfaces

UC3 M e r r i t t

Curation micro-servicesMode Focus Value Service Valence Visibility

Curation

ValueAccretion Annotation

UI / Access control / M

essage queuing

Interoperation

User-facing

Visibility Notification

Utility

Accessibility Access

Application

Derivation Transformation

Selectivity Search

Actionability Index

Stewardship Ingest

Preservation

ContextEpistemology Characterization

Interpretation

Provider-facing

Ontology Inventory

State

Reliability Replication

ProtectionFixity Fixity

Stability Storage

Identity Identity

UC3 M e r r i t t

Design goals

Principle of least surprise

Multiple interface modalities– RESTful HTTP– Command line– Procedural (Java, Perl, Ruby, …)

Linked data

Stable URL references

The file system is the database

http://example-store/

State or content

Storage nodeObjectVersionFile

default/1234/3/xyzstate/

Storage service

UC3 M e r r i t t

“You say micro, I say macro…”

AccessANVLARKBagItCANCheckmDflatERCEZIDGhOSTIngestInventoryLockItN2TNamasteNoidPairtreeReDDRUUStorage

AccessANVLARK

bagit.pl BagItCANcheckm.pl Checkm

DflatERC

EZIDGhOSTIngestInventory

LockItN2T

namaste.pl NamasteNoid

PairtreeReDD

RUUStorage

Service Tool Convention

UC3 M e r r i t t

Development roadmap

First wave Second wave Third wave Fourth wave Fifth wave Sixth wave

Identity Inventory Index Search Notification Annotation

Storage Ingest / Access Fixity Replication Characterization Transformation

IDm / Authn / Authz Metadata standards

Object / collection modeling Semantic interoperability

Policy / business model development

UC3 M e r r i t t

Ingest process flow

Submitting user agent

Submitting user agent IngestIngest

InventoryInventory

StorageStorage

NodeNode

NodeNode

NodeNode

IdentityIdentity

Submit

Create identifier

Identifier

Add version

Get version metadata

Version metadata

Version metadata

Notification

Notification

Version metadata

Get version metadata

Add version

UC3 M e r r i t t

Ingest implementation

Submitting user agent

Submitting user agent

SubmitterSubmitter ConsumerConsumer IngesterIngester StorageStorageQueue

HTML form

ServletImplicitly multi-threaded

ServletImplicitly multi-threaded

DæmonExplicitly multi-threaded

Zookeeper dæmon

Job metadata

Job payload

Submission notification

Ingest notification

Batch or single object

UC3 M e r r i t t

Questions?

silverpipes.jog / firstpresmacomb.org

UC3 M e r r i t t

More information

UC Curation Center (UC3)http://www.cdlib.org/uc3

Micro-service specificationshttps://confluence.ucop.edu/display/Curation

Digital curation grouphttp://groups.google.com/group/digital-curation

UC3Stephen Abrams Erik Hetzner Margaret Low Mark Reyes

Perry WillettPatricia Cruse Greg Janée David Loy Tracy SenecaScott Fisher John Kunze Isaac Rabinovitch Marisa Strong