Curation Micro-Services “It’s a Series of Tubes” Curation Micro-Services “It’s a Series of Tubes”

  • Published on
    20-Jan-2016

  • View
    212

  • Download
    0

Embed Size (px)

Transcript

  • Curation Micro-ServicesIts a Series of TubesCuration Micro-ServicesIts a Series of Tubes

  • The Unix philosophyMake each program do one thing wellTo do a new job, build afresh rather than complicate old programs by adding new featuresExpect the output of every program to become the input to another, as yet unknown, programDesign and build software to be tried earlyDon't hesitate to throw away the clumsy parts and rebuild them D. L. McIlroy et al., Unix time-sharing system forward, Bell System Technical Journal 57:6,part2 (1978): 1902

  • The micro-services philosophy

  • Curation micro-services

    MetaphorsAssumptionsPrinciplesPreferencesPracticesPipelineSafety through redundancyModularityThe small and simple over the large and complexFocus on outcomes, not meansLego bricksMeaning through contextGranularityThe minimally sufficient over the feature ladenComplexity through composition, not additionUtility through serviceOrthogonalityThe configurable over the prescribedPolicy neutral, platform and protocol independentValue through use (and reuse)EmergenceThe proven over the (merely) novelApproach sufficiency through incrementally necessary stepsStewardship is a relayEvolutionEarly prototyping, frequent refactoringParsimonyCode to interfaces

  • Curation micro-services

    ModeFocusValueServiceValenceVisibilityCurationValueAccretionAnnotationUI / Access control / Message queuingInteroperationUser-facingVisibilityNotificationUtilityAccessibilityAccessApplicationDerivationTransformationSelectivitySearchActionabilityIndexStewardshipIngestPreservationContextEpistemologyCharacterizationInterpretationProvider-facingOntologyInventoryStateReliabilityReplicationProtectionFixityFixityStabilityStorageIdentityIdentity

  • Design goalsPrinciple of least surpriseMultiple interface modalitiesRESTful HTTPCommand lineProcedural (Java, Perl, Ruby, )Linked dataStable URL referencesThe file system is the databasehttp://example-store/State or contentStorage nodeObjectVersionFiledefault/1234/3/xyzstate/Storage service

  • You say micro, I say macroAccessANVLARKBagItCANCheckmDflatERCEZIDGhOSTIngestInventoryLockItN2TNamasteNoidPairtreeReDDRUUStorageAccessANVLARKbagit.plBagItCANcheckm.plCheckmDflatERCEZIDGhOSTIngestInventoryLockItN2Tnamaste.plNamasteNoidPairtreeReDDRUUStorageServiceToolConvention

  • Development roadmap

    First waveSecond wave Third waveFourth wave Fifth waveSixth wave IdentityInventoryIndexSearchNotificationAnnotationStorageIngest / AccessFixityReplicationCharacterizationTransformationIDm / Authn / AuthzMetadata standardsObject / collection modelingSemantic interoperabilityPolicy / business model development

  • Ingest process flowSubmitting user agentIngestInventoryStorageNodeNodeNodeIdentitySubmitCreate identifierIdentifierAdd versionGet version metadataVersion metadataVersion metadataNotificationNotificationVersion metadataGet version metadataAdd version

  • Ingest implementationSubmitting user agentSubmitterConsumerIngesterStorageQueueHTML formServletImplicitly multi-threadedServletImplicitly multi-threadedDmonExplicitly multi-threadedZookeeper dmonJob metadataJob payloadSubmission notificationIngest notificationBatch or single object

  • Questions?silverpipes.jog / firstpresmacomb.org

  • More informationUC Curation Center (UC3)http://www.cdlib.org/uc3Micro-service specificationshttps://confluence.ucop.edu/display/CurationDigital curation grouphttp://groups.google.com/group/digital-curationUC3Stephen AbramsErik Hetzner Margaret Low Mark Reyes Perry WillettPatricia CruseGreg Jane David Loy Tracy SenecaScott FisherJohn Kunze Isaac Rabinovitch Marisa Strong