Preserving Digital Content at Scale · • UW Libraries, Archives, State agencies, public libraries (DPLA), campus partners, etc. • Digital Ecosystem • Digital library, IR, licensed

Preserving Digital Content at Scale

CNI-Spring2017-PreservingDigitalContentatScale

LeeKonradAssociateUniversityLibrarianforTechnology,Data,andDigitalLibraryServices

[email protected]

Long-term Digital Asset Management Initiative at the University of Wisconsin-Madison

University of Wisconsin •  UWSystem

•  TwodoctoralinsKtuKonsAdozencomprehensives•  MulKplelibraryDLandIRenvironments

•  UniversityofWisconsin-Madison

•  Libraries,Archives,Museums,ResearchCenters,AdministraKve,etc.

•  UWDigitalCollecKons

•  Contentcontributors•  20yearsofdigitalcollecKonbuilding–100+disparatecollecKons–3Mobjects•  UWLibraries,Archives,Stateagencies,publiclibraries(DPLA),campuspartners,etc.

•  DigitalEcosystem

•  Digitallibrary,IR,licensedcontent,etc.•  100TBofmaterialwarranKngpreservaKon•  10Musesperyear•  Redundantandgeographicallydistributedstorage•  Fedora•  Dspace•  CoordinatedDiscovery




Selected Digital Collection Models


•  ClearneedwithintheLibrariesandArchivesdivisions

•  ClearrisktotheinsKtuKonalinvestmentsindigitalcollecKonbuilding

•  Presumedneedwithrespecttoothercampusstakeholders/enKKes

•  ExcellentcentralbackupITservices–noactualpreservaKonclassservicesofferedoncampus

•  Adhocworkinggroupbeganconsideringtheneedsandrisks–formalizedasLDAM

•  HitachiContentPla]orm–JointprojectofcampusITandLibraries

UW-Madison – Preservation Pilot

UW-Madison and DPN •  DPN–CharterMembers

•  DPNmembershipjointlysponsoredby

•  ViceProvostforLibraries•  ViceProvostforInformaKonTechnologyandCIO

•  Campuspartners

•  LibrariesandArchives•  CIO’soffice•  DivisionofInformaKonTechnology•  LegalServicesandBusinessServices

•  DPNinUWcontext

•  EnvisionjointservicewithLibrariesandcentralITtoprovidearangeofservicespertainingtodigitalcuraKon,assetmanagementandpreservaKon–onramptoDPNand/orpartners

•  TwoteammembershavebeentrainedtodepositmaterialsintoDPN•  FinalizingdepositagreementwithLegalandBusinessservicedivisions



LDAMLong-termDigitalAssetManagementIniKaKve

Libraries - Digital Preservation Framework

•  Frameworkfordevelopingandorganizingdigitalpreserva5onpoliciesfortheUW–MadisonLibraries(LDAM)

•  Stewardshipandpreserva5on•  RequiresinsKtuKonalcommitment•  Sustainablefunding•  Accountability–policy,audit,risk

•  Regulatoryandprogramma5cenvironment

•  Federalandstatelaw•  Copyright,fairuse,etc.•  Recordsreten5onschedules(federal,state,ins5tu5onal)

•  Interna5onalstandardsandbestprac5cesrelatedtodigitalpreserva5on•  Centrallyplannedandmanagedpolicyandsystems-distributedpreserva5onac5vity

•  Technologicalandgeographicredundancy•  Documen5ngworkflows,tomaintainboththeintegrityandiden5tyoftheobjectsundercura5on.•  SystemstomeetorexceedcampusITSecurityrequirements,policies,andprocedures•  Systemsconsideredmission-cri5calinfrastructure,withappropriatelevelsofmonitoring,response,

andrecoverability.


LDAM – Rationale for Consideration of Campus Service

•  Increasingly,scholarlyworkpersistindigitalformonly•  Recognizedneedforenterprisescaleservicetoaccept,maintainandrecoverdigitalassets•  NumerousunitssupportthecreaKonandmaintenanceofdigitalarchives•  Valueinbringingcontentcreators/providerstogethertoconsiderservicesforLDAM


LDAM Initiative - Administration

Execu5veSponsors

ViceProvostforLibrariesandUniversityLibrarianViceProvostforInformaKonTechnologyandChiefInformaKonOfficer

Governance

AdvisoryCommideeWorkingGroup


LDAM – Deliverables Report(forthcoming)

•  DeterminesanddocumentsthecapabiliKesrequiredtosupportthedigitalarchiveneedsoftheUniversity

•  DevelopinventoryanddescripKonofservicesthatcurrentlyprovidedigitalarchiveservices.•  Assessthelevelofmaturityandalignmentofcurrentserviceswithcurrentandfutureneeds

ofkeycampusunitsandthecampusasawhole•  Surveycommunitytodeterminehowpeoplearecurrentlyaddressingdigitalarchivingneeds•  AnalyzegapbetweencurrentservicesandrequiredcapabiliKes•  SurveyanddescribeexternalopportuniKesandhowtheymightfitintoourplans(e.g.DPN,

DuraSpace,etc.)•  Draghigh-levelroadmap,communicaKngcampuscurrentstate,andthestepsnecessaryto

achievethedesiredfuturestate•  MakeiniKalrecommendaKonsforprojects,governanceorotheracKonsnecessaryto

achievethedesiredfuturestateandevolvetheservicesoverKme


LDAM – Phases

•  Planning•  Stakeholderengagements(administraKve)•  PeerconsultaKons•  SurveytoidenKfycommuniKesofneed•  InterviewsandFocusGroups•  QualitaKveAnalysis•  Report(forthcoming)


LDAM – Planning and Process

•  ReviewedkeytoolsanddocumentsinuseatpeerinsKtuKons(e.g.OAIS,DataCuraKonProfileToolkit,datamanagementframeworks…)

•  ReviewedLibrariesPreservaKonFramework•  DevelopedguidingprinciplesfortheLDAMservice(forourselvesandtoinform

stakeholderengagements)•  DevelopedcriteriatofacilitateconversaKonsastonatureofserviceasdisKnctfrom

backup,storage,etc.•  EnvironmentalscanofexisKngcampusservicesandtechnologies


LDAM Service - Guiding Principles - 1

•  Principlesforthecrea5onandopera5onofLongTermDigitalAssetManagementservices

• TheOpenArchivalInformaKonSystem(OAIS)ReferenceModel

• Theservices(s)willbeoperatedwithafocusontheneedsofthecommuniKesthat:

• Createandwishtopreservecontent.OAISProducers• Maywishtoaccessthecontent.OAISConsumers


• ResponsibilityfortheManagementoftheServicewillbeclearlydefined…

• ProvideServicecharterandscope.• ProvidefundingfortheService–determinepricingmodels• EvaluatesServiceperformanceanduKlizaKonagainstdefinedgoals.• AssessesandaddressesriskstotheServiceandholdings.• DetermineorendorsespoliciesrelatedtotheServiceincluding:

• TypeofcontenttheServicemaynot,may,isrequiredtoaccept.• ResolvesconflictinvolvingProducers,ConsumersandServiceAdministraKon.• DevelopsandenforcespoliciesforuseoftheService.E.g.must,should,maypersistcontentwiththeService.



ResponsibilityfortheAdministra5onoftheservice(s)willbeclearlydefinedandshouldinclude.•  Developanddocumentpoliciesandprocedureswhichensure:

•  ThatthecontentispreservedagainsttheconKngenciesspecifiedbyManagement,includingdemiseoftheService.

•  Conformancewithalllegalrequirements(e.g.copyright)thatapplytothecontent.

•  DevelopmentofformalsuccessionandescrowarrangementsiftheServicedisconKnuesoperaKon.

• Non-archivesoTwareshouldnotbereliedupontomaintainRepresentaKonData.

•  ContentwillneverbedeletedunlessaspartofadefinedstrategyapprovedbyManagement.Noadhocdele5ons.



LDAM Service – Criteria and Considerations for Inclusion Balancingcost,risk,andeffort

•  Metadata-levelofdescripKon•  FormatstandardizaKon•  VolaKlity•  Uniqueness•  ValueofcontenttoinsKtuKon•  QuanKty•  Providerprofile(commitmenttoinsKtuKon)•  InsKtuKonalinvestmentandrisk


LDAM - Inventory of Current Campus Services/Technologies

LibraryandUniversityArchivesrepositories:MINDS@UW,UWDigitalCollecKons,HathiTrust

Sharedcampusserivces

ElectronicLabNotebooks-LabNotebookSogwareandStorageforResearchersResearchDataServices-DataManagementPlans,ConsultaKons,EducaKonandTrainingHitachiContentPla]orm-DoITandLibraryDataArchiveServiceAdvancedCompuKngIniKaKve-CampusResearchCompuKngResourcesCampusCompuKngInfrastructure(CCI)-PhysicalandVirtualServerHosKngandDataStorageBuckyBackup-BuckyBackupLite(Diskbased)andBuckyBackupArchive(Tapebased)UWCloudServices-AmazonWebServices,MicrosogAzure,GoogleCloudPla]orm,IBMSogLayerUWCloudFileStorage–Box,GoogleApps,Office365

DepartmentalITServices–SocialScienceCompuKngCooperaKve(SSCC),InsKtuteforClinicalandTranslaKonalResearch(ICTR),DiscoverITStorage,ComputerSystemsLab,DepartmentofPhysics,Biochemistry,BockLabs,RecoveringAnalogandDigitalData(RADD)


LDAM Stakeholder Analysis

University Unit/Area Continuing Studies Education and Social/Behavioral Science IRB Electronic Lab Notebook Graduate School Libraries Risk Management Social Sciences Computing Cooperative (SSCC) UW Archives and Records Management UW Archives and Records Management Video and broadcasting Enrollment Management Information Technology Committee (ITC) MTAG Data Stewardship Council Aquatic Sciences Center Arboretum Biotechnology Center Biotron Discovery to Product Great Lakes Bio Research Center Institute for Molecular Virology Institute on Aging Laboratory of Cell and Molecular Biology Physical Sciences Laboratory Primate Research Center Research Animal Resources Center Space Science Engineering Center Stem Cell & Regenerative Medicine Center University of Wisconsin Press Waisman Center WID Wisconsin IceCube Particle Astrophysics Center Chazen Museum of Art Geology Museum L.R. Ingersoll Physics Museum Zoological Museum Wisconsin State Herbarium Athletics WARF Wisconsin Extension Business Services Wisconsin Historical Society Renk Agribusiness Institute Antarctic Meteorological Research Center Arts Institute Asthma, Allergy and Pulmonary Research Center Biological and Biomaterials Preparation, Imaging and Characterization Laboratory Bolz Center for Arts Administration Cardiovascular Research Center Center for Child and Family Well-Being Center for Climatic Research Center for Community and Nonprofit Studies Center for Cooperatives Center for Culture, History and Environment

Center for Dairy Research

Center for Demography and Ecology Center for Demography of Health and Aging Center for Eukaryotic Structural Genomics Center for Financial Security Center for Health Enhancement Systems Studies Center for Health Systems Research and Analysis Center for Integrated Agricultural Systems Center for Law, Society and Justice Center for Limnology Center for Patient Partnerships Center for Quality and Productivity Improvement Center for Quick Response Manufacturing Center for Rapid Evolution Center for Sustainability and the Global Environment Center for Sustainable Nanotechnology Center for the Humanities Center for the Study of Liberal Democracy Center for the Study of Upper Midwestern Cultures Center for Tobacco Research and Intervention Center for Urban Population Health Center for Women's Health Research Center on Education and Work Collaborative Center for Health Equity Cooperative Institute of Meteorological Satellite Studies Delta Program Drug Development and Development Fee-for-Service Capabilities E-Business Consortium Engine Research Center Environmental Remote Sensing Center McPherson Eye Research Institute Fusion Technology Institute Global Health Institute Great Lakes Bioenergy Research Center Health Innovation Program HealthEmotions Research Institute Holtz Center for Science and Technology Studies Institute for Biology Education Institute for Research in the Humaities Institute for Research on Poverty Regional and International Institute Internet Scout Project J.F. Crow Institute for the Study of Evolution Kohl's Center for Retailing W.M. Keck Laboratory for Biological Imaging McArdle Laboratory for Cancer Research Land Tenure Center Language Institute Lubar Institute for the Study of the Abrahamic Religions Materials Research Science and Engineering Center Max Kade Institute for German-American Studies McBurney Disability Resource Center Molecular and Environmental Toxicology Center Mosse/Weinstein Center for Jewish Studies Nanoscale Science and Engineering Center Nelson Institute for Environmental Studies Zeeh Pharmaceutical Experiment Station Population Health Institute Rheology Research Center Robert M. La Follette School of Public Affairs Sea Grant Institute Social Systems Research Institute Sonderegger Research Center (Interdisciplinary Research in Social Administrative Pharmacy) Synchrotron Radiation Center Trace Research & Development Center University of Wisconsin Survey Center UW Carbone Cancer Center UW Food Research Institute UW Institute for Clinical and Translational Research UW Skin Diseases Research Center Center for Visual Culture Water Resources Institute Wavelet Center for Ideal Data Representation Wisconsin Advanced Internet Laboratory Wisconsin Alzheimer's Disease Research Center Wisconsin Alzheimer's Institute Wisconsin Area Health Education Centers Wisconsin Brain Donor Program Wisconsin Center for Applied Microelectronics Wisconsin Center for Education Research Wisconsin Center for the Advancement of Postsecondary Education Wisconsin Office of Rural Health Wisconsin State Laboratory of Hygiene College of Agricultural and Life Sciences College of Engineering College of Letters & Science Law School School of Education School of Human Ecology School of Medicine and Public Health School of Nursing School of Pharmacy School of Veterinary Medicine International Studies Univesity Communications Wisconsin School of Business

LDAM – Campus Communications and Connections

LDAM Stakeholder Analysis

University Unit/Area Continuing Studies Education and Social/Behavioral Science IRB Electronic Lab Notebook Graduate School Libraries Risk Management Social Sciences Computing Cooperative (SSCC) UW Archives and Records Management UW Archives and Records Management Video and broadcasting Enrollment Management Information Technology Committee (ITC) MTAG Data Stewardship Council Aquatic Sciences Center Arboretum Biotechnology Center Biotron Discovery to Product Great Lakes Bio Research Center Institute for Molecular Virology Institute on Aging Laboratory of Cell and Molecular Biology Physical Sciences Laboratory Primate Research Center Research Animal Resources Center Space Science Engineering Center Stem Cell & Regenerative Medicine Center University of Wisconsin Press Waisman Center WID Wisconsin IceCube Particle Astrophysics Center Chazen Museum of Art Geology Museum L.R. Ingersoll Physics Museum Zoological Museum Wisconsin State Herbarium Athletics WARF Wisconsin Extension Business Services Wisconsin Historical Society Renk Agribusiness Institute Antarctic Meteorological Research Center Arts Institute Asthma, Allergy and Pulmonary Research Center Biological and Biomaterials Preparation, Imaging and Characterization Laboratory Bolz Center for Arts Administration Cardiovascular Research Center Center for Child and Family Well-Being Center for Climatic Research Center for Community and Nonprofit Studies Center for Cooperatives Center for Culture, History and Environment

Center for Dairy Research

Center for Demography and Ecology Center for Demography of Health and Aging Center for Eukaryotic Structural Genomics Center for Financial Security Center for Health Enhancement Systems Studies Center for Health Systems Research and Analysis Center for Integrated Agricultural Systems Center for Law, Society and Justice Center for Limnology Center for Patient Partnerships Center for Quality and Productivity Improvement Center for Quick Response Manufacturing Center for Rapid Evolution Center for Sustainability and the Global Environment Center for Sustainable Nanotechnology Center for the Humanities Center for the Study of Liberal Democracy Center for the Study of Upper Midwestern Cultures Center for Tobacco Research and Intervention Center for Urban Population Health Center for Women's Health Research Center on Education and Work Collaborative Center for Health Equity Cooperative Institute of Meteorological Satellite Studies Delta Program Drug Development and Development Fee-for-Service Capabilities E-Business Consortium Engine Research Center Environmental Remote Sensing Center McPherson Eye Research Institute Fusion Technology Institute Global Health Institute Great Lakes Bioenergy Research Center Health Innovation Program HealthEmotions Research Institute Holtz Center for Science and Technology Studies Institute for Biology Education Institute for Research in the Humaities Institute for Research on Poverty Regional and International Institute Internet Scout Project J.F. Crow Institute for the Study of Evolution Kohl's Center for Retailing W.M. Keck Laboratory for Biological Imaging McArdle Laboratory for Cancer Research Land Tenure Center Language Institute Lubar Institute for the Study of the Abrahamic Religions Materials Research Science and Engineering Center Max Kade Institute for German-American Studies McBurney Disability Resource Center Molecular and Environmental Toxicology Center Mosse/Weinstein Center for Jewish Studies Nanoscale Science and Engineering Center Nelson Institute for Environmental Studies Zeeh Pharmaceutical Experiment Station Population Health Institute Rheology Research Center Robert M. La Follette School of Public Affairs Sea Grant Institute Social Systems Research Institute Sonderegger Research Center (Interdisciplinary Research in Social Administrative Pharmacy) Synchrotron Radiation Center Trace Research & Development Center University of Wisconsin Survey Center UW Carbone Cancer Center UW Food Research Institute UW Institute for Clinical and Translational Research UW Skin Diseases Research Center Center for Visual Culture Water Resources Institute Wavelet Center for Ideal Data Representation Wisconsin Advanced Internet Laboratory Wisconsin Alzheimer's Disease Research Center Wisconsin Alzheimer's Institute Wisconsin Area Health Education Centers Wisconsin Brain Donor Program Wisconsin Center for Applied Microelectronics Wisconsin Center for Education Research Wisconsin Center for the Advancement of Postsecondary Education Wisconsin Office of Rural Health Wisconsin State Laboratory of Hygiene College of Agricultural and Life Sciences College of Engineering College of Letters & Science Law School School of Education School of Human Ecology School of Medicine and Public Health School of Nursing School of Pharmacy School of Veterinary Medicine International Studies Univesity Communications Wisconsin School of Business

•  GovernanceEngagement•  Sharedchargewithcampusgovernancegroups•  MetwithgovernancegroupstoreviewchargeandrequesthelppromoKng

iniKaKve

•  Surveysenttocampuscolleges,schools,divisions,andresearchcenters–seekingdiscussantstoinformpotenKalserviceneeds/requirements

•  In-depthinterviews


LDAM - Face-to-face Interview Roadmap 1.   Pleasetellusabitmoreaboutyourself,yourrole,yourorganizaKon(10minutes)

2.   Pleasetellusmoreaboutthedigitalmaterialsyouareconsideringpreserving.(15minutes)•  WhatvaluedoesthematerialhavethatmakesitacandidateforpreservaKon?•  Whataspectsofthematerials,informaKonaboutthematerial,shouldbepreserved?

3.   Inyourmind,whatconsKtutespreservaKon?(20minutes)•  Howlong?•  Howmanycopies,where?•  Howaccessibleshouldthepreserveddatabe?Towhom?\•  HowimportantisitthatthematerialsbemigratedoverKmetokeepupwithchangesinformatsandtools?•  WhataresomeofthepotenKalusesyouseeforthispreserveddata,wheresomeperson,group,organizaKonorrobotinthe

futuremaywanttorecoverthesematerialsforsomepurpose?

4.   TellusmoreaboutanyregulaKons,policies,guidelines,mandates,etc.thatinfluenceyourpreserva5ondecisionsandac5vi5es.(10minutes)

5.  PreservaKonhascosts.Tellushowyouthinkabouttheresources(people,storage,etc.)requiredtopreparematerialsforpreservaKon,storethem,andmanagethemoverKme.(10minutes)•  Howmuchdoescost(intermsofmoney,Kme,andeffort)influenceyourpreservaKondecisions?•  Wouldacampus-widepreservaKonservicebeofusetoyou?Ifso,whatmightthatlooklike?•  Assumingthatsuchaserviceisunlikelytobe100%centrally-funded,whatkindofcostmodelsforsuchaservicewouldworkfor

you?


LDAM – Findings - 1

•  Weinterviewedcommunitymemberstolearnmoreabouttheirunits’concernsandapproachestodigitalpreservaKonatUW

•  Weconductedin-depthinterviewswithrepresentaKvesfromeightdifferentunitsfromacrosscampus,represenKngarchival,library,administraKveandresearchperspecKves

•  Nomajorsurprises•  Severalexpresseddesireforopennessofmaterialsforphilosophicaland/orcompliancereasons

•  Afewexpressedconcernsabouthowbesttopreservematerialthatcould/shouldnotbeopenforreasonsofprivacy,copyright,orevenheldelsewherealready


•  StorageandbackupsoluKonsvariedwidely•  Centrally-hostedstorage•  Sharedlocaldrives•  CollaboraKvecloudpla]ormssuchasBox•  Externalpla]ormssuchastheOpenScienceFramework

•  Consensusthatrangeofop5onsoTenfeltincompleteandlackedformalizedworkflow

•  Concernsthat5mespentoverseeingthetechnologyneedsoftheirunittookawayfromotherjobresponsibiliKes

•  AllawaretheirLDAMstrategiesdidnotequatetopreserva5onyetallsoughttoensurethattheirmaterialswereaccessibleinperpetuity



•  ScaleofmaterialrequiringLDAMacrosstheseunits•  SeveralunitscitedoveralldatacollecKonsbetween1-5TB•  Oneunitoutsidelibrarycitedholdingsof90TB•  Oneunitgenerates50-100TBperyear

•  Submissionpreferences–annualsubmissiontodailysubmissions

•  Takeaway-Dataisbeingpreservedwhilemetadatamaynotbe.Metadata,suchasformatdefiniKons,isnotbeingsaved,sothemeaningofthedatamaybelost,orthedatamaynotbediscoverable.

•  ConKnuedneedtostudyfullrangeofcampusneed,parKcularlyforthepreservaKoncaseswhereculturalandcampusheritageisatrisk



LDAM Next Steps

•  DevelopiniKalLDAMservicerecommendaKonsandcompletedragofreport

•  SharereportwithsponsorsandreviewrecommendaKons

•  CreateiniKalbudgetmodelandservicelayersforlocalLDAMserviceifserviceisendorsed

•  AddHCPtolocalsuiteofLDAMservicesifpromising

•  PromoteDPNaspartofsuiteofLDAMservicesendorsedbycampus


LDAM - Key Challenges to Campus-level LDAM

•  Sustainableandcommon-goodfundingmodelsandgovernancemodels

•  DefiniKons:clarifyingdisKncKons-storage,archiving,darkarchiving,preservaKon(assetofacKons,people,tools)

•  Riskanalysisvs‘lesser’alternaKves

•  CopyrightandsuccessionofmaterialdepositedintoDPNorthird-partysystems


Next steps with DPN and others

•  SupportandcollaboratewithothersintheworkofthedigitallibraryandpreservaKoncommuniKes

•  DPN•  UKlizethecreditswehaveearnedasapartner–publicdomainmaterialsonly•  Finalizearrangeswithcampuslegal/businessfordeposiKonofaddiKonalmaterialsintoDPN•  FormallyposiKonDPNaspartofcampuspreservaKonstrategyandaddintosuiteofservices

•  LibrarieswilllikelybethepointofentrytoDPNandLDAMservicesforcampusstakeholders•  TesKngandiniKalingestsviaDuraSpace•  DeveloptheworkflowsandbestpracKcesthatwouldbeadoptedbyacampussoluKoninthefuture


Documents

Preserving Digital Content at Scale · • UW Libraries, Archives, State agencies, public libraries (DPLA), campus partners, etc. • Digital Ecosystem • Digital library, IR, licensed