1. BUILDING AN ELECTRONIC RECORDS PROGRAMIN A MAJOR RESEARCH LIBRARYSPECIAL COLLECTIONS:POTENTIAL PATHS TO SUCCESS Michelle Belden January 2012

2. TAKING OUR PULSE:THE OCLC RESEARCH SURVEYOF SPECIAL COLLECTIONS AND ARCHIVES [OCT 2010] 79% said they had born-digital material in their collections, Yet, only 35% could estimate the extent of those materials, and 45% werent sure who was responsible for this material. Undercollected, undercounted, undermanaged, unpreserved, inaccessible -Jackie Dooley of OCLC 3. WHAT ARE WE TALKING ABOUT? Records: Information, in a fixed form, used as a source of informationabout the past Records have content, structure & context Special Collections: Primary Sources (Material that contains firsthand accounts ofevents and that was created contemporaneous to those eventsor later recalled by an eyewitness.) Examples of records in Special Collections: Meeting minutes Letters Diaries Authors manuscripts 4. ELECTRONIC RECORDS Written on magnetic or optical medium, recorded in binarycode, and accessed using computer software & hardware The Board of Trustees Meeting Minutes are online as PDFs People send letters via email People keep diaries via blogs Authors donate manuscripts on hard drives Recently an artist donated her website! 5. ELECTRONIC UNIVERSITY RECORDS For PSU records, we must adhere to a records schedule. We must keep certain documentation for a certain amount oftime, no matter its format. Some examples: Faculty Senate Course Proposals University Web Bulletin Newswires Central Policies & Procedures Manuals University Archivist must be able to reconstructevents/decisions/procedures While demonstrating authenticity, reliability, integrity 6. ELECTRONIC UNIVERSITY RECORDS CASESTUDYThe head of an academic department is complaining to theProvost that he did not approve a course currently beingtaught by a new professor in his department.Course proposals must pass through 3 levels of approval.Course proposals are archived in digital format, and the threelayers of approval are recorded through digital signatures.The Provost asks the University Archivist to retrieve the courseproposal and verify that the department head signed off on it.The course proposal shows that indeed it went through allappropriate approvals. The University Archivist must make thecase that the department heads (digital) signature isauthentic.The University Archivist must also make sure that the version ofthe course proposal signed off on by the department head isthe same version currently being taught. 7. P-RECORDS VS. E-RECORDS Both can take many forms Both can come to us quite messy For e-records: More copies, decentralized Authenticity can be harder to demonstrate Privacy may be harder to guarantee Less stable: viruses, accidental deletion, bit rot, formatsbecome obsolete (remember floppy discs?) However, they are more amenable to batch processingand automated searching Were still talking archives 8. TRADITIONAL ARCHIVAL FUNCTIONSAppraise Acquire Who created it and why? Records Schedule? What does it document? Gift or Purchase? Who might use it? Donor agreement? Does it serve our mission? How to transport? Is it authentic? Is it rare/valuable? Physical condition? Privacy issues? 9. TRADITIONAL ARCHIVAL FUNCTIONS, CONTDAccessionArrange & Describe Establishphysical, administrative & Original Order?intellectual control Series? Survey for formats, extent Sorting?and condition To what level? Check for issues ofprivacy/confidentiality Controlled vocabularies? Re-house? Preliminary description Document accessrestrictions Assign secure location Assign processing priority 10. TRADITIONAL ARCHIVAL FUNCTIONS, CONTDPreserve Make Accessible Appropriate environment Restrictions? Archival supplies Onsite/online Security Outreach Conservation (repair ofindividual items)considered separateAn E-records program will enable us to perform all these functions on all our e-recordson an ongoing basis 11. WE NEED NEW: Staff Models Standards Tools Infrastructure/tech support Policies Workflows Partnerships & A positive attitude towards change 12. STAFF:WHAT SKILLS DOES A DIGITAL ARCHIVIST NEED?http://blogs.loc.gov/digitalpreservation/2011/07/what-skills-does-a-digital-archivist-or-librarian-need/ Knowledge of formats &standards, but also: Adaptability, flexibility Ability to bridge gap betweentechies and not-so-techies Ability to communicate andadvocate for what they do 13. MODELS: REFERENCE MODEL FOR AN OPEN ARCHIVAL INFORMATION SYSTEM (OAIS)An OAIS is the combination of systems and people necessary to preserveselected information over the long term and make it available for aDesignated Community 14. INFORMATION PACKETS 15. OAIS AND TRADITIONAL ARCHIVAL FUNCTIONS 16. MODELS: DIGITAL CURATION CENTRE - LIFECYCLEDigital curation involves maintaining, preserving and adding value todigital research data throughout its lifecycle. 17. STANDARDS UNICODE(character coding system for worldwide interchange of text) Dublin Core(basic set of metadata elements to enable cross-searching) PREMIS(metadata for preservation) XML(set of rules for encoding documents, emphasizingsimplicity, generality and usability) PDF/a(open standard for document exchange, specialized for digitalpreservation) Etc.Cartoon by Rebecca Goldman derangementanddescription.wordpress.com 18. TOOLS Duke Data Accessioner http://www.duke.edu/~ses44/downloads/guide.pdf Copies data, using MD5 checksums Droid & Jhove plug-ins identify file formats Creates XML wrapper Virus Scanner (Symantec) PII Scanner (Identity Finder) TRAC Trusted Repositories Audit & Certification Drambora Digital Repository Audit Method Based on Risk Assessment Archive-IT/WAS (hosted service solutions) Etc. 19. TECHNICAL INFRASTRUCTURE/SUPPORT Hardware: E-records workstation in secure location PC with network access PC with secure (dark) storage Other equipment: Mac would be nice, additional mediareaders (floppy, zip) writeblocker. Automated backup/disaster recovery Discovery System 20. POLICIES Collection Development policies Service Level agreements What kind of storage can we secure? What kind of services will we offer prior to submission? Submission agreements What file formats accepted Ask for non-proprietary, non-lossy, widely adopted (Tiff, PDF) What metadata required How transferred (Web server? Physical disk?) Use agreements Who can access what materials, when, how, and forwhat purposes? 21. WORKFLOWSAccession (traditional)Ingest (electronic) Survey for formats, extent and Survey for file formats, extentcondition(MB/GB/TB? Files/folders?)(papers, photos, maps, linear/cu Scan for virusesbic feet, mold, insects) Scan for PII Run checksum, copy to new Check for issues ofdisc, verify checksumprivacy/confidentiality Preliminary metadata(correspondents, SSNs) Document access restrictions Re-house?(Sensitive data? Need special Preliminary descriptionsoftware/hardware?) Document access restrictions Move to secure digital storage(Certain groups? Donor Assign processing prioritypermission? 50 year hold?) Assign secure location in stacks Assign processing priority 22. PRESERVATIONTraditional Digital Format usually bound Format can depend onwith content, stable encoding or applications e.g. website with stylesheets, documents with MS formatting So, what are we preserving? Just the information or also the look and feel? Bitstream replication, system preservation? Characterization: 25 bytes, 48 characters wide, This is a video from YouTube 23. PRESERVATION, CONTINUEDTraditional Digital Context/relationships When lift bits fromcan often bephysical media howestablished through much contextphysical proximity or can/should youother cuesinclude? Metadata forrelationships, hierarchies 24. PRESERVATION CONTINUEDTraditional Digital Baseline preservation Controlled environmentaccomplished by & proper storage andproviding controlledhandling of physicalenvironment, proper media arestorage & careful important, but all mediahandling. require periodicreformatting 25. PARTNERSHIPS: THE WAY FORWARD Not just I.T.! Need to partner with records creators - and theiradministrative support - early in their processes toencourage the use of archive-friendly formats andthe production of good metadata! 26. CURATION ARCHITECTURE PROTOTYPE SERVICES(CAPS) Based on December 2009 platform review, which revealedinefficiencies & gaps, e.g. no platform for e-records Explored microservices approach to digital curation Based on work by California Digital Library Small, self-contained, independent services Easier to develop, deploy, maintain, enhance, replace. Interoperable: combine for more complex applications.Small things...specialized jobs...only truly powerful when they work inconcert...ZOMG ITS THE SMURFS Michael B. Klein 27. EXAMPLES OF MICROSERVICES Annotate - describe or catalog an object Authenticate - authenticate a user Authorize - authorize a user to access an object Characterize - generate administrative metadata foran object Identify - generate an identifier for an object Inventory - record an objects location on disk Relate - relate two or more objects Store - store an object on a filesystem Verify - check the integrity/checksum/fixity of anobject Version - add a version to an object 28. WHO WAS DOING THIS EXPLORING? Representatives from: Scholarly Communications I-Tech DLT Special Collections Cataloging & Metadata Stakeholders from 4 additional departments/ libraries: Arts & Architecture, Digitization & Preservation, Maps, University Archives 29. PROCESS (OUTREACH & AGILITY) Daily meetings with core team Weekly meetings with stakeholders Constantly incorporating feedback into our workand reformulating long/short term goals Never no just not now Progress tracked immediately on wiki Led to buy-in from stakeholders Developed prototype product in 3 months time 30. CAPS ARCHITECTURE 31. SCREENSHOT (1) 32. SCREENSHOT (2) 33. SCREENSHOT (3) 34. AIMS http://born-digital-archives.blogspot.com/ Born Digital Collections: An Inter-institutional Model for Stewardship UVA, Stanford, Yale, University of Hull Mellon Grant, 13 born-digital collections Framework for accessioning, arrangement & description, discovery Uses Hydra, an open-source technical framework available underApache 2 license Principal platforms are Fedora, Blacklight, Solr, Ruby on Rails One body (digital repository) many heads (feature-rich assetmanagement applications) 35. AIMS, CONTINUED One of AIMS Hydra heads is Hypatia Allows archivists to arrange and describeborn-digital assets.Hypatia: Greek Philosopher in Includes the ability to:Roman Egypt, first notable Drag and drop to arrange, woman in mathematics Return to original order, View file types, Add descriptive metadata, and Apply rights & permissions (high level ofgranularity) 36. ARCHIVEMATICA Artefactual Systems, City of Vancouver, University of BritishColumbia, Rockefeller Center& UNESCO Microservices design pattern Integrated suite of open-source tools that allow users to process digitalobjects from ingest to access. Based on Linux, written in Python. Utilizes METS, PREMIS, Dublin Core. Users monitor and control via a web-based dashboard. Implements media type preservation plans based on an analysis of thesignificant characteristics of file formats. Hydra fork: Rubymatica 37. ARCHIVEMATICA AND OAIS 38. POSITIVE ATTITUDE TOWARDS CHANGE ITS THE ONLY CONSTANTKeep up through:Websites like http://www.dcc.ac.uk/Listservs like [digital-curation]Professional meetings (e.g. SAA, Open Repositories, Code4Lib, DLF)Publications (e.g. American Archivist, Journal of Digital Information, FirstMonday, Digital Preservation Coalitions Technology Watch Reports) 39. FIVE ORGANIZATIONAL STAGESOF PROGRAM DEVELOPMENT: Acknowledge that digital curation is a shared concern Act- initiate digital preservation projects Consolidate- segue from projects to programs Institutionalize- incorporate the larger environment Externalize - embrace inter-institutional collaboration 40. BEWARE CONWAYS LAW: Organizations produce designs that copy theircommunication structures (Do not wait for a single, ultimate solution to emerge. Thepieces of the puzzle are in place to build preservationenvironments DigCCurr 2009) Drawing by Rube Goldberg 41. TAKE THE LEAP Set up new DRA at e-records workstation(Equipped with tools for ingest and access to securestorage) Select a group of records Partner with the creators of those records Agree on some policies to get those recordssubmitted(Policies should include standards) Have your new staff run the selected recordsthrough your new workflows using your new tools Document successes, failures Tweak, try again 42. SPARTAN ARCHIVE AT MICHIGAN STATE 3 year (began April 2010) NHPRC-funded ($250k) project Appraisal, accession & ingest, preservation &management, on-line access for 3 large series from Officeof Registrar: Catalog of Academic Programs Course Descriptions Annual Student Directory Utilizing Integrated Rule Oriented Data Systems (iRODS)distributed data grid solution Project will result in new policies, procedures, institutionalmetadata standards, definitions for SIPs, AIPs and DIPs