de theory and practice of digital preservation

PRESERVING AV COLLECTIONS…

Technological shifts

Changing storage media

Evolving playout equipment

Obsolescence

Nick Gentry, Obsolescence III (2011)

SO, WHAT’S NEW?

Presentator

Presentatienotities

So what’s new, one could argue? After all, AV-archives have been dealing with formatprogression, with obsolescence , with migration and conversion from the beginning of their existence. This ofcourse is caused by the nature of their collections. AV archives need technical equipment to be able to see and hear their content and because this equipment is regularly renewed, the AV formats had to be converted and migrated too. Nothing new therefore.

VOLUMES

Presentator

Presentatienotities

Secondly, the digitalization have caused for a very large increase in the volumes that we archive. There is the digital born content, the daily production of television programmes in our case, that is increasing, because producers simply produrce more and the content is getting larger (think of HD) . And there is the digitization of older content, of our analoge collections that causing digital volume to rise. For example, the digital collections of our institute, Beeld en Geluid…

PLATFORMS

Presentator

Presentatienotities

And because of this networked environment we are servicing many more users than before, and many more different kinds of users, be it professionals, the general public, schools, etc. And these usergroups have not only increased , the users also want different things from the archive, want to access it each in their own way, using many different platforms and devices to doo..

INFRASTRUCTURE

Presentator

Presentatienotities

Then, a third big change, we have become part of a network, of many networks even, internal and external. The archive is not standalone anymore, but connected to the outside world in many ways.

ICT

Presentator

Presentatienotities

But ofcourse a lot of things HAVE changed. First of all the archive has become a sort of IT system. For practically all of the archival processes sophisticated ICT is needed, Collecting, storing, cataloguing, presenting and distributing our content. We highly depend on systems and computers, on ICT with all the risks, organisational en management issues that go with that. Our fims and tapes have become files and datatapes. Our storeroom is a taperobot.

COSTS TOTAL COST OF OWNERSHIP DIGITAL PRESERVATION (TCO)

• Require rethinking the archive’s organization

• Remodelling of archival processes

• Standards, competences , relations with users/donors

• Creating a trusted digital environment

ANOTHER ‘NEW’ CONCEPT: TRUST

Keeper of cultural heritage

Keeper of commercial assets

Preservation as a (paid) service

Legal obligations

Funding

Becoming formally trustworthy

Certification

“A Trusted Digital R epository is one whose miss ion is to provide long-term access to managed digital resources ; that accepts responsibility for the long

term maintenance of digital resources on behalf of its depositors and for the benefit of current and future users; that des igns its systems in accordance with

commodity accepted conventions and standards to ensure the ongoing

management, access and security of materials deposited within it; that

establishes methodologies for system evaluation that meet community expectations of trustworthiness ; that can be depended upon and carries out its

long –term responsibilities to depositors and users openly and explicitely;

and whose policies , practices and performance can be audited and measured”

RLG-OCLC Report : TDR attributes and responsibilities (2002)

Presentator

Presentatienotities

A dormal name for such a trusted environment is a Trusted Digital Repository. The OCLC organization in 2002 decribed the characteristics and obligations of a digital archive with preservation responsibility more precisely and defined this so-called Trusted Digital Repository. Note the terms: Long term access and maintenance On behalf of depositors and users Accepted standards Community expectations of trustworthiness And the fact that the activities have to be done openly and explicitely and can be audited and monitored.

TRUST BETWEEN WHOM?

STAKEHOLDERS OF THE REPOSITORY

The Depositors/ Producers

The Users : Designated Communities

The Archive : Repository

Nick Gentry, Obsolescence IV (2011)

Trusting that those undertaking preservation activities can maintain the integrity and the authenticity of the objects

Presentator

Presentatienotities

So who are the parties, the stakeholders of a TDR and what is the domain of their trust?

TRUST IN THE CONTEXT OF DIGITAL PRESERVATION

Data level

Organisational level

Data level TRUST IN THE CONTEXT OF DIGITAL PRESERVATION

Can I use it? Accessibility

How can I be sure that it hasn’t been tampered with? Authenticity

Do I get out what I put in? Integrity

Presentator

Presentatienotities

Trust in the context of digital preservation circles around three basic questions.

TRUST IN THE CONTEXT OF DIGITAL PRESERVATION Organisation level

Governance & organizational viability (mission statement, preservation strategic plan, collection policy) Organisational structure & staffing (skilled professionals, clear roles and responsabillities) Procedural accountability & preservation policy framework (documenting all processes, decision making & goalsetting, selfassesment, compliance with standards)

Presentator

Presentatienotities


TRUST IN THE CONTEXT OF DIGITAL PRESERVATION Organisation level

Financial sustainability (short & longterm business planning) Contracts, licenses & liabilties (clear & legal contracts with depositors)

STEP

S TO

TRU

STW

ORTH

INES

S Analysis of existing functions, processes and standards

Embedding a new MAM system in existing processes

Adaptation and implementation Mapping to standards,

gapanalysis

OAIS compliant definitions

OAIS compliant preservation workflows

AS IS ANALYSIS BUSINESS CONCEPT IMPLEMENTATION

Presentator

Presentatienotities

The most important tool to help is the OAIS model. The Open Archive Information System. Designed by NASA mostly, as a way to safeguard various digital data that was produced during missions into space, data that ofcourse could never be lost or become inaccessible.

O A I S Open Archive Information system

Reference model for digital archives Designed in 1982 ( NASA (US), ESA (EU), RSA (USSR), NASDA (Japan)

ISO Standard 14721 since 2002

“A framework for procedures that support long term preservation of digital

information “

“A digital archive for long term storage of digital information with a high

cultural and/or historical value. Garantees platform- independent access for

50, 100 years or more”

“An organisation of people and machines that have accepted responsability

to archive digital information and keep this accessible to a target audience”

Definitions O A I S

OAIS, PREMIS, PAIS, TDR- DATA SEAL OF APPROVAL (DSA), PRESTO PRIME, DRAMBORA, SPOT, EBU P- meta, PBCore, VideoMD and AudioMD, NARA reVTMD. ANSI/NISO Z39.87 .. . . . .

Frames of reference

STANDARDS, MODELS, CERTIFICATES

Presentator

Presentatienotities

Not just OAIS, a lot of other standards had assisted us to

Designated Communities: Typologies and Demands V1.0 Designated Communities: Monitoring Mechanisms V1.0 Guidelines for Submission and Order Agreements V1.0 Standard Service Level Agreement V1.0

Frames of reference

PRODUCERS AND CONSUMERS

Information model Digital Archive v1.0 Preservation Metadata Dictionary v1.2

Frames of reference

PROCESSES AND DATA

• Life cycle management :workflows en events • Information Package Definitions SIP, AIP, DIP • Inventory of AV-technical and provenance metadata

Preservation planning

Frames of reference

STORAGE AND PRESERVATION

• Guidelines for Preservation Plans of Action v1.0 • Technology Watch

Storage

• Normative description

1. Security Policy

Frames of reference

ICT RELATED TOPICS

2. Backup Policy 3. Disaster Recovery Planning

STEP

S TO

TRU

STW

ORTH

INES

S Analysis of existing functions, processes and standards

Embedding a new MAM system in existing processes

Adaptation and implementation Mapping to standards,

gapanalysis

OAIS compliant definitions

OAIS compliant preservation workflows

AS IS ANALYSIS BUSINESS CONCEPT IMPLEMENTATION

STEP BY STEP

Presentator

Presentatienotities

Geleidelijke formalisering, completering en structurering van ALLE preservation metadata in ALLE processen d.m.v. Metadatamapping : bepalen (per formaat) waar en in welke bestaande en nieuwe systemen de technische en provenance metadata zitten en hoe ze daar heten, om ze toegankelijk te maken als preserveringsmetadata 2.0 versie Preservation Metadata Dictionary met technische, provenance en rechtenattributen per preserveerbaar AV-formaat (MXF, WAV, PDF, TIFF, DPX) Procesmapping : bestaande en nieuwe workflows en bewerkingen (events) identificeren in termen van OAIS compliant preserveringsfuncties, vastleggen in informatiearchitectuur

MAM AND PRESERVATION

WHAT DOES THE NEW MAM OF NISV OFFER TO PRESERVATION?

• MAM-system can meet a number of ISO 16363 (TDR) requirements

• All the remaining requirements need to be met through policies and work processes → But a MAM can help you a great deal with structuring your workflow management

TRUST IN THE CONTEXT OF DIGITAL PRESERVATION

Data level

Organisational level

Acquisition Selection

Registration Cataloguing Metadata

“content”

Services Education

B2B Marketing

Sales Museum

Communication

“customers and users”

Ingest

Storage Application

management Servicedesk

Digitalisation Digitaal Asset management

Analogue management

“technical

stuff”

COLLECTIONS

INFRASTRUCTURE

& IT

Producers (broadcasters) and heritage collections

Consumers (broadcasters, general public and educational stakeholders)

PROJECT

PORTFOLIO

MGT

R&D

MARKETING AND SALES

Presentator

Presentatienotities


A NEW ORGANISATIONAL MODEL

OAIS FUNCTIONS AT SOUND AND VISION

Ingest ICT, Digitisation dpt., Ingest Dept.

Archival Storage ICT + Ericsson Data Management (Limited coordination ) ICT, Access Dept., Metadata Mgt, afd. Ingest Dept, Development Dept.

Access Order service iMMix, KCC, Access Dept.

Preservation Planning (Ad hoc) ICT Mgt TechWatch and DC Monitoring: by KCC and R&D

Administration (Limited coordination) ICT, KCC, Access Dept. (?)

Presentator

Presentatienotities

Not just OAIS, a lot of other standards had assisted us to

ROLES AND RESPONSIBILITIES

Producers Broadcasters, ECV, heritage collections, Governmental Institutions, a.o.

Consumers Media professionals, Education, General Public Internal users

Management Digital Archive

Unitmanager Archive : strategy / policy Strategic/tactical policy and coordination: Collecton Policy Advisor : Digital Collection Policy, Selection/ Retention Policy Digital Preservation Officer: Preservation Policy, Quality Watch and Certification Information manager: Information policy, Metadata & Digital Life Cycle Management Domain managers : Accountmanagement ,Preservation- and Access Services IP - specialist : Access-, Use- and Rights management data policy

DIGITAL PRESERVATION IS NOT (just) digital storage IS NOT (just) migration IS NOT (just) technology IS NOT (just) format choices IS governing and controlling all activities for long term preservation in a logical and centralised way 1. Starting from the demands of users and

producers (and collectionpolicy)

2. In reality it’s mainly accomplished through integral metadata- and workflowmanagement embedded in a well- structured organisation

Levels of trustworthiness

CERTIFICATION

Basic Certification is granted to repositories which obtain Data Seal of Approval certification (DSA) Extended Certification is granted to Basic Certification repositories which in addition perform a structured, externally reviewed and publicly available self-audit based on ISO 16363 or DIN 31644 Formal Certification is granted to repos itories which in addition to B as ic C ertification obtain full external audit and certification based on IS O 16363 or equivalent DIN 31644.

Presentator

Presentatienotities

Sageguarding and keeping files accessible is not done by digitizing the collections. In fact, once you have you digitized and stored the content, most of the work is still to begin, and what’s more important, will never end. You have to make sure that the content is truly preserved, meaning that is is to be made permanently accessible to users. Storing is merely preserving bytes. Preservation is about being able to permanently reproduce and use the information in those bytes, in current formats, with current technology. Preservation is constant migration. In this way digital sustanability means nothing else than permanent access.

Levels of trustworthiness

CERTIFICATION

Basic Certification is granted to repositories which obtain Data Seal of Approval certification (DSA) Extended Certification is granted to Basic Certification repositories which in addition perform a structured, externally reviewed and publicly available self-audit based on ISO 16363 or DIN 31644 Formal Certification is granted to repos itories which in addition to B as ic C ertification obtain full external audit and certification based on IS O 16363 or equivalent DIN 31644.

Presentator

Presentatienotities

Sageguarding and keeping files accessible is not done by digitizing the collections. In fact, once you have you digitized and stored the content, most of the work is still to begin, and what’s more important, will never end. You have to make sure that the content is truly preserved, meaning that is is to be made permanently accessible to users. Storing is merely preserving bytes. Preservation is about being able to permanently reproduce and use the information in those bytes, in current formats, with current technology. Preservation is constant migration. In this way digital sustanability means nothing else than permanent access.

NEED TO KNOW MORE? Preservation policy: http://publications.beeldengeluid.nl/pub/388 End of 2016: paper with all details as well as links to all the documents.

Data & Analytics

de theory and practice of digital preservation