Archiving2010-ndiippbesser.tsoa.nyu.edu/howard/Talks/archiving2010-ndiipp.pdfSix Strategies for Sustainable Preservation of Born Digital Public Television • Project Background •

6/2/10

1

Besser & van Malssen, Archivng2010, 2/6/2010

Six Strategies for Sustainable Preservation of Born Digital Public Television Howard Besser & Kara Van Malssen

New York University http://dlib.nyu.edu/pdptv/

Six Strategies for Sustainable Preservation of Born Digital Public

Television •  Project Background •  Other Papers/Research •  Six Strategies



NDIIPP Preserving Digital Public Television Project-

•  Background & Goals •  Relevance to other domains •  Various sub-projects

–  Pushing Metadata Gathering Upstream –  Examining user needs for selection/quality –  Intellectual Property Issues –  Repository Design –  Sustainability Issues

•  Creation of American Archive


Background & Goals:

NYU/Public Television Project •  2004-2010 •  $6 million project -- 50% from LC/NDIIPP •  Marry asset management to preservation •  Preserve a broad set of elements (including ancillary material) •  Life-cycle mgmt (add metadata as soon as a clip comes in) •  Establish a community of stakeholders, working together for

preservation (stations, university, librarians, journalists, historians, producers, scholars, …)

•  Build an OAIS Repository •  Explore appropriate file formats, wrappers, METS extensions •  Develop sustainable business model


Project Partners •  Thirteen/WNET & WGBH – Content and production expertise

–  The two largest television stations in the PBS system –  Together produce largest percentage of national programs –  Both have preservation Archives

•  Public Broadcasting Service – More content and network design –  Distributes most of the national programming –  Determines and keeps ‘broadcast’ versions

•  New York University – Facilitation and Resources –  Leadership in designing digital libraries –  Experience in process for setting standards –  Has new Masters Program in Moving Image Archives


Public Television was analog for a long time

6/2/10

2


PBS Remote Tape Storage Facility


Public Television has become Digital for Production & Distribution

•  though some things still need to be streamlined


Programs are shot, assembled and edited digitally. Completed programs are packaged as digital files,

often with many different elements.

Typical AVID Editing Suite

The same huge video files are digitized over and over again for different uses.

A lot of video is not used for broadcast, but goes to the internet, to DVDs and to other media.


Local broadcasting is nearly all via digital systems

Thirteen Master Control manages one high definition, two analog and three digital over-the-air broadcast channels.


Broadcast playback is from a digital server

- no more tapes or tape machines All Thirteen broadcast programs are

played from this server.

It has to talk to the broadcast automation system, the satellite system, and other in-house and external networks.

It is the size of a window air conditioner and holds 1700 hours of material.

Other Papers/Research-

•  Pushing Metadata Upstream – Examination of production workflow

•  Appraisal/Selection •  Intellectual Property •  Repository Design •  Sustainability


6/2/10

3


Pushing Metadata Gathering Upstream: The Problem

TRADITIONALLY… •  Very little metadata required for

preservation accompanies an object to a repository.

•  Archives, libraries and other repositories must create (or re-create) most of the necessary metadata.

•  This requires many manual hours, and significant resources - both time and money.

IN THE DIGITAL WORLD… •  This doesn’t scale up. Repositories

will be unable to continue in this manner, as more metadata than ever is required.


But much of the necessary metadata has already been gathered during production

•  For each element/clip, production team usually notes source, date, place, people, and other descriptive info

•  But this is treated as internal information, and often various parts of the info are distributed among the personal notebooks of different production assistants

•  There is seldom a central location for this info, and the info is seldom turned over to the archive (which later tries to recreate much of it)


When the Archive tries to re-create this info, it is seldom successful

Producers know much more about the content of their productions than the archivists do. Archivists wanting accurate info must go back to the production staff (often years later) to start brainstoriming over the info

“Once the (television) program is finished, it is passed on to the archive or library for safe keeping. Librarians will catalog and classify the content, possibly using a proxy copy, and enter the resulting informative metadata in their database so they can retrieve it in the future. However, rarely if ever is the metadata from the rest of the process passed onto them, except, perhaps, for the title, tape number, and basic technical information about recording formats. It has to be re-created, with all the associated risk of errors and lack of accuracy--not to mention the work and time involved.”

- Cox, Tadic, and Mulder, Descriptive Metadata for Television (2006)


Similar issues w/other content types--E-Journals

•  “The necessary or additional metadata cannot be effectively and satisfactorily produced either as an afterthought post-production process on the publisher’s side or as a pre-ingest conversion activity at the archive’s end. Approaching e-archiving in this fashion leads to distribution delays and a more complex production and distribution scenario, with all the accompanying potential to introduce production delays and errors.”

- Yale University, YEA: The Yale University Archive, One Year of Progress, 2002


Solutions…?

•  Preservation becoming a shared responsibility between content creators, distributors, curators, and preservationists.

•  Partnerships are needed to come to unified solutions.

•  Preservationists seek reliable metadata back upstream in the production workflow...


Workflow in Production Process

•  Site Visits to productions •  Interview Production staff •  Diagrams of Workflow-

6/2/10

4


Public Television Workflow Basically similar to workflows in other fields…


For digital preservation, this shouldn’t be the only place for metadata in the preservation workflow! This is far too late in the cycle!

+


METADATA

METADATA

It also needs to be here!


We need to find ways to push metadata access upstream

•  Digital requires even more metadata than Analog –  As the workflow becomes file-based, the need for robust and

accurate metadata will become critical. File relationships, video codecs, bit rates, and rights information must be explicit, accurate, and immediately accessible. This will require a much deeper level of metadata than is currently captured in tape-based archives.

–  We can’t continue to supply this metadata at ingest; that won’t scale •  Obtaining the necessary metadata at the end of production and

broadcast life cycle is not feasible. Metadata will need to be systematically gathered during the production lifecycle and submitted with the programs to the preservation repository.

Examining user needs for selection/quality

•  Studies of Users and Potential Users – K-12 and Higher Ed instructors – Journalists – Historians and other researchers – Producers of newer works

•  What types of works do they want, and what quality is needed

•  Appraisal & Selection Report Besser & van Malssen, Archivng2010, 2/6/2010

Intellectual Property Report

•  “Intellectual Property and Copyright Issues Relating to the Preservation and Future Accessibility of Digital Public Television Programs” released April 2010


6/2/10

5

Intellectual Property Report (1/2)

•  What is required to keep digital content usable and accessible, particularly in regard to intellectual property issues –  documenting the experiences of the PDPTV team in building the

prototype repository –  examining production and distribution contracts and documents;

conducting IP audits on target programs –  reviewing current research and commentary on relevant technology

and copyright legislation –  monitoring parallel efforts by our colleagues in various fields, such

as publishing, higher education, information technology, and broadcast television in the US and around the world


Intellectual Property Report (2/2)

•  Identifies the context of rights agreements, procedures and economics by which public television programs are produced

•  Provides a detailed explanation of the many types of rights and permissions that can exist for any particular program

•  Discusses the technology sections of the copyright law and their impact on digital preservation

•  Looks at current efforts to amend the laws •  Presents case studies of two different public television series

that had to address major copyright issues before they could be reissued for re-broadcast and non-broadcast distribution


Repository Design

•  Design and develop a preservation repository for born-digital public television content –  Design of preservation environment, technologies

used to support preservation functions –  creation of AIPs for managing complex video files, –  identification of standards needed for various sub

-parts (METS, PBCore, PREMIS), •  (Joe Pawletko)


Sustainability

•  Examine issues of long-term content accessibility and recommend methods for sustaining digital preservation of public television materials

•  Look at technical, operational, & economic requirements for sustainable preservation of file-based television production (Kara van Malssen)


6 Strategies: Our Situation

•  Large, diverse, fragmented system •  No bosses/administrators over entire

system


6 Strategies: Our Report

•  Technical, Operational, and Economic requirements for digital preservation in our environment

•  Ideas based on OAIS and TRAC •  Also discusses

–  Some economic and business models for sustainability

–  Case study of costs related to the development of our initial prototype


6/2/10

6

6 Strategies: Important Issues

•  Saving bits is not enough; need to be discoverable, understandable, and playable

•  Need to go beyond the capabilities of a DAM or CMS, ensuring that we have: –  sound technical infrastructure –  good descriptive metadata –  well articulated policies –  a mission statement supporting that content

•  Interoperating factors constantly at play throughout the lifecycle, and need to be periodically revisited


Six Strategies- •  alignment of stakeholders •  integration of archival-friendly practices in the

digital production workflow •  following OAIS and TRAC guidelines •  the application of appropriate business

models •  clear communication of value •  ongoing re-evaluation of preservation needs

and users expectations Besser & van Malssen, Archivng2010, 2/6/2010

Define and Align Stakeholder Roles and Responsibilities (1/2)

•  No organization or person was previously responsible for preservation; public and corporate libraries only received a fraction of what was produced (only content that remained late in its life-cycle)


Define and Align Stakeholder Roles and Responsibilities (2/2)

•  All stakeholders in the public broadcasting community (producers, broadcasters, viewers, rights-holders, librarians/archivists, documentary-makers) need to work together to answer these questions: –  Who has the incentive to preserve? –  Who can take on the role of preservation? –  At what point in the lifecycle of the digital work do handoffs

take place? –  Who decides what is preserved (given the enormous amount

of material created for each production)? –  Who is responsible for providing descriptive metadata? –  Who can use the materials, and under what conditions?


Integrate Archival-Friendly Practices into the Production Workflow

•  Because we were dealing with many different entities (production units, local stations, national distributors), there was little consistency of metadata, file-naming, or even of compression ratios/image quality

•  First we developed workflows that allowed the new NYU Repository to collect the “best copy” from every source (in 8 different video file formats; metadata exports from 4 diverse DBs)-


Integrate Archival-Friendly Practices NYU collects from multiple sources/qualities/metadata


6/2/10

7

Integrate Archival-Friendly Practices Phase I Workflow and Mapping

very complex


Integrate Archival-Friendly Practices Phase I Workflow and Mapping very complex






•  This obviously was not scalable •  But in 2008 a nightly news show

launched, and we were able to institute good practices around standards and metadata, making these part of their workflow process

•  This resulted in much simpler work at the Repository end-




Integrate Archival-Friendly Practices Phase II Workflow and Mapping very simple

6/2/10

8


WorldFocus •  Nightly news program begun Oct 2008 •  We began working with Workflows six months before

program began •  Had ability to engineer metadata gathering into the

creation/production process

Follow OAIS and TRAC Guidelines for Digital Preservation

•  Trustworthy Repositories Audit and Certification: Criteria and Checklist (TRAC)

•  Digital Repository Audit Method Based on Risk Assessment (DRAMBORA) toolkit

•  Bit Preservation •  Content Accessibility

–  Identified and located –  Interpreted or played back –  Delivered to users

•  Organizational Infrastructure Besser & van Malssen, Archivng2010, 2/6/2010

Identify and apply appropriate organizational & business models

(old models won’t do)

•  Business models we used in analog world are unsustainable (grant funds, one-time expenses) –  Too often we’ve seen projects obtaining funds to

digitize, but not funds to store them, add metadata, build infrastructure, make them available on Web

•  Digital gives us opportunities to increase access, create new value, and enable people to find new uses for content



(need to diversity funding sources)

•  “A sustainable project covers its operating costs through a combination of revenue sources and cost-management strategies and continues to enhance its value based on the needs of the user community.”

-Sustaining Digital Resources: An On-the-Ground View of Projects Today, Ithaka Strategy and Research, 2009



(our first set of internal tasks)

•  Identify the interests, motivations, and roles of stakeholders involved, and how they are related to each other

•  Use this to build business models •  Example

–  predict how an entity, such as a program producer or a repository, might choose to allocate its resources under given economic constraints, and indicate what actions are required by other entities, such as users or public funders, to ensure that preservation occurs


Clearly Communicate Value of Preservation and Ongoing Access to Stakeholders (1/2)

•  “To make the case for preservation, make the case for use… [After all, preservation is a] “derived demand – an activity that people undertake in the service of something else they value. In the case of digital information, people care about the possibility of future access and use, and preservation creates that potential.”

-Sustainable Economics for a Digital Planet, Blue Ribbon Task Force (BRTF) on Sustainable Digital Preservation and Access


6/2/10

9

Clearly Communicate Value of Preservation and Ongoing Access to Stakeholders (2/2)

•  Stakeholders already implicitly know that they value the results of preservation –  Producers can potentially reuse materials for new

productions –  Distributors can reach new audiences and markets –  A university professor teaching a course on human rights

can show footage of actual events to spark student interest –  The public often likes to re-watch old favorite programs

•  But they need to see how a repository can facilitate their own downstream use, and be willing to make efforts to making that repository work (submissions, metadata, cooperation, ...)


Continually re-evaluate workflows, preservation requirements, and user needs (1/2)

•  At various points in the project, we have asked a variety of stake-holders access questions that we needed to shape the system –  What type of material do they want access to? –  What different uses can they think of? –  What quality and format of video and audio is

needed? –  What metadata is most important?


Continually re-evaluate workflows, preservation requirements, and user needs (2/2)

•  In the 6 years of this project, we’ve repeatedly needed to re-examine these issues, and we need to plan on that continuing far into the future, particularly as user expectations change from the other information resources that they use



Preserving Digital Public Television

http://dlib.nyu.edu/pdptv/

NYU Team Howard Besser

Kara van Malssen Joe Pawletko

Documents

Archiving2010-ndiippbesser.tsoa.nyu.edu/howard/Talks/archiving2010-ndiipp.pdfSix Strategies for Sustainable Preservation of Born Digital Public Television • Project Background •