Cni mc donough_preservation

Preservation-Worthy Digital Video: Hope You Brought Your

Checkbook

Jerome McDonoughNew York University

April 12, 2023

Digital Video Basics

A video signal consists of luminance and chrominance informationLuminance – brightness, varying from white to black (abbreviated as Y)Chrominance – color (hue & saturation), conveyed as a pair of color difference signals: R-Y (hue & saturation for red, without luminance) B-Y (hue & saturation for blue, without

luminance)


Where’s the green?Spectral response of the human eye peaks in the green frequencies. The perceived brightness of an item can be constructed using weighted values for its red, green and blue components: Y = 0.299R + 0.587G +0.114B, or

G Y R B 0 3 0 110 5 9. .

.


4:2:2 sampling 4:2:0 sampling

4:1:1 sampling

4:2:2 – High End DV (DigitalBetacam, DVCPro50)

4:2:0 – MPEG 1 & 24:1:1 – DV and DVCAM


Why not 4:4:4 sampling? 720 x 486 resolution = 349,920 pixels per frame 349,920 pixels x 10 bits/sample x 3 samples/pixel =

10,497,600 bits per frame 10,497,600 bits/frame X 29.97 frames/second =

314,613,072 bits per second 314,613,072 bps x 3600 seconds = ~141.58 GB/hour For 1920x1080 HDTV, more like 840 GB/hour

4:2:2 sampling drops that rate by a third with almost no perceptible difference in quality. 4:2:0 and 4:1:1 drop it in half.


MPEG 2 Compression Further subsampling

down sample to 8 bits/sample down sample to 4:2:0 sampling regime

Discrete Cosine Transformation + Requantizing of coefficients from DCT

Variable Length Encoding & Run Length Encoding Interframe compression (motion compensation)

all of which can take a 209 mbps video rate (for 4:2:2 video) and reduce it to around 8 mbps with no apparent visual loss.


Raw digital video is extremely storage and bandwidth intensive.As a result, almost all digital video processing systems employ a mix of lossless and lossy compression mechanisms.

Preservation-Worthy Digital Video

Desired characteristics for digital video we feel we can preserve include: Content can be migrated to new formats

and new media without introducing artifacts Stored in non-proprietary, standard format

which is openly documented Easy to produce derivatives for end-user

distribution Minimize costs of production, distribution &

migration

Sampling, Migration & Artifacts

As in still image digitization, employing lossy compression can lead to artifacting when you migrate.Unlike still images, lossy compression is assumed in almost all video processing technology today.

One River Media Codec Test Image

Images courtesy of Marco Solorio, One River Media

Black Magic 8-Bit 4:2:2 Uncompressed Codec


Black Magic 8-Bit: 10th Generation


One River Codec Test


Digital Voodoo 10-bit Codec


Digital Voodoo 10-bit: 10th Gen.


One River Codec Test


Apple 4:4:4 “None”: 10th Gen.


Sampling, Migration & Artifacts

4:4:4 sampling fulfills the digital promise of perfect copies across generations, butMost video equipment doesn’t actually support it. Most high-end video editing packages on computers do and will store 4:4:4 to disk.Lesson: if you want to store 4:4:4 uncompressed video, prepare to buy a lot of disk (or HSM), and abandon videotape.

Storing 4:4:4 Uncompressed Video

QuickTime – Proprietary, but publicly documented and does the task, and software support is availableMJPEG 2000 – Open Standard (ISO/IEC 15444), supports 4:4:4 uncompressed. Software support iffy, but growing.Material Exchange Format – Open Standard, but software support is weak, and some vendor issues

Storing 4:4:4 Uncompressed Video

Videotape is a non-starter. D1 tape decks for uncompressed video cost $200K, and use 4:2:2.Disk vs. HSM Tape Cost vs. Speed Opportunities to detect bit rot, ability to

migrate, time to produce derivatives

Waiting on grid storage….

Mind Games I: Conversion

NYU has approximately 30,000 hours of moving image material, undigitized, in its special collections. Let’s digitize 1/10 of that.9 Digitization/Editing workstations: $690,0009 conversion staff full time for 1 year: $350,000 425 TB of Disk Storage @ $10k/TB = $4,250,000Grand Total: $5,290,000FYI, according to ARL, that’s nearly half our entire 2002 materials budget

Mind Games II: On-going costs

Assume migration every 10 years. Assume time to migrate = 2x time of source material. 6,000 hours x staff salary = ~$120K / 10 = $12k/year

Assume new derivatives every 5 years, and time to migrate = 2x time of source material 12,000 hours x staff salary = ~$240k / 10 = $24k/year

3% disk loss/year x 425 TB = 12.75 TB replaced/year. Assuming disk prices are halved every two years, for next 10 years we’d have $38,750 total replacement costs, or $3,875/year.Grand Total: ~$40K/year maintenance costs

Mind Games III

On-going maintenance costs for 3,000 hours of video on disk aren’t particularly bad.Initial conversion costs, however, are nightmarish.If you don’t spend the money, however, your digital video is unlikely to prove any more preservation-worthy than analog.

NYU Costs: Capture Hardware

Complete system cost: ~$125,000.00

NYU Costs: Conversion Personnel

Currently conversion takes approximately 8 hours for every hour of tape.Minimum personnel cost of ~$150/hour (staff time + benefits) for conversionHope to lower conversion time with practice (and better equipment), but at best, probably around $100/hour of tape.

NYU Costs: Repository Hardware

Sun Enterprise 15K w/L700 Tape Backup: ~$400K/yearSun T3 Disk Arrays (10 TB): ~$100K ($10K/TB)

NYU Costs: Repository Personnel

Fractional part of NYU ITS Unix SysAdmin, Network Support Specialist, Tape Backup support, equivalent to about 1 FTE~$75K/year

NYU Costs: Summary

About $475K/year to keep our server alive and happy; $10K to add another terabyteAbout $125K to add a new video capture/editing workstationAbout $100-150 per hour of video to pay staff capture/conversion costs

The Good vs. the Perfect

Good: 1 hour of video dubbed to Digital Betacam (w/duplicate master) and converted to DVD & MPEG4 streaming derivatives $150 staff time + $70 (2 digibeta tapes) + $3 (DVD-

R blank w/case) + $24.75 (disk for MPEG4) = $247.75

Perfect: 1 hour of video converted to 4:4:4 uncompressed on disk (w/replicated backup) w/DVD & MPEG4 streaming derivatives $150 staff time + $2,860 (disk for master and

backup) + $3 (DVD-R) + $24.75 (disk for MPEG4) = $3,037.75

Affordable Perfection

Disk prices from 1992 to 2000 fell at about 45% per year.If that holds, by 2010, the 1 TB which costs us $10,000 today will cost $276.Storing 143 GB (one hour of 4:4:4 uncompressed video w/audio) will cost $39.50 ($4.50 more than a 60 minute Digital Betacam tape today).

Technology

Cni mc donough_preservation