Digitization Basics for Libraries, Archives, and Museums

Digitization Basics for Libraries, Archives and Museums

Martin R. Kalfatovic | 8 June 2016 | Chapel Hill, NC

DIGITIZATIONBASICSforLIBRARIESARCHIVESMUSEUMSand

Presentation CoverSingle portrait image

ARCHIVESMUSEUMSLIBRARIESand

1

A bit about meAbout this sessionWhy do we digitize?Context: Libraries, Archives and Museums (LAMs)Introduction

01.1 IntroductionJust a bit about me ...@[email protected]/in/martinkalfatovic an inordinate fondness for Dodos

01.2.1 IntroductionAbout this sessionTaking a broad overview of standards, lingo, hardware, software and planning considerations, this session will get everyone current and on a level playing field to proceed through the subsequent topics.

The session will help you establish the foundational vocabulary to both enrich your SEI experience and increase your capacity to communicate with your colleagues about the basics of digital reformatting.

The session will introduces the practices, standards, and challenges evident across the spectrum of cultural heritage institutions acquiring, managing, and providing access to digital collections. The session considers the digital curation life-cycle as well as lightly touching on funding and aggregators.

About this session1. Lecture/Discussion

2. Vocabulary Building

3. Exercises 01.2.2 Introduction

01.3 IntroductionContextLibraries | Archives | MuseumsIn principle, the work of art has always been reproducible. Objects made by humans could always be copied by humans. Replicas were made by pupils in practicing for their craft, by masters in disseminating their works, and, finally, by third parties in pursuit of profit. But the technological reproduction of artworks is something new. Having appeared intermittently in history, at widely spaced intervals, it is now being adopted with ever-increasing intensity.Das Kunstwerk im Zeitalter seiner technischen Reproduzierbarkeit. Walter Benjamin (1936)

01.3 IntroductionContextLibraries | Archives | MuseumsDigitization has magnified our ability to reproduce art, books, and even objects, with increasing rapidity, ease, and added functionality.

01.4 IntroductionProvide online access to collectionsMake digitized material and metadata available through online catalogs AND for reuse on other platforms.Maximize value to the largest audience in new and creative ways.Advance the preservation by reducing wear and tear on the originals.

Why do we digitize?Based on NARA strategic plan

01.4 IntroductionWhy do we digitize?Based on NARA strategic planProvide access to those materials that can no longer be accessed in their original format.Maximize the efficient and effective use of resources to carry out digitization and achieve cost-saving benefits whenever possible. Improve our service to customers by responding to their evolving expectations

2

Lets all count in binary!Analog vs. Binary : Wave vs. SampleBytes vs. BitsBasics

02.1.1 BasicsBASE 10 (Decimal)0 1 10 100 1,000 10,000 100,000 1,000,000

BASE 2 (Binary)0 1 2 4 8 16 32 64 128 256 512 1,024 2,048 5,096

BASE 3 (Ternary) not on the test!0 1 2 3 9 27 81 243 729 2,187 6,561 19,683BasicsLets All Count in Binary!

02.1.2 BasicsBasicsLets All Count in Binary!1 - 12 - 103 - 114 - 1005 - 1016 - 1107 - 1118 - 10009 - 100110 - 1010 11 - 101112 - 110013 - 110114 - 111015 - 111116 - 1000017 - 1000118 - 1001019 - 1001120 - 10100 91 - 101101192 - 101110093 - 101110194 - 101111095 - 101111196 - 110000097 - 110000198 - 110001099 - 1100011100 - 1100100

02.1.3 Basics

BasicsLets All Count in Binary!

02.1.4 Basics

BasicsLets All Count in Binary!

1ExercisesExercise 1: Lets Convert Between Binary and Decimal

02.2 BasicsAnalog vs. Binary : Wave vs. Sample

By Hyacinth - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=30716342By Hyacinth - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23867344

02.3 BasicsBytes vs BitsBITA bit is the smallest unit of information that can be stored or manipulated on a computer; it consists of either zero or one. AKA a bit a binary digit, especially when working with the 0 or 1 values.

BYTEA byte is how many bits are needed to represent letters of the alphabet and other characters. For example, the letter "A" would be 01000001. 8 bits = 1 byte

WORDGroups of 4 Bytes (translated into Hexadecimal Base 16!),e.g. 4B 4A 57 00 = K J W

02.4 BasicsNaming the Bytes

1 Bit = Binary Digit8 Bits = 1 Byte1000 Bytes = 1 Kilobyte 1000 Kilobytes = 1 Megabyte 1000 Megabytes = 1 Gigabyte 1000 Gigabytes = 1 Terabyte 1000 Terabytes = 1 Petabyte 1000 Petabytes = 1 Exabyte1000 Exabytes = 1 Zettabyte 1000 Zettabytes = 1 Yottabyte 1000 Yottabytes = 1 Brontobyte1000 Brontobytes = 1 Geopbyte

https://flic.kr/p/Knb8k

IDIGITIZATION VOCABULARYVocabulary I: Introduction and Basics

Term: Binary


Definition:

In mathematics and digital electronics, a binary number is a number expressed in the binary numeral system or base-2 numeral system which represents numeric values using two different symbols: typically 0 (zero) and 1 (one). The base-2 system is a positional notation with a radix of 2. Because of its straightforward implementation in digital electronic circuitry using logic gates, the binary system is used internally by almost all modern computers and computer-based devices. Each digit is referred to as a bit.Term: Binary


Term: Analog


Definition:

An analog signal has a theoretically infinite resolution. In practice an analog signal is subject to electronic noise and distortion introduced by communication channels and signal processing operations, which can progressively degrade the signal-to-noise ratio (SNR). In contrast, digital signals have a finite resolution.Term: Analog


Term: Byte


Definition:

The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit of memory in many computer architectures. The size of the byte has historically been hardware dependent and no definitive standards existed that mandated the size. The de facto standard of eight bits is a convenient power of two permitting the values 0 through 255 for one byte. Term: Byte


Term: Digitization


Definition:

The process of recording an analog signal in a digital form. In relation to content of this site, it describes the process of translating analog signal data emanating from an object (light or sound) into a digitally encoded format. Audio, still and moving images are commonly digitized for increased access or for preservation purposes.Term: Digitization


Term: Brontobytes


Definition:

The prefixbronto, as used in the termbrontobyte, has been used to represent anything from 1015to 1027 bytes, most often 1027Term: Brontobytes

3Get ready!Prioritization: Some things are more equal than othersTools to help you Get Ready and PrioritizeCopyright!

Measure Twice, Cut Once

03.1 Measure Twice Measure Twice, Cut OnceGet ready!Staffing ResourcesAcknowledge that digitizing for public access is a significant business process that crosses multiple business units. Develop a separate human resource plan to support this digitization business process. IT InfrastructureAlong with staffing, require an IT plan to support digitization that includes bandwidth, storage, the ability to share images and metadata across business units, among other requirements.

03.1 Measure Twice Policy and Guidance for Digitization ActivitiesPromulgate policy and guidance that provides further implementation direction as business units begin implementing the strategyTechnical Digitization StandardsDevelop technical digitization requirements for the approaches outlined above to ensure uniformity and standardization.Funding StrategiesSeek out and explore other options and relationships to digitize and make content availableMeasure Twice, Cut OnceGet ready!

03.2 Measure Twice Candidates for digitization projects will be prioritized according to established criteria for significance and use. Candidates for digitization projects will be prioritized in order to achieve a demonstrated high priority preservation benefit for the agency. Funding is available or likely to be available and sustainable for the project.Measure Twice, Cut OncePrioritization: Some things are more equal than others

03.3 Measure Twice Starting upTools to help you Get ready & PrioritizeDigitization PlansDigital Asset Management PlanWeb Access Plan

03.4 Measure Twice Copyrighthttps://xkcd.com/14/

Sometimes I just can't get outraged over copyright law ...

CopyrightTo promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.

Article I, Section 8, Clause 8 of the United States Constitution

but most of the time I am but still 03.4 Measure Twice

4

Staffing a digitization projectHowWho

04.1 StaffingStaffing a digitization projectDepending on the size of the institution, staff members may fill a number of roles. Also, do not forget that in addition to your regular staff, your volunteers, interns, and student help can participate in the digitization process (with the proper training and supervision).

04.2 StaffingStaffing a digitization projectHowIn-house staffingOutsourcingHybrid approach

04.3 StaffingStaffing a digitization projectWhoDirector / CEOProject ManagerCuratorTechnical StaffConservatorScanning Operators

04.3.1 StaffingStaffing a digitization projectDirector / CEOAs with any LAM activity, the overall responsibility for all functions ultimately rests with the director. Strong leadership and vision for digitization is necessary for a successful program.

04.3.2 StaffingStaffing a digitization projectProject ManagerManage goals, expectations, identify further staffing, equipment, liaison between departments and staff, create workplans and associated documents, manages funds.

04.3.3 StaffingStaffing a digitization projectCuratorOr, the person in charge of a collection. In addition to their responsibilities of caring for the collections, curators are also generally responsible for the display of the objects in coherent and informative or educational ways.

04.3.4 StaffingStaffing a digitization projectTechnical StaffDatabase development, web/database integration, CGI (Common Gateway Interface) script writing, Perl programming, and related activities that simplify the process of getting objects to the scanning operations and the resulting files in a usable state.

04.3.5 StaffingStaffing a digitization projectConservatorDepending upon the types of collections, consultation with the preservation / conservation staff in varying degrees will be necessary to determine if (and how) the items can be digitized and/or photographed.

04.3.6 WhoStaffing a digitization projectScanning OperatorsScanning, photography, handling materials (packing, shipping) and other such skills are just a few that may be required of staff doing the actual conversion.

5

Getting TechnicalPixel tricksColor and not colorFile formatsCost and implementation factorsFile namingCompressionBundling file formats

05.1 Getting TechnicalGetting TechnicalPixel TricksPixelAn abbreviation of picture element, this term may refer to a component of either a digital image or a digital sensor. In the case of a digital image, the pixel is the smallest discrete unit of information in the image's structure. In the case of the sensor in a scanner or digital camera, a pixel is the smallest photosensitive component or cell providing a response to light (or photons).

05.1.1 Getting TechnicalGetting TechnicalWhy do we care?Remember back to wave vs. sample? Pixels can be thought of as those elements of the samples that fall within the wave.

Getting Technical 05.1.1 Getting TechnicalAnalog vs. Binary : Wave vs. Sample

2ExercisesExercise 2: Worksheet: Calculate PPI

Exercise 2: Worksheet: Calculate PPI53

05.1.3 Getting TechnicalGetting TechnicalPixel ParametersSampling Frequency

This parameter measures the physical pixel count in pixels per inch (ppi), pixels per mm, etc. This parameter informs us about the size of the original and also provides part of the data needed to determine the level of detail recorded in the file.

Raster Still Images for Digitization A Comparison of File Formats. FADGI. 2014

05.1.3 Getting TechnicalGetting TechnicalPixel ParametersSharpening

Sharpening artificially enhances details to create the illusion of greater definition. There are three major sharpening processes in a typical imaging pipeline: capture sharpening (through camera setting adjustment), image sharpening in post processing, and output sharpening for print or display purposes. Sharpening is usually implemented through image edge enhancement, such as filtering techniques using unsharp masks and inverse image diffusion.


05.1.3 Getting TechnicalGetting TechnicalSharpening

Basic imageAutosharpenExtreme sharp

05.1.3 Getting TechnicalGetting TechnicalPixel ParametersReproduction Scale Accuracy

This parameter measures the relationship between the size of the original object to the size of that object in the digital image. This parameter is measured in relation to the pixels per inch (ppi) or pixels per mm (ppmm) of the original digital capture.


05.2 Getting TechnicalBitonalGrayscaleColorAdditive colorSubtractive colorGetting TechnicalColor and Not Color

05.2.1 Getting TechnicalGetting TechnicalColor and Not ColorRGB Color

GrayscaleBitonal

05.2.2 Getting TechnicalGetting TechnicalColor and Not Color: CMYK

The CMYK color model (process color, four color) is a subtractive color model, used in color printing, and is also used to describe the printing process itself. CMYK refers to the four inks used in some color printing: cyan, magenta, yellow, and key (black). Though it varies by print house, press operator, press manufacturer, and press run, ink is typically applied in the order of the abbreviation.

By Viliam Furk - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=39316936 (upper)

By SharkD at English Wikipedia Later version uploaded by Jacobolus, Dacium at en.wikipedia. - Transferred fromen.wikipediato Commons., Public Domain, https://commons.wikimedia.org/w/index.php?curid=3791468 (lower)

05.2.3 Getting TechnicalGetting TechnicalColor and Not Color: RBGThe RGB color model is an additive color model in which red, green, and blue light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green, and blue.

The main purpose of the RGB color model is for the sensing, representation, and display of images in electronic systems, such as televisions and computersWikipedia https://en.wikipedia.org/wiki/RGB_color_model

05.2.4 Getting TechnicalGetting TechnicalColor and Not Color: RBG vs. CYMKBy RGB_CMYK_4.jpg: Annette Shacklett derivative work: Marluxia.Kyoshu [Public domain], via Wikimedia Commons

A comparison of RGB and CMYK color spaces. The image demonstrates the difference between the RGB and CMYK color gamuts. The CMYK color gamut is much smaller than the RGB color gamut, thus the CMYK colors look muted. If you were to print the image on a CMYK device (an offset press or maybe even a ink jet printer) the two sides would likely look much more similar, since the combination of cyan, yellow, magenta and black cannot reproduce the range (gamut) of color that a computer monitor displays. This is a constant issue for those who work in print production. Clients produce bright and colorful images on their computers and are disappointed to see them look muted in print.

05.3 Getting TechnicalMaster / Archival / AccessImage file typesGetting TechnicalFile Formats

Getting TechnicalFile FormatsMasterA digital file (images, video, audio) which has been stored in its original captured state. These master files are also referred to as master copies, preservation masters or preservation copies.

05.3.1 Getting Technical

Getting TechnicalFile FormatsArchivalA file that is composed of one or more computer files along with metadata. Archive files are used to collect multiple data files together into a single file for easier portability and storage, or simply to compress files to use less storage space. Archive files often store directory structures, error detection and correction information, arbitrary comments, and sometimes use built-in encryption. 05.3.1 Getting Technical

Getting TechnicalFile FormatsAccess Often used to for low resolution images, (thumbnails, screen images) that are made available the Internet. See also delivery copy and surrogate image. This could be an identical copy of the original file or perhaps a lower quality version with a smaller file size. Sometimes called delivery or surrogate or derivative. 05.3.1 Getting Technical

RAWTIFFJPEGJPEG 2000PNGPDFGIF

Getting TechnicalImage file types 05.3.2 Getting Technical

File FormatsThree FactorsCostSystem ImplementationSustainability

Raster Still Images for Digitization A Comparison of File Formats. FADGI. 2014 05.4 Getting Technical

File FormatsCost FactorsImplementation CostCost of Software ToolsCost of equipment needed to produce filesStorage CostNetwork CostOngoing Cost of ProductionCost of Providing AccessCost of Preservation Processing

Raster Still Images for Digitization A Comparison of File Formats. FADGI. 2014 05.4.1 Getting Technical

File FormatsImplementation Factors

Raster Still Images for Digitization A Comparison of File Formats. FADGI. 2014Level of difficulty/complexityTechnical ComplexityToolset ComplexityAvailability of toolsEase and accuracy for OCREase and accuracy of File validationEase and accuracy of monitoring of quality 05.4.2 Getting Technical

File FormatsSustainability FactorsDisclosure AdoptionTransparencySelf-DocumentationNative Embedded Metadata CapabilitiesEmbedded Metadata Capabilities Through ExtensionLevel of Work Necessary to Embed Native MetadataLevel of Work Necessary to Embed Metadata Through ExtensionGeo-referencing MetadataLevel of Effort to Embed Geo-referencing MetadataImpact of PatentsTechnical Protection Mechanisms

Raster Still Images for Digitization A Comparison of File Formats. FADGI. 2014 05.4.3 Getting Technical

3Exercises

Exercise 3: Deep Dive: FADGI File Format Sheet

File FormatsFile Naming (Zhngmng | ) 05.5 Getting TechnicalThe Master replied, What is necessary is to rectify names. So! indeed! said Tsze-lu. You are wide of the mark! Why must there be such rectification?

Therefore a superior man considers it necessary that the names he uses may be spoken appropriately, and also that what he speaks may be carried out appropriately. What the superior man requires is just that in his words there may be nothing incorrect.From The Analects of Confucius, Book 13, Verse 3 (James R. Ware, translated in 1980):

File FormatsFile Naming: Some GuidelinesSemantic NamesThere is meaning encoded in the name, like:

BCA_03_04_00_145

Syrnium fulvescens from Biologia Centrali-Americana). Aves. Volume IV (1879-1904) by Osbert Salvin and F. DuCane Godman (4th volume of the 3rd part of Biologia Centrali-Americana, plate 145 05.5.1 Getting Technical

BCA_03_04_00_145.jpg

File FormatsFile Naming: Some GuidelinesPracticalUse barcodes, accession numbers, etc. 39088002738714-0003

Wiener Farbenkabinet (1794) 05.5.2 Getting Technical39088002738714-0003.jpg

File FormatsFile Naming: Three PartsPrefixOrdinal PositionSuffix 05.5.3 Getting Technical39088002738714-0003.jpg

File FormatsFile Naming: Three PartsPrefix

Bca_03_04_00

This is the part thats either semantic or practical 05.5.3 Getting TechnicalOrdinal

145

Position of the item in relation to a compound objectSuffix

jpg

File typeBca_03_04_00_145.jpg

File FormatsFile Naming: Last Thoughts 05.5.3 Getting TechnicalStick with three letter extension for the suffix (.tif, .jpg, .jp2, .png)Keep file names the same length (padding with Zeros not spaces!)Better to be consistent than right!

05.6 TechnicalFile FormatsCompression / Lossy / Lossless

Some file format types, specifically JPEG, JPEG 2000, and TIFF, allow you to compress the file size.

Compression leads to loss of data (since you are making the files smaller).

Be aware of this data loss when compressing files.

05.7 TechnicalFile FormatsBundling file formats

When you need to move many associated files around, you may wish to bundle them to pull them all together into one package. Common formats are:

ZIPTARRAR7z

IIDIGITIZATION VOCABULARYVocabulary II: Getting Technical

Term: Pixel


Definition:

An abbreviation of picture element, this term may refer to a component of either a digital image or a digital sensor. In the case of a digital image, the pixel is the smallest discrete unit of information in the image's structure. In the case of the sensor in a scanner or digital camera, a pixel is the smallest photosensitive component or cell providing a response to light (or photons). Term: Pixel


Term: Pixilation


Definition:

A term used to describe the an abrupt and unnatural transition over and edge feature. Also referred to as "staircasing" because of the jagged and abrupt transition.Term: Pixilation


Term: PPI


Definition:

PPI stands for pixels per inch, commonly used in describing the resolution capabilities of an imaging device such as a scanner or the resolution of a digital image. The terms DPI (dots per inch) and PPI are used somewhat interchangeably today.Term: PPI


Term: Pixel Dimensions


Definition:

Pixel Dimensions are the horizontal and vertical measurements of an image expressed in pixels. The pixel dimensions may be determined by multiplying both the width and the height by the DPI. A digital camera will also have pixel dimensions, expressed as the number of pixels horizontally and vertically that define its resolution (e.g., 2,048 by 3,072). Calculate the DPI achieved by dividing a document's dimension into the corresponding pixel dimension against which it is aligned.Term: Pixel Dimensions


Term: Color


Definition:

Color is the visual perceptual property corresponding in humans to the categories called red, blue, yellow, etc. Color derives from the spectrum of light (distribution of light power versus wavelength) interacting in the eye with the spectral sensitivities of the light receptors. Color categories and physical specifications of color are also associated with objects or materials based on their physical properties such as light absorption, reflection, or emission spectra. Term: Color


Term: Bit depth (image)


Definition:

The number of bits used to represent each pixel in an image. The term can be confusing since it is sometimes used to represent bits per pixel and at other times, the total number of bits used multiplied by the number of total channels. For example, a typical color image using 8 bits per channel is often referred to as a 24-bit color image (8 bits x 3 channels). Color scanners and digital cameras typically produce 24 bit (8 bits x 3 channels) images or 36 bit (12 bits x 3 channels) capture, and high-end devices can produce 48 bit (16 bit x 3 channels) images. A grayscale scanner would generally be 1 bit for monochrome or 8 bit for grayscale (producing 256 shades of gray). Bit depth is also referred to as color depth.Term: Bit depth (image)


Term: CMYK


Definition:

A subtractive color model used in printing that is based on cyan (C), magenta (M), yellow (Y) and black (K). These are typically referred to as process colors. Cyan absorbs the red component of white light, magenta absorbs green, and yellow absorbs blue. In theory, the mix of the three colors will produce black, but a black ink is used to increase the density of black in a print.Term: CMYK


Term: RGB


Definition:

An additive color model based on the three primary colors of red (R), blue (B) and green (G).Term: RGB


Term: Grayscale


Definition:

Grayscale is a range of monochromatic shades from black to white. Therefore, a grayscale image contains only shades of gray and no color.Term: Grayscale


Term: Bitonal


Definition:

A bitonal image is represented by pixels consisting of 1 bit each, which can represent two tones (typically black and white), using the values 0 for black and 1 for white or vice versa.Term: Bitonal


Term: Continuous tone


Definition:

Generally referring to pictorial images where there is a non-broken range of tones from white to black that may have every shade of gray represented. There are theoretically an infinite number of tones. Traditional photography (photochemical photography) produces continuous tone images. When reformatting pictorial items, it is important to distinguish continuous tone originals from printed halftones, since these two classes are likely to require different strategies and methods for making the digital images.Term: Continuous tone


Term: Color Space


Definition:

The choice of colorspace determines how many unique colors are potentially possible in your digital file, and how fine the gradations are between shades of color.Each colorspace was designed for a specific purpose, none is superior to the others for all applications. However, FADGI recommends selecting an appropriate colorspace from the recommendations in the charts in this document.Term: Color Space


Term: OCR


Definition:

Optical Character Recognition (OCR) is a technology that allows dots or pixels representing machine generated characters in a raster image to be converted into digitally coded text. In addition to recognizing and coding text, OCR programs attempt to recognize and code the structural elements of a document page, such as columns and non-text graphical elements. Term: OCR


Term: Archival master file


Definition:

File that represents the best copy produced by a digitizing organization, with best defined as meeting the objectives of a particular project or program. These objectives differ from one content category to another and the specifications to be recommended at this Web site (forthcoming) will be tailored to fit a variety of common categories and objectives. In some cases, an archive may produce more than one archival master file.Term: Archival master file


Term: Master File Format


Definition:

The choice of master file format is a decision which affects how your digitized materials can be used and managed. There is no one correct master file format for all applications, all format choices involve compromises between quality, access and lifecycle management. The FADGI star system tables list the most appropriate master file formats for each imaging project type. Selection of the most appropriate format within these recommended choices is an important decision that should be consistent with your long term archive strategy.Term: Master File Format


Term: Access File Format


Definition:

FADGI anticipates continual evolution in the availability of access file formats, each new format designed to provide specific advantages over others for a specific application. Care should be taken when selecting access formats to insure long term viability.Term: Access File Format


Term: Vector graphics


Definition:

Vector graphics is the use of polygons to represent images in computer graphics. Vector graphics are based on vectors, which lead through locations called control points or nodes. Each of these points has a definite position on the x and y axes of the work plane and determines the direction of the path; further, each path may be assigned a stroke color, shape, curve, thickness, and fill.Term: Vector graphics


Term: Bundling file format


Definition:

These formats encapsulate their constituent files and, save for a directory that provides the filenames, do not describe the content and the relationships that may obtain between files. (In this, they differ from what are often called wrappers.) Archetypes include ZIP, StuffIt, and TAR, the latter associated with the UNIX operating system. Simple bundling formats tend to be generic, i.e., they may be used for a wide range of content types.Term: Bundling file format

6

Do it yourself (DIY)

06.1 DIYDo It Yourself!Describing the Collection

06.1.1 DIYDo It Yourself!Describing the Collection


The sheer number of metadata standards in the cultural heritage sector is overwhelming, and their inter-relationships further complicate the situation. This visual map of the metadata landscape is intended to assist planners with the selection and implementation of metadata standards

Seeing Standards: A Visualization of the Metadata Universe by Jenn Riley


Each of the 105 standards listed here is evaluated on its strength of application to defined categories in each of four axes: community, domain, function, and purpose. The strength of a standard in a given category is determined by a mixture of its adoption in that category, its design intent, and its overall appropriateness for use in that category.

Seeing Standards: A Visualization of the Metadata Universe by Jenn Riley

06.2 DIYDo It Yourself!Digitization Tools

06.2.1 DIYDo It Yourself!Scanner vs. Camera

The Worlds First Digital Camera (1975) by Kodak and Steve Sasson

06.2.1 DIYDo It Yourself!Camera

At the heart of a digital camera is the sensor. The size and density of the sensor determines the pixel count of the resulting image.

The sensor in combination with an optical lense creates the digital image.

By C-M - own Image, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=2150801

06.2.3 DIYDo It Yourself!Scanner

A scanner is a much different device; the sensor array is long and thin.

By moving across the target at varying speeds and angles, higher or lower resolution outputs can be generated.By Scanner_a_plat_fonctionnement.png: User:Jean-noderivative work: Pluke (talk) - Scanner_a_plat_fonctionnement.png, FAL, https://commons.wikimedia.org/w/index.php?curid=17009063

06.2.4 DIYDo It Yourself!A Note on Sensors

CCD vs CMOS

Charge-Coupled Devices vs. Complementary MetalOxideSemiconductor

Both types of sensor accomplish the same task of capturing light and converting it into electrical signals.

By Filya1 - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=6304562

06.3 DIYDo It Yourself!Digitization Equipment

There are many options for equipment to create digital files from collections objects:

ScannersSlide/Negative scannersSpecialized toolsThe Worlds First Digital Camera (1975) by Kodak and Steve Sasson

06.3.1 DIYDo It Yourself!Digitization Equipment: Flatbed Scanners

25-50 Page Automated Document FeederFlat Bed ScanningSupport for either TWAIN and/or ISIS interface driversUSB or SCSI InterfaceSupport for largest expected documentsDuplex (Automatic) scanning (two sides at one pass)Optical 200 x 200 - 600 x 600 Dots Per Inch (DPI)

06.3.2 DIYDo It Yourself!Digitization Equipment: MFM / Slide / Neg

Remember to keep in mind your original size with doing transparencies and negatives. You will need to scan at a much higher PPI to create an image closer in size to the original for display or printing

Microfilm scannersSlide & Negative scanners

06.3.4 DIYDo It Yourself!Specialized Tools

Digital camerasScanning back camerasHigh speed book scanners3D scanners

06.3.4.1 DIYDo It Yourself!Digital Cameras

Digital cameras come if a vast array of sizes, shapes, and formats.

DT ATOMFrame footprint: 24.625 wide x 31 deepWith table top: 25.625 wide x 31.25 deepHeight with Column: 65.5Light Arm Span: 78.5

06.3.4.1 DIYDo It Yourself!Digital Cameras

DT BC100MaterialsFrame is made from black anodized extruded aluminum, custom brackets are made from black anodized aircraft grade aluminum.Overall Dimensions7H x 65 W x 5DFootprint6W x 5DGlass Platen Dimensions24.9 x 17.48 on each sideBook Binding Limitations6 bindingWorking Table Height30Accessory & Monitor Shelf DimensionsSide shelves are 19 x 34 black laminateCompressor.5 HP with a 6.3 gallon tank

06.3.4.2 DIYDo It Yourself!Scanning Back Cameras

Scanning back cameras provide generally higher resolution by replacing the sensor array with a sensing device that scans across the image created by the camera lens.

More megapixel imagesSlower scanning times

XF Phase OneDT RCamDT RG3040 Images from Digital Transitionshttp://dtdch.com/

06.3.4.3 DIYDo It Yourself!High Speed Book Cameras

For high throughput, robotic scanners are available for some materials, such as books.

06.3.4.4 DIYDo It Yourself!3D Equipment

There are a wide variety of 3D digitization tools and processes. What type of equipment and process you use should be carefully thought out with the end-use of the digitization in the forefront.

Smithsonian 3D Imaging Team

06.4 DIYDo It Yourself!Where to do it: The Gray Room

Having dedicated workspace for your digitization is, of course, optimal. In reality, digitization will occur wherever it is most practical.

Still, if you have the luxury of dedicated space, here are some guidelines on building it out.Internet Archive, San Francisco

06.4.1 DIYDo It Yourself!FADGI Space Guidelines

The working environment should be painted/decorated a neutral, matte gray with a 60% reflectance or less to minimize flare and perceptual biases.FADGI Guidelines


Monitors should be positioned to avoid reflections and direct illumination on the screen.

FADGI Guidelines

Smithsonian Libraries Scanning Room


ISO 12646 requires the room illumination be less than 32 lux when measured anywhere between the monitor and the observer, and the light a color temperature of approximately 5000K. Consistent room illumination is a fundamental element of best practice in imaging. Changes in color temperature or light level from a window, for example, can dramatically affect the perception of an image displayed on a monitor.FADGI GuidelinesSmithsonian Libraries Scanning Room


Each digitization station should be in a separate room, or separated from each other by sufficient space and with screening to minimize the light from one station affecting another. It is critically important to maintain consistent environmental conditions within the working environment.

FADGI GuidelinesInternet Archive, San Francisco


Care should be taken to maintain the work environment at the same temperature and humidity as the objects are normally kept in. Variations can cause stress to some materials and in severe cases may damage the originals. The use of a datalogger in both imaging and storage areas is highly recommended. FADGI GuidelinesSmithsonian Libraries Scanning Room

06.5 DIYDo It Yourself!Quality Control

Quality Control (QC), or Quality Assurance (QA), is key to maintaining the overall quality and fidelity of any digitization project. Differing levels of QC may be needed for the type of project and materials being digitized.

In large scale projects, 100% QC will rarely be feasible.http://www.sil.si.edu/imagegalaxy/imagegalaxy_imageDetail.cfm?id_image=7403

4Exercises

Exercise 4: Quality Control Sampling

06.6 DIYDo It Yourself!Storage

Now that weve created all this data, we need to store it somewhere

https://flic.kr/p/86miWv

06.6.1 DIYDo It Yourself!Storage

Short TermMedium TermLong Term

https://flic.kr/p/86miWv

06.6.2 DIYDo It Yourself!Primary magnetic storage

DiskettesHard disks (both fixed and removable)High capacity floppy disksDisk cartridgesMagnetic tapeSmithsonian Data Centerhttps://flic.kr/p/cGFn2f

06.6.3 DIYDo It Yourself!Primary optical storage

Compact Disk Read Only Memory (CD ROM)Digital Video Disk Read Only Memory (DVD ROM)CD Recordable (CD R)CD Rewritable (CD RW)

Smithsonian Data Centerhttps://flic.kr/p/cGFn4E

06.6.3 DIYDo It Yourself!Solid-state storage

Solid-state storage is a type of non-volatile computer storage that stores and retrieves digital information using only electronic circuits, without any involvement of moving mechanical parts. (Wikipedia)

Examples:SSDFlash drive

Internet Archivehttps://flic.kr/p/dnDS11

06.6.4 DIYDo It Yourself!Acronyms

DASNASSANDAM

Internet Archivehttps://flic.kr/p/8Ms4QV

06.6.4.1 DIYDo It Yourself!Direct-attached storage (DAS)... is a traditional mass storage, that does not use any network. This is still a most popular approach. This retronym was coined recently, together with NAS and SAN. (Wikipedia)Internet Archivehttps://flic.kr/p/8Ms4QV

06.6.4.2 DIYDo It Yourself!Network-attached storage (NAS)

is mass storage attached to a computer which another computer can access at file level over a local area network, a private wide area network, or in the case of online file storage, over the Internet.. (Wikipedia)

Internet Archivehttps://flic.kr/p/8Ms4QV

06.6.4.3 DIYDo It Yourself!Storage area network (SAN)

... is a specialized network, that provides other computers with storage capacity. The crucial difference between NAS and SAN is the former presents and manages file systems to client computers, whilst the latter provides access at block-addressing (raw) leve. (Wikipedia)Internet Archivehttps://flic.kr/p/8Ms4QV

06.6.4.3 DIYDo It Yourself!Digital asset management (DAM)

consists of management tasks and decisions surrounding the ingestion, annotation, cataloguing, storage, retrieval and distribution of digital assets. (Wikipedia)Internet Archivehttps://flic.kr/p/8Ms4QV

06.6.5 DIYDo It Yourself!Digital Preservation

http://www.xkcd.com/1683/

06.6.5 DIYDo It Yourself!Digital Preservation

https://xkcd.com/242/

There are two kinds of preservationists: those who have lost data and those who will.Minimum Digitization Capture Recommendations (2013)

06.7 DIYDo It Yourself!Recap of Scanning

Scan at best resolution you can afford to storeManuscripts and text: 300 ppiPhotographs: 400-800 ppiGraphic materials: 600-800 ppiMaps: 600 ppi (up to 36) or 300-400 pp (greater than 36)Calibrate monitor and scanning devices Smithsonian Libraries Scanning Room

06.8 DIYDo It Yourself!Recap of Process

Create master (uncompressed) fileFor analog content: Scan/sampleFor born-digital content: ConvertName the file in a consistent wayPerform quality control;edit as neededSave master on stable, long-term storageCreate derivative or access fileShare access files as neededhttp://xkcd.com/730/

06.9 DIYDo It Yourself!Digitization Life-cycle

Create master (uncompressed) fileFor analog content: Scan/sampleFor born-digital content: ConvertName the file in a consistent wayPerform quality control;edit as neededSave master on stable, long-term storageCreate derivative or access fileShare access files as neededBiodiversity Heritage Library Digitization Life-Cycle

06.10 DIYDo It Yourself!Better, Faster, Cheaper!

Now that weve mastered digitization. How do we scale it to 100s, 1000s, millions! of objects?

Smithsonian Natural History Museum

06.10 DIYDigitization Program Office (Smithsonian)

The old prioritization methodology focused on whittling down collections to a manageable size.

160

THE OLD PARADIGM

This seems like aninteresting and feasiblesubset



The new prioritization methodology is focused more on the physical characteristics of collections; identifying large, homogenous collections.162

THE NEW PARADIGM

These homogenous subsets have rapid-capture technologies available


These are known as digistreetsThe opportunity is the massive economies of scale that industrial-scale digitization provides.

163

IIIDIGITIZATION VOCABULARYVocabulary III: Do It Yourself!

Term: Array


Definition:

Any orderly arrangement of individual sensor elements. In digital imaging, there are primarily three array types; two dimensional or area arrays, one dimensional or linear arrays, and tri-linear arrays consisting of three consecutive linear arrays of red, green, and blue sensitive sensor elements.Term: Array


Term: Ambient light


Definition:

Light existing in the environment that is not produced by the imaging system. Ambient light can be natural or artificial light. Ambient light is generally uncontrolled and can be highly variable, posing a possible risk to image quality. The level of ambient light should be minimized in relation to the level of light produced by the imaging system.Term: Ambient light


Term: Calibration


Definition:

The comparison of instrument performance to a standard of higher accuracy. The standard is considered the reference and the more correct measure. Calibrations should be performed against a specified tolerance.Term: Calibration


Term: Exif


Definition:

Exchangeable image file format (Exif) describes a metadata set to accompany TIFF, JPEG, and RIFF WAV formatted image files. Exif was prepared by the Technical Standardization Committee on AV & IT Storage Systems and Equipment and is Published by the Japan Electronics and Information Technology Industries Association (JEITA ). The Exif 2.2 specification (JEITA CP-3451) is in nearly universal use by camera manufacturers. Term: Exif


Term: IPTC Metadata


Definition:

Embedded metadata used for image management. IPTC metadata is primarily composed of descriptive, administrative, and rights metadata, as opposed to the technical nature of Exif. IPTC metadata was developed and is controlled by the IPTC.Term: IPTC Metadata


Term: METS


Definition:

The Metadata Encoding and Transmission Standard (METS) is a metadata standard for encoding descriptive, administrative, and structural metadata regarding objects within a digital library, expressed using the XML schema language of the World Wide Web Consortium (W3C). The standard is maintained as part of the MARC standards of the Library of Congress, and is being developed as an initiative of the Digital Library Federation (DLF).Term: METS


Term: Aspect ratio


Definition:

The relationship between the horizontal and vertical dimensions of an image. The horizontal dimension is normally listed first. For example, a 4 (vertical) by 6 inch (horizontal) print has an aspect ratio of 3:2.Term: Aspect ratio


Term: Artifact (defect)


Definition:

General term to describe a broad range of undesirable flaws or distortions in digital reproductions produced during capture or data processing. Some common forms of image artifacts include noise, chromatic aberration, blooming, interpolation, and imperfections created by compression, among others. In digital sound recordings, the effect of lossy compression is often cited as accounting for audible artifacts, although several other types of artifacts may also be present.Term: Artifact (defect)


Term: Aliasing


Definition:

A sampling effect that leads to spatial frequencies being falsely interpreted as other spatial frequencies.Term: Aliasing


Term: Compression, lossless


Definition:

Data compressed using a lossless compression technique will allow the decompressed data to be exactly the same as the original data before compression, bit for bit.The compression of data is achieved by coding redundant data in a more efficient manner than in the uncompressed format. The Compression ratios that can be achieved with lossless compression are generally much lower than those that can be achieved using lossy compression techniques.Term: Compression, lossless


Term: Compression, lossy


Definition:

Data compressed using a lossy compression technique results in the loss of information. The decompressed data will not be identical to the original uncompressed data. Conservative lossless compression can result in a form of lossy compression referred to as visually lossless compression.Term: Compression, lossy


Term: Compression, visually lossless


Definition:

A form or manner of lossy compression where the data that is lost after the file is compressed and decompressed is not detectable to the eye; the compressed data appearing identical to the uncompressed data.Term: Compression, visually lossless


Term: Compression ratio


Definition:

The ratio of a files uncompressed size over its compressed size. A file compressed ten-fold over its uncompressed size would be described as having a ten-to-one compression, expressed as 10:1. Some formats such as JPEG and JPEG 2000 allow the user to specify the compression ratio.Term: Compression ratio


Term: FADGI


Definition:

Federal Agencies Digitization Guidelines Initiative. A collaborative effort by federal agencies formed as a group in 2007 to define common guidelines, methods, and practices to digitize historical content in a sustainable manner. Term: FADGI

7

A Little Bit about Smithsonian Rapid Capture and 3D Imaging

07.1 SmithsonianA Little Bit about Smithsonian Rapid Capture

Courtesy of Keri Thompson, SIL Courtesy of Karen Weiss, AAA feat. Archives of American Art case study

What Is Rapid Capture?Rapid Capture is more than just taking a digital picture of an object or specimen. . .

Rapid Capture Workflows are comprehensive, end-to-end digitization workflows.

Rapid Capture workflows follow a collection object or specimen from its shelf in permanent storage all the way to its potential destination as a virtual object online, available for access by the public.

07.1 Smithsonian

- Many of you have seen our Rapid Capture Projects during Open Houses and as of now, 5 units have been deeply immersed in Rapid Capture through our Pilot Projects. . .

But for those that havent had the opportunity to see Rapid Capture in action, let me give a quick definition of what were doing. . .

First and foremost. . .196

THE NEW PARADIGM

These homogenous subsets have rapid-capture technologies available

These are known as digistreetsThe opportunity is the massive economies of scale that industrial-scale digitization provides.

197

123456ABC

123456ABC

CDIS

DAMS HotFolder IngestDAMSTMS

IDS

StagingRapid Capture Digitization: Object & Data WorkflowBarcode put in Filename and/or IPTC Title field:

Object PathData PathDataMatrixBarcode

Generates Derivative Media ImageGenerates Derivative Metadata

123456ABC123456ABC123456ABD123456ABD123456ABE123456ABE

Rapid Capture workflows follow a collection object or specimen from its shelf in permanent storage, to its digitization, to storage of the image in DAMS and of the collection records in your units CIS, and finally, all the way to its potential destination as a virtual object online, available for access by the public.198

Show what rapid capture looks like

Rapid Capture In ActionNMAH NumismaticsHillery York, NMAH Collection Mgr, moves objects from staging to the capture station

In addition to the various types of objects weve digitized at each pilot project, weve also integrated various SI collections management systems into our rapid capture workflows to include SIRIS at Gardens, TMS at NMAAHC & Freer Sackler and MIMSY-XG at American History.

Additionally, during each pilot project weve introduced new techniques, technologies and processes. For example at Gardens we introduced new tools that support Quality Control in the digitization workflow and at American History we integrated Transcription Center into the Rapid Capture workflow.

In our next pilot project with Natural History well integrate the EMu Collection Management System into our workflow, and well also introduce barcoding into the rapid capture workflow

199

Rapid Capture ImpactAccess:From the shelf to the public in less than 24 hours.

Throughput:Flat objects: 100,000+ to 1.8M per yearNon-flat objects/specimens: 30,000 to 60,000 per year

07.1 Smithsonian

The impact of these RCPP have shown us several things. . .

- In typical Rapid Capture workflows, its possible for an object or specimen to go from permanent storage, where it hasnt been seen by the public in years, decades or perhaps ever, to publicly accessible Smithsonian websites in as little as a few hours!

- And because Rapid Capture workflows are fine tuned, improved & continuously optimized at every step of the way, depending on the collection object or specimen, high quality digital assets can be generated at throughput rates of 150 objects per day to 700 to 1,300 specimens, and upwards to 6,000 objects or specimens per day!200

What weve learnedMoving fast requires a holistic approach.

Object handling, cleaning, etc.Dedicated hardware,Quality controlNetwork / Systems

07.1 Smithsonian

To take advantage of the opportunities presented by comprehensive, end-to-end rapid capture workflows, weve learned we need to take a holistic approach to our workflows; workflows which include not only the digitization process itself, but the object or specimen handling that comes before it and the movement of mass amounts of data that come after it.201

From Human-Driven Systems

07.1 Smithsonian

Weve shown what we can do with Human Driven Rapid Capture Workflows. . . 202

To Conveyor-Driven Systems

07.1 Smithsonian

and the DPO is actively researching technologies available in the digitization community from around the world. . .203

Rapid Capture Conveyor Powered

07.1 Smithsonian

To include semi-automated conveyor based digitization systems that have demonstrated their efficiency with high volume, flat collections.

As well as robotic systems that can digitize small objects such as insects and large objects such as painting at extremely high resolutions.

The bottom line is were constantly looking for ways to expand our capabilities, improve efficiencies and reduce costs all while maintaining the highest quality levels.

204

07.2 SmithsonianA Little Bit about Smithsonian 3D Imaging

8

ResourcesOverview of Resources

08 ResourcesYou Are Not Alone!There are a wealth of resources to help with digitization project of all types Resources

08 ResourcesDigital Library FederationStrategy meets practice at the Digital Library Federation (DLF). Through its programs, working groups, and initiatives, DLF connects the vision and research agenda of its parent organization, the Council on Library and Information Resources (CLIR), to an active and exciting network of practitioners working in digital libraries, archives, labs, and museums. DLF is a place where ideas can be road-tested, and from which new strategic directions can emerge.

Resourceshttps://www.diglib.org/

08 ResourcesMuseums and the WebThe Museums and the Web Bibliography comprises all papers published on MW conference websites or in annual selected proceedings. Entries can be filtered by year and are listed alphabetically by the primary author's name. Clicking a paper title shows details including an abstract and a live URL link if appropriate. Clicking an author's name lists all papers by that author. This bibliography is a work in progress as we standardize all entries..

Resources

http://www.museumsandtheweb.com/

08 Resources Federal Agencies Digitization Guidelines InitiativeFederal Agencies Digitization guidelines Initiative. Formed as a group in 2007 to define common guidelines, methods, and practices to digitize historical content in a sustainable manner. Two separate working groups were formed.

The Federal Agencies Still Image Digitization Working Group will concentrate its efforts on image content such as books, manuscripts, maps, and photographic prints and negatives.

The Federal Agencies Audio-Visual Working Group is focusing its work on sound, video, and motion picture film.

Resourceshttp://www.digitizationguidelines.gov/

9

Last Thoughts

http://xkcd.com/1685/

THANKYOU!QUESTIONS?


THANKSTOSarah Osborne BenderSmithsonian Digitization Program OfficeGnter Waibel Adam Metallo Vincent Rossi Richard Naples (Smithsonian Libraries)Keri Thompson (Smithsonian Libraries)Jacqueline Chapman (Smithsonian Libraries)
