Moving Media StorageTechnologiesApplications & Workflows

for Video and Media Server

Platforms

Karl Paulsen

CONTENTS

Acknowledgments xv

Introduction xvii

Chapter 1 Information, Data, and Video 1

Data Structure 1Information 2

Storing the Moving Image 3

Digital Video Recording 4

Analog Component Video 5

Colorimetry and Conversion 11A Digital Media World: Going Forward 12Further Readings 13

Chapter 2 Digital Disk Recording 15

Recording Moving Images 16Dawn of the Video Disk 17Fundamentals of Digital Disk Recording 18Core Technologies for Digital Disk Recording 21

DDR, DVR, or VideoServer 28

Comparing DDRs to VideoServers 31

Clip Servers 32DVR Marketplaces 35Home Media Network Disk Recorders 36Audio 37

Further Readings 38

Chapter 3 Video Media Servers 39

Carrying Over from the DDR 39Universal Server Approach 41

VideoServers as Media Servers 46The Future of VideoServers 66

Communications Protocols Used in Video Serving Platforms 66

Further Readings 71

Chapter 4 Hard Drive Anatomy 73

Magnetic Recording History 74

Drive Components 77

Disk Performance Optimization 85

Disk Scheduling 85

Access Time Components 87

Servicing and Scheduling Criteria 88

Capacity Factors, Transfer Time and Obstacles 95Read-Write Heads 97

Advanced Format Sector Technology 102

Superparamagnetic Limit 106

Heat-Assisted Magnetic Recording. 107

Further Reading 108

Chapter 5 Buses, Ports, and Interfaces 109

Carrying and Transferring Data 110

Bus Types 110

Storage Bus Standards 112

Storage Module Device 112

Serial Storage Architecture 114

High-Performance Parallel Interface 115

IDE 116

Parallel ATA 116

SCSI 117

Further Readings 124

Chapter 6 SATA, SAS, and RAID 125

Transitioning from Parallel to Serial Technologies 126Serial Attached SCSI (SAS) 127SATA 133RAID 139Alternative Disk Sets and Configurations 155Error Prevention Methods 159

Conclusions 163Further Readings 163

Chapter 7 Solid State Disks 165

Solid State Storage Evolution 166

Dynamic Random Access Memory 167

Flash Memory 167

Flash Memory Functionality 168

Portability and Green Perspectives 182

Supplementing Video on Demand with SSD 183

Further Readings 185

Chapter 8 Optical Storage Media 187

Defining Removable Media 188

Optical Media Definitions 188

Blu-rayDisc 195

Holographic Storage 202

Care and Handling 210

Further Readings 211

Chapter 9 Interchange, Transcoding, and Wrappers 213

Moving Toward Non-Real Time 214

Compressed Video 216

Issues with Interchange 217

Media File Vocabularies and Terminologies 218

Content 229

Material exchange Format (MXF) 229

Task Force for MXF 230

Operational Patterns 233

Specialized Operational Pattern Atom 234

Items and Packages 236

Essence Container 238Conversion and Transcoding 239

Converting Audio and Video Essence 240

Transcode Pipeline 244

Multiplatform Conversion and Distribution 246

Examples and Conclusions 252

Further Readings 253

Chapter 10 IP Storage 255

Internet Protocol 255Delivering Packets by IP 256Connected or Connectionless 256IP Address Classes 257Addresses

257

Domain Naming 258Ethernet 259Internet Engineering Task Force 262IP Storage 262iSCSI 263iSCSI RFCs 264iSCSI Protocol 264

Initiator-Target Messages 264Transfer Directions 265

Network Address Authority (NAA) . 265iFCP 265

Gateway Protocol to TCP/IP 266Provisions 266Fabric 266iFCP Services 268

Transport Services 268

Other 268

Security 268

Fibre Channel Over IP (FCIP) 269

Complementary 270

Nominal Distances 270

FCIP Link 271

Clarification 272

Internet Storage Name Service (iSNS) 272

iSNS Architectural Components 273

iSNS Protocol (iSNSP) 273

iSNS Client 273

iSNS servers 274

iSNS Database 274

What iSNS Does 275

Unified Discovery and Management 276

Fiber Channel Discovery 277

Zoning 277

Using SCN Functions 277Distributed Intelligence 278IP Storage Support 278

Quality of Service for IP Storage Networks 278Class of Service (CoS) 278Traffic Prioritization 279

TOS Precedence 279Differentiated Services 279

Performance Predictability 280Traffic Shaping 280

Theory and Research 280Variable Delay 281

Throttling Decisions 281Resource Reservation Protocol (RSVP) 282

Multiprotocol Label Switching (MPLS) 283MPLS Transport Profile 284

InfiniBand 285IP Storage Considerations 290IP Switches and Routers 291

Determining the Need 291

Logical Separation Points 291

Security Considerations 292Ports and Connectivity 292Other Features 293

Management Tools 293Acceleration and Compression 294

Security 294

Authentication 294

Fibre Channel—Security Protocol (FC-SP) 295

Device Identity 296Authorization 298

Encryption 298

Auditing 298

Integrity Checking 298

Quality of Service (QoS) 298Further Readings 299

Chapter 11 Object-Based Storage 301

Introduction: Objects 302

Storage Categories 307

Storage Interface Progression 310Parallel Heading 313

Applications of OSD 321Further Readings 325

Chapter 12 Archives, Backups, and Linear Tape 327

Organizational Requirements for Data 328

Legal Requirements for Archive or Backup 329

Repacking and Repurposing 330Added Operational Requirements 331

Backup or Archive: There are Differences 332

Backups 332

Acting like Archives 333Data Backups—Not Archives 333Internal Application-Specific Backups 334Archive and Transfer Management—Nonlinear Editing 335Differentiation 335

Archiving for Discovery 336Data Archiving 337

Tape or Disk 338Pros and Cons 339

Self-Protecting Storage Systems 340Disk Archiving Pluses 340

Tape Archiving Pluses 341

Storage Retention Period 341

Disadvantages of Tape 341

Legacy Tape Archive Problems 342

Standards-Based Archive Approaches 342Common Backup Formats 343

Determining the Right Archiving Method 343Cloud Data Protection 344Cloud Benefits 344

Scaling 345Services Provided 345

Holographic Storage for the Archive 345RAIT 347

Mirroring—Level 1 RAIT 349Data Striping—Level 3 RAIT 349

Data Striping—Level 5 RAIT 349

Not for Random Access 349

Automated Tape Libraries (ATL) 350

Stackers, Autoloaders, or Libraries 351

Server and Archive Components 351

Evolution in Digital Tape Archives 352

Tape Technology Basics 353

Tape Metrics 353Linear Serpentine 354

Helical Scan Recording 355

Historical Development of Data Tape 355

QIC and DAT 356

Travan 356

8 mm Recording Technologies 356

Tape Error Detection 358

Tape Media Formats and Fabrication Parameters 359

Metal Particle to Advanced Metal Evaporated 359

Advanced Intelligent Tape (AIT) 360

Super and Turbo AIT 361

Digital Data Storage (DDS) 362

Data Storage Technology (DST) 362

Digital Tape Format (DTF) 363

Advanced Tape Materials 364

Digital Linear Tape (DLT) 365

Error Detection- 365

Adaptive Cache Buffering 366

Compression Algorithms 366

Duty Cycle 366

DLT1 Drive 367

Super DLT 367

RoadmapsforSDLT 368

Linear Tape Open (LTO) - 369

Genealogy 369

Ultrium—First Generation 370

LTO Generation 1 370

LTO Generation 2 370LTO Generation 3 370LTO Generation 4 371LTO Generation 5 371LTFS 371

LTO Program Roadmap 372

Security WORM 373

Why Use Tape? 373Data Archiving and Tape Storage 375

Preparing for the Archive 376Future Standards 377

Chapter 13 Metadata and Media Asset Management 379

Media Management Systems 380Media Assets and Content 380

Content Management System 381

Web-Based CMS 382

Digital Asset Management 382

Asset Management System 383

Media Asset Management 384

Tape-Based MAM 384

Drawbacks to Tape-Based Systems 386

MAM for VideoServer Implementations 387Lack of Media Asset Integration 388

Collaboration Issues 388Metadata 388

Media-Centric Metadata 389

Structural Metadata 391

Descriptive Metadata 391

Interoperability 393Dublin Core 393

PBCore 394

Usage Metadata 396

Schemas 396

Standards, Practices, and Policies for Metadata 397

Storing Metadata 397

Metadata Services 398

Media Application Services 398

Messaging Services for MAM 400

Intelligent Integrated MAM 401

Workflow Management 402

Collaboration among Multiple Users 402

Tool Set Integration 402

Centralized Database 403

Multi-Tier Architecture 403

Scalability 404

Storage for MAM 404

Latency 405

Interfacing with Production Storage 405

Archive 407

Further Readings 408

Chapter 14 Clustering 409

Why Use Clusters? 409

Physical Components 411

Capabilities and Types of Clustering 412

Cluster Scaling Characteristics 415

Shared Structures 416

Managing Storage Clusters for Scalability 420

Grid Computing 422

Summary 428

Further Readings 428

Chapter 15 Networked Storage .. 429

Networked Storage—Fundamentals 430Networks 436

Network-Attached Storage 442

Direct Access File System 443

Storage Area Network 444

Fibre Channel SAN Components 460

Next Generation SANs 467

Further Readings 467

Chapter 16 High-Performance and Shared Storage 469

Applications for High-Performance Storage 470

Size Matters 474

Intelligent Adaptive Storage 475

Storage Accelerators 476

Storage Appliance Technologies 477

Real-World Performance 490

Shared Storage Implementations 498

Storage Scaling and Management 510

Further Readings 510

Chapter 17 Scalability, Extensibility, and Interoperability 511

Scalability 511

Flexibility 516

Reliability 520

Accessibility and Availability 526

Extensibility 533

Interoperability 539

Science and Technology 541

Further Readings 541

Chapter 18 Storage Management 545

Storage System Dependencies 546

Storage Management Concepts 549

Electronic Media Management 554

Fragmentation 554

SAN Management 560

NAS Management 564

Snapshots 571

Data Deduplication 574

Further Readings 579

Chapter 19 Resilience, Fault Tolerance, and Virilization 581

Resilience 582

Storage Tiering 583

Protecting Data and Storage 595

Visualization in File-Based Storage 605

Fault Tolerance 610

Conclusion 617

Further Readings 617

Index 619

Companion website: www.elsevierdirect.com/companions/9780240814483