Archive eXchange Format AXF

An Open Format for Universal Content Transport,!Storage and Long Term Preservation!

!Brian Campanotti!

CTO!Front Porch Digital!

What is AXF? •  AXF is a universal format for the encapsulation,

storage, transport and preservation of file assets !

•  AXF is like an advanced ZIP which encapsulates any number of files, metadata and a ubiquitous file system!

•  AXF packages can exist on block based storage as well as any file system on any media type !

•  AXF is the first open format targeted at the storage, preservation and transport of file based assets!

Origins of AXF •  Developed based on the demanding needs of the

media and entertainment industry!

o Very large files and file collections!o Complex and important file relationships!o Fundamental connection between assets and metadata!o Randomly accessible at every levels!o Valuable assets which must be preserved!

•  Although developed in the M&E space, the benefit of AXF is easily extended to any file based application !

•  AXF is currently at the heart of various storage, preservation and cloud transport applications today!

Where Did We Come From? •  Proprietary systems are rampant in M&E!o Many data formats on media!o Numerous interface protocols!o Same system type is required for recovery!o Media migration is not easy !

•  Ever increasing danger of orphaned archives!

•  High costs of implementation and operation!o Individualized system integration requirements!o Transfer costs resulting from inability to interchange

media!

User Demands •  Simplified movement of material between different storage

and archive systems!

•  Flexibility in changing storage vendors, technology and middleware solutions!

•  Limitless scale (number of files and size) including the ability to span across media!

•  Universally accessible leveraging stand alone utilities!

•  Support for all media types – now and into the future!

•  Highly resilient with in-built redundancy!

•  Universal despite the storage or transport technology!

AXF Technology

Storage and Preservation •  We must take steps to ensure long term accessibility

to valuable file-based assets stored in digital archives in all industries!

•  Key goals of an ideal storage format!o Ensure long term accessibility!o Self-describing assets and self-describing media!o Encapsulation to maintain important metadata/file

relationships!o Scalability for any number of elements of any size and type!o Standardized regardless of storage media technology!o Transportability and compatibility between systems!o Preservation (OAIS) features such as fixity, provenance,

etc.!o Streaming support to enable on-the-fly processing!

•  What choices are there?!

Tape ARchive Format •  Tape ARchive (TAR) format has been around for

many decades!

•  There is no true universal TAR implementation but many “customized” ones !

•  TAR is a legacy format which cannot support many of the core functionality required in the M&E space !

•  TAR does not address many of the storage format goals we have outlined !

•  Do we have any other choices?!

Linear Tape File System •  The Linear Tape File System (LTFS) is a simple file system

for linear data tape !

•  LTFS makes data tapes appear as “removable storage”!

•  There are no standards bodies which currently endorse LTFS although some work is underway within SNIA!

•  Unfortunately LTFS has some significant limitations with respect to the goals previously outlined !

•  LTFS is very useful for the physical “transport” of content but not for long term storage or preservation !

•  But why?!

Linear Tape File System •  LTFS offers no media encapsulation and relies on simple

folder hierarchies to form important asset relationships!

•  LTFS does not scale well due to lack of support for spanning across storage media – a significant problem in M&E!

•  LTFS only supports modern data tape technologies and is not applicable to any other storage technologies!

•  Neither LTFS nor TAR achieve 100% of our long term storage and preservation goals!

•  What other choices are there?!

Archive eXchange Format •  AXF is a file collection “wrapper” which can

encapsulate any number of files of any type and size !

•  AXF does not overlap with MXF or other “media wrapper” approaches as these are simply files inside AXF objects!

•  AXF brings the same level of universal transport and interoperability to storage as MXF has for media !

•  It was designed to support all storage technologies – now and into the future !

•  It is IT-centric and not tied to media applications alone!

What is an Object Store? •  Key to media and entertainment asset storage and

preservation is the concept of an Object Store !

•  An Object Store encapsulates all related files (essence, metadata, ancillary files, etc.) into a single package maintaining critical asset relationships !

•  CSM solutions are Object Stores by definition (HSM systems are NOT!)!

•  This concept becomes even more important when we extend the CSM functionality into the cloud and asset transport!

A Media Object!

CSM as an Object Store

File Collection!

Metadata!

+

+

Source Information!

Archive!

Cloud!

Disk and Flash!© 2012 Front Porch Digital Inc.!

AXF Characteristics •  Unlimited Storage Support!

o Any number of files, any size of files and media spanning !

•  Resilience to Media Damage and Corruption!o Redundancy in all structures; payload can be independently

recovered !

•  Support for Media with and without a File System!o Raw data tape, LTFS data tape, spinning disk, flash media,

optical, etc.!

•  Support for Any File System or Operating System!o Embedded file system abstracts the underlying operating

and file system!

AXF Characteristics •  Self describing Objects and Self Describing Media!

o Enables simple transport of objects and media between systems !

•  Object Versioning and Collection Support!o Supports complex relationships between objects; additions,

updates, etc.!

•  Support for All File Types – Not Just Media Files!o IT-centric implementation based on experience in M&E big

data handling !

•  Streaming and File Based Asset Transport and Delivery!o Support for streaming de-encapsulation; in-path checksums

for structures and files; cloud storage and delivery applications!

Archive eXchange Format Structured!

Unstructured!

Metadata File Payload

Ancillary Files

Proprietary!

Open!

Asset Compone

nts

Provenance! Fixity! Context! Reference!Access Control!Preservation

Elements Universal Storage Agnostic File System!File System

AXF Object!

© 2012 Front Porch Digital Inc.!

Universal Storage Format

AXF!Object!

Spinning Disk!(NTFS, FAT, etc.)!

Data Tape!(Block Based, LTFS, etc.)!

Solid State Disk!(NTFS, FAT, etc.)!

Flash Media!(NTFS, FAT, etc.)!

Future Storage Technologies!

DVD!(UDF, etc.)!

Blu-Ray!(BDFS, UDF, etc.)!

Holographic!(UDF, etc.)!

© 2012 Front Porch Digital Inc.!

Basic AXF Overview

AXF

Obj

ect H

eade

r!AX

F M

ediu

m Id

entifi

er!

AXF Object!1!

AXF Object!2!

AXF Object!N!

Met

adat

a Co

ntai

ner!

Met

adat

a Co

ntai

ner!

File!

1!

AXF

File

Foo

ter!

File!

2!

AXF

File

Foo

ter!

File!

N!

AXF

File

Foo

ter!

AXF

Obj

ect F

oote

r!

Metadata Payload

File Payload

© 2012 Front Porch Digital Inc.!

Inside an AXF Object

AXF on Block Based Media

AXF

Obj

ect

Head

er!

Met

adat

a Co

ntai

ner!

Met

adat

a Co

ntai

ner!

File

1!

AXF

File

Foo

ter!

File

2!

AXF

File

Foo

ter!

File

N!

AXF

File

Foo

ter!

AXF

Obj

ect F

oote

r!

Constructing an AXF Object

File Collection!

Metadata!

+

+

Source Information!

Archive!

Cloud!

Disk and Flash!

© 2012 Front Porch Digital Inc.!

An AXF Object!

Handling an AXF Object

•  Block alignment for increased resiliency!•  Provenance records track the entire lifecycle of object!•  On-the-fly fixity verification on all files and structures!•  Spanning across media for limitless scale (eg. data tape spanning)!•  Random access to all metadata and files!•  Fully contained AXF indices enable quick re-indexing!•  Self-describing objects and media allow quick transport between systems!

Archive!

Cloud!

Disk and Flash!© 2012 Front Porch Digital Inc.!

AXF

Obj

ect H

eade

r!

Met

adat

a Co

ntai

ner!

Met

adat

a Co

ntai

ner!

File

1!

AXF

File

Foo

ter!

File

2!

AXF

File

Foo

ter!

File

N!

AXF

File

Foo

ter!

AXF

Obj

ect F

oote

r!

An AXF Object!

Transporting an AXF Object

•  On-the-fly encapsulation and de-encapsulation!•  AXF objects are re-aligned to the block/chunk size of target storage devices!•  Provenance metadata automatically tracks all actions on the AXF object!•  On-the-fly fixity verification on all files and structures!•  Self-describing objects and media allow quick transport between systems!•  Important file and metadata relationships are always maintained!

© 2012 Front Porch Digital Inc.!

AXF

Obj

ect H

eade

r!

Met

adat

a Co

ntai

ner!

Met

adat

a Co

ntai

ner!

File

1!

AXF

File

Foo

ter!

File

2!

AXF

File

Foo

ter!

File

N!

AXF

File

Foo

ter!

AXF

Obj

ect F

oote

r!

An AXF Object!

AXF

Obj

ect H

eade

r!

Met

adat

a Co

ntai

ner!

Met

adat

a Co

ntai

ner!

File

1!

AXF

File

Foo

ter!

File

2!

AXF

File

Foo

ter!

File

N!

AXF

File

Foo

ter!

AXF

Obj

ect F

oote

r!

An AXF Object!

WAN$

AXF Benefits •  AXF Objects can scale to any size and encapsulate any

number of files with full support for media spanning !

•  No need to upgrade existing storage infrastructures!

•  AXF ensures long term compatibility and resiliency with self-describing features for both AXF objects and media !

•  AXF overcomes all the technical, operational and functional limitations of TAR and LTFS!

•  AXF is an IT-centric implementation and is not limited to media files alone (documents, imaging data, etc.)!

•  AXF satisfies all of our ideal storage format goals as well as enables effective transport of complex file collections!

AXF in the Industry •  Manufacturers are beginning to release products which fully

support AXF today!

•  Implementations are based on the SMPTE standards body draft technical specifications which is nearing completion !

•  Universal exchange tools such as AXF Explorer can be used to access AXF objects stored on data tape, disk, etc.!

•  AXF is also the basis for cloud-based asset transport, archive and preservation services being offered – AXF is not just for storage!!

•  Very important milestone for the media, archive and preservation communities and offers real-world proof of AXF!

•  Front Porch Digital is leading AXF innovation as its primary inventor!

The AXF Standard •  AXF is a SMPTE 31FS initiative which will standardize the

way media Objects (and any other file collections) are transported, stored and preserved!

•  The SMPTE AXF committee was founded in 2006 because of industry demand and reactive manufacturers!

•  AXF draft standard (ST2034 Part 1) is nearing completion and is expected to enter ballot stage this year!

•  Part 2 of the standard will be an RP on AXF applications !

•  Many users and vendors are actively participating in committee activities!

OpenAXF.org •  OpenAXF.org is the community portal for the AXF

initiative !

•  The website includes news, documentation, videos and whitepapers providing additional technical details on AXF!

•  This is the focal point for the AXF community – so please sign up now!!

•  SMPTE is always looking for active participants (users, manufacturers, etc) to help bring other perspectives to AXF!

•  Visit smpte.org for more information!

Thank You!

For questions or comments please contact me at

brian.campanotti@fpdigital.com