Sony Pictures Entertainment Digital Backbone File ... pool for 2012 and no specified throughput requirements. 2. Expanded storage pool to 500TB usable to support future projects TBD

This document is solely for the use of Sony Pictures Entertainment (SPE) personnel. No part of it may be circulated, quoted, or reproduced for distribution outside the SPE organization without prior written approval from IBM.

© Copyright IBM Corporation 2009

Sony Pictures Entertainment

Digital Backbone File Management and Infrastructure

Nov 3, 2009

2

DBB File Management

Sony Pictures Digital Backbone | November 3, 2009 | Confidential © Copyright IBM Corporation 2009

Agenda

DBB recap

Workflow and throughput requirements

Approaches

Virtual File Repository (VFR)

Server and storage infrastructure options

Pricing

3

DBB File Management


IBM and SPE have been growing the motion picture DBB for more than a year and today we’re discussing HSM

“2012 Project”1

Q2 2008

Finalized Workload4

Q4 2009

• Grew to 500 TB • 2 PB LTO

DBB “2012 Expansion”2

Q2 2009TSM/HSM Sandbox3

Q3 2009

• 380 TB Storage• GPFS• Bladecenter

• Bladecenter• 32 TB raw storage• GPFS TSM/HSM testing

Today’s Discussion

• Validated GPFS-TSM/HSM policy works

• Write speed to tape below expectations

• Backbone storage resource constraint

• File management limitation

• Pipeline workflow flexibility

• Colorworks volume req’mt increased

• Additional TV project ingest

• Colorworks request to write directly to tape accessible by DBB

1. Storage pool for 2012 and no specified throughput requirements.

2. Expanded storage pool to 500TB usable to support future projects TBD. Backbone would grow over 5 years to 24PB of activity. Model for volume of 30TB/day or 350MB/sec for a 24-hour day.

3. Leveraged existing 2012 sandbox to test TSM/HSM integration with GPFS.

4. Full documentation on next 2 pages of this document. 1.8GB/sec throughput.

4

DBB File Management


In October, SPE finalized its bandwidth requirements for the DBB

5

DBB File Management


At SPE’s request, IBM presented an alternative workflow to support Colorworks direct write to LTO

6

DBB File Management


IBM has identified three approaches, and we will focus on VFR with TSM today

Approach Pros Cons

Virtual File Repository (VFR) with TSM

Improves tape speed write performance

Provides pipeline flexibility Leverages GPFS and TSM

VFR is a services offering

TSM/HSM with disk caching Improves tape speed write performance

Standard IBM software Inexpensive at low volume

Scalability limitations of TSM/HSM database

Limitation of one TSM server in GPFS file system

Limited pipeline flexibility – doesn’t support alternate workflow

High Performance Storage Subsystem (HPSS)

High-end data movement services offering

Exceeds SPE’s current scalability requirements

Expensive Implementation resource

constraint Services offering



Virtual File Repository and TSM

8

DBB File Management


VFR in combination with TSM/archive will provide the following services

VFR makes all the files look like they are in a file system– VFR simplifies and virtualizes the TSM interface

VFR aggregates 1000s of files into larger TSM objects

VFR mapfiles store information about all of the individual files that

comprise the larger object

The large objects are archived via standard TSM software

Supports copy and dir/ls

Existing Sony scripts request a set of files for retrieval using the

VFR API

VFR mapfiles are used to locate files on TSM managed tape to

restore them

9

DBB File Management


The Virtual File Repository (VFR) provides improved performance and functionality

Improved performance

– A group of files are written as a single object on tape

– Larger objects written to tape at a faster rate

– Reduction in load on TSM database Robust design

– The map between tape objects and disk files is stored with the files, both on disk and on tape, and can also be saved independently

Selected file(s) retrieval – saves time and space

– Individual files can be retrieved without retrieving the entire group Enhanced functionality

– Files can be restored to a different location than the saved location, significantly reducing time and space

– Move content directly to tape and bypass the digital backbone storage system – frees ups space on the backbone and reduces bandwidth required

Utilize standard TSM / archive software – provides the movement to tape

10

DBB File Management


Virtual File Repository

SonyScripts

VFR API

TSM

TSM tape

GPFSBackbone

VFR DataMovers

Sony scripts interact with the files through the VFR API, requesting files for applications

VFR moves files between GPFS and TSM.

VFR stores a sequence of files as one TSM object

VFR retrieves single files, partial sequences of files, or an entire sequence of files from TSM

TSMDatabase

SonyApplications

GPFS CW

11

DBB File Management


It is possible to deploy VFR by January 2010

Phase 1: Enable reading/writing to tape – Store entire directories from disk to tape

– Retrieve entire directories from tape to disk

– Invoke data movers from the command line

– Migration of data to tape via a manually invoked VFR script

Phase 2: VFR API and partial file recall– Retrieve subsets of the directories from tape to

disk

– Invoke data movers from Sony scripts (VFR API)

– Plan for skills transfer to SPE

– Plan for transition support to SPE

2 weeks afterContract Signing and pre-reqs are installed

Requires collaboration with Sony engineers

12

DBB File Management


Workflow database application for scheduling, tracking, tracing, and auditing of data movers

Open Tape support for export and archive. These tapes are self describing without reading the entire tape. They contain an XML index file.

VFR can be expanded to provide additional functionality

13

DBB File Management


TSM and VFR incremental infrastructure requirements are minimal

2 TSM servers – reuse the existing 3550 M2s

– Add 2 fiber channel HBAs

VFR code will run on existing blades



Server and storage infrastructure options

15

DBB File Management


Based on SPE’s revised volumes, IBM modeled the storage requirements

Understand aggregate storage performance requirements

Design a system to meet those requirements

Reuse existing assets

Modeling Objectives

Assumptions: 2 MB block size Varied throughput till one of the system

components began to be stressed

Measures: Internal FC utilization External Host Adapter utilization Hard Drive utilization Processor utilization PCI Bus utilization

Modeling Parameters

This is where most bottlenecks

occurred

16

DBB File Management


IBM’s modeling produced design recommendations

Loop optimizations DS4800 performs best with disk drawers in even multiples of 4 DS5000 performs best with disk drawers in even multiples of 8

Performance optimizations Increasing quantity and speed of Host Adapters had biggest impact, as would be

expected for a large block size workload SATA disk drives provide sufficient bandwidth if present in sufficient quantity

GPFS performance with mixed subsystems Parallel file workload runs at the speed of the slowest subsystem in the cluster Storage controllers should be balanced

Metadata workloads GPFS metadata and TSM/VFS database should be on separate, high

performance fibre channel subsystems

17

DBB File Management


Moving the metadata to a separate subsystem improves throughput and response time

Coresident vs Offloaded Metadata

0

50

100

150

200

250

300

350

400

450

500

DS48004 x 4 Gb HAs

224 disks (current)Metadata coresident

DS48004 x 4 Gb HAs

224 disks (current)Metadata separate

Th

rou

gh

pu

t in

MB

/sec

0

5

10

15

20

25

30

Res

po

nse

Tim

e in

ms

Coresident metadataLarge block responsetime 35% worse

Coresident metadataThroughput at least 10% worse

Coresident metadataMetadata responsetime 80% worse

18

DBB File Management


IBM identified four options, all of which leverage the existing assets

1. Keep current disk config, move GPFS metadata onto a separate subsystem 2 x DS4800 with 14 drawers of 1 TB SATA drives

1 x DS5100 with 12 drawers of 1 TB SATA drives

1 x DS3400 with 2 drawers containing 14 x 450 GB 15K FC drives for GPFS metadata

2. Rebalance current disk config, move GPFS metadata onto a separate subsystem 2 x DS4800 with 12 drawers of 1 TB SATA drives (a recommended config)

1 x DS5100 with 16 drawers of 1 TB SATA drives, addnl disk attach feature is added (a recommended config)

1 x DS3400 with 2 drawers containing 14 x 450 GB 15K FC drives for GPFS metadata

3. 2 x DS5100; rebalance all existing drawers; Upgrade SAN with 8 Gb/s blades 2 x DS5100 with 20 drawers of 1 TB SATA drives,

1 DS5100 upgraded w addnl disk attach feature and 8 x 8 Gb/s are added 1 DS5100 net new with addnl disk feature and 8 x 8 Gb/s are added

1 x DS4800 reused with 1 drawer of 16 x 450 GB 15K FC drives for GPFS metadata

4. 2 x DS5300; rebalance all existing drawers; Upgrade SAN with 8 Gb/s blades 2 x DS5300 with 20 drawers of 1 TB SATA drives,

1 DS5300 upgraded from DS5100 w addnl disk attach feature,16 x 8 Gb/s are added; performance upgrade 1 DS5300 net new with addnl disk feature and 8 x 8 Gb/s are added

1 x DS4800 reused with 1 drawer of 16 x 450 GB 15K FC drives for GPFS metadata

19

DBB File Management


IBM made a number of assumptions in its design

Assumptions Services to reconfigure the disk drawers are included Existing disk subsystems are located in the same location they will

be repurposed Existing SAN Director can be upgraded with additional blades Existing SAN edge switches can continue to be utilized

The workflow provided by SPE does not include the following: Disaster Recovery Local copy (synchronous) services or remote copy (asynchronous) Non-production environments

– Test, Development, Quality Assurance, Etc Process change; head room for growth, etc Workloads different from that modeled Other

Not included No tape upgrades included Cabling and cables not included Remote Support Manager for DS5000 not included

20

DBB File Management


Overall System Performance and PricesDisk Performance Upgrade Options

0

1000

2000

3000

4000

5000

6000

7000

2 x DS4800 14drawers

DS5100 12 drawers


DS5100 12 drawers


Upg DS5100 16drawers

Upgraded 2 xDS5100 20 drawers

2 x DS5300 20 drawers

Th

rou

gh

pu

t M

B/s

ec

DS4800 A: 4 x 4 Gb/s HA current

DS4800 B: 4 x 4 Gb/s HA current


DS5100 A': 8 x 4 Gb/s HA; addnl disk opt

DS5100 A'': 8 x 8 Gb/s HA; addnl disk opt

DS5100 B'': 8 x 8 Gb/s HA; addnl disk opt

DS5300 A: 16 x 8 Gb/s HA; addnl disk opt

DS5300 B: 16 x 8 Gb/s HA; addnl disk opt

Est. Sony DBB original throughput reqt

Option 1 - only add GPFS metadata ctlr

Option 2 - add metadata ctlr & re-

balance disks

Option 3 - add metadata ctlr; upgr

to DS5100 & 8 Gb/s SAN


to DS5300 &8 Gb/s SAN

$13,662 $13,662

Base Case

$191,957

$225,256

21

DBB File Management


Overall System Performance and Prices

Options 1 and 2 are not recommended due to potential limited headroom

Disk Performance Upgrade Options

0

1000

2000

3000

4000

5000

6000

7000


DS5100 12 drawers


DS5100 12 drawers


Upg DS5100 16drawers

Upgraded 2 xDS5100 20 drawers

2 x DS5300 20 drawers

Th

rou

gh

pu

t M

B/s

ec


DS4800 B: 4 x 4 Gb/s HA current


DS5100 A': 8 x 4 Gb/s HA; addnl disk opt

DS5100 A'': 8 x 8 Gb/s HA; addnl disk opt

DS5100 B'': 8 x 8 Gb/s HA; addnl disk opt

DS5300 A: 16 x 8 Gb/s HA; addnl disk opt

DS5300 B: 16 x 8 Gb/s HA; addnl disk opt

Est. Sony DBB original throughput reqt

Option 1 - only add GPFS metadata ctlr

Option 2 - add metadata ctlr & re-

balance disks


to DS5100 & 8 Gb/s SAN


to DS5300 &8 Gb/s SAN

$13,662 $13,662

Base Case

$191,957

$225,256

Est. Sony DBB with alternative workflow/

throughput reqt

22

DBB File Management


In all four options nearly all assets are reused

1. All disks and controllers reused DS3400 added for metadata

2. All disks and controllers reused One existing DS5100 upgraded

DS3400 added for metadata

3. One new DS5100 and upgrade current DS5100 to have 8 x 8Gb/sec Host Adapters and addnl drive attachment (up to 448) All SATA drives and drawers reused

One DS4800 reused, one not reused

4. One new DS5300 and upgrade current DS5100 to DS5300 DS5300s to have 16 x 8 Gb/sec Host Adapters and 16 GB cache

All SATA drives and drawers reused

One DS4800 reused, one DS4800 not reused

1 set of 8 x 4 Gb/s adapters in the DS5100 not reused

23

DBB File Management


DS4000/5000 Upgrade Options

Upgrades that change the controller only of the DS4000 or DS5000 series are disruptive, but not destructive

– DS5100 can replace a DS4800 in place. This requires an outage but the data remaining on the disk is unchanged

– DS5100 can be upgraded to DS5300 in place. This requires an outage but the data remaining on the disk is unchanged

Reconfiguration of disk drawers is disruptive and destructive– Data must be migrated or backed up and restored

– Migration can be done non-disruptively but requires interim space for the migration or backup

Recommendation is to do controller and SAN upgrade and reconfiguration of disks prior to production

24

DBB File Management


Projected Library Capacity Requirements

Additional Required Frames

SLOTS L53 D53 D53 S54 S54 S54 S54 D53Exisitng Slot Count 2423 287 408 408 1,320 1,320 1,320 1,320 440Required Slot Count 6,701 6,823

10 Feet20 Feet

MONTHS WEEKSTOTAL WEEKS

70TB+35TB PER WEEK

TOTAL TB STORED

18 4.5 81 105 8,505

Assumptions:

• 18 months of stored data at 105 TB of data per week.

• 8.5 PB of storage in month 18.

• The stored data is not compressible.

• The media will fill to 70% of the native capacity.

• LTO4 (560 GB consumed capacity) requires 10,600 slots.

• Current library has 2,423 slots.

• The plan includes moving to LTO5 (1.2 TB consumed capacity) upon its introduction.

• An undetermined amount of LTO4 media will remain in the library.

IBM recommendation:

Add three S54 and one D53 frames (totaling 6,700 slots) upon the introduction of LTO5.



Pricing

26

DBB File Management


TSM and VRF initial costs

TSM SW and installation services included TSM uses existing x3550 servers VFR uses existing servers (Blade Center) Cabling and cables not included Support and maintenance not included

TSM and VFR TSM Services 69,000$ TSM SW 40,000$ VFR services 100,000$

TSM and VFR subtotal 209,000$

27

DBB File Management


Storage Options Pricing Summary

Storage Requirements Option 1 - GPFS metadata ctlr only 13,662$ Option 2 - reconfig drawers 31,912$ Option 3 - 5100s w 8 Gb/s 191,957$ Option 4 - 5300s w 8 Gb/s 225,256$

28

DBB File Management


Our review uncovered licensing and maintenance requirements on the current system

Required ISL license for switches added (current trial license expired) Maintenance on SAN B16 switches added (currently no

maintenance on these switches)

Licensing and Maintenance of Current SystemPurch temp license for ISL 1,260$ Maint on SAN B16 2,100$

Subtotal 3,360$

29

DBB File Management


Optional Considerations

Data movers (x3650) recommended to: – Isolate the data movement requirements of DBB

– Provide additional Backbone ingest and request servicing

Maintenance on the 28 EXP drawers attached to the DS4800s is currently only 9x5. This should be upgraded to 24x7.

The tape library currently only has room for 2-3 months of data at the projected storage rates. Cables and installation will be required. IBM can provide Relocation of disk from Imageworks to Stage 6 not included. IBM can provide BladeCenter SAN infrastructure can be upgraded to 8 Gb/s

Optional DBB ConsiderationsData Mover servers x3650 27,222$ Maint upgrade on EXPs to 24/7 26,891$ Tape Library 120,392$ Cables (est) 5,850$ HS21 HBA upgrade to 8 Gb/s 5,247$ BladeCenter H switch upgrade 4,457$

Optional subtotal 190,059$

30

DBB File Management


Sony DBB – Recommended Next Steps

• Schedule move of imageworks backbone to the Colorworks location

VFR and TSM:

• Finalize agreement for VFR work

• Phase 1: Begin ASAP

• Phase 2: Can be done in parallel – collaboration with Sony engineers

• Purchase and install TSM software

• “Option B” to install disk cache and TSM/HSM if implementation delays occur

Equipment to support new workload / workflow:

• Select storage option – order / install (some options require more time)

• Other: data movers, maintenance, ISL

Target date for completion: January 1st, 2010 – ready for Spiderman 4

Documents

Sony Pictures Entertainment Digital Backbone File ... pool for 2012 and no specified throughput requirements. 2. Expanded storage pool to 500TB usable to support future projects TBD