19. April 2013

ddn.com©2012 DataDirect Networks. All Rights Reserved.

19. April 2013

Toine [email protected]

Big Data Evolution


Top500 – Storage, Nov 2012

DDN proudly powers…

50% 5 Of the TOP10

55% 11 Of the TOP20

54% 27 Of the TOP50

50% 50 Of the TOP100 30% 148 Of the TOP500over 50% Of the TOP100 GB/sover 70% Of the Lustre Sitesover 65% Of the GPFS Sites


Accelerating Accelerators

DDN is the leading provider of affordable, high-availability storage for the next generation of particle physics research.

DDN Supplied over 40PB of Storage to the LHC Community in the last 4 years


LHC Customer Base

• Tier 0– CERN-LHCb (1*S2A9900 SFA10K,

100TB)

• Tier 1– SARA/NIKHEF (13*S2A9900, 6 PB)– KIT (10*S2A9900, 1*SFA10K, 20PB)– IN2P3 (7*DCS9550, 1.5PB)– PIC (2*S2A9900, 2.4PB)– INFN-CNAF (5*S2A9900, 1*SFA10K,

10PB)

• Tier 2– DESY (2*S2A9900, 2*SFA10K, 3PB)– NBI (1*S2A6620, 60*2TB)– INFN-PISA (2*S2A9900, 1*SFA12K,

1PB)– INFN-PADOVA (1*S2A9900, 240TB)– IFCA (1*S2A9900, 1.2PB)– TRIUMF (2*DCS9900, 0.6PB)– SFU (1*S2A9900, 1PB)– UNIV. ALBERTA (1*S2A9550, 100TB)– UNIV. VICTORIA (1*S2A9900, 500TB)– SCINET (2*S2A9900, 1PB)– McGill UNIV. (2*SFA10K, 1PB)


DDN Portfolio

5

• EXAScaler™• 10Ks of Clients• 1TB/s+, HSM• NFS, CIFS

• Storage Fusion Architecture Storage Appliances

• WOS® 2.5• 256 Billion Objects

• GeoReplicated• Cloud Foundation

• Mobile Cloud Access

• 40GB/s/1.7M IOPS• 1,680 Drives: 2 Racks• Embedded Computing

• SAS • SATA • SSD

• 12K

• DirectMon• Enterprise Platform

• Management

• 10GB/s, 600K IOPS• 60 Drives in 4U; 396 Drives in 20U

• Embedded Computing (tba)

• 7700

• Flexible Media Configuration

• Parallel File Storage

• Analytics

• Block

• Cloud • Storage

• GRIDScaler™• 1Ks of Clients• 1TB/s+, HSM• NFS, CIFS

• Storage Fusion Xcelerator (SFX) Flash Acceleration

• SFX • Read

• SFX • Write

• SFX • Context• Commit

• SFX • Instant• Commit

• Cloud • Tierin

g

• Filesystems, • customer applications

• Embedded systems


SS8460 – Highest Density Enclosure

• 84 Drives – SSD, SAS, SATA - in 4 rack units

• Up to 336 TB (84 x 4)


SFA12K-40 (Block Appliance)

Highly Parallelized SFA Storage Processing Engine

Active/Active Storage Design35-40GB/s Read & Write Speed

Up to 6.7PB of Disk2.4+ Million Burst IOPS

700K+ Random Spinning Disk IOPS1.7M Sustained Random SSD IOPS64GB+ Mirrored Cache (Protected)

RAID 1/5/6Intelligent Block Striping

DirectProtect™GUI, SNMP, CLI, API

16 x FDR IB Host-Ports8RU Height

• 240Gb/s• Cache Link

• 32-64GB High-Speed Cache

• 32-64GB High-Speed Cache

• SFA Interface Virtualization

• SFA Interface Virtualization

• 960Gb/s Internal SAS Storage Management Network

• 16 x FDR InfiniBand Host Ports

• SFA RAID 5,6

• RAID 5,6

• SFA RAID 1

• 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • P• RAID 5,6

• Q• RAID 6

• 1 • 2 • 3 • 4

• 1 • 1m

• Q• RAID 6

• P• RAID 5,6

• Internal SAS Switching

• Internal SAS Switching

• 40 GB/s Read & Write Speed

ddn.com©2012 DataDirect Networks. All Rights Reserved. DDN Confidential

Big Data Solution Shapes

8

► Data separate from compute

► Data inside compute

► Data Separate from Compute

► Compute inside the Data

► Compute and Data all over the place


Traditional Parallel Filesystems

9

► Data separate from compute

• Storage Fusion Architecture™ • [Core Storage S/W Engine]

• In-Storage Processing™ Engine & DMA Driver

•D

irect

Mon

™: I

nfra

stru

ctur

e M

anag

emen

t

• ‘Scaler File System Family

• Low-Latency Connect: FC, IB, Memory

• Interrupt-Free Storage Processing

• ReACT™ Adaptive Cache Technology

• DirectProtect™ Data Integrity Management

• Quality of Service Engine

• Storage Fusion Fabric™

• Storage Fusion Xcelerator (SFX)


High Single Client Performance

10

► Data Separate from Compute

10

• 1-Trillion Row Big Data

Queries in less than

20s.

• Best Runtime

Ever for Drug Discovery, Warranty,

Risk Analytics

• Up to 570% faster FSI

back-testing and risk

management

http://support.sas.com/rnd/scalability/grid/SASGridDDN.pdf

http://www.ddn.com/pdfs/Vertica_Results_Solution_Brief.pdf

http://www.stacresearch.com/node/13285


Move Compute to the Data

11

► Data inside compute

• Full Data Protection Offload

• Innovative I/O Node Data Pipelining

• End:End RDMA for Hadoop Shuffle

• 8+2 Data Protection w/ Real-Time I/O

• Hybrid HDD & SSD Configuration

• 300%+ Density; Flexible Scaling


Embedded Systems

12

► Compute inside the Data

• Multi-core CPU Application Processor (AP)

Back-End Storage

Enclosures

•Fi

le S

erve

r

• Dedicated• I/O Bridge

• Multi-core CPU RAID Processor (RP)

• Memory Pointers• (Virtual Disks)

• Multi-Threaded Real-Time• RAID Engine, Hypervisor

• Dedicated• I/O Bridge

• Cache• Memory

Filesystem Clients

• High Speed Bus

•Fi

le S

erve

r

•Fi

le S

erve

r

• Application• Memory

• Virtual Disk• Block

Driver

• Dedicated PCI-e I/O

• ……

ddn.com©2012 DataDirect Networks. All Rights Reserved. DDN Confidential13

► Compute and Data all over the place


Simplified Data Lifecycle

• Process• Ingest

• Distribute• Store

14 06/2012

ddn.com©2012 DataDirect Networks. All Rights Reserved.15

WOS: WEB Object Store

► Not POSIX-based► Not RAID-based► No Spare Drives► No inode references, no FAT, no

extent lists► No more running fsck► No more volume management► Not based on single-site/box

architecture► 3 commands only:

PUT, GET, DELETE

ddn.com©2012 DataDirect Networks. All Rights Reserved. Confidential - To Be Shared Under NDA Only16


• Zone 2• Zone 1

WOS Puts & Gets

• App/Web Servers

• A file is uploaded to the application or web server.

• Application makes a call to the WOS client to store (PUT) a new object

• Application returns file to user.

• The WOS client stores the object on a node. Subsequent objects are automatically load balanced across the cloud.

• LAN/WAN

• Database

• The WOS client returns a unique Object ID which the application stores in lieu of a file path. The application registers this OID with the content database.

• Application makes a call to the WOS client to read (GET) the object. The unique Object ID is passed to the WOS client.

• A user needs to retrieve a file.

• The WOS client automatically determines what nodes have the requested object, retrieves the object from the lowest latency source, and rapidly returns it to the application.

• The system then replicates the data according to the WOS policy, in this case the file is replicated to Zone 2.

• OID = 5718a36143521602

• OID = 5718a36143521602


DDN | WOS®

18

•W

OS

Clu

ster

Man

agem

ent

• ObjectAssure™ Erasure Coding• Replication Engine

• WOS Policy Engine

• De-clustered Data Management

• Self-Healing Object Storage Clustering

• Latency-Aware Access Manager

• WOS Core• [Peer:Peer Object Storage]

iRODS

•C

onne

ctor

s

WOS API• C++, Python, Java, PHP, •HTTP, REST interfaces•PUT, GET, DELETE

NFS

• API-based• Integrate applications and

devices more robustly

• Policy driven• Manage truly via policy, rather

than micromanaging multiple layers of traditional filesystems

6/8/12

• Object Placement

• Global, Peer:Peer• Distribute data across 100s of

sites in one namespace

• Self-Healing• Intelligent Data Management

system recovers from failures rapidly and autonomously

CIFS

• S3 & WebDAV APIs

• IOS Smartphones and Tablets

• Multi-tenancy, Reporting and Billing

• Object ID Management

• Data Protection• Replicate and/or Erasure Coding

• Small files, large files, streaming files

• Low seek times to get data• WOS caching servers for massive

streaming data


DDN | WOS™ Deployment & Provisioning

DDN | WOS building blocks are easy to deploy & provision – in 10 minutes or less

• Provide power & network for the WOS Node• Assign IP address to WOS Node

& specify cluster name (“Acme WOS 1”)• Go to WOS Admin UI. WOS Node appears

in “Pending Nodes” List for that cluster

• San Francisco

• New York• London• Tokyo • Simply drag

new nodes to any zone to extend storage

• NoFS

• Drag & Drop the node into the desired zone• Assign replication policy (if needed)

• It’s that simple to add 90TB (30x3)• to your WOS cluster!


WOS Screenshots

Confidential - To Be Shared Under NDA Only20


• iRODS

• Server

• ICAT DB

• ..

• iRODS

• Server

• Site 3

• Site 2

• WOS Clust

er

• SFA 10K

• Site 1

• iRODS

• Server

• ICAT DB

• ..

• iRODS

• Server

• iRODS

• Server

• JBOD

• ICAT DB

• ..

• iRODS

• Server

Big Data ConnectediRODS Integration

Site 1

iRODSServer

ICAT DB

.. iRODSServer

iRODSServer

JBODICAT DB

.. iRODSServer

iRODSServer

ICAT DB

.. iRODSServer

Site 3

Site 2

WOS Cluster

SFA 10K

► Now: iRODS/WOS compound resource

► E-iRODS will bring a composable resources• Now iRODS uses WOS as a global storage system• iRODS Talks to all WOS IP addresses via REST

interface• WOS handles the replica management

► Interconnect Parallel Filesystem, Analytics and Cloud storage with Policy-based management


Enable High Speed Edge Computing with GRIDScaler-WOS Bridge

• Offload – Offload/archive GRIDScaler files to WOS to free up space & improve performance

• Distribute & Federate – Replicate & federate files across other sites for collaboration & disaster protection

• Collaborate – NFS users at remote sites can review & update files at local LAN speeds & share with GRIDScaler users

• GRIDScaler• GRIDSca

ler


Local Ingest Remote DistributionArchive & Distribution Use Case

• WS

• WOS Zone 1

• GPFS

• WOS Policy driven replication

• WOS REST

• or WOSLIB

• Samba or native CIFS

• Analysis

• Native GPFS or nfs

• WOS Zone 2

• WOS REST

or WOSLI

B

• Viewing App

• NFS or CIFS

• WOS Access NFS

WOS Zone 2

WOS REST or WOSLIB

Viewing App

NFS or CIFS

WOS Access NFS

• GPFS / WOS Access DB Sync

• WOS Access

• Ingest to GS for analysis, GS w/connector distributes to WOS for viewing/processing, GS to WOS DB Sync federates GS & WOS

• Compute Cluster

GS Ingest Process Flow, GS to WOS connector1. Raw data “A” ingested into GS via WS2. Analysis App processes raw sequencer data, writes

resultant file “B” to WOS via GS-WOS Bridge3. WOS Bridge DB synchronizes with WOS Access Sites 4. WOS Access DB syncs across sites which federates the

NFS view across all sites Users utilizing review and/or processing applications @ remote sites can access the resultant files.

• 2

• 1

• 2

• 1• 2

• 4

• 3

• 4

• GPFS –WOS DB Sync

• 3

• 3

• A• B

• B

• B

• Compute Cluster

• 4


Big Data Connected

Confidential - To Be Shared Under NDA Only24

► Use GPFS HSM features to create candidate file lists

► Fast data movements into object store via 10Gbe/IB

► Maintain metadata and file stubs in GPFS

► Data in WOS moves according to policy


WOS: Performance Comparison

-

10,000,000,000

20,000,000,000

30,000,000,000

40,000,000,000

50,000,000,000

60,000,000,000

• World’s Fastest

POSIX FS

• World’s Fastest

POSIX FS• (new:

2H11)

• Google!

• EMC: http://reg.cx/1P1E• Lustre; http://wiki.lustre.org/images/1/16/LUG08_Cray_HPCS.pdf• GPFS: http://www.spscicomp.org/ScicomP13/Presentations/IBM/GPFSGunda.pdf• Megastore: http://highscalability.com/blog/2011/1/11/google-megastore-3-billion-writes-and-20-billion-read-transa.html

EMC Atmos Lustre GPFS Megastore WOS 2.0Reads/Day 500,000,000 864,000,000 2,764,800,000 20,000,000,000 55,472,947,200

Writes/Day 500,000,000 864,000,000 2,764,800,000 3,000,000,000 23,113,728,000

ddn.com©2012 DataDirect Networks. All Rights Reserved.26

Summary

► Scaling Up and Out and Deep across the Big Data Landscape requires a range of building block shapes

► Managing the complexity of global distribution requires an alternative model for data management

► Connecting Big Data solutions is the next challenge

► More details: www.ddn.com

http://cern.ch/Computing.Seminars/2013/0131

Documents

19. April 2013