IP-Based Object Drives Now Have a Management …...OpenStack Swift • Industry File-Sharing Protocols: NFS, SMB, AFP and FTP • Several data protection schemes and cluster topologies

IP-Based Object Drives Now Have a Management Standard

April 20, 2017 10:00 am PT

Live Webcast

Today’s Presenters

Enrico SignorettiOpenIO

Alex McDonaldCo-Chair SNIA CSI, NetApp

David SlikCo-Chair, SNIA Cloud TWG

NetApp

Erik RiedelDell EMC

2

SNIA at a glance

160 unique member

companies

3,500 active contributing

members

50,000 IT end users & storage

pros worldwide

Learn more: snia.org/technical @SNIA 3

SNIA Legal Notice

  The material contained in this presentation is copyrighted by the SNIA unless otherwise noted.   Member companies and individual members may use this material in presentations and literature under

the following conditions:   Any slide or slides used must be reproduced in their entirety without modification   The SNIA must be acknowledged as the source of any material used in the body of any document

containing material from these presentations.   This presentation is a project of the SNIA.   Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or

should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.

  The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

  NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

4

Agenda

  Object Drive Overview, David Slik   Some Products & Observations, Enrico Signoretti   Experiments & Experiences, Erik Riedel

5

David Slik – NetApp

What are IP Based Drives?

  Interface changed from SCSI based to IP based (TCP/IP, HTTP)   Channel (FC/SAS/SATA) interconnect moves to Ethernet network   Want to get involved? Join the SNIA Object Drive TWG at: https://members.snia.org/apps/org/workgroup/objecttwg/

7

What is driving the market?

  A number of scale out storage solutions expand by adding identical storage nodes incrementally

  Typically use an Ethernet interface and may be connected directly to the Internet

  Open source examples include:   Scale out file systems

  Hadoop’s HDFS   Lustre

  Ceph   Swift (OpenStack object storage)

  Commercial examples also exist

8

Who would buy IP Based Drives?

  System vendors and integrators   Enables simplification of the software stack

  Hyperscale Data Centers   Using commodity hardware and open source software

  Enterprise IT   Following the Hyperscale folks

9

Traditional Block Storage Device

  Interconnect via SATA/SAS/PCIe

  Locally addressable   connected directly or via local

fabric to storage servers   Traditional block protocols

(SATA/SAS/NVMe)   Block addressable   designed for reliable transport   Long lived local communication   Coordinated concurrent

accesses

10

Traditional Drive

SATA/SAS/PCIe

ATA/SCSI /NVMe

Disk and/or NVM

IP Based Drive

  Interconnect via Ethernet   Globally addressable   provide storage services over

network protocols directly to clients

  IP Based Protocol   Higher-level storage services   Error tolerant protocol (e.g.,

TCP/IP)   Transitory global communication   Independent concurrent

accesses

IP Based Drive

Ethernet

Storage Services

Disk and/or NVM

11

IP-Based Drive Management

  The Object Drive TWG produced a specification for scalable management of IP Based Drives

  Based on the RedFish management specification from DMTF https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.0.1.pdf

  Uses Odata (OASIS) for RESTful interface   Minimizes and simplifies the management

of resources   SNIA standard specifies common features

and references the other standards

12

Management Activities for Object Drives

  As a device, how do I connect to a network?   How do devices physically negotiate to connect to a

network?   How do devices configure themselves to talk TCP/IP over

a network?   Address assignment, name resolution, time services, etc.   How do devices discover where they physically are

located?

13


  As a manager, how do I discover devices?   & discover what devices are available to manage?   & distinguish my devices from other peoples devices?

  How do I find out   where these devices are located?   how these devices are connected?   if any of this changes?

14


  As a manager, how do I configure devices?   CPU Firmware?   drive firmware?   the network?   … & select the drive application?

15


  As a manager, how do I keep devices secure?   find out when security updates are available?   push security updates to devices?   maintain operations during updates?

  How do I keep my devices up to date?   tell when firmware updates are available?   push firmware updates to my devices?   maintain operations during updates?

16


  How do I know when things are failing?   How do I

  monitor environmental health?   monitor drive hardware health?   monitor drive data health?   monitor drive application health?   tell when something is unhealthy?   tell when something is about to fail?   tell when something has failed?   identify what needs to be done to replace a failed device?

17

IP Based Drive Management

  Specification is now a SNIA Technical Position (Standard) IP-Based Drive Management Specification v1.0

  IP-Based Drive Characteristics and Requirements   Describes the physical form factors, electrical and link layer requirements   Has a Taxonomy of various possible drive types with protocol and other

information   IP-Based Drive Management

  Describes the device discovery and management   Assignment of IP address   Discovery of Basic Services   Redfish based management

18

Using Redfish to manage drives

  The following services are used (but not limited to these):   Account Service   Session Service   Chassis Collection   Manager Collection   Computer System Collection   Update Service (recommended)

  “ChassisType” property is “IPBasedDrive”   The Redfish implementation should support the Redfish standard

Drive entity.

19

Three mockups available

  What are the mockups?   Examples of management interfaces to an IP Based drive system   Redfish schemas   Located at http://www.snia.org/object-drives

  What mockups are available   Simple IP Based drive mockup

  Single drive   Dual network connections

  IP Based drive array mockup   Single manager   Multiple drives arranged hierarchically

20

Enrico Signoretti

• Dual-core ARM-v8 CPU • RAM, flash memory, 2 * 2.5gb/s Ethernet links

• 3W power consumption and HDDs Power management

• Supports 8,10,12 TB HDDs

Hyper Scalable Storage

Nano-node

22

• N+1 power supplies and cooling units

• Chassis Management • 2x 6-port 40gb/s Ethernet switches for front-end and back-to-back expansion

• Up to 96 hot-swap nano-nodes

No Single Point of Failure

SLS 4U96

23

• N+1 power supplies and cooling units

• Chassis Management • 2x 6-port 40gb/s Ethernet switches for front-end and back-to-back expansion

• Up to 96 hot-swap nano-nodes

SLS 4U96

No Single Point of Failure

24

• Standard Object APIs to leverage natively the platform: OpenIO REST/HTTP, Amazon S3 and OpenStack Swift

•  Industry File-Sharing Protocols: NFS, SMB, AFP and FTP

• Several data protection schemes and cluster topologies • Ease of Use. GUI, APIs, CLI • Lightweight backend design • Grid for Apps: event-driven framework for Serverless

computing

Same software, same capabilities

SDS

25

Erik Riedel

SCALE

27

Gen1 (2008) 1TB

Gen2 (2010) 2,3TB

Gen3 (2012) 3,4,6TB

Gen4 (2014) 6TB

Gen5 (2015) 8TB

Gen5+ (2016) 10TB Gen6 (2017)

12TB

24 EMC Atmos Hardware Guide

Configurations

Figure 3 Promo Code 1 Front (tray pulled out)

Atmos

ECS

Scale

28

srvs encs disks TB drv TB rack

gen1 16 16 240 1TB 240TBgen2 6 24 360 2TB 720TBgen3 8 8 480 4TB 1.9PBgen4 8 8 480 6TB 2.9PBgen5 8 8 480 8TB 3.8PB

gen5+ 8 8 784 10TB 7.8PBgen6 4 8 784 12TB 9.4PB

Mechanicals

29

back-to-back

trays

drawers modular

modular

sleds

drawers

29

Density

2012 Disks (raw) @ 3TB Disks (protected) Racks @ 480 disks5 PB 1,700 disks 2,700 disks 6 racks

20 PB 6,700 disks 11,000 disks 23 racks50 PB 17,000 disks 27,000 disks 56 racks

2014 Disks (raw) @ 6TB Disks (protected) Racks @ 480 disks5 PB 830 disks 1,300 disks 3 racks


2016 Disks (raw) @ 10TB Disks (protected) Racks @ 780 disks5 PB 500 disks 670 disks 1 rack


Updated from “Long-Term Storage”, presented at Library of Congress Workshop in September 2012

30

Scale Out 3.

8PB/

8n48

0d

23 PB petabytes 48 nodes

2,880 disks

RU NILEDENSE40 GbE39 10GbE38 10GbE37 Servers4Node3635 Servers4Node3433

Voyager60Disk

32313029

Voyager60Disk

28272625

Voyager60Disk

24232221

Voyager60Disk

20191817

Voyager60Disk

16151413

Voyager60Disk

1211109

Voyager60Disk

8765

Voyager60Disk

4321 NotUsed 31

FLEXIBILITY

32

Experiments in Flexibility (Goal)

320 TB 4 node 40 disks

RU NILEDENSE40 GbE39 10GbE38 10GbE37 Server4Node3635 Server4Node3433 Voyager

60Disk32313029 Voyager






60Disk8765 Voyager

60Disk4321 NotUsed

3.8PB 8 nodes 480 disks

RU NILEDENSE40 GbE39 10GbE38 10GbE37 Empty3635 Server4Node3433

Blank32313029

Blank28272625

Blank24232221

Blank20191817 Voyager



10Disk8765 Voyager

10Disk4321 NotUsed

1.9 PB 8 nodes 240 disks

640 TB 8 nodes 80 disks








10Disk8765 Voyager

10Disk4321 NotUsed








30Disk8765 Voyager

30Disk4321 NotUsed

RU NILEDENSE40 GbE39 10GbE38 10GbE37 Server2Node3635 Server4Node3433

Blank32313029






10Disk8765 Voyager

10Disk4321 NotUsed

RU NILEDENSE40 GbE39 10GbE38 10GbE37 Server1Node3635 Server4Node3433

Blank32313029

Blank28272625





10Disk8765 Voyager

10Disk4321 NotUsed



5 10GbE4 1GbE3 Server4Node21 KineDc12d

7 10GbE6 10GbE5 1GbE4 Server4Node32 KineDc12d1 KineDc12d



15 10GbE14 10GbE13 1GbE12 Server4Node1110 KineDc12d9 KineDc12d8 KineDc12d7 KineDc12d6 KineDc12d5 KineDc12d4 KineDc12d3 KineDc12d2 KineDc12d1 KineDc12d


RU ECSFLEX40 10GbE39 10GbE38 GbE37 Server4Node3635 Server4Node3433 Server4Node3231 Server4Node3029 Server4Node2827 Server4Node2625 Empty2423 Empty2221 Empty2019 Empty1817 Empty1615

Titan/KineDc84Disk

1413121110

Titan/KineDc84Disk

98765

Titan/KineDc84Disk

4321

2.0 PB 24 nodes 252 disks 33

Experiment – SAS Switching (2012)

34

40 turtle

39 rabbit

38 left right

37 lehi murray

36 layton logan

35 orem ogden

34 provo sandy

33 eight31

29 seven27

25 six23

21 five19

17 four15

13 three11

09 two07

05 one03

01 EMPTY

left right

SERVER

Voy 1A

LCC A

PB EB

EAPAVoy 1B

LCC A

PB EB

EAPAVoy 2A

LCC A

PB EB

EAPAVoy 2B

LCC A

PB EB

EAPAVoy 3A

LCC A

PB EB

EAPAVoy 3B

LCC A

PB EB

EAPAVoy 4A

LCC A

PB EB

EAPAVoy 4B

LCC A

PB EB

EAPA

SERVER

SAS 16Port

SAS 16 Port

SERVER SERVER

Expansion/Secondary

Primary

KEY

SVC Node

SVC Node

1:120 Disk Node with HA SAS and 2 Service/Management Nodes

Experiment – Kinetic (2014)

35

Application File System DB

POSIX File System

Volume Manager Driver

Devices SAS Interface

SMR, Mapping Cylinder, Head, Sector

Drive HDA

Storage Server RAID

Battery Backed RAM CACHE

FC

SAS

from Seagate material “Kinetic Open Storage - Enabling Break-through Economics in Scale-out Object Storage”

PRODUCT SHEET

8OWUD�(IÀFLHQW�(WKHUQHW�'LUHFW�+''�6WRUDJH�6\VWHP�GHOLYHULQJ�2EMHFW� VWRUDJH�WR�&ORXG�DQG�%LJ�'DWD� 'HSOR\PHQWV

7KH�1HZLV\V�('$��SDYHV�WKH�ZD\�IRU�WKH�ODWHVW�GHYHORSPHQWV�LQ�6WRUDJH�WHFKQRORJ\�WDUJHWLQJ�(WKHUQHW�GLUHFW�+DUG�'LVN�'ULYH�WHFKQRORJ\��2SWLPL]HG�IRU�2EMHFW�VWRUDJH��WKH�('$��LPSOHPHQWV�DQ�XOWUD�HIÀFLHQW�VWRUDJH�SODWIRUP�IRU�&ORXG�DQG�%LJ�'DWD� GHSOR\PHQWV��,W�LV�LGHDO�IRU�VFDOH�RXW�DQG�GLVWULEXWHG�VWRUDJH�VROXWLRQV�

8QOLNH�WUDGLWLRQDO�VWRUDJH�ER[HV��WKH�('$��SURYLGHV�UHGXQGDQW�(WKHUQHW�IDEULFV�WKDW�FRQQHFW�GLUHFWO\�WR�WKH�GLVN�GULYHV��HOLPLQDWLQJ�PDQ\�OD\HUV�RI�RYHUKHDG�DQG�HQDEOLQJ�QHZ�OHYHOV�RI�VWRUDJH�VFDODELOLW\��:KHWKHU�GHSOR\HG�LQ�&ORXG�LQVWDOODWLRQV��IRU�%LJ�'DWD�RU�WKH�WUDGLWLRQDO�GDWD�FHQWHU��WKH�('$��GHOLYHUV�REMHFW�VWRUDJH�DW�XQSUHFHGHQWHG�HIÀFLHQF\�

7KH�ÀUVW�GLVN�SURGXFW�WR�OHYHUDJH�WKH�('$��LV�WKH�QHZ�6HDJDWH�.LQHWLF�2SHQ�6WRUDJH�GULYH��

:LWK�XS�WR��[��µ�6HDJDWH�.LQHWLF�2SHQ�6WRUDJH�GULYHV�SHU��8�HQFORVXUH��WKH�LQGXVWU\�XQLTXH�1HZLV\V�('$��LV�DQ�XOWUD�GHQVH��VSDFH�DQG�SRZHU� VDYLQJ��FRPSOH[LW\�UHGXFLQJ�VWRUDJH� VROXWLRQ��7KH�1HZLV\V�('$��ÀWV�QLFHO\�LQWR�D�VWDQGDUG��µ�ZLGH��P�GHHS��UDFN�WKDW�HDVLO\�FUHDWHV�D��GULYHV�8�REMHFW�VWRUDJH�EXLOGLQJ�EORFN��:LWK��7%�GULYHV��WKLV�FDQ�GHOLYHU��3HWDE\WH�SHU�VWDQGDUG��8�UDFN��DQG�FDQ�HDVLO\�VFDOH�RXW�EH\RQG�WKDW�

Product Highlights

Newisys® EDA-46054U-60 3.5” Ultra DenseEthernet Drive Storage Array

� ��2SWLPL]HG�IRU�(WKHUQHW�'LUHFW�� +''V��VXFK�DV�WKH�QRYHO�6HDJDWH�� .LQHWLF�2SHQ�6WRUDJH�'ULYHV� ��,GHDO�EXLOGLQJ�EORFN�IRU�2EMHFW�� 6WRUDJH�GHSOR\PHQWV� ��5HGXFHV�FRPSOH[LW\�DQG�LPSURYHV�� HIÀFLHQF\�IRU�&ORXG�DQG�%LJ�'DWD�� 2EMHFW�6WRUDJH�LQVWDOODWLRQV� ��)XOO�IHDWXUHG��KLJKO\�DYDLODEOH�� KLJK�SHUIRUPDQFH�2EMHFW�6WRUDJH

Product Features� ��)RXU��*E(�FRQQHFWLRQV�WR�WKH�� GDWDFHQWHU� ��5HGXQGDQW��*E(�FRQQHFWLRQV�WR�� HDFK�RI�WKH��+''V� ��'XDO��UHGXQGDQW��KRW�SOXJJDEOH�� (WKHUQHW�6ZLWFK�DQG�0DQDJHPHQW�� (60��PRGXOHV�� 'XDO��UHGXQGDQW��KRW�SOXJJDEOH�� KLJK�HIÀFLHQF\�SRZHU�VXSSO\�DQG�� IDQ�XQLWV��368�� 5HGXQGDQW�KRW�SOXJJDEOH��V\VWHP�� EORZHUV�LPSOHPHQWHG�LQ�WKH�368V�� 0RGXODU�GHVLJQ�LQFUHDVHV�SURGXFW�� FRQÀJXUDWLRQ�ÁH[LELOLW\� ��6WDQGDUG�FKDVVLV�FXVWRPL]DWLRQ�� DQG�EUDQGLQJ�DYDLODEOH

('$��'XDO�(WKHUQHW�6ZLWFK�DQG�0DQDJHPHQW�

• Seagate Kinetic Ethernet drive –  4TB in October 2014 (2x 1 GbE network) –  8TB in September 2015 (2x 2.5 GbE network)

•  Newisys EDA-4605 Enclosure

–  60-disk –  dual 10 GbE

controllers –  4x 10 GbE

uplinks

Experiment – Kinetic (2014)

36

Experiment – Kinetic 2nd Generation (2017)

37

RU ECSFLEX40 10GbE39 10GbE38 GbE37 Rinjin

4Blade3635 Rinjin

4Blade3433 Rinjin

4Blade323130

Titan/KineDc14Disk

2928272625

Titan/KineDc14Disk

2423222120

Titan/KineDc14Disk

1918171615

Titan/KineDc14Disk

1413121110

Titan/KineDc14Disk

98765

Titan/KineDc14Disk

4321

12 nodes 504 disks 240 cores

4,032 TB raw 160 Gbps


4Blade3635 Rinjin

4Blade3433 Rinjin

4Blade323130

Titan/KineDc14Disk

2928272625

Titan/KineDc14Disk

2423222120

Titan/KineDc14Disk

1918171615

Titan/KineDc14Disk

1413121110

Titan/KineDc14Disk

98765

Titan/KineDc14Disk

4321


4Blade3635 Rinjin

4Blade3433 Rinjin

4Blade323130

Titan/KineDc84Disk

2928272625

Titan/KineDc84Disk

2423222120

Titan/KineDc84Disk

1918171615

Titan/KineDc84Disk

1413121110

Titan/KineDc84Disk

98765

Titan/KineDc84Disk

4321

  Parts List   9x Rinjin servers (36 nodes)

  2x 10 GbE per node SFP+

  6x 10 GbE data switches (Arista 64-port SFP+)

  3x 1 GbE mgmt switches (Arista 48-port Cat6)

  18x Titan enclosures (dual controller, 4x 10 GbE uplinks)

  84 * 6 + 14 * 15 = 714

  714x Kinetic/8TB drives

  SFP+ twinax cables (data)

  Cat6 cables (mgmt)

10 Gb Data Switch VLAN 2 (private storage) VLAN 1 (public)

I/O Connectivity – Data Path (2017)

38

Node 1

Kinetic DAE

Node 2 Node x

Kinetic DAE Kinetic DAE

…

…

customer network

LAG

38

1Gb Mgmt Switch VLAN 2 (diag) VLAN 1 (private mgmt)

Management Connectivity (2017)

39

Node 1

Kinetic DAE

Node 2 Node x

Kinetic DAE Kinetic DAE

…

…

deployment network

SUMMARY

40

Software-Defined Storage

  Scale-out storage is all about density (PB/rack) and cost ($/TB)   achieved by simplicity   less components, less cables   less code, less layers

  Many deployments need flexibility   start small, grow large   adjustable compute/storage ratios   purchase-time choice is good; dynamic choice is even better

  Ethernet drives offer this flexibility & scalability

41

Today’s Speakers

  Object Drive Overview, David Slik   Some Products & Observations, Enrico Signoretti   Experiments & Experiences with Object Drives, Erik Riedel

42

After This Webcast

  Please rate this webcast. We value your feedback   This webcast and a copy of the slides will be on the SNIA Cloud Storage website and available on-demand

http://www.snia.org/forum/csi/knowledge/webcasts   A Q&A from this webcast, including answers to questions we couldn't get to today, will be on the SNIACloud blog

http://www.sniacloud.com/   Follow us on Twitter @SNIACloud

43

Thank you.

Documents

IP-Based Object Drives Now Have a Management …...OpenStack Swift • Industry File-Sharing Protocols: NFS, SMB, AFP and FTP • Several data protection schemes and cluster topologies