IBM Systems

Next generation Ethernet connect to All FLASH: iSERand NVMeF

Subhojit RoySenior Technical Staff Member, IBM

IBM Systems

§ Flash growth and dimensions of change

§ Shared SAN storage requirements

§ What is iSER?

§ What is NVMeF?

§ All FLASH on SAN – roadmap

§ What are we doing at IBM?

Agenda

| 2

IBM Systems

Flash Growth and Dimensions of Change…

StorageSWArchitecture

StorageMedia

Layer2Interconnect(L2)

HDDà SASSSDsàPCIe NVMe àPM(3DXP)

KernelModeà UserMode(SPDK)

FC8G/16Gà Eth10Gà FC32Gà EthRDMA25/40/50/100G

UpperLevelProtocols(ULP) FCPà iSER &NVMeoF (Eth&FC)

StorageHWArchitecture AFServerà AF-HyperConvergedà AF-Arraysà AFDisaggregatedStorage

WorkloadsTraditionalEnterpriseWorkloadsàNewageFlashworkloads

(Tier0:SAPHANA,RTAnalytics,Tier1:OLTP,VDI,SocialMediaApps)

ü Flashisdrivingchange ü Changeismultidimensional

ü Changewouldnotbeallatonce.Onedimensionatatime

IBM Systems

Shared SAN storage requirements

Ø Multipath (for high availability)

Ø Reservation (SCSI Persistent Group Reservation)

Ø vVols (offload Copy Services to Storage Controller)

Ø Data movement/migration without involving host CPU (XCOPY, ODX)

Ø Unmap: Optimize space deallocation on Thin Provisioned storage

Ø Atomic Compare And Write (CAW): Alternative to SCSI3 Reservations

Ø Error Handling: Abort Task, Abort Task Set, LUN Reset, Target Reset etc.

| 4

5

What is iSER?

OperatingSystem/Applications

SCSILayer

FCP iSCSI

TCP

IP

NICDriver

iSER

iWARP Driver RoCE DriverFiberChannelDriver

OFED IB verbs

InfinibandAdapterDriver

Sof

twar

e D

river

sH

ardw

areFCHBA NIC iWARP rNIC RoCE rNIC IBHCA

FC iSCSI iSER

Ω

IBM Systems

iSER: Current state of affairs

6

LowLatency,LowCPUUtilization(Eliminatescopiesto/fromTCP/IPbuffers)

NoChangestoiSCSIadministration(vSphere, Widows, OpenStack work asis)

VendorandTechnologyIndependent(works oniWARP, RoCE &Infiniband HCAs)

WorksonStandardEthernetequipment(10G and25/50/100G switches)

Enterpriseapplicationsjustwork!(vVols, Clustering, Multipath etc.)

SuitableforAll FLASHoverHighSpeedEth(10,25,40,50,100Gbps andbeyond)

Nodisruptiontoadministrationmodel

Fitswell intoSoftwareDefined Storage(SDS)paradigm

CostSavings

Idealforsharedstorage(bothforFLASHandHDD)

iSER is iSCSI with RDMA Data path

iSER is ready for Shared All FLASH SAN storage today!

7

What is NVMeF?

OperatingSystem/Applications

SCSI

iSCSI

iSER

iWARP Driver RoCE Driver

OFED IB verbs

InfinibandAdapterDriver

Sof

twar

e D

river

sH

ardw

are

iWARP rNIC RoCE rNIC IBHCA

iSER

NVMeF

iWARP Driver RoCE DriverFCAdapterDriver

iWARP rNIC RoCE rNIC FCHBA

NVMeF

IBM Systems

NVMeF: Current state of affairs

8

LowLatency,LowCPUUtilization(Primarilyscuts downonHosts/wstack)

NewAdministrativeModel

VendorandTechnologyIndependent(works oniWARP, RoCE &Fiber Channel)

WorksonStandardEthernetequipment(10G and25/50/100G switches)

Applicationsmustchangetoexploitparallelism

SuitableforAll FLASHoverHighSpeedEth(10,25,40,50,100Gbps andbeyond)

ChangestovSphere,Openstack etc.

YesandNo!Needcommonuserspacelayer2

CostSavings

Applicationsneedtotransformyet

NVMeF is still evolving to adapt to Shared Storage Applications!

IBM Systems

HDD

< 2010

SCSI

FC

HDD/SSD

2012-2016…

SCSI

FC

HDD

2010+….

SCSI

iSCSI/Eth

….

Flash/NVMe

2015-….

SCSI

FC

….

Flash/3DXP

2017

NVMeoF

NVMf/RDMA

…. ….

SCSI/FC ruled the

Enterprise Shared Storage World

iSCSI started

penetration in low end

market

PCIe NVMeFlash debuted

as high performance

storage

….

Flash environments

used SCSI with FC as an interconnect

Flash/NVMe

2016-….

SCSI

iSER/RDMA

iSER came as an alternative

to FC for connecting

external Flash

NVMeoFspecification

and technology matured for

Tier0 usecase

….Timeline ForMaturity

Technology

AdoptionStatus

Flash/3DXP

SCSI/NVMeoF

iSER/NVMf

NVMeoF/Eth RDMA

matured for Shared Storage Usecase

UsecaseDetails

1990s – 2010 : HDD/SCSI/FC Rule

2007- 2016 SSD/Flash media evolution, maturity

2015 - … NVM evolution

2016 to 2020 – iSER/SCSI

2019/2022 –iSER/NVMeF

2019….

All Flash SAN roadmap!

IBM Systems

Host HostHost Host Host

Host SAN

HostSVC

HostSVC

HostSVC

HostSVC

Vdisk 1 Vdisk 1 Vdisk 1 Vdisk 1

Device SAN

RAIDCtlr

RAIDCtlr

RAIDCtlr

RAIDCtlr

RAIDcontrollerLUNS

Clustering of Nodes on iSER

Host attach over iSER

Ø Host Attach over iSER –Linux, VMWare

Ø Clustering over iSER

What are we doing at IBM?

IBM Systems 11

iSER vs Fibre channel

I/O iSER (40Gb) Fibre Channel(16Gb)

Read4KiB 50(us) 80(us)

Write4KiB 139(us) 195(us)

Read64KiB 95(us) 196(us)

Write64KiB 209(us) 337(us)

iSER:Fiber Channel benefitsminus theadditional costs

IBM Systems

Thank You!

12