IBM® Edge2013 - A Brief History of SVC and Storwize Family

© 2013 IBM Corporation

A Brief History of SVC & Storwize FamilyWhat, How and Why?

Barry Whyte – IBM Master Inventor – Virtual Storage Performance Architect

June 2013


A Brief History of SVC and Storwize Family

2

Session goals and welcome

This session presents a brief history of IBM SAN Volume Controller (SVC) and IBM Storwize family.

Starting back in 1999, it will explain why IBM decided to develop SVC, and the research projects that lead to the clustering architecture and the I/O processing architectures.

The session goes on to describe the benefits of storage virtualization and how the software and hardware have evolved over time to where we are today, with the same software running across twelve different hardware platforms in as many years.

From a field-hardened code base to a modern storage appliance/controller architecture that is the fastest thing in anyone's SAN -- even with today's SSD devices – you'll learn why these IBM offerings lead the way in storage virtualization.



Professor Steven Hawking's seminal work...

By the end of this session, I'll show how we :

– We are 14+ years ahead of the recent software defined storage trend

– Set several trends in the storage industry• First to use and prove commodity hardware as a storage platform• First to implement integrated Flash technology

A Brief History ...

Recently added a time-machine to the Storwize family

– Without the need for a Flux-capacitor...



4

From Direct Attached to expanding SANs

A Brief History of SVC and Stowize Family



Storage Industry – IBM circa 1995 - Direct Attached Storage

Before the mid 1990's most storage systems were direct attached

IBM Serial Storage Architecture– Developed by the IBM UK Hursley Storage team (who develop the Storwize Family)– Dual serial disk loops per adapter – 96 drives per loop - 160MB/s

• Spatial re-use – only seen now in todays SAS topologies• 8 way server attachment• 2 way RAID architecture

– Over 95% attachment rate to IBM RS6000 server platform (Power Systems)– Original storage disk adapters in IBM Enterprise Storage System (ESS)

• Evolved into IBM DS8000 with FC-AL interfaces using same adapter firmware– Returned over $2 billion revenue between 1996 and 2003

SANs started to make direct attach obsolete, when many hosts could attach to one or more disk system



Moore’s Law

60-100% CGR

60-100% CGR

25% CGR

25% CGR

Gbi

t / s

q uar

e in

c h

'Ye canny change the laws of physics, Captain!



Back in 1999 we could see the second slow down was coming

SAN complexity had grown “out the box”Storage itself was becoming a commodityStoage islands – wrong place – wasted capacity?

At the same time :– Custom storage function silicon development was too slow and costly – 3yrs etc– Commodity (x86) silicon was proving to be enterprise ready

Questions we asked :– How can we tame heterogenous vendor SANs?– How can we consolidate the storage functions?– How can we make better use of what customers already have?– How can we ride the x86 technology wave for years to come?

State of the Storage Industry – circa 1999



More questions... can something help answer them all?

In addition to the technology questions, several business and operational questions were also being asked :

– The continual growth in data center costs– Inability of IT organization to respond quickly to business demands– Poor availability or service levels– Lack of skilled staff for storage admin tasks– Poor asset utilisation

So while SANs had opened the potential for more flexibility– In reality, they opened up another set of problems

Could we radically change the way we think about storage?



Storage Virtualization – an often over-abused term

Server virtualization, wasn't really around in 1999, other than on mainframe systems

But what if we could virtualize all those disk systems– Consolidate the disparate capacity

• Make better use of whats already there• Help slow down the continual need to “buy more disk”

– Provide mechanism to migrate around downtime– Implement a common set of storage functions

But also make sure we answered the technology questions– Use a commodity hardware base– Ride Intels technology wave– Write the intelligence into the software

We'd need a flexible – future proof software architecture

+ Software Function+ Software Function



10

Host Zone

Storage Zone

Host Zone

Storage Zone

In-band Appliance• Caching and scale-out nodes enhances performance• Consolidates Copy Services• Grows with business needs

Array Based• No additional HW required• Not scalable - limited by box performance• Consolidates Copy Services

Out-of-band – Switch Based• Potentially scales to large networks• Array-based or Switch-based Copy Services

Storage Zone

Host Zone

Storage Virtualization – The Approach War



IBM Research – 1999 Storage Software Architecture

1999 IBM Systems Journal– The software architecture of a SAN storage

control system

Cluster of processing nodes

Abstraction layers– Interface layer

• Abstract the I/O from the protocol

• Common platform I/O (plio) definition

– Peer to peer comms• Abstract inter-node comms

– Common Configuration API• Object manipulation

– Common cluster API model• Cluster state interfaces

Stack of I/O components– Each implements a specific,

encapsulated storage function– Upper/Lower common plio interface

I/O Component

I/O Component

I/O Component

I/O Component

Clusterin g

Configur ation

Peer C

o mm

unic ations

Interface Layer

User Interfaces



12

Cache

FlashCopy

Replication

SCSI Target

Mirroring

Thin Provisioning

Virtualization

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Compression


The Birth of SVC



All virtualized storage common across entire cluster (visible to all nodes)– Controller volumes (LUNs) grouped into common attribute storage poolsC

Control Enclosure

Storage pools of similar performing / RAIDed LUNs

All storage pools visible to all SVC nodes

SVC nodes deployed as active/active caching pairs

Virtual volumes provisioned from striped capacity from a given storage pool

Associated with a caching node pair

IBM SVC – Inband Storage Virtualization Concept ~2000



IBM SVC v1.1.0 – 2003

2000 – Hursley development begins work on turning the Almaden research work into an in-band virtualization appliance.

By July 2003 the first release was ready for GA

– 4-way node clustering– FlashCopy and MetroMirror

functionality – Online storage migration

Initial controller support– IBM DS and FAStT product ranges– EMC DMX and Clariion

Latest Hardware (4F2)– X335 Server Base– 2x 1core 2.8Ghz Xeon (Prestonia)– 533MHz FSB– 4 GB Cache– 4x 2Gbit Fibre Channel

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication (MM)

SCSI Target

Virtualization

SCSI Initiator

Supported Hardware

SVC ● 4F2 Node



IBM SVC – v2.1.0 and v3.1.0 - 2004/2005

Support for up to 8-way node clusteringOS and controller support ++

2004 Latest Hardware (8F2)– X336 Server Base– 2x 1core 3.0Ghz Xeon (Irwindale)– 800MHz FSB– 8 GB Cache– 4x 2Gbit Fibre Channel– ~ 25% IOPs boost (FSB)

2005 Same as above except: (8F4)– 4x 4Gbit Fibre Channel– 2x GB/s boost

Internal code changes – DV (DA)-V– Political debates

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication (MM)

SCSI Target

Virtualization

SCSI Initiator

Supported Hardware

SVC ● 4F2 Node● 8F2 Node● 8F4 Node



IBM SVC – v4.1.0 - 2006

Continued OS and controller support ++

First major new set of functional enhancements

Multiple-target FlashCopy– Up to 256 targets from single volume– No overhead (only really 1 copy unless split

again)

GlobalMirror– Added asynchronous replication functions– Using same replication layer– Sync or Async at create time Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

SCSI Initiator

Supported Hardware

SVC ● 4F2 Node● 8F2 Node● 8F4 Node



IBM SVC – 2007 – Multi-core CPUs

Late 2006 saw the evolution away from the GHz race– More cores per chip– Early samples of Intel Woodcrest dual core CPU packages– Fundamental change to they way we had previously

allocated resources

Cores, Threads and Spinlocks– SVC up till now had run two threads, one pre CPU– Spinlocks relatively low impact– While mutli-threaded, base platform code

– With four threads, spinlocks, or rather spinlock contention wasting ~15% CPU cycles – idle cores...

Without giving too much IP away– Binding of threads, ports and cores– Increased threading, batching while locked...



SVC and Storwize Family Architecture

Since 2007 each node has at least 4 cores– Piggy-back on Intel development– Latest and greatest PCIe bus/memory

Generally run a single “fibre thread” per core– Queues of lightweight fibres– No interrupts or kernel code– Thread polls hardware from user mode

One or more “driver polling functions” per thread– Each port has its own polling function– Some management functions – like PCIe driver– Some kernel functions – like iSCSI

Object ownership maybe distributed across cores– Attempt to avoid context switch between threads if possible

Fibre Thread (bound to a single core)

FC Port polling fn

SAS Port polling fn

iSCSI Port polling fn

Fibre Queue

Fibre

Fibre

Fibre

Fibre

Fibre

Fibre

...



IBM SVC – v4.2.0 - 2007


First release of new multi-thread changes– 2x IOPs boost– 33% boost to existing nodes

Enhanced FlashCopy– Incremental– Cascaded multi-target

Latest Hardware (8G4)– X3350 Server Base– 2x 2core 2.33Ghz Xeon (Woodcrest)– 1333MHz FSB– 8 GB Cache– 4x 4Gbit Fibre Channel

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

SCSI Initiator

Supported Hardware

SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node



IBM SVC – v4.3.0 - 2008


Forwarding for asymmetric access

Volume Mirroring– Two copies of a single volume stored on

potentially disparate storage controllers– Like LVM mirroring in the SAN– Migration using split-mirror

Thin Provisioning– Fine grained thin provisioning (32KB - 256KB)– Use volume mirroring for thick to thin migrate– Mix within pools of thick and thin– Volume and pool usage warnings

New hardware – 8A4 – Entry Edition

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

SCSI Initiator

Supported Hardware

SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node● 8A4 Node

Mirroring

Thin Provisioning

Forwarding

Forwarding



21

Cache

FlashCopy

Replication

SCSI Target

Mirroring

Thin Provisioning

Virtualization

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Compression


The 1,000,000 IOPs Era (but we were first!)



22

2008 - Quicksilver - Scale-out Flash with SVC

Quicksilver was an SVC and Flash technology demonstration in 2008– Proved the capability of a scale-out clustered storage system– Proved SVC stack was Flash ready... already

Achieved 1.2 million IOPS (70/30 4KB), at under 1 ms response time

Technology demo used– 14 current generation SVC nodes (8G4)– 32 System X servers with Flash PCIe cards

– Each of these running a cutdown SVC virtualizing local block Linux /sd devices

– Presenting the /sd devices as “mdisks” to the 14 node SVC cluster



IBM SVC – v5.1.0 - 2009

Continued OS and controller support ++First integrated Flash support in virtualizer

– Integrated SAS HBA / driver• What could we use that for in the future...

– First single system capable of more than 1M IOPS– Low latency key factor – 50us overhead

Up to four Cluster Partnerships for Replication iSCSI – 1Gbit

– SVC software upgrade strategy– All users, all hardware now iSCSI capable

Stretched Cluster (Split Cluster) HAZero detect on new CF8 HardwareLatest Hardware (CF8)

– X3350M2 Server Base– 1x 4core 2.4Ghz Xeon (Nehelan)– QPI replaces FSB– 24 GB Cache– 4x 8Gbit Fibre Channel

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

SCSI Initiator

Supported Hardware

SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node● 8A4 Node● CF8 Node

Mirroring

Thin Provisioning

Forwarding

Forwarding



24

Cache

FlashCopy

Replication

SCSI Target

Mirroring

Thin Provisioning

Virtualization

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Compression


Introducing the Family



What we need is an Enterprise stack for all...

++

=

If you built it.. they will come...

Software function – everyone wants it– What if we could take what we've done for the last

ten years, and make it available to all..– SVC is just software running on a server– But what about drives, and RAID

• Remember that SAS driver• Remember our SSA history

Build a dual-controller integrated disk system

Time to market...– Just a year away...

Lessons learnt...– One code base, one build, one binary...



IBM SVC, Storwize – v6.1.0 - 2010


RAID– Taken from SSA RAID firmware– SAS already added for CF8 Flash Drives

• New SAS network topologiesEnclosure Services

– SES integration, RASEasyTier

– Optimal use of 5% Flash capacityPCIe Driver

– Inter-node communications inside enclosure

Latest Hardware – V7000

• Dual node SVC inside 2U

Enclosure with disks

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

Supported Hardware

SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node● 8A4 Node● CF8 Node

Mirroring

Thin Provisioning

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Supported Hardware

Storwize● V7000

Drive



IBM SVC, Storwize – v6.2.0, v6.3.0 - 2011


10Gbit Ethernet– iSCSI 10Gbit support on SVC and V7000

Storwize V7000 – 480 drive clusters

Storwize V7000 Unified– NFS, CIFS, SMB, HTTP, SCP file modules

Latest Hardware (SVC CG8)– X3350M3 Server Base– 1x 6core 2.4Ghz Xeon (Westmere)– QPI 6.4GT– 24 GB Cache– 4x 8Gbit Fibre Channel

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

Supported Hardware

SVC ● 8F2 Node● 8F4 Node● 8G4 Node● CF8 Node● CG8 Node

Mirroring

Thin Provisioning

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Supported Hardware

Storwize● V7000● V7000 Unified

Drive



Compression

IBM SVC, Storwize – v6.4.0, v6.4.1 - 2012


FCoE Support– Using existing 10Gbit HBAs

Volume Migration– NDVM – Volume move between IO/Groups

without disruption

960 Drive V7000 Clusters

Real-time Compression– Temporal Locality– (I promised you a time machine)

Storwize V3500, V3700– Entry level controller

Legend

Virtual

Mdisk

Cache

FlashCopy

Replication

SCSI Target

Virtualization

Supported Hardware

SVC ● 8F2 Node● 8F4 Node● 8G4 Node● CF8 Node● CG8 Node

Mirroring

Thin Provisioning

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Supported Hardware

Storwize● V3500● V3700● V7000● V7000 Unified

Drive



29

Cache

FlashCopy

Replication

SCSI Target

Mirroring

Thin Provisioning

Virtualization

Forwarding

Forwarding

RAID

SCSI Initiator

Forwarding

Easy Tier

Compression

A Brief Future of SVC and Storwize Family

The Future...



2003 We were Here →

Cache

FlashCopy

Virtualization

1st Hardware model3 Features :

Metro Mirror, FlashCopy and Migration (Virtualization)

1 Block protocol :Fibre Channel

Potential to be:– Extensible– Flexible– Augmentable

Was the architecture right?

Configur ation

Peer C

o mm

unic ations

Interface Layer

Clusterin g

SCSI Initiator

SCSI Target

Replication (MM)

Fibre Channel



2013 We are Here →

Compression

Cache

FlashCopy

Virtualization

Mirroring

Thin Provisioning

Forwarding

Forwarding

RAID

Easy Tier

12 New hardware models running the same software*

8 new major features :Global Mirror, Volume Mirrroing, Thin Provisioning, Stretched Cluster, EasyTier, Compression, RAID, Global Mirror with Change Volumes

6 Major and 17 Minor Software releases :

(1.1, 1.2, 2.1, 2.1.1, 3.1, 4.1, 4.1.1, 4.2, 4.3, 4.3.1, 5.1, 6.1, 6.2, 6.3, 6.4, 6.4.1 … 7.1)

4 new block protocols:iSCSI, FCoE, PCIe, SAS

Extensible, flexible, augmentable - YES!

Get the platform and software architecture and philosophy right on day one, and there are no limits or boundaries

* Only 4F2 hardware limited to running no later than 5.1 Software due to 32bit CPU

Configur ation

Peer C

o mm

unic ations

Interface Layer

Clusterin g

SCSI Initiator

Forwarding

SCSI Target

Replication

Fibre Channel

iSCSI

FCoE

SAS

PCIe



SVC and Storwize Family – Into the second decade

CG8 Hardware Enhancements - 1H13– 2x FC ports (16x 8Gbit per node pair)– 12 core CPU per node

• 3x Compression throughput

FlashSystem + SVC– Continues the evolution of SVC as the ultimate performance and function platform– Greater than 1M Production IOPs in half a rack

More information this week :– SV-1038A Optimizing Virtual Storage Performance– SV-1218A IBM Storage Virtualization Product Directions– PU-1670A IBM SAN Volume Controller and Storwize Family: What's New

Documents

IBM® Edge2013 - A Brief History of SVC and Storwize Family