Moore’s Law and Big Data

Andreas Bechtolsheim

Chairman Arista NetworksCo-Founder DSSD

Agenda

• Moore’s Law

• Server Technology

• Storage Technology

• Big Data Architecture

• Use Cases

Moore’s Law is Alive and Well

Dr. Moore’s Prediction 1965

1965 Prediction: Density would double every year

10/29/09 12:49 PMhttp://upload.wikimedia.org/wikipedia/commons/0/00/Transistor_Count_and_Moore%27s_Law_-_2008.svg

Page 1 of 1

40 Y

1,000,000X

2X/2Y

Moore’s Law 1971-2011: 2X/2Y

Overall Roadmap Technology Characteristics —2008 Update 7

Average Industry

"Moores Law“ :

2x Functions/chip Per 2 Years

2007 ITRS Product Technology Trends -

Functions per Chip

1.E-02

1.E-01

1.E+00

1.E+01

1.E+02

1.E+03

1995 2000 2005 2010 2015 2020 2025

Year of Production

Pro

du

ct

Fu

nc

tio

ns

/Ch

ip

[ G

iga (

10^

9)

- b

its, tr

an

sis

tors

]

Flash Bits/Chip (Gbits)Multi-Level-Cell (4bitMLC)

Flash Bits/Chip (Gbits)Multi-Level-Cell (2bitMLC)

Flash Bits/Chip (Gbits)Single-Level-Cell (SLC )

DRAM Bits/Chip (Gbits)

MPU GTransistors/Chip- high-performance (hp)

MPU GTransistors/Chip- cost-performanc (cp)

2007 - 2022 ITRS Range

Average Industry

"Moores Law“ :

2x Functions/chip Per 2 Years

Average Industry

"Moores Law“ :

2x Functions/chip Per 2 Years

2007 ITRS Product Technology Trends -

Functions per Chip

1.E-02

1.E-01

1.E+00

1.E+01

1.E+02

1.E+03

1995 2000 2005 2010 2015 2020 2025

Year of Production

Pro

du

ct

Fu

nc

tio

ns

/Ch

ip

[ G

iga (

10^

9)

- b

its, tr

an

sis

tors

]

Flash Bits/Chip (Gbits)Multi-Level-Cell (4bitMLC)

Flash Bits/Chip (Gbits)Multi-Level-Cell (2bitMLC)

Flash Bits/Chip (Gbits)Single-Level-Cell (SLC )

DRAM Bits/Chip (Gbits)

MPU GTransistors/Chip- high-performance (hp)

MPU GTransistors/Chip- cost-performanc (cp)

2007 - 2022 ITRS Range

Figure ORTC2 ITRS Product Function Size Trends:

MPU Logic Gate Size (4-transistor); Memory Cell Size [SRAM (6-transistor); Flash (SLC and MLC), and

DRAM (transistor + capacitor)]--Updated

THE INTERNATIONAL TECHNOLOGY ROADMAP FOR SEMICONDUCTORS: 2008 UPDATE

100X

12Y

Silicon Roadmap 2012 - 2024

What can one do with 100 Billion Transistors per chip?

• Lot’s of CPU Cores • 100s of high-speed cores per CPU chip

• Lots of Memory Bits • PB Flash, Multi-TB DRAM Systems

• Lots of I/O and Network Bandwidth • 10s of Gigabytes/sec per CPU

Intel 4S Server Board (2015)

• 4 CPU Sockets

• 72 Cores

• 3+ GHz

• 48 DIMMs

• 3 TByte DRAM

• 1400W Power

SunFire E25K (2005)• 72 SPARC Cores

• 1.5 GHz

• 1.15 TByte DRAM

• 76”H, 33”W, 64”D

• 2900 lbs

• 28.7 KWatt

Less throughput than today’s single board 4S X86 Server

0

250

500

750

1000

2010 2012 2014 2016 2018 2020 2022

100X Performance Gain from 2012 to 2024

64-bit CPU Cores over Time

Average increase 30%/year

0

0.25

0.5

0.75

1

2010 2012 2014 2016 2018 2020 2022

Declining Cost of Computing

Average decline ~ 30%/year

• Moore’s Law 1971-2024 • Density has increased 2X every 2 Years • Million-fold improvement 1971-2011 • Another 100X expected 2012-2024 • 100 Million Fold Increase 1971-2024

• Moore’s Law is not ending, but slowing down • Economic and Physics limits on the horizon

Moore’s Law Summary

• Economics are getting more Challenging • Scaling is not “Free”

• Geometric Scaling is Hitting Physics Limits • 10nm, 7nm, 5nm, ???

• Certain Things Don’t Scale Well • For example Flash Bits

Moore’s Law Challenges

However there are always creative solutions to problems

Flash Scaling Issue• Flash does not scale well below 15 nm

• Not enough electrons left to store bits • Charge loss due to shallow trapped electrons • Reduced write and read endurance

• Solution: 3D Stacking • Older process technology with many layers • Enables 256 Gbit (MLC), 512 Gbit (TLC) • More electrons per bit improves endurance

Toshiba BiCS 3D-NAND Stack

DRAM Memory Scaling

© 2014 IBM Corporation4Flash Memory Summit 2015

DRAM & Flash Scaling – Density Trends

DRAM scaling at sub 20nm node for 12/16Gbit – bit cost reduction vs increased technology complexity & fab investment requirements

Flash scaling continues via 2D > 3D NAND transition – enabling 256Gb & higher density MLC/TLC Flash

DRAM Memory Scaling

© 2014 IBM Corporation4Flash Memory Summit 2015

DRAM & Flash Scaling – Density Trends

DRAM scaling at sub 20nm node for 12/16Gbit – bit cost reduction vs increased technology complexity & fab investment requirements

Flash scaling continues via 2D > 3D NAND transition – enabling 256Gb & higher density MLC/TLC Flash

Up to 16 3D NAND Die/Package

19

0

256

512

768

1024

2013 2014 2015 2016 2017

NAND Density/Package (GB)

2X

2X

2X

2X

Combination of TLC, 3D NAND, 16 Stack Die/Package

20

DRAM vs FLASH Pricing

Intel 3D X-Point NV Memory

• Read Performance

• Comparable to DRAM

• Write endurance • Much better than Flashbut not unlimited (10^7)

• Density

• Higher than DRAMbut lower than Flash

• Cost • Lower than DRAMbut higher than Flash

Intel 3DXP The Real Thing

Intel has researched PCM for 45 Years!

Intel 3X Xpoint Form Factors

DDR4 DIMM Formfactor Supports mixed DRAM/3DXP

Memory ConfigurationUp to 512 MB/DIMM

NVMe PCIe FormFactor Supports High-SpeedStorage Applications"Up to 4 TB/Container

How to Build Very High-performanceScalable Storage Compute Systems

The Memory Storage Hierarchy

“ Ideally one would desire an indefinitely large memory capacity such that any particular … word would be immediately available. … It does not seem possible physically to achieve such a capacity. We are therefore forcedto recognize the possibility of constructing a hierarchy of memories, each of which has greater capacity than the preceding but which is less quickly accessible.”

Preliminary Discussion of the Logical Design of an Electronic Computing Instrument Arthur Burks, Herman Goldstine and John von Neumann, 1946

Today’s Memory Storage Hierarchy

CPU Caches

DRAM / 3DXP

Flash Storage

Hard Disk

Optical / Tape

nanosecond Latency

10s of nanosecond Latency

Flash Storage

Disk

10 second Latency

100 microsecond Latency

10 millisecond Latency

• Data lives on Disk or Flash!• Deep Storage Hierarchy!• Apps use small % of data!• Legacy I/O Stacks (FC/SCSI)

• Data lives in persistent memory!• Many CPUs surround and use it!• Big Data Apps use entire datasets!• Need to rethink I/O Stack

Big DataCPU

PBs

TBsGBs

Memory

DiskTape

Old Compute-Centric Model New Data-centric Model

PetaByte High-Perf. Storage

CPU

CPUCPU

CPU CPU

CPU

CPU

CPU

Big Data Apps require rethinking of storage architecture

A New Data-centric Storage Model

Rack-scale DSSD Flash Storage

Copyright 2016 DSSD+EMC, Inc. Proprietary/Confidential: NDA Use Only

D5 Summary

22

36X 4T FMs (100TB usable)

96X Gen3 x4 Client Ports

Dual hot-swap controllers

Dual hot-swap I/O modules

Dual service modules

2+2 hot-swap power and fan

Flood Datapath

10 Million IOPS, 100 GByte/sec IO100 usec Latency, 100 TByte Storage

Rack-Scale Shared NVMe Storage

Copyright 2016 DSSD+EMC, Inc. Proprietary/Confidential: NDA Use Only

Rack with SSDs

3

Compute Node

SSDsControl+Data Traffic

Compute Node

SSDsControl+Data Traffic

Compute Node

SSDsControl+Data Traffic

Compute Node

SSDsControl+Data Traffic

Copyright 2016 DSSD+EMC, Inc. Proprietary/Confidential: NDA Use Only

Rack Scale Flash with DSSD

4

Compute

NodeControl Traffic

Compute

NodeControl Traffic

Compute

NodeControl Traffic

Compute

NodeControl Traffic

PCIe3 Data

PCIe3 Data

PCIe3 Data

PCIe3 Data

10/40Gb Uplink

10/40Gb Uplink

10M IOPS

100 GB/s

100 TB

100μs

Rack-scale NVMe Storage is bigger,faster, and more reliable than

local Flash Storage

NAND

System Call

Application

Libraries

Filesystem

Device Driver

( SAS/SATA )

PCIe HBA

Volume Manager

Flash Controller

NAND Flash

Kernel

Hardware

SoftwareApplication

Libraries

PCIe Link Card

DSSD Controller

DSSD Flash Module

5 ms H

DD

200µs SSD

20-100µs

25 µs OS

25 µs OS

User DMA Port

Rack-scale Direct Access Storage

100 usec Latency, All the Time

Copyright 2016 DSSD+EMC, Inc. Proprietary/Confidential: NDA Use Only

Hadoop I/O on HDD, SSD, and DSSD

11

DSSD Confidential

33

Petabyte Flash Capacity

Petabyte/sec IO

Fastest Storage System

on the planet

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

CM0

CM1

CM

CM

SM1A

SM1B

SM0A

SM0B

0 17 18 35

NSF/TACC DELL/DSSD Storage Cluster

Direct Support for Hadoop/HDFS

Copyright 2016 DSSD+EMC, Inc. Proprietary/Confidential: NDA Use Only

HDFS Architecture

10

HDFS

Filesystem

Buffer Cache

Device Driver

SATA Controller

Disk

HDFS

Filesystem

Buffer Cache

Device Driver

HDFS

PCIe SSD

PCIe

SATA

PCIe

PCIe

Hadoop

Kernel

Motherboard

Traditional PCIe SSD DSSD✓✘ ✘

DSSD HDFS Plugin delivers 2X faster I/O to the application

Application Level Benefits

• 10X Improvements in HDFS Performance

• 3X Improvement in Oracle Benchmarks

• Consistent 100 usec Latency to Application

• Shared Data Avoids Need for Replication

• Enterprise-grade Data Integrity and Security

Use Cases

Oil & Gas: Extraction Simulation Grids

Financial: Transaction Modeling

Government: Design

Life Sciences & Research

Fraud Detection Risk Analytics Predictive ModelingReal Time Analytics on Streaming Data

Financial Modeling

Ingest Market Data

Analyze in near real-time

Correlate with Historical Data

Order of Magnitude Speed-upover existing solutions

Customer Quotes

“Cloudera has tested the DSSD D5 appliance in our lab, and we've seen an order of magnitude increase in performance. It’s the fastest HBase cluster we’ve ever tested.” Mike Olson, Chief Strategy Officer at Cloudera !“DSSD D5 has fundamentally changed our business by eliminating unnecessary software, hardware and pre-processed batch jobs. With DSSD, we’re able to support applications and analytics at never-before-seen speeds.” Brian Dougherty, Chief Technical Architect, CMA !"No other supplier has a storage array equivalent in performance to the D5; it stands alone." Chris Mellor, The Register

Summary: Technology Drivers

Bigger Flash Memory

Faster CPUs

Faster Switch Chips

Denser Memory

Data Centric System Architecture

Big DataCPU

PBs

TBsGBs

Memory

DiskTape

PetaByte High-Perf. Storage

CPU

CPUCPU

CPU CPU

CPU

CPU

CPU

Combination of NVMe Flash, 3D XPoint,High-performance Interconnect and new

Software APIs solve the Data Deluge

Narrow the Data Analysis Gap

Time

Data being collectedThe G

ap

Data that can be analyzed

Conclusions

1. Rapid Growth in Real-time and Historical Data!

2. Many Opportunities to Monetize such Data!

3. Reducing Latency to Applications is Key!

4. Rack-scale Storage provides 10X Speedup !

5. Expect Further Performance Improvements