NVIDIA Confidential

Exascale An Innovator’s Dilemma

Jen-Hsun Huang, CEO SC11, Seattle, Washington | Nov. 15, 2011

"Generally, disruptive innovations were

technologically straightforward,

consisting of off-the-shelf components

put together in a product architecture

that was often simpler than prior

approaches. They offered less of what

customers in established markets

wanted and so could rarely be initially

employed there. They offered a

different package of attributes valued

only in emerging markets remote from,

and unimportant to, the mainstream.”

NVIDIA Confidential

A History of Amazing Advances

1988 1998 2009

TERAFLOPS

PETAFLOPS

ZETTAFLOPS

GIGAFLOPS

EXAFLOPS

Cray T3E-1200 0.9 TF

Cray XT-5 “Jaguar” 1.8 PF, 7 MW

Cray Y-MP8 2.7 GF, 0.15 MW

Advances with Dennard Scaling

1988 1998 2009

TERAFLOPS

PETAFLOPS

ZETTAFLOPS

GIGAFLOPS

EXAFLOPS

Cray T3E-1200 0.9 TF

Cray XT-5 “Jaguar” 1.8 PF, 7 MW

Cray Y-MP8 2.7 GF, 0.15 MW

½ L every 4 yrs 8X – same P

1.68X per/W CAGR

End of Dennard Scaling

1988 1998 2009

TERAFLOPS

PETAFLOPS

ZETTAFLOPS

GIGAFLOPS

EXAFLOPS

Cray T3E-1200 0.9 TF

Cray XT-5 “Jaguar” 1.8 PF, 7 MW

Cray Y-MP8 2.7 GF, 0.15 MW

½ L every 4 yrs 8X @ 4X P

1.19X per/W CAGR

Supercomputing is Power Limited

1988 1998 2009

TERAFLOPS

PETAFLOPS

ZETTAFLOPS

GIGAFLOPS

2012 2035

EXAFLOPS

Cray T3E-1200 0.9 TF

Cray XT-5 “Jaguar” 1.8 PF, 7 MW

Cray Y-MP8 2.7 GF, 0.15 MW

70 PF, 20 MW

100 PF, 20 MW

1 EF, 20 MW

CPU-only “Titan” 6 PF, 8.6 MW

2022 2019

CPUs Fast But Complex

Optimized for single-threaded performance

~50X energy to schedule instruction than the operation

~20X energy to move data across chip than the calculation

Super Efficient Processors Needed

Many simple processors with minimal overhead

Locality reduces data movement energy

Poor single-threaded performance

Innovator’s Dilemma!

PRINCIPLE #1 Companies depend on customers

and investors for resources.

Clayton M. Christensen (1997) The Innovator’s Dilemma:

When New Technologies Cause Great Firms to Fail

PRINCIPLE #2 Small markets don’t solve growth

needs of large companies.

Clayton M. Christensen (1997) The Innovator’s Dilemma:

When New Technologies Cause Great Firms to Fail

GPU Computing “New Market” Disruption

Disruptive technologies underperform established products in mainstream markets.

Cheaper, smaller, and frequently more convenient.

Clayton M. Christensen (1997) The Innovator’s Dilemma:

When New Technologies Cause Great Firms to Fail

Nagasaki University Professor Tsuyoshi Hamada’s Homemade supercomputer

GPU Computing

GPUs with CUDA >350,000,000

Toolkit Downloads >1,000,000

Active Developers >120,000

Universities Teaching GPU Computing >475

HPC OEMs offer GPU Clusters 100%

World’s First

Whole H1N1 Virus Simulation

Lifecycles of fish in Australia

University of Melbourne

Stars and galaxies 12.5B years ago

University of Groningen

Neural networks in a self-learning robot

The University of Plymouth

Directives

65x in 2 Days 5.6x in 5 Days 4.7x in 4 Hours

2012 2022

PETAFLOPS

2019 2035

The Road to Exascale

70 PF, 20 MW

100 PF, 20 MW

1 EF, 20 MW

CPU-only “Titan” 6 PF, 8.6 MW

GPU-Accelerated “Titan”

20 PF, 8.6 MW

1 EF, 20 MW

ZETTAFLOPS

2012

ZETTAFLOPS

PETAFLOPS

2035

The Road to Exascale

70 PF, 20 MW

100 PF, 20 MW

1 EF, 20 MW

CPU-only “Titan” 6 PF, 8.6 MW

GPU-Accelerated “Titan”

20 PF, 8.6 MW

1 EF, 20 MW

1 EF, 20 MW

2022 2019

2012

ZETTAFLOPS

PETAFLOPS

2035

The Road to Exascale

70 PF, 20 MW

100 PF, 20 MW

1 EF, 20 MW GPU-Accelerated “Titan”

20 PF, 8.6 MW

1 EF, 20 MW

100 EF, 20 MW

2022 2019

CPU-only “Titan” 6 PF, 8.6 MW

Huge Markets Support GPU

Mobile

1,000M

PCs

400M

Console

20M

Workstation

5M

5 Watts “Teraflops”

1997 2019

ASCI Red @ Sandia Labs

DEMO: BATTLEFIELD 3

100 Watts “Tens” of Teraflops

2004 2019

Red Storm @ Sandia Labs

DEMO: ASSASSIN’S CREED

1000 Watts

2006 2019

“Hundreds” of Teraflops

Blue Gene @ LLNL

DEMO: MAYA

DEMO: FUMEFX

One Disruption Enables Another

‘Super’ Computing From Super Computers to Super Phones