The way toward peta-flops - Fujitsu · Title: The way toward peta-flops Author: Dr. Pierre Lagier, Chief Technology Officer, Fujitsu Systems Europe Subject: ISC-2011 Created Date:

The way toward peta-flops

ISC-2011

Dr. Pierre LagierChief Technology OfficerFujitsu Systems Europe

Copyright 2011 FUJITSU Limited

DESIGN CONCEPTSWhere things started from

Copyright 2011 FUJITSU Limited2

New challenges and requirements

Optimal sustained flops per watt!

!

!

!

Low operating cost through efficient cooling, floor space and weight

Mission critical ready with high reliability and availability

Scalability towards 100000’s processors


Petascale supercomputer design concepts

HPC centric

• Design first a true HPC processor rather than adapt the technologies around a commodity processor

• Most efficient technologies fitting together toward HPC efficiency

Environmentally efficient

• Optimal sustained flops per watt

• Flexible mixed water/air cooling capabilities

• High density integration to reduce floor space

Mission Critical

• Longest mean time between interrupt (MTBI) and mean time between failure (MTBF)

• Shortest mean time before restart production

• High throughput capabilities


DESIGN OUTLINEBuilding a peta-flops class computer


Processor

SPARC64TM VIIIfx architecture

Multi-cores technology

256 Floating point registers per core

32KB(I)+32KB(D) 2 Way L1 cache per core

6MB Shared L2 cache

Inter-core hardware synchronisation

Application access to cache management

High performance per watt

128 GFlops

58 Watts peak

Water cooling

Low current leakage of the CPU

Low power consumption and low failure rate of CPUs

Processor

HPC engine

Low –Power

Reliable


Memory hierarchy

Memory

High

throughput

Single CPU

SPARCfx

ICC

L1

32KB(I)+32KB(D)

L2

6 MB

Main memory

16 / 32 / 64GB

64 GB/s


On board interconnect controller (ICC)

ICC

Rich functions

High BW &

Low latency

SPARCfx

ICC

Rich functions

User level RDMA

Hardware barrier and reduction offload engine

Low latency and high-throughput

Full crossbar router with redundant direct paths

Dedicated data path with CPU

Direct command issuing to ICC enables low latency – user space interconnect DMA hardware registers memory mapped through Linux kernel driver extension


System Board

Compact design

4 x compute nodes per system board

Hybrid cooling

Direct 6D Mesh/Torus (4 ICC’s)

32 x DIMMs for memory

dimensions 526mm x 481mm x 46mm

Low processor temperature

Easy replacement

Remote power on/off of each system board

No need to stop a rack to replace a system board

Motherboard

Compact

Maintenance


Tofu Interconnect (1)

Interconnect

Fast

Reliable

Efficient

Technology

Very fast node to node communication,5GB/s x 2 (bi-directional)

Low latency, less than 2 µs point to point hardware latency

Global hardware barrier, less than 10 µs to synchronise all compute nodes of petaflops class machine

Integrated MPI support for collective operations

Topology

6D Torus / Mesh physical node addressing (x, y, z, a, b, c)

Logical 3D Torus partitioning (x, y, z) with 3 additional communication paths (a, b, c)

y

x

z

c

b

a


Rack integration

High density integration

24 multi-node system boards

96 compute nodes

12 Tflops of compute power

6 I/O nodes per rack

48 GB/s I/O throughput

“Tofu” Integrated switchless 6D Torus/Mesh interconnect

Flexible mixed water and air cooling

Easy access to all components with very fast hardware replacement

Few minutes to replace a system board

Rack

Density

Cooling

Maintenance


Deployment

Inside a rack

Z axis of 3D Torus

8 mesh, 12 nodes each with a,b,c axis

Between racks

X,Y axis of 3D Torus

Interconnect

[X-Y] between

racks

[Z][a-b-c]

inside a rack

X

Y

Zc

b

a

8 mesh


RIKEN NATIONAL PROJECTBreaking 8 peta-flops sustained

All images courtesy of RIKENCopyright 2011 FUJITSU Limited13

Facilities in Kobe (1)

Main computer building

Research unit building

Power supply and cooling unit building

Electricpower supply

~65m

~65m

Courtesy of RIKEN

Kobe site ground plan


Facilities at Kobe (2)

Cooling towers

Centrifugal chillerCourtesy of RIKENPower Supply and Cooling Unit Building


Inside the computer room

The Next-Generation Supercomputer installed in the facility


1717 Copyright 2011 FUJITSU Limited

Documents

The way toward peta-flops - Fujitsu · Title: The way toward peta-flops Author: Dr. Pierre Lagier, Chief Technology Officer, Fujitsu Systems Europe Subject: ISC-2011 Created Date: