Interconnection network network interface and a case

study

Network interface design issue• The networking requirement user’s perspective

– In-order message delivery– Reliable delivery

• Error control• Flow control

– Deadlock free• Typical network hardware features– Arbitrary delivery order (adaptive/multipath routing)– Finite buffering– Limited fault handling

• How and where should we bridge the gap?– Network hardware? Network systems? Or a

hardware/systems/software approach?

The Internet approach

– How does the Internet realize these functions?• No deadlock issue• Reliability/flow control/in-order delivery are done at the TCP

layer?• The network layer (IP) provides best effort service.

– IP is done in the software as well.

– Drawbacks:• Too many layers of software• Users need to go through the OS to access the communication

hardware (system calls can cause context switching).

Approach in HPC networks

• Where should these functions be realized?– High performance networking• Most functionality below the network layer are done

by the hardware (or almost hardware)– This provide the APIs for network transactions

• If there is mis-match between what the network provides and what users want, a software messaging layer is created to bridge the gaps.

Messaging Layer

• Bridge between the hardware functionality and the user communication requirement– Typical network hardware features

• Arbitrary delivery order (adaptive/multipath routing)• Finite buffering• Limited fault handling

– Typical user communication requirement• In-order delivery• End-to-end flow control• Reliable transmission

Messaging Layer

Communication cost

• Communication cost = hardware cost + software cost (messaging layer cost)– Hardware message time: msize/bandwidth– Software time:

• Buffer management• End-to-end flow control• Running protocols

– Which one is dominating?• Depends on how much the software has to do.

Network software/hardware interaction -- a case study

• A case study on the communication performance issues on CM5– V. Karamcheti and A. A. Chien, “Software

Overhead in Messaging layers: Where does the time go?” ACM ASPLOS-VI, 1994.

What do we see in the study?• The mis-match between the user requirement

and network functionality can introduce significant software overheads (50%-70%).

• Implication?– Should we focus on hardware or software or

software/hardware co-design?– Improving routing performance may increase

software cost• Adaptive routing introduces out of order packets

– Providing low level network feature to applications is problematic.

Summary from the study

• In the design of the communication system, holistic understanding must be achieved:– Focusing on network hardware may not be

sufficient. Software overhead can be much larger than routing time.

• It would be ideal for the network to directly provide high level services.– The newer generation interconnect hardware tries

to achieve this.

Case study

• IBM Bluegene/L system• InfiniBand

Interconnect Family share for 06/2011 top 500 supercomputers

Interconnect Family Count Share % Rmax Sum (GF) Rpeak Sum

(GF) Processor Sum

Myrinet 4 0.80 % 384451 524412 55152

Quadrics 1 0.20 % 52840 63795 9968

Gigabit Ethernet 232 46.40 % 11796979 22042181 2098562

Infiniband 206 41.20 % 22980393 32759581 2411516

Mixed 1 0.20 % 66567 82944 13824

NUMAlink 2 0.40 % 107961 121241 18944

SP Switch 1 0.20 % 75760 92781 12208

Proprietary 29 5.80 % 9841862 13901082 1886982

Fat Tree 1 0.20 % 122400 131072 1280

Custom 23 4.60 % 13500813 15460859 1271488

Totals 500 100% 58930025.59 85179949.00 7779924

Overview of the IBM Blue Gene/L System Architecture

• Design objectives• Hardware overview– System architecture– Node architecture– Interconnect architecture

Highlights

• A 64K-node highly integrated supercomputer based on system-on-a-chip technology– Two ASICs

• Blue Gene/L compute (BLC), Blue Gene/L Link (BLL)

• Distributed memory, massively parallel processing (MPP) architecture.

• Use the message passing programming model (MPI).• 360 Tflops peak performance• Optimized for cost/performance

Design objectives

• Objective 1: 360-Tflops supercomputer– Earth Simulator (Japan, fastest supercomputer

from 2002 to 2004): 35.86 Tflops• Objective 2: power efficiency– Performance/rack = performance/watt *

watt/rack• Watt/rack is a constant of around 20kW• Performance/watt determines performance/rack

• Power efficiency: – 360Tflops => 20 megawatts with conventional

processors– Need low-power processor design (2-10 times better

power efficiency)

Design objectives (continue)

• Objective 3: extreme scalability– Optimized for cost/performance use low

power, less powerful processors need a lot of processors• Up to 65536 processors.

– Interconnect scalability– Reliability, availability, and serviceability– Application scalability

Blue Gene/L system components

Blue Gene/L Compute ASIC

• 2 Power PC440 cores with floating-point enhancements– 700MHz– Everything of a typical superscalar processor• Pipelined microarchitecture with dual instruction fetch,

decode, and out of order issue, out of order dispatch, out of order execution and out of order completion, etc

– 1 W each through extensive power management

Blue Gene/L Compute ASIC

Memory system on a BGL node

• BG/L only supports distributed memory paradigm.• No need for efficient support for cache coherence on

each node.– Coherence enforced by software if needed.

• Two cores operate in two modes:– Communication coprocessor mode

• Need coherence, managed in system level libraries

– Virtual node mode• Memory is physical partitioned (not shared).

Blue Gene/L networks

• Five networks.– 100 Mbps Ethernet control network for diagnostics,

debugging, and some other things.– 1000 Mbps Ethernet for I/O– Three high-band width, low-latency networks for data

transmission and synchronization.• 3-D torus network for point-to-point communication• Collective network for global operations• Barrier network

• All network logic is integrated in the BG/L node ASIC– Memory mapped interfaces from user space

3-D torus network

• Support p2p communication• Link bandwidth 1.4Gb/s, 6

bidirectional link per node (1.2GB/s).

• 64x32x32 torus: diameter 32+16+16=64 hops, worst case hardware latency 6.4us.

• Cut-through routing• Adaptive routing

Collective network• Binary tree topology, static

routing• Link bandwidth: 2.8Gb/s• Maximum hardware latency: 5us • With arithmetic and logical

hardware: can perform integer operation on the data– Efficient support for reduce, scan,

global sum, and broadcast operations

– Floating point operation can be done with 2 passes.

Barrier network

• Hardware support for global synchronization.• 1.5us for barrier on 64K nodes.

IBM BlueGene/L summary

• Optimize cost/performance– limiting applications.– Use low power design

• Lower frequency, system-on-a-chip• Great performance per watt metric

• Scalability support– Hardware support for global communication and barrier– Low latency, high bandwidth support