A Scalable, Commodity Data Center Network Architecture

Mohammad Al-Fares, Alexander Loukissas, Amin Vahdat

Presented by Gregory Peaker and Tyler Maclean

Overview

• Structure and Properties of a Data Center• Desired properties in a DC Architecture• Fat tree based solution• Evaluation of fat tree approach

Common data center topology

Problem With common DC topology

• Single point of failure• Over subscription of links higher up in the

topology– Typical over subscription is between 2.5:1 and 8:1– Trade off between cost and provisioning

Properties of solutions

• Compatible with Ethernet and TCP/IP• Cost effective– Low power consumption & heat emission– Cheap infrastructure– Commodity hardware

• Allows host communication at line speed– Over subscription of 1:1

Cost of maintaining switches

Review of Layer 2 & Layer 3• Layer 2– Data Link Layer

• Ethernet• MAC address

– One spanning tree for entire network• Prevents looping• Ignores alternate paths

• Layer 3– Transport Layer

• TCP/IP– Shortest path routing between source and destination– Best-effort delivery

FAT Tree based Solution

• Connect end-host together using a fat tree topology – Infrastructure consist of cheap devices• Every port is the same speed

– All devices can transmit at line speed if packets are distributed along existing paths

– A k-port fat tree can support k3/4 hosts

Fat-Tree Topology

Problems with a vanilla Fat-tree

• Layer 3 will only use one of the existing equal cost paths

• Packet re-ordering occurs if layer 3 blindly takes advantage of path diversity– Creates overhead at host as TCP must order the

packets

FAT-tree Modified

• Enforce special addressing scheme in DC– Allows host attached to same switch to route only

through switch– Allows inter-pod traffic to stay within pod– unused.PodNumber.switchnumber.Endhost

• Use two level look-ups to distribute traffic and maintain packet ordering.

2 Level look-ups

• First level is prefix lookup– Used to route down the topology to endhost

• Second level is a suffix lookup– Used to route up towards core– Diffuses and spreads out traffic– Maintains packet ordering by using the same

ports for the same endhost

Diffusion Optimizations

• Flow classification– Eliminates local congestion– Assign to traffic to ports on a per-flow basis

instead of a per-host basis• Flow scheduling– Eliminates global congestion– Prevent long lived flows from sharing the same

links– Assign long lived flows to different links

Results: Heat & Power Consumption

Implementation• NetFPGA:• 4 Gigabit Ports, 36 Mb SRAM• 64MB DDR2, 3GB SATA Port

• Implemented elements in Click Router Software• Two Level Table• Initialized with preconfigured information

• Flow Classifier• Distributes output evenly across local ports

• Flow Report + Flow Schedule• Communicates with central schedule

Evaluation• Purpose: measure bisection bandwidth

• Fat-Tree: 10 machines connected to 48 port switch• Hierarchical: 8 machines connected to 48 port switch

Results

Related Work• Myrinet – popular for cluster based supercomputers• Benefit: low latency• Cost: proprietary, host responsible for load balancing

• Infiniband – used in high-performance computing environments• Benefit: proven to scale and high bandwidth• Cost: imposes its own layer 1-4 protocol• Uses Fat Tree

• Many massively parallel computers such as Thinking Machines & SGI use fat-trees

Conclusion• The Good: cost per gigabit, energy per gigabit is

going down• The Bad: Datacenters are growing faster than

commodity Ethernet devices• Our fat-tree solution• Is better: technically infeasible 27k node cluster using 10

GigE, we do it in $690M• Is faster: equal or faster bandwidth in tests• Increases fault tolerance• Is Cheaper: 20k hosts costs $37M for hierarchical and

$8.67M for fat-tree (1 GigE)• KO’s the competing data center’s