Global Networking Services Team, Global Foundation Services, Microsoft Corporation

Routing Design for Large Scale Data Centers:

2

3

Problem Statement

4

100s thousands of servers 10G NICs

Aware of the network Explicit parallelism Example: Web Index computation

5

Query Background

Query

Background

6

The simpler the better

Single protocol Simple behavior Wide vendor support

7

What We Started With

8

Folded on diagram

ECMP Based

9

10

11

Why BGP over IGP

12

Better vendor interoperability Less state-machines, data-structures etc

Use for unequal-cost Anycast load-balancing solution

13

BGP RIB structure is simpler compared to link-state LSDB Clear picture of what sent where (RIBIn, RIBOut)

E.g. link failures have limited propagation scope More stability due to reduced event flooding domains

14

Not a problem with automated configuration generation

Is not our primary goal anyways, few seconds are OK Practical convergence in less than a second

15

The New Approach

16

Broadcast storms Hard to troubleshoot

Bandwidths scales up, and not out

17

18

19

No need to buy higher-radix boxes Cheaper infrastructure

No interworking/redistributions etc

20

Details and Design Choices

21

We rely on ECMP for routing Needed for Anycast prefixes

Simplifies path hiding at WAN edge (remove private AS) Simplifies route-filtering at WAN edge (single regexp)

22

Allow AS in

23

AS_PATH Multipath Relax

Allow AS In

Fast eBGP Fall-over

Remove Private AS

24

Otherwise: Route Black-Holing on link failure!

25

Otherwise: Route Black-Holing on link failure!

26

This made it perfect choice for us!

27

Questions?