Configuration & Routing of Clos Networks

v

Running the House That CLOS Built Configuration of CLOS Fabrics

Dinesh Dutt, Chief Scientist

17th September, 2014

Cumulus Networks: Making Networks Accessible

Mission To enable high capacity networks that are easy to deploy and affordable helping customers realize the full promise of the software-defined data center.

Vision Bringing the Linux Revolution to Networking

September 17, 2014 cumulusnetworks.com 2

A Key Transformation

cumulusnetworks.com 3

SPINE

LEAF

CORE

L2 ACCESS

L3 AGGREGATION

September 17, 2014

Simplifications Wrought By L3

§ Fewer protocols to configure §  Single routing protocol vs many L2 protocols §  No FHRP required §  Standard, inter-operable protocols

§ Fewer protocols to troubleshoot

§ You can traceroute across the network


Agenda

§ Intro

§ Configuration & Automation

§ PTM

§ Outro

cumulusnetworks.com 5 September 17, 2014


• Intro

Picture courtesy Nanoer.com @flickr

September 17, 2014

Routing vs Bridging

§ Forwarding based on IP address, not MAC

§ Loop-free protection in data plane and control plane §  Not just determined by a control protocol (STP)

§ Routing protocol assumes peer is down and not reachable if it doesn’t hear from it §  STP takes ports into forwarding if it doesn’t hear from the

other end

§ Routing protocols can work with simple and sophisticated use cases


Basic Parts To A Routing Routing Protocol

§ Three basic parts §  Who do I communicate with (neighbor, peer etc.) §  What do I tell them (IP prefixes usually) §  Tuning the conversation (timers, various protocol

specific knobs)

§ But first, who am I ? §  Router ID


Comparing STP vs Routing Configuration


§  auto br0

§  iface br0 inet static

§  bridge-ports swp1 swp2 swp3

§  bridge-stp on

§  mstpctl-treeprio 32768

§  mstpctl-portautoedge swp1=yes

§ 

§  router ospf §  router-id 0.0.0.1 §  interface swp1 §  ip ospf area 0.0.0.0 §  ip ospf network point-to-point §  interface swp2 §  ip ospf area 0.0.0.0 §  ip ospf network point-to-point

§  router bgp 65535 §  bgp router-id 0.0.0.7 §  neighbor 1.2.3.4 remote-as 65534 §  neighbor 1.2.3.4 activate §  redistribute connected

September 17, 2014

Who Am I Who Do I Talk To What Do I Tell Them Fine Points Of Conversation

The Nice Thing About Routing Protocols…

§ Link state vs Distance Vector §  In distance vector, you tell your neighbors what you know

about the world •  More precisely, a node provides a summarized view of the world to its

neighbors and each neighbor then examines the summaries from all its available neighbors to pick the best paths to use for communication

§  In link state, you tell the world about your neighbors •  More precisely, every node has an exact copy of what every other

node said and uses this to compute reachability to each of the other nodes

•  Comparatively chatty protocol •  Network design to configure protocol to reduce chattiness



• Configuring CLOS Networks

September 17, 2014

Sample Topology


CLUSTER or POD CLUSTER or POD

INTER-CLUSTER SPINE

T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

September 17, 2014

Using OSPF


§  Popular link-state protocol used in enterprises

§  Separate versions for IPv4 and IPv6

§  Chattiness is reduced by defining neighborhoods (called areas)

§  All nodes in a neighborhood have a precise copy of the info from the other neighbors in the neighborhood

§  Information outside the neighborhood is summarized by the border nodes

§  M1& M2 are the border nodes for T1 and T2. Similarly M3 and M4 for T3 and T4.

§  Two level neighborhood hierarchy support only

T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

September 17, 2014

Using OSPF (example configuration)


§  router ospf §  ! On M1 §  log-adjacency-changes detail §  router-id 0.0.1.1 §  passive-interface lo §  interface swp1 §  ip ospf area 0.0.0.1 §  ip ospf network point-to-point §  interface swp2 §  ip ospf area 0.0.0.1 §  ip ospf network point-to-point §  interface swp3 §  ip ospf area 0.0.0.0 §  ip ospf network point-to-point §  interface swp4 §  ip ospf area 0.0.0.0 §  ip ospf network point-to-point §  interface lo §  ip ospf area 0.0.0.0 §  ip ospf network loopback

T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

area 0.0.0.0

Area 0.0.0.1. Area 0.0.0.1.

September 17, 2014

Using BGP


§  Swiss-army knife of routing

§  Powers the Internet

§  Quite a simple path-vector (variant of distance vector) protocol

§  Configuration can be made hairy, but can be kept simple

§ Multi-protocol support

§  Used without any other routing protocol in play in the DC

T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

September 17, 2014

Using BGP


§  router bgp 65400 §  ! M1 config §  ! As defined by draft-lapukhov-bgp-..txt §  bgp log-neighbor-changes §  bgp router-id 0.0.0.1 §  neighbor 10.10.1.1 remote-as 64512 §  neighbor 10.10.1.1 activate §  neighbor 10.10.2.1 remote-as 64513 §  neighbor 10.10.2.1 activate §  neighbor 10.10.10.1 remote-as 65534 §  neighbor 10.10.10.1 activate §  neighbor 10.10.11.1 remote-as 65534 §  neighbor 10.10.11.1 activate §  maximum-paths 64

T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

September 17, 2014


• Simplifying The Configuration

September 17, 2014

Key Characteristics of Any Automatable Configuration

§ Cookie cutter configuration §  As little node-specific variation as possible

•  Nothing more than a single IP address, node name, for example §  As little duplication of information as possible

•  Specifying IP addresses of interfaces AND in OSPF/BGP network statements

§  As much configuration as necessary, not more


OSPF Unnumbered Interfaces

§ IP links are configured to have an IP address at either end in traditional network design §  Adds to node-specific config §  Adds to IP addresses to be allocated §  IPv6 is very cool with link local address §  If these addresses are advertised, they can become attack vectors

§ OSPFv2 has a concept called unnumbered interfaces §  Single IP address per node, usually assigned to loopback interface,

inherited by all the others

§ Simplifies interface configuration, if not routing configuration


Simplifying BGP Configuration

§ BGP uses TCP as its transport which implies the need for an IP address in the specification

§ Violates multiple cookie cutter configuration metrics: §  IP address specified in multiple places (neighbor

statement, IP interface definition etc.) §  Configuration all become node specific


Unnumbered BGP Configuration


T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

§  router bgp 65400 §  ! M1 config §  ! As defined by draft-lapukhov-bgp-..txt §  bgp router-id 0.0.0.1 §  neighbor swp1 remote-as 64512 §  neighbor swp2 remote-as 64513 §  neighbor swp3 remote-as 65534 §  neighbor swp4 remote-as 65534 §  maximum-paths 64

September 17, 2014

Unnumbered BGP: More Details

§ Uses IPv6 RA to determine link local address of remote peer

§ Sets up session over link local address

§ Works well for IPv6 today, IPv4 in the works

§ Interoperates with other implementations that can’t support this mechanism, but can peer over link local address


Using iBGP For Further Simplification

§ ASN forces routing configuration to be node-specific

§ iBGP can be used to force configuration to be a simple three template model: §  One set of configuration each for leaf, pod-level spine

and inter-pod level spine §  All leaves get the same configuration §  All pod-level spines get the same configuration etc.


Using iBGP to Get Rid of ASN


T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

§  router bgp 65400 §  ! M1 config §  bgp router-id 0.0.0.1 §  neighbor TOR peer-group §  neighbor TOR remote-as 64500 §  neighbor TOR route-reflector-client §  neighbor TOR next-hop-self-all §  neighbor TOR activate §  neighbor swp1 TOR §  neighbor swp2 TOR §  neighbor IPS peer-group §  neighbor IPS remote-as 64500 §  neighbor IPS next-hop-self §  neighbor IPS activate §  neighbor swp3 IPS §  neighbor swp4 IPS §  maximum-paths ibgp 64

September 17, 2014


• PTM (Prescriptive Topology Manager)

September 17, 2014

Sorting through the Cables

§ Networking cabling complexity §  As networks scale, the number of cables increases §  How do we ensure cabling correctness in an easy way?


Expected

Topology Design

Network Blueprint

Cable Install

Observed

Steady State Failures Re-

Cabling September 17, 2014

Network Cabling Errors

§ Improper cabling causes §  Reachability issues §  Unpredictable and low

performance


--Alexander Pope

“To err is human”

September 17, 2014

Prescriptive Topology Manager (PTM)

§  Define expected topology using DOT language §  Verify connectivity per topology plan using LLDP §  Take dynamically defined actions based on match of expected & actual


Graph G { S1:p1 – M1:p3; S1:p2 – M2:p3; S1:p3 – M3:p3; S1:p4 – M4:p3; S2:p3 – M3:p4; S2:p4 – M4:p4; M1:p1 – T1:p1;

... M4:p2 – T4:p2;

}

Topology Graph

T1 T2 T3 T4

M1 M2 M3 M4

S1 S2

September 17, 2014

Prescriptive Topology Manager (PTM) – Work Flow


Graphviz

Topology.dot Configuration Management

Tool

Matching design workflow

NOT matching design workflow

September 17, 2014

Prescriptive Topology Manager (PTM) – Work Flow (contd)


• Notify engineers • Work around problem

PTMd

LLDP Topology.dot

neighbors

•  Configure interfaces •  Add self to monitoring…

Matching design workflow

NOT matching design workflow

September 17, 2014

PTM Pre-defined Hookups

§ With Quagga §  Quagga acts as PTM client §  Listens to link-state and BFD notifications


cumulus@S1:~$ sudo vtysh -‐c 'conf t' -‐c 'ptm-‐enable' cumulus@S1:~$ sudo vtysh -‐c 'show interface swp1' Interface swp1 is up, line protocol is up PTM status: pass index 3 metrix 1 mtu 1500 flags: <UP,BROADCAST,RUNNING,MULTICAST> HWaddr: 00:02:00:00:00:11 inet 21.0.0.2/24 broadcast 21.0.0.255 inet6 fe80::202:ff:fe00:11/64 cumulus@S1:~$

PTM: Running State

§ ptmctl (contd) §  Shows expected and observed topology §  Comparison test results – pass or fail


root@cumulus:~# ptmctl ------------------------------------------ port cbl BFD BFD status status peer ------------------------------------------ swp13 pass pass 51.0.0.41 swp14 pass pass 52.0.0.41 swp15 pass pass 53.0.0.41 swp20 fail N/A N/A

ptmd: Running At A Git Near You

§ Open source, published under Eclipse Public License (EPL)

§  h6ps://github.com/CumulusNetworks/ptm



• PTM 2.0

September 17, 2014

Mapping Gods Eye View To Box Specific Config

§ A fundamental problem with networking configuration is the inability to map network-wide view to box-specific configuration

§ PTM provides a network-wide configuration mechanism


Generate Routing Configuration From PTM


Graph G { S1:p1 – M1:p3; S1:p2 – M2:p3; S1:p3 – M3:p3; S1:p4 – M4:p3; S2:p3 – M3:p4; S2:p4 – M4:p4; M1:p1 – T1:p1;

... M4:p2 – T4:p2;

}

§  router bgp 65400 §  ! M1 config §  ! As defined by draft-lapukhov-bgp-..txt §  bgp router-id 0.0.0.1 §  neighbor swp1 remote-as 64512 §  neighbor swp2 remote-as 64513 §  neighbor swp3 remote-as 65534 §  neighbor swp4 remote-as 65534 §  maximum-paths 64

A Summing Up

§ CLOS topology allows for a simplified configuration

§ Routing configuration can be fairly simple

§ Coupled with PTM, BGP unnumbered interface and other like innovations can make configuring L3 networks a snap


Resources available

§ Cumulus WB: http://cumulusnetworks.com/cumulus-workbench/

§ cldemo: github.com/CumulusNetworks/cldemo

§ Website: cumulusnetworks.com

§ Blog: cumulusnetworks.com/blog

§ Twitter: @CumulusNetworks @ddcumulus


CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners.

§ Thank You!


Bringing the Linux Revolution to Networking

September 17, 2014

Software

Configuration & Routing of Clos Networks