BRKMPL-2108 © 2012 Cisco and/or its affiliates. All rights ...d2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKMPL-2108.pdf · How Cisco helped with the ... All rights reserved. Cisco

© 2012 Cisco and/or its affiliates. All rights reserved. BRKMPL-2108 Cisco Public

Global MPLS WAN Redesign Case Study BRKMPL-2108

2


Assumptions and Disclaimers

Participants should have ‒ A solid base knowledge of IP routing over a WAN

‒ Basic knowledge of VRFs and IP Tunnels

‒ Basic understanding of MP-BGP, MPLS control/forwarding plane

ASN Numbers depicted are for representative purposes only

This discussion will not cover the encryption devices, except to note their existence in the topology

While the diagrams depict Internet connectivity, again this is for representative purposes only

3


Agenda

Introduction

Legacy Network Infrastructure

Change Impetus and Requirements

WAN Virtualization Options Considered

End to End Design

Proof-of-Concept Testing

Migration Strategy

Conclusion and Lessons Learned

4


Who and Where?

Government institution ‒ World-wide private network

‒ 50+ nodes on Private WAN

Cisco Advanced Services ‒ Dedicated team on-site for over 7 years delivering Network Optimization Service

(NOS)

‒ Performing Design Review, Software Strategy, Integration Testing, Network Troubleshooting

Adam Callis ‒ CCIE #18125 Service Provider / Routing & Switching

‒ MPLS, BGP, VoIP, and Telepresence

5


What?

Global MPLS WAN Redesign

How Cisco helped with the Private WAN redesign ‒ Reviewed legacy design against current and future network requirements

‒ Reviewed new network virtualization requirements

‒ Proposed MPLS over GRE solution

‒ Designed and executed proof of concept testing

‒ Identified required hardware upgrades

‒ Creation and testing of the migration strategy

‒ Development of the baseline production configurations

‒ Support of production network migrations

6

Legacy Network Infrastructure


7609

Evolution of the WAN

8

2621XM

7507

7513

3600

7513

7507

2621XM

Internet

OC3

DS3 DS3 DS3

DS3

DS1 DS1

DS1

DS1

Private Line Core Circa 1999-2004

8950

8950

8950

8850

7609

7609

7606

7609

7606

7609

7609

7606

Internet

Backup

OC192

OC192

OC192

OC48

OC48

Primary

ATM MGX Core Circa 2004-2009

FastEth

Internet

Provider MPLS VPN

7609

7606 7606

7606

Provider MPLS Core Circa 2009 - Present

GRE

GRE

GRE GRE

FastEth

FastEth

GigE

GigE

FastEth

GigE

GigE

GigE

GigE

GigE


Private Line Core – Key Points

No network virtualization deployed ‒ One of over thirty built by customer

‒ Costly dedicated hardware and circuits

‒ Requires independent NMS tools

‒ Simplify security posture

Dedicated bandwidth between sites ‒ Doesn't allow for sharing bandwidth between

networks

‒ Simplifies QoS policies per network

9

2621XM

7507

7513

3600

7513

7507

2621XM

Internet

OC3

DS3 DS3 DS3

DS3

DS1 DS1

DS1


ATM MGX Switched Core – Key Points

Virtualization of core bandwidth ‒ High bandwidth core circuits (OC192,OC48)

‒ Share core circuits between WANS

‒ ABR VC's allow bursting when available

Cisco 7600 with ATM SPAs used to create VRF-Lite private IP transport core ‒ Provides GigE / FastEth handoff

‒ Supports QoS on ATM Links

IPSec VPN Device (IVD) required ‒ IVD tunnels over ATM Core

‒ IVD needs static routes loaded (Management pain point)

10

8950

8950

8950

8850

7609

7609

7606

7609

7606

7609

7609

7606

Internet

Backup

OC192

OC192

OC192

OC48

OC48

Primary

FastEth

FastEth

FastEth

GigE

GigE

FastEth


Provider MPLS Core – Key Points

Ethernet IP handoff ‒ Lower cost interfaces

‒ Allows for 802.1q virtualization

‒ Up to 10 Gig per port supported

‒ Compatible with existing IVD's

Provider QoS SLA ‒ Utilizes Diffserv Code Points to classify traffic

‒ Non-realtime bandwidth contract 50Mbps (per site)

‒ Mark down packets when exceeding contract rate

‒ Best effort queue available for large file transfers

7609

Internet

Provider MPLS VPN

7609

7606 7606

7606

GRE

GRE

GRE GRE

GigE

GigE

GigE

GigE

GigE

11


Initial Virtualization (VRF-Lite)

New user community requirement for connectivity

‒ Quick turnaround was required

‒ GRE Tunnels in new VRF built to key nodes requiring access

‒ FR Encapsulation on leased lines deployed to enable DLCI separation

VRF-Lite shortcomings identified

‒ Network outages when provisioning new paths on IVD's

‒ Time consuming process for tunnel deployment

12

7606

7609

7606

7606

7606

Internet AS 701

7609

3945

1941

1841 1841

7606

3845 1841

Global GRE

VRF-A GRE

DS3 FR

DS3 FR

DS1 FR

DS1 FR

GigE 802.1q

GigE 802.1q

GigE 802.1q

Provider MPLS VPN

Change Impetus and Requirements


New Virtualization Needs

New user community needing widespread network connectivity

Cost savings edict from CIO ‒ No more building parallel networks!

‒ Utilize as much existing hardware as possible

Security ‒ Must maintain data separation

‒ Must maintain control plane separation

Scalability ‒ Rapid deployment of new VRF's as new communities of interest want to join the

network

14


Capitalizing on the Technology Shift

Migrate from leased lines to IP based transport where available ‒ Cost Savings

Less hub equipment required to terminate service

Circuits now remain local to SP network

‒ Increased network efficiency by sharing bandwidth

Shift from ATM to Ethernet Service ‒ Ethernet interfaces widely available and typically lower cost then ATM card

‒ Improved network efficiency by removing the ATM "Cell Tax"

15


ATM MGX Equipment EoS Announcement

16

End of Sale Notices Entire Deployed MGX Core announced EoS 30/Jul/2010


Network Improvement Requirements

The "Hand Grenade" Test ‒ Fully redundant hardware configuration

‒ Redundant network paths (for core sites)

Easy to manage

Support for line rate transfers (100M and 1G)

Support for full MTU packets (1500 bytes)

Utilize tunneling protocol to limit source / destination IP addresses

QoS capable of shaping and queuing per destination

Symmetric routing to/from Active/Active firewalls

BGP routing for any inter-as peering

17

WAN Virtualization Options Considered


Policy Based Isolation

Well understood solution

Easy to implement

No capital expenditure required

19

Private VLANS in Campus layer 2 network

Shared layer 3 routing table

Access Control Lists (ACL) applied to each router

Scalability

‒ ACLs managed on 50+ routers not feasible

Lack of Virtualization

‒ IP Addresses cannot overlap

Prone to human error


VRF Lite Isolation

Improves security and stability by creating separate control plane per vrf

Widely supported across platforms Simple configuration Familiar CLI command structure

20

Creates isolated routing table per VRF

Utilizes 802.1q, Frame-Relay, or ATM for Layer 2 separation

IP Tunnels (GRE / DMVPN) extend VRF over IP cloud

Scalability ‒ May cause high CPU utilization for Per

VRF routing processes (eg: OSPF) ‒ Excessive configuration when adding new

nodes ‒ Elongated provisioning time for new VRFs

No support for Layer 2 VPN


MPLS over GRE – Point to Point Tunnels

Extends MPLS over IP Core Supported on most platforms Per-Tunnel H-QoS supported

‒ Shape tunnel ‒ Prioritize shaped traffic

21

Static GRE Tunnels built over IP Core between PEs

OSPF and LDP enabled on tunnel interfaces

IPv4 + VPNv4 iBGP sessions built over tunnels

Complexity ‒ Layered troubleshooting effect on uptime ‒ New CLI commands to learn

Scalability ‒ Requires unique loopbacks per tunnel ‒ Mesh of GRE tunnels as nodes are added

Hardware Dependency for 7600's ‒ SIP-400 w/ 2x1GE SPA or ES+ Required

Static GRE Tunnels

PE

PE PE

PE

IP Core


MPLS over mGRE

Extends MPLS over IP Core Scalability

‒ No tunnel related configuration (IGP, LDP) ‒ Adding nodes only requires iBGP neighbor ‒ Simplified troubleshooting resulting in

more uptime Supported on most platforms

22

No GRE Tunnels configured

All MPLS signaling exchanged over iBGP mesh

Packets auto encapsulated with VPN Label and GRE Header based on BGP Next-Hop

Hardware Dependency for 7600's ‒ SIP-400 w/ 2x1GE SPA or ES+ Required

No Per-Tunnel H-QoS GRE packets always sourced from

BGP Loopback ‒ Creates corner case limitation for our

Customer scenario

iBGP Full Mesh Peering

PE

PE PE

PE

IP Core


Decision to Use Static GRE Tunnels

The WAN virtualization direction came down to Static GRE tunnels vs mGRE. Static GRE tunnels were selected for the following reasons

Software support

‒ At the time of testing and implementation, mGRE wasn't supported on code deployed to legacy ISR routers

‒ Customer testing requirements create 9 month wait for new software deployment

Corner case implementation requirement to integrate with IVD's ‒ Requirements dictate redundant IVDs at core sites ‒ IVD's can only have a single next-hop per destination ‒ To support redundancy with these IVD's, multiple loopbacks for GRE endpoints

are required. This is NOT supported in the current mGRE implementation

23

End to End Design


Design Overview Provider provisioned MPLS VPN

provides core IP transport

Redundant uplink and IVD to Provider MPLS Service

Point to point GRE tunnel overlay through external IVD's

Consolidation of leased lines to tail sites onto 3945's at core sites

Customer provisioned MPLS VPN over GRE and leased lines

CE devices all layer 2 switches

Redundant firewalls to Internet

25

3945 P

1841 PE 1941 PE

7609 P/PE

Provider MPLS VPN Service

eBGP

GigE

GigE

7609 P/PE

DS1

GRE

eBGP

3945 P

GigE

1841 PE 1941 PE

DS1 DS1

DS1

CE Switch CE Switch

FastEth GigE

7609 P/PE

GRE

eBGP

3945 P

GigE

1841 PE 1941 PE

DS1 DS1

CE Switch CE Switch

FastEth GigE

CE Switch CE Switch

FastEth GigE

GRE

7606 7606

Internet

GRE eBGP eBGP

eBGP eBGP

eBGP eBGP


Provider MPLS VPN Service Overview

Use of service is mandated where available

Similar to a private IP service commercially available ‒ OC-192 Core

‒ Supports only Ethernet (1 Gig or 10 Gig) handoff

Standard Layer 3 VPN service

Multicast VPN Support

QoS SLAs guarantee bandwidth based on agreed upon DSCP markings ‒ Non-Realtime

‒ Realtime

‒ Best Effort (Discard Eligible)

26


Core Site Design

Re-Use of 7600's as WAN Edge router ‒ Minimal hardware investment (Line cards only)

‒ Highly available chassis

‒ Provides user access ports via Catalyst cards

‒ PE node for local servers / clients

‒ P node for downstream serial aggregation and tail sites

3945 used to aggregate leased line circuits (Leased Line Tail Circuits) ‒ Reduced per port cost

‒ Frees up high speed slots in 7600 for ES+ modules

27

3945 P

1841 PE 1941 PE

7609 P/PE


GigE

GigE

DS1 DS1

CE Switch CE Switch

FastEth GigE

Serial Aggregation Router


Leased Line Tail Site Design

Layer 2 Switch deployed as CE ‒ VLAN Separation between user communities

‒ PE Node will act as default gateway for user subnets

‒ Saves on power, space, cooling, and cost

Small Branch Office (Up to 25 People) ‒ 1941 ISR-G2 router deployed as PE

‒ DS1 Uplink to closet serial aggregation router

Larger Campus Offices (Over 25 People) ‒ 3945 ISR-G2 router deployed as PE

‒ Multiple DS1 (Bonded with ML-FR) or DS3 Uplink

28

3945 P

1941 PE 3945 PE

DS1 DS3

CE Switch CE Switch

GigE GigE

Serial Aggregation Router



GRE Tunnels over Provider MPLS VPN

Jumbo Frame Support

‒ Avoid fragmentation

‒ Increase performance

RFC 1918 Address Assignment

‒ Conserve routable IPv4 space

‒ Require unique loopbacks per GRE tunnel for 7600

H-QoS policy applied to GRE tunnel

‒ Shaper to prevent overrunning remote site

‒ Queuing to protect real time traffic (Voice / Video)

LDP Enabled on GRE tunnels that can support MPLS

7609

7609

Interface IP

Loopback500 10.255.0.1/32

GigabitEthernet0/0/0 172.16.0.1/30

Tunnel500 10.127.0.1/30

Interface IP

Loopback500 10.255.0.2/32

GigabitEthernet0/0/0 172.16.0.5/30

Tunnel500 10.127.0.2/30

eBGP

eBGP GRE

29


IGP Design Principles

OSPF with single area 0.0.0.0 deployed network wide ‒ Simple to configure ‒ Single area required to support traffic engineering tunnels (future need) ‒ OSPF database not large as it only contains routes to loopback addresses

OSPF Network type set to "point-to-point" on /30 Ethernet interfaces ‒ Avoids DR election reducing network convergence time

Passive interface default and MD5 authentication deployed ‒ Ensures only authorized devices can establish OSPF adjacency

OSPF interface costs set based on round trip latency measured (utilizing ping) at installation ‒ Customer applications are highly sensitive to latency, goal is to deliver lowest

latency possible

30


BGP Design Principles

Full iBGP mesh between four core sites Core sites enabled as route reflectors for downstream tail sites

‒ Eliminates the need to full mesh every node in iBGP network

MD5 Authentication enabled on all peers ‒ Prevent someone from spoofing BGP session

Peer groups configured for core and route reflector clients ‒ Eliminates repetitive configuration ‒ Ensures conformity of configuration

Symmetry is required through firewalls ‒ No state information shared between firewalls

Enable use of Standard and Extended Communities ‒ Standard communities utilized in route decision process ‒ Extended communities required for MPLS VPN service

31


BGP Community Usage – Default Route

Inbound from Internet

‒ Only default route necessary from Internet

Reduces BGP Table size

‒ Communities set to denote exit point and firewall learned through

‒ MPLS Routers peer with both east and west coast route reflectors

‒ MPLS Routers match east or west firewall community and adjust local preference accordingly to prefer that exit path

32

RR - P/PE

3945 P

GigE

1841 PE 1941 PE

DS1 DS1

CE Switch CE Switch

FastEth GigE

INET-GW

Internet

eBGP Per-VRF

eBGP

0.0.0.0/0

0.0.0.0/0 701:1

0.0.0.0/0 701:1 65500:1

Set Local Pref 120


BGP Community Usage – Internal Routes

Outbound to Internet

‒ PE router advertises route with community denoting which firewall traffic should return through

‒ INET-GW adjusts local preference to prefer one firewall over another

‒ INET-GW removes private AS

‒ INET-GW aggregates routes (if possible)

‒ INET-GW adjusts the AS-PATH length to influence return traffic from Internet

33

RR - P/PE

3945 P

GigE

1841 PE 1941 PE

DS1 DS1

CE Switch CE Switch

FastEth GigE

INET-GW

Internet

eBGP

eBGP

192.168.10.0/24 65501:1

192.168.10.0/24 65501:1

192.168.10.0/24 AS PATH Adjustment

Proof of Concept Testing


Lab Baseline

Cisco Equipment

Cisco 7606

‒ SUP720-3BXL

‒ ES+ WAN Interface

‒ SIP-400 w/2x1GE V2 SPA WAN Interface

Cisco ISR (1841, 3845)

Cisco ISR-G2 (1941)

Cisco ASA 5505

Software Baseline

12.2(33)SRE2 (7600)

15.1(2)T2 (1941, 3845)

12.4(22)T3 (1841)

Other Tools

Agilent Test Set (Not shown)

35


Test Case Overview

Basic Functionality ‒ Basic end to end connectivity between workstations within VRF

Failover Testing ‒ Testing the failure of uplinks and redundant hardware to validate network

redundancy

Configuration Complexity ‒ No Pass/Fail Criteria, just documented for future reference

Migration strategy validation ‒ Passing Criteria: User migration can be achieved with minimal downtime

36


Test Case Sample: Basic Functionality Testing

37

PE PE IP Core

Single GRE tunnel between two PE nodes

Establish OSPF, LDP, iBGP over GRE tunnel

Create a VRF on both PEs

Generate ICMP traffic between workstations

0% Packet Loss

Latency consistent with underlying transport (< 5 ms for lab)

Migration Strategy


Pre-Migration Requirements

All egress interfaces from 7600 that carry GRE packets must be either ‒ SIP-400 w/2x1GE v2 SPA

‒ ES+ Module

All 7600 routers running 12.2(33)SRE2 or later

3rd Party IVD's must have routes for remote destinations programmed in advance

New circuit paths must be tested and accepted ‒ Large Packets with DF-Bit to confirm MTU support

‒ Extended ping to validate packet loss

‒ Extended ping to confirm QoS settings from provider VPN

39


Migrating the Network – The Plan

Two maintenance windows This is to allow validation of the IVD behavior

prior to actual cutover ‒ Initial Configuration (Over several days)

Building GRE Tunnels Establishing OSPF adjacencies Configuring and validating OSPF costs Document Latencies of each GRE tunnel

‒ Actual Migration (One night) Enable LDP, VPNv4 BGP peering Disable Per-VRF IPv4 BGP peering

40

7609 7609 Global GRE

GigE GigE

7609 7609 GigE GigE


Migrating the Network – The Reality

Situation ‒ Core sites missing proper hardware

SIP-400 w/2x1GE SPA or ES+ ‒ IOS Code not yet upgraded on core routers ‒ Network virtualization requirements cannot be

delayed ‒ ISR / ISR-G2 can support MPLS AS-IS

Work around ‒ Extend legacy VRF-Lite peering between key

core routers GRE tunnel per VRF

‒ Move forward with MPLS deployment on leased line circuits to tail sites

Implication ‒ Migrations timeline significantly extended

41


GigE GigE

7609 7609 GigE GigE

VRF A - GRE


Next-hop Limitations Deploying Workaround

Non-MPLS nodes need BGP next-hop reachable within VRF. ‒ Next-hop self doesn't apply to iBGP relationships ‒ Achieved by route-map next-hop manipulation

MPLS nodes need BGP next-hop to be in

global table for their island ‒ Achieved by route-map next-hop manipulation

VPNv4 Peering must be contained within island

‒ Prevents PE from attempting to impose VPN label that cannot be reached

‒ No LSP path between islands


GigE GigE

7609 7609 GigE GigE

VRF A - GRE

3945

802.1q

1941

DS1 MPLS

3945

1941

DS1 MPLS

802.1q

MPLS Islands

VRF-Lite

42


Network Migrations – Where Are We Now? Where Are We Going Next? Where are we now?

‒ All new tail sites being deployed are utilizing MPLS for network virtualization

‒ Most existing tail sites have been migrated to MPLS

Primarily sites that didn't require virtualization are left unvirtualized

‒ Some core sites still not migrated

Lack of funding to procure proper SIP/SPA combo required

Where are we going next?

‒ Working with IVD manufacturer to overcome limitation that forced static GRE

‒ Migrating any tail site requiring virtualization

‒ Once hardware and software installed on 7600's, completion of core configurations

43

Lessons Learned


Deploy MPLS over mGRE Instead of Static Point to Point Tunnels Common Issues with static tunnels

‒ Static tunnels add configuration complexity (eg: OSPF, LDP, etc) ‒ It's easy to inadvertently mis-configure static tunnels and cause 7600 into

software forwarding state ‒ Static tunnels require manual MTU adjustment, a simple omission of this

configuration can cause detrimental performance impacts on network ‒ Adding a new node requires significant configuration

Problems we faced ‒ Several tunnel mis-configurations caused packet fragmentation and reassembly

to be done in software on 7600, eventually causing protocol adjacencies to fail due to high CPU utilization

‒ When adding a new node a typo caused the GRE tunnel to be punted to software and sent the CPU to 100%

45


Software Upgrade Lessons Learned

Procure spare CF cards and load them with proper IOS in centralized location, then ship to site for installation with return envelope for old card ‒ Ensures consistency of image being deployed

‒ Ensures sufficient flash space will be present at time of upgrade

‒ Allows limited skill set personnel to be your remote hands onsite

Utilize internal CF adapter on SUP720 ‒ Reduces possibility of local site tech "borrowing" your CF Card needed for IOS

Image

Perform IOS Upgrades well in advance of network migration ‒ Any problems such as insufficient memory can be identified and remedied before

the upgrade is actually required

46


General Design Lessons Learned

DNS will save you a lot of time ‒ Adding the point to point subnets to the DNS server allowed made traceroute

more useable when troubleshooting paths

Utilize RFC 1918 space for internal addresses (Loopbacks / P2P) ‒ Conserves your public space ‒ Allows you to more easily make the IP addresses mean something to the network

administrator (ex: Encoding Building Number into 3rd Octet)

Expect your "temporary workaround" to end up being semi-permanent ‒ Once virtualization requirement was met, remote engineers became focused on

next big requirement leaving network in a state of migration

Plan for "never" ‒ Each time I heard "We will never need X" I took a note and planned on supporting

it as inevitably we would need to support those features.

47


BGP Design Lessons Learned

Document and publish your BGP Communities in use ‒ While it made perfect sense what you were doing in the lab, you will forget why

your setting that community when your troubleshooting.

Enable soft-reconfiguration inbound on all peers ‒ During troubleshooting it is often valuable to soft clear BGP sessions

When mixing VRF-Lite and MPLS, pay close attention to next-hop

Don't forget that in order to utilize MD5 on BGP through a firewall requires special configuration

48


Network Management Lessons Learned

Centralize management of network to small group of qualified engineers ‒ Having technicians that are less then qualified dispersed around the country leads

to confusion, configuration errors, and ultimately unplanned downtime

‒ A small qualified group can be trained and highly familiar with the proper configurations. This will help prevent unplanned outages

Ensure your NMS platforms are included in your global routing table and are the first to be migrated ‒ Some router management functions are not well supported if inside a VRF.

Placing all management (TACACS, NTP, Syslog, SNMP, etc) in global table ensures maximum support

‒ Routers respond very slowly when they cannot reach TACACS+ for command authorization (if configured) exasperating the migration time

49


Migration Lessons Learned

Wait to migrate until ENTIRE network can be migrated ‒ Software incompatibilities and hardware availability forced us to migrate from the

outside (tail sites) inward toward the core creating MPLS islands

‒ Created many next-hop resolution issues

‒ Massive confusion among NOC engineers when troubleshooting

Validate ALL network paths, even previously known good paths ‒ A provider MTU had been mis-configured preventing our full frame packets from

transmitting when the DF-Bit was set

‒ A QoS policy shaper was mis-configured on provider causing all traffic to be treated as best-effort by default vs non-realtime by default

50


Related Sessions

MPLS ‒ BRKMPL-2102 – Deploying MPLS-based IP VPNs

‒ BRKMPL-2109 – MPLS Solutions for Cloud Networking

‒ BRKMPL-3101 – Advanced Topics and Future Directions in MPLS

WAN Virtualization ‒ BRKRST-2045 – Network Virtualization Design Concepts over the WAN

51

Questions?


Complete Your Online Session Evaluation Give us your feedback and you

could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.

53


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-demand session videos, networking, and more!

Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus

‒ Twitter: https://twitter.com/#!/CiscoLive

‒ LinkedIn Group: http://linkd.in/CiscoLI

54


Documents

BRKMPL-2108 © 2012 Cisco and/or its affiliates. All rights ...d2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKMPL-2108.pdf · How Cisco helped with the ... All rights reserved. Cisco