Multilayer CampusMultilayer Campus Architecture and Design ... · Multilayer CampusMultilayer Campus Architecture and Design Principlesand Design Principles ... Design Principles

Multilayer CampusMultilayer Campus Architecture and Design Principlesand Design Principles

© 2008 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKRST-203114457_04_2008_c2 1

Enterprise-Class AvailabilityR ili t C C i ti F b iResilient Campus Communication FabricCampus Systems Approach to High Availability

Network-level redundancy

System-level resiliencyUltimate Goal……………..100%

y y

Enhanced management

Human ear notices the

Next-Generation AppsVideo conf., Unified Messaging,Global Outsourcing, E-Business Wireless Ubiquityu a ea ot ces t e

difference in voice within 150–200 msec—10 consecutive G711 packet loss

E Business, Wireless Ubiquity

Mission Critical Apps.Databases, Order-Entry,consecutive G711 packet loss

Video loss is even more noticeable

CRM, ERP

Desktop Apps

200 msec end-to end-campus convergence

Desktop AppsE-mail, File & Print

APPLICATIONS DRIVE REQUIREMENTS

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2BRKRST-203114457_04_2008_c2

APPLICATIONS DRIVE REQUIREMENTS FOR HIGH AVAILABILITY NETWORKING

Next Generation Campus DesignU ifi d C i ti E l tiUnified Communications Evolution

VoIP is now a mainstream technology VoIP is now a mainstream technology

Ongoing evolution to the full spectrum of Unified Communications

High Definition Executive Communication Application requires High-Definition Executive Communication Application requires stringent Service-Level Agreement (SLA)

Reliable Service—High Availability Infrastructure

Application Service Management—QoS


AgendaAgenda

Multilayer CampusData Center Services

Block Multilayer Campus

Design Principles Foundation Services Foundation Services Campus Design

Best Practices IP Telephony

ConsiderationsSiSiSiSi

QoS Considerations Security

ConsiderationsSiSiSiSiSiSi SiSi

Considerations Putting It All Together

Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi


Summary Distribution Blocks

High-Availability Campus DesignStructure Modularity and HierarchyStructure, Modularity, and Hierarchy

Access

SiSi SiSi SiSi SiSi SiSi SiSiDistribution

SiSi SiSiCore

SiSi SiSiSiSi SiSi

SiSi SiSiDistribution SiSi SDistribution

Access


Data CenterWAN InternetAccess

Hierarchical Campus NetworkHierarchical Campus NetworkStructure, Modularity and Hierarchy

Not This!!SiSi

SiSi

SiSi

SiSi SiSi SiSi

SiSi SiSi

SiSi

Server Farm

SiSi SiSi SiSi


WAN Internet PSTN

Hierarchical Network DesignHierarchical Network DesignWithout a Rock Solid Foundation the Rest Doesn’t Matter

Access Offers hierarchy—each layer has specific role

Modular topology building blocks

SiSi SiSiDistribution

Modular topology—building blocks Easy to grow, understand, and

troubleshoot Creates small fault domains

SiSi

Core

Creates small fault domains—clear demarcations and isolation

Promotes load balancing and redundancy SiSi

SiSi

Distribution

y Promotes deterministic traffic

patterns Incorporates balance of both Layer 2

SiSi SiSiDistribution p y

and Layer 3 technology, leveraging the strength of both

Utilizes Layer 3 routing for load balancing fast convergence


Building BlockAccess balancing, fast convergence,

scalability, and control

Access LayerAccess Layer

It’s not just about connectivity

Feature Rich EnvironmentIt s not just about connectivity

Layer 2/Layer 3 feature rich environment; convergence, HA, security, QoS, IP multicast, etc. Core

SiSiSiSi Intelligent network services: QoS,

trust boundary, broadcast suppression, IGMP snooping

Intelligent network services: PVST+, gRapid PVST+, EIGRP, OSPF, DTP, PAgP/LACP, UDLD, FlexLink, etc.

Cisco Catalyst integrated security features IBNS (802.1x), (CISF):

DistributionSiSi SiSi

features IBNS (802.1x), (CISF): port security, DHCP snooping, DAI, IPSG, etc.

Automatic phone discovery, conditional trust boundary Accessconditional trust boundary, power over Ethernet, auxiliary VLAN, etc.

Spanning tree toolkit: PortFast, UplinkFast BackboneFast LoopGuard

Access


UplinkFast, BackboneFast, LoopGuard, BPDU Guard, BPDU Filter, RootGuard, etc.

Distribution LayerDistribution Layer

Availability load balancingPolicy, Convergence, QoS, and High Availability

SiSiSiSi

Availability, load balancing, QoS and provisioning are the important considerations at this layer

Corey

Aggregates wiring closets (access layer) and uplinks to coreuplinks to core

Protects core from high density peering and problems in access layer


problems in access layer

Route summarization, fast convergence, redundant Accesspath load sharing

HSRP or GLBP to provide first hop redundancy

Access


p y

Core LayerCore Layer

Backbone for theScalability, High Availability, and Fast Convergence

SiSiSiSi

Backbone for thenetwork—connects network building blocks Core

Performance and stability vs. complexity—less is more in the core

Aggregation point for distribution layer


Separate core layer helps in scalabilityduring future growth Accessduring future growth

Keep the design technology-independent

Access


Do I Need a Core Layer?Do I Need a Core Layer?It’s Really a Question of Scale Complexity and ConvergenceNo Core Fully meshed distribution layers

Ph i l bli

Scale, Complexity, and Convergence

Physical cabling requirement

Routing complexity Second Building Block–4 New LinksBlock–4 New Links

4th Building Block12 New Links24 Links Total

3rd Building Block8 New Links24 Links Total

8 IGP Neighbors12 Links Total

5 IGP Neighbors


Do I Need a Core Layer?Do I Need a Core Layer?It’s Really a Question of Scale Complexity and ConvergenceDedicated Core Switches Easier to add a module Fewer links in the core

Scale, Complexity, and Convergence

2nd Building Block Easier bandwidth upgrade Routing protocol peering

reduced Equal cost Layer 3 links

2nd Building Block8 New Links

q yfor best convergence

4th Building Block4 New Links

16 Links Total3rd Building Block

4 New Links16 Links Total3 IGP Neighbors

12 Links Total3 IGP Neighbors


Design Alternatives Come Within a Building (or Distribution) BlockBuilding (or Distribution) Block

Layer 2 Access Routed Access Virtual Switching System

Access

SiSi SiSi SiSi SiSiDistribution

SiSi SiSiCore

SiSi SiSiSiSi SiSi

SiSi SiSiDistribution SiSi SDistribution

Access



Layer 3 Distribution InterconnectionLayer 3 Distribution Interconnection

Tune CEF load balancingLayer 2 Access—No VLANs Span Access Layer

Tune CEF load balancing Match CatOS/IOS EtherChannel

settings and tune load balancing Summarize routes towards core SiSi SiSi

Core

Limit redundant IGP peering STP Root and HSRP primary

tuning or GLBP to load balance Layer 3gon uplinks

Set trunk mode on/no-negotiate Disable EtherChannel

Point to

Point Link

Layer 3

SiSi SiSi

Distribution

unless needed Set port host on access

layer ports:Di bl T ki

VLAN 120 Voice

Disable TrunkingDisable EtherChannelEnable PortFast

RootGuard or BPDU-GuardVLAN 20 Data10.1.20.0/24

VLAN 140 Voice

VLAN 40 Data10.1.40.0/24

Access


VLAN 120 Voice10.1.120.0/24 Use security features

VLAN 140 Voice10.1.140.0/24


Tune CEF load balancingLayer 2 Access—Some VLANs Span Access Layer


settings and tune load balancing Summarize routes towards core

SiSi SiSi Core

Limit redundant IGP peering STP Root and HSRP primary or

GLBP and STP port cost tuning to load balance on uplinks

Layer 2

Distributionload balance on uplinks Set trunk mode on/no-negotiate Disable EtherChannel

unless needed

TrunkSiSi SiSiDistribution

RootGuard on downlinks LoopGuard on uplinks Set port host on access

Layer ports:Layer ports:Disable TrunkingDisable EtherChannelEnable PortFast

RootGuard or BPDU-GuardVLAN 120 Voice

10.1.120.0/24

VLAN 20 Data10.1.20.0/24

VLAN 140 Voice10.1.140.0/24

VLAN 40 Data10.1.40.0/24

Access


VLAN 250 WLAN10.1.250.0/24

RootGuard or BPDU-Guard Use security features

Routed Access and Virtual Switching SystemVirtual Switching SystemEvolutions of and Improvements to Existing Designs

SiSi SiSi CoreSiSi SiSi

DistributionVSS Link

P-t-P Link

Layer 3SiSi SiSiDistributionNew

Concept

VLAN 20 Data10.1.20.0/24

VLAN 120 Voice10.1.120.0/24

VLAN 20 Data10.1.20.0/24

VLAN 140 Voice10.1.140.0/24

VLAN 40 Data10.1.40.0/24

AccessVLAN 40 Data10.1.40.0/24VLAN 120 Voice10.1.120.0/24VLAN 140 Voice10 1 140 0/24


See RST-3035—Advanced Enterprise Campus Design Alternatives: Routed Access and Virtual Switching System (VSS)

10.1.140.0/24VLAN 250 WLAN10.1.250.0/24

Virtual Switch System (VSS)Virtual Switch System (VSS)Hub and Spoke VLANs can Span Access Layer


settings and tune load balancingSiSi SiSi Core

settings and tune load balancing Summarize routes towards core Set trunk mode on/nonegotiate Use PaGP and Multi-Chassis VSS LinkUse PaGP and Multi Chassis

EtherChannel RootGuard on downlink (MEC) LoopGuard on uplink (MEC)

New ConceptSiSi SiSi

Distribution

Set port host on access Layer ports:

– Disable trunkingDisable EtherChannelE bl P tF tEnable PortFast

RootGuard or BPDU-Guard on access ports

Use security features VLAN 120 Voice10 1 120 0/24

VLAN 20 Data10.1.20.0/24

VLAN 140 Voice10 1 140 0/24

VLAN 40 Data10.1.40.0/24

Access


VLAN 250 WLAN10.1.250.0/24

y 10.1.120.0/24 10.1.140.0/24

Virtual Switching System 1440Network System Virtualization

Core/Distribution Data Center Access

Network System Virtualization

SiSi SiSi SiSi SiSiSiSi SiSi SiSi SiSiSiSi SiSi SiSi SiSi

Features

Network System Virtualization

Benefits of VSSIncreased Operational Efficiency

i Si lifi d N t k

Inter-Chassis Stateful Switch Over (SSO)

via Simplified Network

Boost Non-stop Communication


( )Multi-Chassis EtherChannel

(MEC)Scale the System Bandwidth

Capacity to 1.4 Tbps

AgendaAgenda



Design Principles Foundation ServicesFoundation Services Campus Design






Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi



Foundation ServicesFoundation Services

Layer 1 physical things Layer 1 physical things

Layer 2 redundancy—spanning tree

Layer 3 routing protocols

Trunking protocols—(ISL/.1q)

Unidirectional link detection

Load balancinggEtherChannel link aggregation

CEF equal cost load balancing HSRP

First hop redundancy protocolsVRRP, HSRP, and GLBP

SpanningTreeRouting


Best Practices—Layer 1 Physical ThingsLayer 1 Physical Things

Use point to point Use point-to-point interconnections—no L2 aggregation points between nodes

Use fiber for best SiSi SiSi SiSi SiSi SiSi SiSi

convergence (debounce timer)

Tune carrier

Layer 3 Equal Cost Links


SiSiSiSi Tune carrier delay timer

Use configuration on SiSi SiSiSiSiSiSiUse configuration on the physical interface not VLAN/SVI when possible

SiSiSiSi


possibleData CenterWAN Internet

Redundancy and Protocol InteractionLi k N i hb F il D t tiLink Neighbour Failure Detection

Indirect link failures are harder Hellos Indirect link failures are harder to detect

With no direct HW notification of link SiSi

Hellos

loss or topology change convergence times are dependent on SW notification

Indirect failure events in a bridged

SiSi

SiSi

Hub

Indirect failure events in a bridged environment are detected by Spanning Tree Hellos

BPDU In certain topologies the need for TCN

updates or dummy multicast flooding (uplink fast) is necessary for

SiSi

BPDUs

(uplink fast) is necessary for convergence

You should not be using hubs in a high il bilit d i

SiSi

S

Hub


availability design SiSi

Redundancy and Protocol InteractionLi k R d d d F il D t tiLink Redundancy and Failure Detection

Direct point-to-point fiber provides for fast Cisco IOS Throttling: C i D l Ti Direct point-to-point fiber provides for fast

failure detection IEEE 802.3z and 802.3ae link negotiation

define the use of Remote Fault Indicator and

3 Carrier Delay Timer

define the use of Remote Fault Indicator and Link Fault Signaling mechanisms

Bit D13 in the Fast Link Pulse (FLP) can be set to indicate a physical fault to the

2 Linecard Throttling: Debounce Timer

be set to indicate a physical fault to the remote side

Do not disable auto-negotiation on GigE and 10GigE interfaces 110GigE interfaces

The default debounce timer on GigE and 10GigE fiber linecards is 10 msec

f

1

The minimum debounce for copper is 300 msec

Carrier-Delay

1

Remote IEEE SiSi SiSi


3560, 3750 and 4500—0 msec6500—leave it set at default

Fault Detection Mechanism

Si Si

Redundancy and Protocol InteractionL 2 d 3 Wh U R t d I t fLayer 2 and 3—Why Use Routed Interfaces

Configuring L3 routed interfaces provides for faster convergence Configuring L3 routed interfaces provides for faster convergence than an L2 switch port with an associated L3 SVI

L2L3

1. Link Down

SiSiSiSi

L2SiSiSiSi

L3

1. Link Down

2. Interface Down

3. Autostate

4 SVI Down

2. Interface Down

3. Routing Update~ 8 msec ~ 150-200

21:32:47.813 UTC: %LINEPROTO-5-UPDOWN: Line

4. SVI Down

5. Routing Update

21:38:37.042 UTC: %LINEPROTO-5-UPDOWN: Line

8 msec loss

150 200 msec loss

21:32:47.813 UTC: %LINEPROTO 5 UPDOWN: Line protocol on Interface GigabitEthernet2/1, changed state to down21:32:47.821 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet2/1, changed state to down21:32:48.069 UTC: %LINK-3-UPDOWN: Interface Vlan301,

protocol on Interface GigabitEthernet3/1, changed state to down21:38:37.050 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet3/1, changed state to down21:38:37.050 UTC: IP-EIGRP(Default-IP-Routing-


,changed state to down21:32:48.069 UTC: IP-EIGRP(Default-IP-Routing-Table:100): Callback: route, adjust Vlan301

Table:100): Callback: route_adjust GigabitEthernet3/1

Best Practices—Spanning Tree ConfigurationSpanning Tree Configuration

Only span VLAN across Same VLAN Same VLAN Same VLAN Only span VLAN across

multiple access layer switches when you have to!

Use Rapid PVST+ for bestLayer 2 Loops

Use Rapid PVST+ for best convergence

More common in the d

SiSi SiSi SiSi SiSi SiSi SiSi

data center Required to protect against

‘user side’ loopsLayer 3 Equal

Cost LinksLayer 3 Equal

Cost LinksSiSiSiSip

Required to protect against operational accidents (misconfiguration or SiSi SiSiSiSiSiSi(misconfiguration or hardware failure)

Take advantage of the spanning tree toolkit

SiSiSiSi


spanning tree toolkitData CenterWAN Internet

Multilayer Network DesignMultilayer Network DesignLayer 2 Access with Layer 3 Distribution

SiSi SiSi SiSi SiSi

Vlan 10 Vlan 20 Vlan 30 Vlan 30 Vlan 30 Vlan 30

Each access switch hasunique VLANs

At least some VLANs span multiple access switches

No layer 2 loops Layer 3 link between distribution No blocked links

Layer 2 loops Layer 2 and 3 running over

link between distribution


No blocked links link between distribution Blocked links

Optimizing L2 ConvergenceOptimizing L2 Convergence

Rapid-PVST+ greatly improves the restoration times for any VLAN thatPVST+, Rapid PVST+ or MST Rapid-PVST+ greatly improves the restoration times for any VLAN that

requires a topology convergence due to link UP

Rapid-PVST+ also greatly improves convergence time over backbone fast for any indirect link failures

35

for any indirect link failures

PVST+ (802.1d)Traditional spanning tree ec

)

25

30UpstreamDownstream

p gimplementation

Rapid PVST+ (802.1w)Scales to large size a

Flow

s (s

e15

20DownstreamScales to large size

(~10,000 logical ports)

Easy to implement, proven, scales

MST (802 1s) stor

e D

ata

5

10 MST (802.1s)

Permits very large scale STP implementations (~30,000 logical ports) Ti

me

to R

e


0PVST+ Rapid PVST+

Not as flexible as Rapid PVST+

T

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a00807b0670.shtml

Layer 2 HardeningLayer 2 HardeningSpanning Tree Should Behave the Way You Expect LoopGuard

Place the root where you want it

Root primary/secondary macroSTP Root

the Way You Expect LoopGuard

Root primary/secondary macro

The root bridge should stay where you put it

RootGuard

SiSiSiSi

RootGuardRootGuardLoopGuardUplinkFastUDLD

LoopGuard

UDLD

Only end-station traffic should be seen on an edge port

UplinkFastg p

BPDU GuardRootGuardPortFast

BPDU Guard or RootGuard

PortFast


Port-securityPortFast

Port Security

Best Practices—Layer 3 Routing ProtocolsLayer 3 Routing Protocols

Typically deployed in distributionTypically deployed in distribution to core, and core to core interconnections

Used to quickly re-route d f il d d /li k hilaround failed node/links while

providing load balancing over redundant paths

Build triangles not squares for


g qdeterministic convergence

Only peer on links that you intend to use as transit



SiSiSiSi

Insure redundant L3 paths to avoid black holes

Summarize distribution to core to limit EIGRP query diameter or SiSi SiSiSiSiSiSito limit EIGRP query diameter or OSPF LSA propagation

Tune CEF L3/L4 load balancing hash to achieve maximum tili ti f l t th

SiSiSiSi


utilization of equal cost paths(CEF polarization) Data CenterWAN Internet

Best Practice—Build Triangles Not SquaresBuild Triangles Not Squares

Triangles: Link/Box Failure Does NOT Squares: Link/Box Failure Requires

Deterministic vs. Non-DeterministicTriangles: Link/Box Failure Does NOT

Require Routing Protocol ConvergenceSquares: Link/Box Failure Requires

Routing Protocol Convergence

SiSi SiSi SiSi SiSi

SiSiSiSiSiSiSiSi

Model A Model B

SiSiSiSi

Layer 3 redundant equal cost links support fast convergence Hardware based—fast recovery to remaining path

Model A Model B


Convergence is extremely fast (dual equal-cost paths: no need for OSPF or EIGRP to recalculate a new path)

Best Practice—Passive Interfaces for IGPPassive Interfaces for IGPLimit OSPF and EIGRP Peering Through the Access Layer Limit unnecessary peering using

passive interface:F VLAN i i l t

RoutingUpdates

Distribution SiSiSiSi

Through the Access Layer

Four VLANs per wiring closet 12 adjacencies totalMemory and CPU requirements increase with no real benefitincrease with no real benefitCreates overhead for IGP

OSPF Example: EIGRP Example:

Access

Router(config)#routerospf 1Router(config-router)#passive-interfaceVlan 99

Router(config)#routereigrp 1Router(config-router)#passive-interfaceVlan 99

Router(config)#routerospf 1Router(config-router)#passive-interface default

Router(config)#routereigrp 1Router(config-router)#passive-interface default


Router(config-router)#no passive-interface Vlan 99

Router(config-router)#no passive-interface Vlan 99

Why You Want to Summarize at the Distributionat the Distribution

It is important to force No SummariesQ i G B d th C

Limit EIGRP Queries and OSPF LSA PropagationIt is important to force summarization at the distribution towards the core

For return path traffic an OSPF EIGRP t

Queries Go Beyond the CoreRest of Network

CoreOSPF or EIGRP re-route is required

By limiting the number of peers an EIGRP router must query or

SiSi SiSi

q ythe number of LSAs an OSPF peer must process we can optimize this re-route

EIGRP example:

Distribution

EIGRP example:

SiSiSiSiinterface Port-channel1description to Core#1ip address 10.122.0.34 255.255.255.252ip hello-interval eigrp 100 1ip hold-time eigrp 100 3ip summary-address eigrp 100

Access


10.1.2.0/2410.1.1.0/24

ip summary-address eigrp 100 10.1.0.0 255.255.0.0 5

Why You Want to Summarize at the Distributionat the Distribution

It is important to forceReduce the Complexity of IGP Convergence

SummariesSt Q i t th CIt is important to force

summarization at the distribution towards the core

For return path traffic an OSPF EIGRP t i i d

Rest of NetworkCore

Stop Queries at the Core

or EIGRP re-route is required By limiting the number of peers

an EIGRP router must query or the number of LSAs an OSPF

SiSi SiSi

|peer must process we can optimize his re-route

For EIGRP if we summarize at the distribution we stop queries

Summary:10.1.0.0/16 Distribution

the distribution we stop queries at the core boxes for an access layer ‘flap’

For OSPF when we summarize SiSiSiSi

at the distribution (area border or L1/L2 border) the flooding of LSAs is limited to the distribution switches; SPF now deals with

Access


;one LSA not three 10.1.2.0/2410.1.1.0/24

Best Practice—Summarize at the DistributionSummarize at the Distribution

Best practice—summarize atGotcha—Distribution-to-Distribution Link Required Best practice—summarize at

the distribution layer to limit EIGRP queries or OSPF LSA propagation SiSi SiSi

Corep p g

Gotcha:Upstream: HSRP on leftdistribution takes over when

Summary:10.1.0.0/16

SiSi SiSi

distribution takes over whenlink fails

Return path: old router still advertises summary to core

SiSiSiSi

Distribution

Return traffic is dropped on right distribution switch

Summarizing requires a link

SiSiSiSi

between the distribution switches

Alternative design: Use the access layer for transit 10.1.2.0/2410.1.1.0/24

Access


Use the access layer for transit

Provide Alternate PathsProvide Alternate Paths

What happens if fails? What happens if fails? No route to the core

anymore?Si l P th

SiSiSiSi Core

Allow the traffic to go through the access?

Do you want to use your access it h t it d ?

Single Pathto Core

switches as transit nodes?How do you design for scalability if the access used for transit traffic?

SiSiSiSiDistribution

Install a redundant link to the core

Best practice: install Best practice: install redundant link to core and utilize L3 link between distribution Layer A B

Access


bet ee d st but o aye(summarization—coming)

EIGRP Design Rules in the Campus L th T l P id dLeverage the Tools Provided

The greatest advantages of EIGRP areThe greatest advantages of EIGRP are gained when the network has a structured addressing plan that allows for use of summarization and stub routers when appropriate

10.10.0.0/16

routers when appropriate

EIGRP provides the ability to implement multiple tiers of summarization and route filtering SiSi SiSisummarization and route filtering

Minimize the number and time for query response to speed up convergence

10.10.0.0/17 10.10.128.0/17SiSi SiSi

convergence

Summarize distribution block routes upstream to the core

If routing in the access configureSiSiSiSi SiSiSiSi

If routing in the access configure all access switches as EIGRP stub routers

If routing in the access layer filter


If routing in the access layer filter routes sent down to access switches

OSPF Design Rules in the CampusWh A th A ?Where Are the Areas?

Area design based on addressArea 100 Area 110 Area 120

Area design based on address summarization

Area boundaries should define buffers between fault domains

Summarize routes from the distribution block upstream into the core

Minimize the number of LSAs and routes in the core


routes in the core Reduce the need for SPF calculations

due to internal distribution block changes SiSiSiSi

Area 0

ABR for a regular area forwards Summary LSAs (Type 3)ASBR summary (Type 4)Specific externals (Type 5)

SiSi SiSiSiSiSiSi

Specific externals (Type 5)

Stub area ABR forwardsSummary LSAs (Type 3)Summary default (0.0.0.0)

SiSiSiSi


A totally stubby area ABR forwardsSummary default (0.0.0.0)

Data CenterWAN Internet

Equal Cost Multi PathEqual Cost Multi-Path

Depending on the traffic flow patterns and IP SiSiSiSi

Optimizing CEF Load-SharingDepending on the traffic flow patterns and IP Addressing in use one algorithm may provide better load-sharing results than another

Be careful not to introduce polarization in a multi-

SiSiSiSi

30% of

70% ofBe careful not to introduce polarization in a multi

tier design by changing the default to the same thing in all tiers/layers of the network

SiSi

of Flows

of Flows

C t l t 4500 L d Sh i O ti

SiSiSiSiLoad-Sharing

SimpleOriginal Src IP + Dst IP

Universal* Src IP + Dst IP + Unique ID

Include P t

Src IP + Dst IP + (Src or Dst Port) + Unique ID

Catalyst 4500 Load-Sharing Options

Default* Src IP + Dst IP + Unique ID

Catalyst 6500 PFC3** Load-Sharing OptionsSiSiSiSi

Load-Sharing Full Simple

Port

SiSi

Load-Sharing Simple

Full Src IP + Dst IP + Src Port + Dst Port

Full Exclude Port Src IP + Dst IP + (Src or Dst Port)

Simple Src IP + Dst IP

Full Simple Src IP + Dst IP + Src Port + Dst Port


SiSiFull Simple Src IP Dst IP Src Port Dst Port

* = Default Load-Sharing Mode** = PFC3 in Sup720 and Sup32 Supervisors

CEF Load BalancingCEF Load Balancing

Redundant Paths IgnoredAvoid Underutilizing Redundant Layer 3 Paths

CEF polarization: without some tuning CEF will select the same path left/left or

Redundant Paths Ignored

pright/right

Imbalance/overloadcould occur


Default L3 Hashcould occur

Redundant paths are ignored/underutilized

L RCore g

The default CEF hash ‘input’ is L3

SiSi SiSi

LR

Default L3 Hash

We can change the default to use L3 + L4 information as ‘input’ to the hash derivation

SiSiSiSiRDistribution

Default L3 Hash


input to the hash derivation

CEF Load BalancingCEF Load Balancing

All Paths UsedAvoid Underutilizing Redundant Layer 3 Paths

The default will for Sup720/32 and latest hardware (unique ID added to

All Paths Used

( qdefault). However, depending on IP addressing, and flows imbalance could occurSiSi SiSi

DistributionL3/L4 Hash

Alternating L3/L4 hash and L3 hash will give us the best load balancing res lts

RLRLCore

balancing results

Use simple in the core and full simple in the distribution

SiSi SiSi

RL

Default L3 Hash

full simple in the distribution to add L4 information to the algorithm at the distribution and maintain differentiation

SiSiSiSi

RLDistributionL3/L4 Hash


and maintain differentiation tier-to-tier

Best Practices Trunk ConfigurationBest Practices—Trunk Configuration

Typically deployed onTypically deployed on interconnection between access and distribution layers

Use VTP transparent mode802.1q Trunks

Use VTP transparent mode to decrease potential for operational error

Hard set trunk mode to on and


encapsulation negotiate off for optimal convergence

Change the native VLAN to Layer 3 Equal


Cost LinksSiSiSiSi

something unused to avoid VLAN hopping

Manually prune all VLANS t th d d

SiSi SiSiSiSiSiSi

except those needed Disable on host ports:

CatOS: set port host

SiSiSiSi


Cisco IOS: switchport hostData CenterWAN Internet

VTP Virtual Trunk ProtocolVTP Virtual Trunk Protocol

Centralized VLAN Centralized VLAN management

VTP server switch F

Set VLAN 50

TrunkPass

Through Update

propagates VLAN database to VTPclient switches

FServer

Trunk Trunk

A Transparent

Ok, I Just L t

Runs only on trunks Four modes:

Trunk Trunk

Ok, I Just Learnt

Learnt VLAN 50!

Server: updates clientsand serversClient: receive updates—

ClientTrunk

BClientVLAN 50!

Dropcannot make changesTransparent: let updates pass through

Off C

Drop VTP

Updates


Off: ignores VTP updates Off

DTP Dynamic Trunk ProtocolDTP Dynamic Trunk Protocol

Automatic formation of Automatic formation of trunked switch-to-switch interconnection

O l b t k

On/OnTrunk

SiSi SiSi

On: always be a trunkDesirable: ask if the other side can/willAuto: if the other sides asks I will Auto/DesirableSiSi SiSi

Off: don’t become a trunk

Negotiation of 802.1Q or ISL encapsulation

Trunk

pISL: try to use ISL trunk encapsulation802.1q: try to use 802.1q encapsulation

Off/OffNO Trunk

SiSi SiSi

pNegotiate: negotiate ISL or 802.1q encapsulation with peerNon-negotiate: always use

SiSiSiSi


encapsulation that is hard set Off/On, Auto, DesirableNO Trunk

Optimizing Convergence: Trunk TuningOptimizing Convergence: Trunk Tuning

DTP negotiation tuning improves link up convergence timeTrunk Auto/Desirable Takes Some Time DTP negotiation tuning improves link up convergence time

CatOS> (enable) set trunk <port> nonegotiate dot1q <vlan>

IOS(config-if)# switchport mode trunk

2 5

IOS(config-if)# switchport nonegotiate

2

2.5

Seco

nds

1

1.5

onve

rge

in S

Two Seconds of Delay/Loss T d A

SiSi

0.5

Tim

e to

Co

Voice Data

Tuned Away


0Trunking Desirable Trunking Nonegotiate

Trunking/VTP/DTP Quick SummaryTrunking/VTP/DTP—Quick Summary

VTP Transparent should be used; there is a trade offVTP Transparent should be used; there is a trade off between administrative overhead and the temptation to span existing VLANS across multiple access layer switches

Emerging technologies that do VLAN assignment b (IBNS NAC t ) i i VLANby name (IBNS, NAC, etc.) require a unique VLAN database per access layer switch if the rule: A VLAN = A Subnet = AN access layer switch is going to be followed

One can consider a configuration that uses DTP ON/ON and NO NEGOTIATE; there is a trade off between performance/HA impact and maintenance and operations implicationsand operations implications

An ON/ON and NO NEGOTIATE configuration is faster from a link up (restoration) perspective than a desirable/desirable alternative. However, in this configuration DTP is not actively monitoring theconfiguration DTP is not actively monitoring the state of the trunk and a misconfigured trunk is not easily identified.

It’s really a balance between fast convergence


y gand your ability to manage configuration and change control …

Best Practices UDLD ConfigurationBest Practices—UDLD Configuration

Typically deployed Typically deployed on any fiberoptic interconnection

Use UDLD aggressive mode for best protection

Fiber Interconnections


Turn on in global configuration to avoid operational



Fiber Interconnections

SiSiSiSiavoid operational error/“misses”

Config example SiSi SiSiSiSiSiSiConfig exampleCisco IOS:udld aggressive

SiSiSiSi



Unidirectional Link DetectionUnidirectional Link Detection

Highly-available networks require UDLD toProtecting Against One Way Communication

Highly available networks require UDLD to protect against one-way communication or partially failed links and the effect that they could have on protocols like STP and RSTP

Primarily used on fiberoptic links where patch panel errors could cause link up/up with mismatched transmit/receive pairs

SiSi

Each switch port configured for UDLD will send UDLD protocol packets (at L2) containing the port’s own device/port ID, and the neighbor’s device/port IDs seen by UDLD on that port

Are You ‘Echoing’

M H ll ?device/port IDs seen by UDLD on that port

Neighboring ports should see their own device/port ID (echo) in the packets received from the other side

My Hellos?

If the port does not see its own device/port ID in the incoming UDLD packets for a specific duration of time, the link is considered


unidirectional and is shutdownSiSi

UDLD Aggressive and UDLD ‘Normal’UDLD Aggressive and UDLD Normal

SiSi SiSi

Timers are the same—15 second hellos by default

SiSi SiSi

Timers are the same 15 second hellos by default

Aggressive Mode—after aging on a previously bi-directional link—tries 8 times (once per second) to reestablish connection then err-disables port

UDLD—Normal Mode—Only err-disable the end where UDLD detected other end just sees the link go downdetected other end just sees the link go down

UDLD—Aggressive—err-disable BOTH ends of the connection due to err-disable when aging and re-establishment of UDLD


g gcommunication fails

Best Practices—EtherChannel ConfigurationEtherChannel Configuration

Typically deployed inTypically deployed in distribution to core, and core to core interconnections

Used to provide linkUsed to provide link redundancy—while reducing peering complexity

Tune L3/L4 load balancing


ghash to achieve maximum utilization of channel members

Deploy in powers of 2 (2, 4, or 8)Layer 3 Equal


Cost LinksSiSiSiSi

Match CatOS and Cisco IOS PAgP settings

802.3ad LACP for interop SiSi SiSiSiSiSiSipif you need it

Disable unless neededCatOS: set port host

SiSiSiSi


pCisco IOS: switchport host Data CenterWAN Internet

Understanding EtherChannelLi k N ti ti O ti PA P d LACPLink Negotiation Options—PAgP and LACP

Packet Aggregation Protocol Link Aggregation Protocol

On/OnChannel

SiSi SiSi

On/OnChannel

SiSiSiSi

On/OffNo Channel

SiSi SiSi

Channel

On/OffNo Channel

SiSi SiSi

Auto/DesirableChannel

SiSi SiSi

Active/PassiveChannel

SiSi SiSi

Off/On, Auto, Desirable

SiSiSiSi

Channel

Passive/PassiveN Ch l

SiSiSiSi

No Channel No Channel

On: always be a channel/bundle memberActive: ask if the other side can/willPassive: if the other side asks I will

On: always be a channel/bundle memberDesirable: ask if the other side can/willAuto: if the other side asks I will


Passive: if the other side asks I willOff: don’t become a member of a channel/bundle

Auto: if the other side asks I willOff: don’t become a member of a channel/bundle

PAgP TuningPAgP TuningPAgP Default Mismatches

Matching EtherChannel Configuration on Both Sides Improves Link Restoration Convergence Timesp gset port channel <mod/port> off

5

6

7

rge

in

As Much As

2

3

4

e to

Con

ver

Seco

nds

As Much As Seven Seconds of Delay/Loss Tuned Away

0

1

2

Tim

e

PAgP Mismatch PAgP Off


PAgP Mismatch PAgP Off

EtherChannels or Equal Cost MultipathEtherChannels or Equal Cost Multipath10/100/1000 How Do You Aggregate It?

SiSiSiSiCore

1010GE and GE and 1010GE channelsGE channels

T i lT i l 44 11

Distribution

Typical Typical 44::11Data OverData Over--

SubscriptionSubscription

SiSi SiSi

Distribution

Typical Typical 2020::11

AccessData OverData Over--SubscriptionSubscription


EtherChannels or Equal Cost MultipathEtherChannels or Equal Cost MultipathReduce Complexity/Peer Relationships

More links = more routing peer relationships and associated overhead

EtherChannels allow you to reduce peers by creating single logical interface to peer over


On single link failure in a bundleOSPF running on an IOS-based switch will reduce link cost and re-route traffic



SiSiSiSi will reduce link cost and re route traffic

OSPF running on a hybrid switch will not change link cost and may overload remaining links

SiSi SiSiSiSiSiSi

EIGRP may not change link cost and may overload remaining links

SiSiSiSi

SiSiSiSi


Data CenterWANWAN InternetInternet

EtherChannels or Equal Cost MultipathEtherChannels or Equal Cost MultipathWhy 10-Gigabit Interfaces

More links = more routing peer relationships and associated overhead

EtherChannels allow you to reduce peers by creating single logical interface to peer over


However, a single link failure is not taken into consideration by routing protocols. Overload



SiSiSiSi out g p otoco s O e oadpossible.

Single 10-Gigabit links address both problems IncreasedSiSi SiSiSiSiSiSi both problems. Increased bandwidth without increasing complexity or compromising routing protocols ability to select

SiSiSiSi


g p ybest path.Data CenterWANWAN InternetInternet

EtherChannels Quick SummaryEtherChannels—Quick Summary

For Layer-2 EtherChannels: Desirable/Desirable is the recommendedFor Layer 2 EtherChannels: Desirable/Desirable is the recommended configuration so that PAgP is running across all members of the bundle insuring that an individual link failure will not result in an STP failure

For Layer-3 EtherChannels: One can consider a configuration that usesFor Layer 3 EtherChannels: One can consider a configuration that uses ON/ON. There is a trade-off between performance/HA impact and maintenance and operations implications.

An ON/ON configuration is faster from a link-up (restoration) perspective g p ( ) p pthan a Desirable/Desirable alternative. However, in this configuration PAgP is not actively monitoring the state of the bundle members and a misconfigured bundle is not easily identified.

Routing protocols may not have visibility into the state of an individual member of a bundle. LACP and the minimum links option can be used to bring the entire bundle down when the capacity is diminished.

OSPF has visibility to member loss (best practices pending investigation) EIGRP does notOSPF has visibility to member loss (best practices pending investigation). EIGRP does not…

When used to increase bandwidth—no individual flow can go faster than the speed of an individual member of the linkB t d t li i t i l i t f f il (i li k t) d d i


Best used to eliminate single points of failure (i.e. link or port) dependencies from a topology

Best Practices First Hop RedundancyBest Practices—First Hop Redundancy

Used to provide a resilient Used to provide a resilient default gateway/first hop address to end-stations

HSRP VRRP and1st Hop Redundancy

HSRP, VRRP, and GLBP alternatives

VRRP, HSRP and GLBPid illi d ti


provide millisecond timersand excellent convergence performanceVRRP if d



SiSiSiSi

VRRP if you need multivendor interoperability

GLBP facilitates uplink SiSi SiSiSiSiSiSi

load balancing Preempt timers need

to be tuned to avoid

SiSiSiSi


black-holed traffic Data CenterWAN Internet

First Hop Redundancy with VRRPFirst Hop Redundancy with VRRP

A group of routers function R1—Master, Forwarding Traffic; R2,—BackupIETF Standard RFC 2338 (April 1998) A group of routers function

as one virtual router by sharing one virtual IP address and one

g pVRRP ACTIVE VRRP BACKUP

IP: 10.0.0.254MAC: 0000.0c12.3456vIP: 10.0.0.10

IP: 10.0.0.253MAC: 0000.0C78.9abcvIP:

virtual MAC address

One (master) router performs packet

vMAC: 0000.5e00.0101 vMAC:

R1 R2performs packet forwarding for local hosts

The rest of the routers act as “back up” in case

SiSiSiSi

Distribution-AVRRP Active

Distribution-BVRRP Backup

act as back up in case the master router fails

Backup routers stay idle f f

Access-a

as far as packet forwarding from the client side is concerned IP: 10.0.0.1

MAC 01IP: 10.0.0.2MAC 02

IP: 10.0.0.3MAC 03


MAC: aaaa.aaaa.aa01GW: 10.0.0.10ARP: 0000.5e00.0101



First Hop Redundancy with HSRPFirst Hop Redundancy with HSRP

A group of routers functionR1—Active, Forwarding Traffic;

R2—Hot Standby, IdleRFC 2281 (March 1998) A group of routers function

as one virtual router by sharing one virtual IP address and one

HSRP ACTIVE HSRP STANDBYIP: 10.0.0.254MAC: 0000.0c12.3456vIP: 10.0.0.10

IP: 10.0.0.253MAC: 0000.0C78.9abcvIP:

virtual MAC address

One (active) router performs packet

R1

vMAC: 0000.0c07.ac00 vMAC:

R2performs packet forwarding for local hosts

The rest of the routers provide “hot standby” in

SiSiSiSi

Distribution-AHSRP Active

Distribution-BHSRP Backup

provide hot standby in case the active router fails

Standby routers stay idle f f

Access-a

as far as packet forwarding from the client side is concerned IP: 10.0.0.1

MAC 01IP: 10.0.0.2MAC 02

IP: 10.0.0.3MAC 03


MAC: aaaa.aaaa.aa01GW: 10.0.0.10ARP: 0000.0c07.ac00



Why You Want HSRP PreemptionWhy You Want HSRP Preemption

Spanning Tree Root and Spanning Tree Root and HSRP Primary aligned

When Spanning Tree Root is re introduced traffic will

SiSiSiSi Core

is re-introduced, traffic will take a two-hop path to HSRP Active

SiSiSiSiDistribution

Spanning Tree Root

HSRPActive

HSRPActive Spanning Tree

RootHSRP Preempt

HSRP Preemption will allow HSRP to follow Spanning Tree topology Access

Without Preempt Delay HSRP Can Go Active Before Box Completely Ready to


Without Preempt Delay HSRP Can Go Active Before Box Completely Ready to Forward Traffic: L1 (Boards), L2 (STP), L3 (IGP Convergence)

standby 1 preempt delay minimum 180

First Hop Redundancy with GLBPFirst Hop Redundancy with GLBP

All the benefits of HSRP R1- AVG; R1, R2 Both Forward TrafficCisco Designed, Load Sharing, Patent Pending All the benefits of HSRP

plus load balancing of default gateway utilizes all available bandwidth

GLBP AVG/AVF, SVF GLBP AVF, SVFR1 AVG; R1, R2 Both Forward Traffic

IP: 10.0.0.254MAC: 0000.0c12.3456vIP: 10 0 0 10

IP: 10.0.0.253MAC: 0000.0C78.9abcvIP: 10.0.0.10

A group of routers function as one virtual router by sharing one virtual IP

vIP: 10.0.0.10vMAC: 0007.b400.0101

vIP: 10.0.0.10vMAC: 0007.b400.0102

R1sharing one virtual IP address but using multiple virtual MAC addresses for traffic forwarding

SiSiSiSi

Distribution-AGLBP AVG/

Distribution-BGLPB AVF, SVFg

Allows traffic from a single common subnet to go through multiple redundant

Access-aAVF, SVF

through multiple redundant gateways using a single virtual IP address

IP: 10.0.0.1MAC 01

IP: 10.0.0.2MAC 02

IP: 10.0.0.3MAC 03


MAC: aaaa.aaaa.aa01GW: 10.0.0.10ARP: 0007.B400.0101



First Hop Redundancy with Load BalancingLoad Balancing

Each member of a GLBP redundancy group owns a unique virtual MAC addressCisco Gateway Load Balancing Protocol (GLBP)

Each member of a GLBP redundancy group owns a unique virtual MAC address for a common IP address/default gateway

When end-stations ARP for the common IP address/default gateway they are given a load balanced virtual MAC address

Host A and host B send traffic to different GLBP peers but have the same default gateway

2

vIP10.88.1.10

GLBP 1 ip 10.88.1.10vMAC 0000.0000.0001

GLBP 1 ip 10.88.1.10vMAC 0000.0000.0002R1 R2

10.88.1.0/24.1 .2ARP

Reply

.5.4

ARPs for 10.88.1.10 ARPs for 10.88.1.10A B


ARPs for 10.88.1.10Gets MAC 0000.0000.0001 Gets MAC 0000.0000.0002

Optimizing Convergence: VRRP HSRP GLBPVRRP, HSRP, GLBP

VRRP not tested with sub-second timers and all flows go throughMean, Max, and Min—Are There Differences?

SiSiSiSi

VRRP not tested with sub second timers and all flows go through a common VRRP peer; mean, max, and min are equal

HSRP has sub-second timers; however all flows go through same HSRP peer so there is no difference between mean, max, and minp , ,

GLBP has sub-second timers and distributes the load amongstthe GLBP peers; so 50% of the clients are not affected by anuplink failure

e

Distribution to Access Link FailureAccess to Server Farm

0.81

1.2

to C

onve

rge

VRRP HSRP GLBP50% of Flows Have ZERO

Loss W/ GLBPGLBP Is 50%

Better

0.20.40.6

in S

econ

ds


0Longest Shortest AverageTi

me

If You Span VLANS Tuning RequiredIf You Span VLANS, Tuning Required

Both distribution switches act as default gatewayBy Default, Half the Traffic Will Take a Two-Hop L2 Path Both distribution switches act as default gateway Blocked uplink caused traffic to take less than optimal path

CoreDistribution-AGLBP Virtual MAC 1

Distribution-BGLBP Virtual

MAC 2

CoreLayer 3

SiSiSiSi

MAC 1 MAC 2DistributionLayer 2/3

F: ForwardingAccessAccess F: ForwardingB: Blocking

Access-bAccess-a

AccessLayer 2AccessLayer 2


VLAN 2VLAN 2

AgendaAgenda



Design principles Foundation Services Foundation Services Campus Design






Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi



Daisy Chaining Access Layer SwitchesDaisy Chaining Access Layer Switches

Return Path Traffic Has a 50/50 Chance of Being ‘Black Holed’Avoid Potential Black Holes

Return Path Traffic Has a 50/50 Chance of Being Black Holed

Core

50% Chance That Traffic Will Go Down Path with

No Connectivity

SiSiSiSiCore

Layer 3

Distribution-A Distribution-B

Layer 3 LinkNo Connectivity

SiSiSiSi

DistributionLayer 2/3

Access-cAccess-a Access-n

AccessLayer 2AccessLayer 2


VLAN 2VLAN 2 VLAN 2

Daisy Chaining Access Layer SwitchesDaisy Chaining Access Layer Switches

Stackwise/Stackwise-Plus technology eliminates the concernNew Technology Addresses Old Problems Stackwise/Stackwise-Plus technology eliminates the concern

Loopback links not required

No longer forced to have L2 link in distribution

If you use modular (chassis-based) switches, these problems are not a concern

HSRP ActiveForwarding

SiSi

Layer 3

HSRP Standby

Forwarding

Catalyst 3750-E SiSi


or Catalyst 2975

What Happens if You Don’t Link the Distributions?Link the Distributions?

STPs slow convergence canSTPs slow convergence can cause considerable periods of traffic loss

STP could cause

STP Secondary Root and HSRP

StandbyCore

STP could cause non-deterministic traffic flows/link load engineering

STP convergence will SiSiSiSiHellos

STP Root and HSRP Active

gcause Layer 3 convergence

STP and Layer 3 timers are independent B

2F 2

VLAN 2VLAN 2

Unexpected Layer 3 convergence and re-convergence could occur

Even if you do link the distribution

2

Access-bAccess-a

VLAN 2yswitches dependence on STP and link state/connectivity can cause HSRP irregularities and unexpected state transitions

Traffic Dropped Until

MaxAge Expires Then

Traffic Dropped Until Transition to Forwarding;


unexpected state transitions Expires Then Listening and

Learning

Forwarding; As much as 50

Seconds

What if You Don’t?

Aggressive HSRP

What if You Don t?Black Holes and Multiple ‘Transitions’ …

Aggressive HSRP timers limit blackhole #1

Backbone fast limits time (30 seconds)

STP Root andHSRP Active

STP Secondary Root and

HSRP Standby

CoreCoreLayer 3

time (30 seconds) to event #2

Even with Rapid PVST+ at least one second

HSRP Active (Temporarily)

SiSiSiSi

HellosDistribution

Layer 2/3

before event #2

MaxAgeF: ForwardingAccessAccess MaxAge

Seconds Before Failure Is Detected…Then Listening and Learning

B: Blocking

Access-bAccess-a

Layer 2Layer 2

VLAN 2VLAN 2

Blocking link on access-b will take 50 seconds to move to forwarding traffic black hole until HSRP goes active on standby HSRP peer

and Learning


After MaxAge expires (or backbone fast or Rapid PVST+) converges HSRP preempt causes another transition

Access-b used as transit for access-a’s traffic

What if You Don’t?

802 1d: up to

What if You Don t?Return Path Traffic Black Holed …

802.1d: up to 50 seconds

PVST+: backbone fast 30 seconds

Core

STP Root andHSRP Active

STP Secondary Root and

HSRP Standby

CoreLayer 3

Rapid PVST+:address by the protocol (one second)

HellosSiSiSiSi

DistributionLayer 2/3

second)

F: ForwardingAccessAccess gB: Blocking

Access-b

Layer 2Layer 2

Access-a

VLAN 2VLAN 2

Blocking link on access-b will take 50 seconds to move to forwarding


Blocking link on access b will take 50 seconds to move to forwarding return traffic black hole until then

Asymmetric Routing (Unicast Flooding)Asymmetric Routing (Unicast Flooding)

Affects redundant Affects redundant topologies with shared L2 access

One path upstream Asymmetric One path upstream and two paths downstream

AsymmetricEqual CostReturn Path

CAM Timer Has CAM table entry ages out on standby HSRP

Downstream

Upstream PacketUnicast to

Active HSRP

CAM Timer Has Aged out on

Standby HSRP SiSi SiSi

Without a CAM entry packet is flooded to all ports

DownstreamPacket

Flooded

VLAN 2VLAN 2

in the VLAN

VLAN 2 VLAN 2


VLAN 2VLAN 2 VLAN 2 VLAN 2

Best Practices Prevent Unicast FloodingBest Practices Prevent Unicast Flooding

Assign one unique Assign one unique data and voice VLAN to each access switch

Traffic is now only Asymmetric Traffic is now only flooded downone trunkA it h

AsymmetricEqual CostReturn Path

Access switch unicasts correctly;no flooding toall ports

DownstreamPacket

Flooded on

Upstream PacketUnicast to

Active HSRPSiSi SiSi

all ports If you have to:

Tune ARP and CAM aging timers; CAM

Flooded on Single Port

VLAN 2

aging timers; CAM timer exceeds ARP timerBias routing metrics t l VLAN 3 VLAN 4 VLAN 5


VLAN 2to remove equal cost routes

VLAN 3 VLAN 4 VLAN 5

AgendaAgenda



Design Principles Foundation Services Foundation Services Campus Design






Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi



Building a Converged Campus NetworkBuilding a Converged Campus Network

Access layerInfrastructure Integration, QoS, and Availability

Access layerAuto phone detectionInline powerQ S h d li

Access

QoS: scheduling, trust boundary and classificationFast convergence

Di t ib ti l


Distribution

Distribution layerHigh availability, redundancy, fast convergence

Layer 3 Equal Cost

Layer 3 Equal Cost

SiSi

CorePolicy enforcementQoS: scheduling, trust boundary and classification

LinksLinks SiSiSiSi

SiSi SiSiSiSiSiSiDistribution Core

High availability, redundancy, fast convergence

SiSiSiSi

SiSi SiSiSiSiSiSi

A

Distribution


QoS: scheduling, trust boundary Data CenterWAN Internet

Access

Infrastructure IntegrationInfrastructure Integration Extending the Network Edge

Switch Detects IP Phone and Applies Power

CDP Transaction Between Phone and Switch

IP Phone Placed in Proper VLAN

DHCP Req est and Call Manager Registration

Phone contains a three-port switch that is configured in conjunction

DHCP Request and Call Manager Registration

Phone contains a three port switch that is configured in conjunction with the access switch and CallManager

1. Power negotiation

2. VLAN configuration

3. 802.1x interoperation

4 QoS configuration


4. QoS configuration

5. DHCP and CallManager registration

Infrastructure Integration: First StepInfrastructure Integration: First Step

Cisco pre standard devices initially receive 6 3 wattsPower Requirement Negotiation Cisco pre-standard devices initially receive 6.3 watts

and then optionally negotiate via CDP

802 3af devices initially receive 12 95 watts unless PSE 802.3af devices initially receive 12.95 watts unless PSE able to detect specific PD power classification

Class Usage Minimum Power Levels Output at the PSE

Maximum Power Levels at the Powered Device

0 Default 15.4W 0.44 to 12.95W

1 Optional 4.0W 0.44 to 3.84W

2 Optional 7.0W 3.84 to 6.49W

3 Optional 15.4W 6.49 to 12.95W

4Reserved for

Future Treat as Class 0Reserved for Future Use: a Class 4 Signature Cannot Be Provided by a


Use Compliant Powered Device

Enhanced Power NegotiationEnhanced Power Negotiation

PD Plugged in

802.3af Plus Bi-Directional CDP (Cisco 7970)PD Plugged in

Switch Detects IEEE PD

PD I Cl ifi d

PSE—Power Source EquipmentCisco 6500,4500,

3750 3560 PD Is Classified

Power Is Applied

3750, 3560

Phone Transmits a CDP Power NegotiationPacket Listing Its Power Mode

PD—Powered Switch Sends a CDP Response

with a Power Request

Based on Capabilities Exchanged

Device Cisco 7970

Using bi-directional CDP exchange exact power

p gFinal Power Allocation Is Determined


Using bi directional CDP exchange exact power requirements are negotiated after initial power-on

Design Considerations for PoEDesign Considerations for PoE

Switch manages power by what is allocated not by what is currently usedPower Management Switch manages power by what is allocated not by what is currently used

Device power consumption is not constant

A 7960G requires 7W when the phone is ringing at maximum volume and q p g grequires 5W on or off hook

Understand the power behavior of your PoE devices

Utilize static power configuration with caution

Dynamic allocation: power inline auto max 7200power inline auto max 7200

Static allocation: power inline static max 7200

Use power calculator to determine power requirementshttp://www.cisco.com/go/powercalculator


Discover Cisco Enhanced PoE at the World of Solutions

Infrastructure Integration: Next StepsInfrastructure Integration: Next Steps VLAN, QoS and 802.1x Configuration

PC VLAN = 10(PVID)

Phone VLAN = 110(VVID)

Native VLAN (PVID) No Configuration Changes

Needed on PC

802.1Q encapsulation with 802.1p Layer 2 CoS

During initial CDP exchange phone is configured with a Voice VLAN ID (VVID)

Phone also supplied with QoS configuration via CDP TLV fields

Additionally switch port currently bypasses 802.1x authentication


for VVID if detects Cisco phone

AgendaAgenda









Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi



Best Practices Quality of ServiceBest Practices—Quality of Service

Must be deployed end-to- Must be deployed end-to-end to be effective; all layers play different but equal roles

Ensure that mission criticalEnd to End QoS

Ensure that mission critical applications are not impacted by link or transmit queue congestion


queue congestion

Aggregation and rate transition points must enforce QoS policies



SiSiSiSi

enforce QoS policies

Multiple queues with configurable admission SiSi SiSiSiSiSiSi

criteria and schedulingare required

SiSiSiSi



Transmit Queue CongestionTransmit Queue Congestion128k Uplink10/100m Queued

WANR

p10/100m Queued

Router

100 Meg in 128 Kb/S out—Packets Serialize in Faster than They Serialize outP k t Q d Th W it t S i li t Sl Li k

100 Meg Link1 Gig Link Queued

Packets Queued as They Wait to Serialize out Slower Link

100 Meg Link1 Gig Link Queued

Access SwitchDistribution Switch

1 Gig In 100 Meg out—Packets Serialize in Faster than They Serialize outQ S S


Packets Queued as They Wait to Serialize out Slower Link

Auto QoS VoIP Making It EasyConfigures QoS for VoIP on Campus SwitchesAuto QoS VoIP—Making It Easy …

Access-Switch(config-if)#auto qos voip ?cisco-phone Trust the QoS marking of Cisco IP Phonecisco phone Trust the QoS marking of Cisco IP Phonecisco-softphone Trust the QoS marking of Cisco IP SoftPhonetrust Trust the DSCP/CoS marking

Access-Switch(config-if)#autoqosvoipcisco-phone( g ) q p pAccess-Switch(config-if)#exit

!interface FastEthernet1/0/21srr-queue bandwidth share 10 10 60 20srr-queue bandwidth shape 10 0 0 0 Mls qos trust device cisco-phoneMls qos trust cos


auto qosvoipcisco-phone end

AgendaAgenda









Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi



Best Practices Campus SecurityBest Practices—Campus Security

New stuff that we will cover! e stu t at e co eCatalyst Integrated Security Feature Set!

Dynamic Port Security, DHCP Snooping, Dynamic ARP Inspection, IP Source Guard

Things you already know—

End-to-End Security

we won’t cover…Use SSH to access devices instead of Telnet Enable AAA and roles-based access control (RADIUS/TACACS+) for the CLI on all devices Enable SYSLOG to a server Collect and


Enable SYSLOG to a server. Collect and archive logsWhen using SNMP use SNMPv3 Disable unused services:

no service tcp-small-servers i d ll

SiSiSiSi

no service udp-small-servers Use FTP or SFTP (SSH FTP) to move images and configurations around—avoid TFTP when possible Install VTY access-lists to limit which addresses

t d CLI iSiSiSiSi

SiSi SiSiSiSiSiSi

can access management and CLI services Enable control plane protocol authentication where it is available (EIGRP, OSPF, BGP, HSRP, VTP, etc.)Apply basic protections offered by implementing RFC2827 filt i t l d i b d WAN Internet


RFC2827 filtering on external edge inbound interfaces

WAN Internet

For More Details, See BRKSEC-2002 Session, Understanding and Preventing Layer 2 Attacks

BPDU GuardBPDU Guard

Problem:Prevent Loops via WLAN (Windows XP Bridging) Problem:

WLAN APs do notforward BPDUs

STP LoopFormedforward BPDUs

Multiple Windows XPmachines can create al i th i d VLAN

BPDU GuardDisables Port

loop in the wired VLANvia the WLAN

Solution: BPDU Solution:

BPDU Guard configuredon all end-station switch

Generated

on all end station switch ports will prevent loopfrom forming Win XP

BridgingEnabled

Win XPBridgingEnabled

BPDUDiscarded


EnabledEnabled

Problem: Prevalence of Rogue APsProblem: Prevalence of Rogue APsExample: 59 APs in Seven Miles in SJ Commute

The majority of WLAN deployments are unauthorized by well intended employees (rogueby well intended employees (rogue APs)—many are insecure

A daily drive to work taken within the car at normal speeds with Insecurecar at normal speeds with an IPAQ running a freeware application (mix of residences and enterprises)

APs

Insecure enterprise rogueAP’s are a result of:

– Well intentioned staff install dueto absence of sanctioned WLAN deploymentto absence of sanctioned WLAN deployment

– An infrastructure that is not “wireless ready” to protectagainst rogue AP’s

59 APs Found


War Chalking

Basic 802 1x Access ControlBasic 802.1x Access ControlControlling When and Where APs Are Connected

Who Are You?

I Am Joe Cisco

CatOS Configuration Exampleset dot1x system-auth-control enableset dot1x guest-vlan 250set radius server 10.1.125.1 auth-port

802.1x Enabled on “user” Facing

PortsAuthorized

User

1812 primaryset radius key cisco123set port dot1x 3/1-48 port-control autoo ts

Who Are You? Cisco IOS Configuration Exampleradius-server host 10.1.125.1radius-server key cisco123

No 802.1xH Rogue AP

D on Authorized WLAN AP Ports

aaa new-modelaaa authentication dot1x default group radiusaaa authorization default group radius

DisabledHere

A h i d AP

WLAN AP Ports aaa authorization config-commandsdot1x system-auth-control

Cisco IOS Per-Port configuration


Authorized AP int range fa3/1 - 48dot1x port-control auto

Securing Layer 2 from Surveillance Attacks

00 0 00 Only 3 MAC

Cutting off MAC-Based AttacksSurveillance Attacks

00:0e:00:aa:aa:aa00:0e:00:bb:bb:bb

Only 3 MAC Addresses Allowed on

the Port: Shutdown250,000 ShutdownBogus MACs

per Second

SOLUTION:Port Security Limits MAC Flooding

Attack and Locks down Port and Sends an SNMP Trap

“Script Kiddie” Hacking Tools Enable Attackers Flood Switch CAM Tables

PROBLEM:SOLUTION:

Sends an SNMP TrapAttackers Flood Switch CAM Tables with Bogus Macs; Turning the VLAN into a “Hub” and Eliminating Privacy

Switch CAM Table Limit Is Finite

switchport port-security switchport port-security maximum 10 switchport port-security violation restrict


Switch CAM Table Limit Is Finite Number of Mac Addresses

switchport port-security aging time 2 switchport port-security aging type inactivity

DHCP SnoopingDHCP SnoopingProtection Against Rogue/Malicious DHCP Server

1

DHCP Server

1000s of DHCP R t tRequests to Overrun the DHCP Server

2

DHCP requests (discover) and responses (offer) tracked

R t li it t t t d i t f li it D S tt k Rate-limit requests on trusted interfaces; limits DoS attacks on DHCP server

Deny responses (offers) on non trusted interfaces; stop malicious


Deny responses (offers) on non trusted interfaces; stop malicious or errant DHCP server

Securing Layer 2 from Surveillance AttacksSurveillance Attacks

Dynamic ARP inspectionProtection Against ARP Poisoning Dynamic ARP inspection

protects against ARP poisoning (ettercap, dsnif arpspoof)

SiSiGateway = 10.1.1.1

MAC=Adsnif, arpspoof)

Uses the DHCP snooping binding table

G i ARPg

Tracks MAC to IP from DHCP transactions

Gratuitous ARP

Gratuitous ARP10.1.1.50=MAC_B

Rate-limits ARP requests from client ports; stop port scanning

Gratuitous ARP10.1.1.1=MAC_B

scanning

Drop BOGUS gratuitous ARPs; stop ARP

i i /MIM tt k Att k 10 1 1 25 Vi ti 10 1 1 50


poisoning/MIM attacks Attacker = 10.1.1.25MAC=B

Victim = 10.1.1.50MAC=C

IP Source GuardIP Source Guard

IP source guardProtection Against Spoofed IP Addresses IP source guard

protects against spoofed IP addresses

SiSiGateway = 10.1.1.1

MAC=Ap

Uses the DHCP snooping binding tablep g g

Tracks IP address to port associationsport associations

Dynamically programs port ACL to drop traffic

Hey, I’m 10.1.1.50 !

port ACL to drop traffic not originating from IP address assigned i DHCP Att k 10 1 1 25 Vi ti 10 1 1 50


via DHCP Attacker = 10.1.1.25 Victim = 10.1.1.50

Catalyst Integrated Security FeaturesCatalyst Integrated Security Features

ipdhcp snoopingIP S G d

Summary Cisco IOSipdhcp snooping

ipdhcp snooping vlan 2-10

iparp inspection vlan 2-10

!

IP Source Guard

Dynamic ARP Inspection

interface fa3/1

switchport port-security

switchport port-security max 3

DHCP Snooping

Port Security

Port security prevents MAC flooding attacks

switchport port-security violation restrict

switchport port-security aging time 2

switchport port security aging typeg DHCP snooping prevents client

attack on the switch and server Dynamic ARP Inspection adds

switchport port-security aging type inactivity

iparp inspection limit rate 100

ipdhcp snooping limit rate 100y psecurity to ARP using DHCP snooping table

IP source guard adds security t IP dd i

ip verify source vlandhcp-snooping

!

Interface gigabit1/1


to IP source address using DHCP snooping table

ipdhcp snooping trust

iparp inspection trust

AgendaAgenda



Design principles Foundation ServicesFoundation Services Campus Design



QoS Considerations Security Considerations

SiSiSiSiSiSi SiSi

Putting It All Together Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi


Distribution Blocks

Hierarchical CampusHierarchical Campus

Access

DistributionSiSi SiSi SiSi SiSi SiSi SiSi

CoreSiSi SiSi

DistributionSiSi SiSi SiSi SiSi

Access

SiSi SiSi




Tune CEF load balancingLayer 2 Access—No VLANs Span Access Layer


settings and tune load balancing Summarize routes towards core SiSi SiSi

Core

Limit redundant IGP peering STP Root and HSRP primary

tuning or GLBP to load balance li k Layer 3on uplinks

Set trunk mode on/nonegotiate Disable EtherChannel

unless needed

Point to

Point Link

Layer 3

SiSi SiSi

Distribution

unless needed Set port host on access

layer ports:Disable Trunking

VLAN 120 Voice

sab e u gDisable EtherChannelEnable PortFast

RootGuard or BPDU-GuardUse security features

VLAN 20 Data10.1.20.0/24

VLAN 140 Voice

VLAN 40 Data10.1.40.0/24

Access


VLAN 120 Voice10.1.120.0/24

Use security features VLAN 140 Voice10.1.140.0/24


Tune CEF load balancingLayer 2 Access—Some VLANs Span Access Layer


settings and tune load balancing Summarize routes towards core

SiSi SiSi Core

Limit redundant IGP peering STP Root and HSRP primary or

GLBP and STP port cost tuning to load balance on uplinks

Layer 2

Distributionto load balance on uplinks Set trunk mode on/nonegotiate Disable EtherChannel

unless needed

TrunkSiSi SiSiDistribution

RootGuard on downlinks LoopGuard on uplinks Set port host on access

Layer ports:Layer ports:Disable TrunkingDisable EtherChannelEnable PortFast

RootGuard or BPDU-GuardVLAN 120 Voice

10.1.120.0/24

VLAN 20 Data10.1.20.0/24

VLAN 140 Voice10.1.140.0/24

VLAN 40 Data10.1.40.0/24

Access


VLAN 250 WLAN10.1.250.0/24

RootGuard or BPDU-Guard Use security features

Routed Access and Virtual Switching SystemVirtual Switching SystemEvolutions of and Improvements to Existing Designs

SiSi SiSi CoreSiSi SiSi

DistributionVSS Link

P-t-P Link

Layer 3SiSi SiSiDistributionNew

Concept

VLAN 20 Data10.1.20.0/24

VLAN 120 Voice10.1.120.0/24

VLAN 20 Data10.1.20.0/24

VLAN 140 Voice10.1.140.0/24

VLAN 40 Data10.1.40.0/24

AccessVLAN 40 Data10.1.40.0/24VLAN 120 Voice10.1.120.0/24VLAN 140 Voice10 1 140 0/24


See RST-3035—Advanced Enterprise Campus Design Alternatives: Routed Access and Virtual Switch System (VSS)

10.1.140.0/24VLAN 250 WLAN10.1.250.0/24

High Availability Campus DesignSimplified with VSSSimplified with VSS

Access

SiSi SiSi SiSiSiSi SiSi SiSi SiSi SiSi SiSi

Distribution SiSi SiSi SiSi SiSi SiSi SiSi

SiSiCore SiSi SiSiSiSi SiSi

SiSi

SiSi

SiSi

SiSi SiSiSiSi SiSi

SiSi SiSiDistribution SiSi SiSiSiSi SiSi


Access


Data CenterWAN InternetData CenterWAN InternetAccess Data CenterWAN Internet

SmartPorts—Predefined Configurations

Access-Switch#show parser macro briefdefault global : cisco-globaldefault interface: cisco-desktoppdefault interface: cisco-phonedefault interface: cisco-switchdefault interface: cisco-routerdefault interface: cisco-wireless

SiSiSiSi

Access-Switch(config-if)#$ macro apply cisco-phone $access_vlan 100 $voice_vlan 10

Access-Switch#show run int fa1/0/19!

SiSi SiSi

!interface FastEthernet1/0/19switchport access vlan 100switchport mode accessswitchport voice vlan 10switchport port-security maximum 2switchport port-securityswitchport port-security aging time 2switchport port-security violation restrictswitchport port-security aging type inactivityswitchport port security aging type inactivitysrr-queue bandwidth share 10 10 60 20srr-queue bandwidth shape 10 0 0 0 mls qos trust device cisco-phonemls qos trust cosmacro description cisco phone


macro description cisco-phoneauto qosvoipcisco-phone spanning-tree portfastspanning-tree bpduguard enableend

AgendaAgenda



Design principles Foundation ServicesFoundation Services Campus Design



QoS Considerations Security Considerations

SiSiSiSiSiSi SiSi

Putting It All Together Summary

SiSi

Distribution Blocks

SiSi SiSi SiSi


Distribution Blocks

SummarySummary

Off hi h h lAccess

Offers hierarchy—each layer has specific role

Modular topology—building blocks

DistributionSiSi SiSi SiSi SiSi SiSi SiSi Easy to grow, understand,

and troubleshoot Creates small fault domains—

Clear demarcations and

Layer 3 Equal Cost

Links

Layer 3 Equal Cost

LinksCore

SiSi SiSi

Clear demarcations and isolation

Promotes load balancing and redundancy

s

DistributionSiSi SiSi SiSi SiSi

Promotes deterministic traffic patterns

Incorporates balance of both Layer 2 and Layer 3

Access

SiSi SiSitechnology, leveraging the strength of both

Utilizes Layer 3 Routing for load balancing, fast


Data CenterWAN InternetAccessg,

convergence, scalability, and control

Q and A


Hierarchical Network DesignHierarchical Network DesignWithout a Rock Solid Foundation the Rest Doesn’t Matter

Access

Distribution SiSi SiSi

Core

HSRPDistribution

SiSi SiSi

SpanningTreeRouting

DistributionSiSi SiSi


TreegAccess

Building Block

Reference MaterialsReference Materials

High Availability Campus Design Guidehttp://www.cisco.com/go/srnd High Availability Campus Design Guide

High Availability Campus Convergence Analysis

High Availability Campus Design Guide—Routed Access EIGRP and OSPF



Optimal RedundancyOptimal RedundancyWhen Is More Less?

Core and distribution engineered with redundant nodes

Access

redundant nodesand links toprovide maximum redundancy and optimal convergence


optimal convergence Network bandwidth

and capacity engineered to

ith t d d

CoreRedundan

tNodesSiSi SiSi

withstand nodeor link failure

120–200ms to converge around


converge around most events

AccessData CenterWAN Internet



Single Points of TerminationSingle Points of TerminationSSO/NSF Avoiding Total Network Outage

AccessL2 = SSO

L3 = SSO/NSF


CoreSiSi SiSi

The access layer is candidate for supervisor redundancy L2 access layer SSO L3 access layer SSO and NSF Network outage until physical replacement or reload vs.


one to three seconds

Supervisor Processor RedundancySupervisor Processor Redundancy

Active/standby supervisorsStateful Switch Over (SSO)

Active/standby supervisors run in synchronized mode

Redundant supervisor is in ‘hot-standby’ mode

SP RP PFCstandby mode

Switch processors synchronize L2 port state information, (e.g., STP, 802.1x, 802.1q)

Active Supervisor

( g , , , q) PFCs synchronize L2/L3 FIB,

NetFlow and ACL tables DFCs are populated with L2/L3

SP RP PFC DFCs are populated with L2/L3

FIB, NetFlow andACL tables

Standby Supervisor

Line Card—DFC

Line Card—DFC


Line Card—DFC

e Ca d C

Non Stop Forwarding (NSF)Non-Stop Forwarding (NSF)

DFC enabled line cards continue toNSF Recovery

DFC enabled line cards continue to forward based on existing FIB entries

Following SSO recovery and activation of standby Sup synchronized PFC

ti t f d t ffi b dSiSiSiSi

continues to forward traffic based on existing FIB entries

“Hot-Standby” MSFC RIB is detached from the FIB isolating FIB fromfrom the FIB isolating FIB fromRP changes

“Hot-Standby” MSFC activates routing processes in NSF recovery mode

MSFC re-establishes adjacency indicating this is an NSF restart

Peer updates restarting MSFC with it’s routing information

Restarting MSFC sends routing updates to the peer

C No Route Flaps During Recovery


RIB reattaches to FIB and PFC and DFCs updated with new FIB entries

No Route Flaps During Recovery

Non Stop Forwarding (NSF)Non-Stop Forwarding (NSF)

Two roles in NSF neighbor gracefulNSF Capable vs. NSF Awareness

Two roles in NSF neighbor graceful restart

– NSF Capable– NSF Aware

NSF-Aware

SiSiSiSi An NSF-Capable router is ‘capable’ of continuous forwarding while undergoing a switchover

An NSF Aware router is able to assist

SiSi

An NSF-Aware router is able to assist NSF-Capable routers by:

– Not resetting adjacency– Supplying routing information for verification

after switchover SiSiafter switchover

NSF capable and NSF aware peers cooperate using Graceful Restart extensions to BGP, OSPF, ISIS and NSF-CapableEIGRP protocols


Design Considerations for NSF/SSODesign Considerations for NSF/SSO

NSF is intended to provide Neighbor Loss NoNSF and Hello Timer Tuning?

NSF is intended to provide availability through route convergence avoidance

Fast IGP timers are intended to

Neighbor Loss, No Graceful Restart

SiSiSiSi

Fast IGP timers are intended to provide availability through fast route convergence

In an NSF environment deadIn an NSF environment dead timer must be greater than SSO Recovery + RP restart + time to send first hello

Switches running Native IOS–OSPF 2/8 seconds for hello/dead–EIGRP 1/4 seconds for hello/hold

Switches running Hybrid–OSPF 3/12 seconds for hello/dead–EIGRP 2/8 seconds for hello/hold


EIGRP 2/8 seconds for hello/hold

Design Considerations for NSF/SSODesign Considerations for NSF/SSO

Redundant topologies with equal costWhere Does It Make Sense?

Redundant topologies with equal cost paths provide sub-second convergence

NSF/SSO provides superior availability in environments with

?availability in environments with non-redundant paths

6

4

5 RP Convergence Is Dependent

on IGP and Tuningst V

oice

SiSi SiSi

3

4 on IGP and Tuning

ds o

f Los

1

2

Seco

nd

SiSi


0 Node Failure

NSF/SSOLink Failure

OSPFConvergence

Documents

Multilayer CampusMultilayer Campus Architecture and Design ... · Multilayer CampusMultilayer Campus Architecture and Design Principlesand Design Principles ... Design Principles