1 © 2001, Cisco Systems, Inc. All rights reserved. © 2002, Cisco Systems, Inc. All rights reserved. 12K Support Training

1© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.

12K Support Training

© 2001, Cisco Systems, Inc. All rights reserved. 2© 2001, Cisco Systems, Inc. All rights reserved. 2© 2002, Cisco Systems, Inc. All rights reserved. 2

Goals

Deepen global 12k support expertise through architecture discussion and hands-on troubleshooting


Agenda

• 12K Product Overview

• System Architecture

• Forwarding Architecture

• Services and Applications

• Troubleshooting


Module I – Overview and System Architecture


Part I - 12K Product Overview


12K Architecture Overview

• Fully distributed, multi-gigabit IP Router

RP provides routing and control services

Line cards perform IP forwarding

• Advanced QoS capabilities

• Bandwidth scalable (OC12, OC48, OC192)


Cisco 12008 Product Highlights

• Crossbar switch fabric architecture

• 8 slot card cage (7 for interfaces)

• Components:Switch Fabric Cards (SFC)

Clock and Scheduler Cards (CSC)

Route Processor (RP)

Line Cards (LCs)



• Crossbar switch fabric architecture

• 12 slot card cage (11 for interfaces)



Route Processor (RP)

Line Cards (LCs)



• Switching performance16 Slot System, 2.5Gbps switching capacity/slot – can support 10Gb LCs if fabric is upgraded

• Increased number of linecards

• Configuration2 Interface Shelves16 slots1 Fabric Shelf, with 5 slots2 Alarm cards – 1 top shelf, 1 bottom shelf


• Switching performance16 Slot System each with 10Gbps switching capacity/slot

• Supports 10G linecardsSupport for existing 12k line cards Slots are wider to accommodate 10 Gb LCs

• Configuration2 interface shelves16 slots1 fabric shelf, with 5 slots2 Alarm cards – 1 top shelf, 1 bottom shelf

Cisco 12416 - Product Overview


• 10 X OC192 capable slots• 8 Slots are wider to accommodate

10 Gb Lcs• 2 x Legacy slots (narrower slots 8 and 9)• 7 card fabric – 2 CSCs & 5 SFCs




• 6 slot card cage 1 narrow slot dedicated for RP

5 for redundant RP and Line Cards



1 or 2 Route Processors (RP)

Up to 5 Line Cards (LCs)

1 or 2 Alarm Cards

• 1/3 rack height


Cisco 12404 GSR Product Highlights

• 4 slot card cage

1 narrow slot for RP

3 10G capable slots

• Components:1 Consolidated Fabric Card : CSC-4

(CSC, SFC, Alarm built in)Route Processors (RP)

Up to 3 Line Cards (LCs)

FABRIC IS NOT RESILIENT


• Switching performance16 Slot System each with 40Gbps switching capacity/slot

• Supports 20G and future 40G linecardsSupport for existing GSR line cards Slots are wider to accommodate 10/20Gb LCsRequires PRP

• Configuration2 interface shelves16 slots1 fabric shelf, with 5 slots2 Alarm cards – 1 top shelf, 1 bottom shelf

Cisco 12816 - Product Overview


• 10 X 40Gb capable slots• 8 Slots are wider to accommodate

10/20 Gb LcsRequires PRP

• 2 x Legacy slots (narrower slots 8 and 9)• 7 card fabric – 2 CSCs & 5 SFCs



Part II - 12K System Architecture


12K Components


System Components

• Route Processor

• Switching fabric

• Line cards

• Power/Environmental Subsystems

• Maintenance BUS


• • • • • •Switch Fabric

Switch Fabric

Maintenance Bus

PowerSuppliesPower

SuppliesFan/Blower

SystemFan/Blower

System

12k Architecture - Components

Line CardLine Card

Line CardLine Card

Line CardLine Card

Route Processor

Route Processor

Line CardLine Card

Line CardLine Card

Line CardLine Card

Route Processor

Route Processor


Route Processor

• Boots and manages line cards

• Provides and coordinates routing services

• Builds, distributes, and maintains FIB

Adjacency table, FIB table, MPLS label table

• Provides out-of-band console/aux ports

• Provides intelligence behind system monitoring and access


RP - System Monitor/Controller

• • • • • •

Line CardLine Card

Line CardLine Card

Line CardLine Card

LineCardLineCard

Switch Fabric

Switch Fabric

Line CardLine Card

Line CardLine Card

Line CardLine Card

RouteProcessor

RouteProcessor

Maintenance Bus

Routing Protocol UpdatesProcess-level Traffic

System health monitoringInterface Status Msgs

Statistics

Temperature,Voltage,

Current MonitoringPower

SuppliesPower

SuppliesFan/Blower

SystemFan/Blower

System


Line Cards

• Perform all packet switching

• Statistics collection and reporting

• Run IOS

• Six different forwarding architectures


• Out-of-band communications channel to linecards

• 1 Mbps - 2 wire serial interface

• Based on Controller Area Network (CAN) 2.0 Spec. (ISO 11898)

http://www.can-cia.org/can/

• A daughter card on each linecard having it’s own CPU w/ integrated CAN controller, A/D converter and other peripherals, dual CAN interface, SRAM, Flash and Serial EEPROM.

CSCs and BusBoard can proxy and/or multiplex MBUS signals for power supplies

• Control pins reach into LED, Serial ID EEPROM, DC/DC power converter, clock select FPGA, temp sensor, voltage sensor

Very large set of functions

PowerSupply

PowerSupply

Fan/BlowerSystem

Fan/BlowerSystem

• • • • • •Multigigabit Crossbar

Fabric

Multigigabit Crossbar

Fabric

Line CardLine Card

Route ProcessorRoute Processor

Line CardLine Card

Line CardLine Card

Line CardLine Card

Line CardLine Card

Line CardLine Card

Route Processor

Route Processor

SchedulerScheduler

Maintenance Bus

MBUS


MBUS Functions

• Power and boot LC

• Device Discovery

• RP arbitration

• OIR management

• Environmental monitoring

• Diagnostics download

• LC console access

Via “attach” command

• Logging


Alarm Cards

• LED display for fabric card status

• External alarm connection

• Power conversion/supply for 5v MBUS power plane

On the 12008, this functionality is on the CSC.


Switch Fabric - Overview

• Provides the data path connecting the LCs and the RP

• Active CSC card provides the master clock for the system

• Everything traverses fabric in Cisco cell.

- Data is 8B/10B encoded

• Two components

- Clock & Scheduler Cards (CSC)

- Switch Fabric Cards (SFC)


ciscoCell

• Packet are chopped into ciscoCells before they are sent across the switching fabric.

• A ciscoCell is 64bytes of data consisting of 48bytes of IP payload and 8bytes of header and 8bytes of CRC.


• Scheduler (SCA)

Handles scheduling requests and issues grants to access the crossbar switching fabric

• Cross-bar (XBAR)

Sets the fabric lines for transmissions following the scheduling decision

Clock Scheduler Card (CSC)


• Each fabric card provides a slice of the Cisco cell data path

• Up to 5 data paths are available – for up to 4+1 redundancy

• The 5th data path carries an XOR of other streams

Used for recovery of a errored stream No 5th path = no recovery capability

• ‘Grants’ travel exclusively between the LC and the active CSC using separate communication lines

Never traverse the SFC cards

Fabric Redundancy


Scheduling Algorithm (“ESLIP”)

• RequestEach input LC makes request to output highest priority queued cell (unicast or multicast)

• GrantEach destination LC grants the request to the highest priority request

• AcceptEach input LC selects the highest grant

• TransmitXBAR set and cells transmitted


ESLIP Illustrated

SwitchFabricSwitchFabric

SchedulerScheduler

DRR

Each line card utilizesDRR to select a setof packets from theVoQs.

Request is sent to Scheduler on CSCto obtain a grant

Request

Scheduler (for eachoutput) selectshighest priority packet fromrequests and determines if outputcan grant request

Grant

Schedulersend multiple grants(for multiple outputs)to slot

Slot select the highestgrant and accepts theconnection

Accept


Bootup process


Startup/Boot Process

• Initial Power On

• RP Boot Process

• Clock Scheduler Boot Process

• Line Card Boot Process

• Switch Fabric Boot Process

• Fabric Initialization

• IOS Download


Initial Power On

• When the chassis is powered on, the Mbus module on each card is powered on.

• After the Mbus module powers on its processor it boots from a module on EEPROM.

• Card power up order varies depending on linecard type.


RP Boot Process

• Mbus module powers first

• Board logic starts, image begins booting and Mbus code is loaded to the Mbus module

• The CPU, Memory controller ASIC, cell-handler ASICs and FIA ASICs are then issued power for startup

• RP arbitration process is executed using the Mbus

• Master RP instructs Line Cards and Switch Fabric Cards to power on.

• RP waits for Line Cards to power and finish booting


Switch Fabric Card Startup/Boot

• Master RP instructs each SFC Mbus module to power on at the same time the Line Card Mbus modules are told

• SFC obtains clock the same way each LC does

• The SLI ASICs and XBAR initialize and power up

• SFC Mbus code is downloaded from the RP

• All cards are now powered on but not usable


Line Card Startup/Boot

• Each LC Mbus module powers up after being told to do so by the RP

• Clock selection takes place

• The Line Card CPU is powered on and boots

• Mbus module code is loaded

• The Line Cards CPU notifies the RP it has booted

• Switch Fabric access is not available yet


Line Card IOS Downloads

• The Line Card may already have enough code in its flash to become operational on the Switch Fabric, or it may require an Mbus download.

• Only enough code for the Line Card to become operational on the fabric will be loaded using the Mbus.

• Once all cards are operational on the fabric, the fabric is initialized and the main IOS software is downloaded.


IPC Services


IPC Overview

• The 12k is a distributed multiprocessor system. The processors communicate via IPC … an essential architectural service

• IPC has a reliable (acknowledged) and unreliable mode of transport (with or without sequence number or notification). The application uses an appropriate method.


IPC Clients

• Applications (clients) can build their own queue structures and feed the IPC queue/cache as well as choose to block or not until an ACK or imbedded response is received.

e.g. … CEF uses a multi-priority queue and it’s own cache in front of the IPC queue (controlled by “ip cef linecard ipc memory”) … it’s got it’s own message handling routines defined in the same registry as direct IPC interrupt or process level message handling.

Many (most) applications use the CEF packaging (XDR) message types and queues as an interface to IPC.

e.g. … route-map updates and acl updates to linecards

• Applications are also responsible for being “well-behaved”.

• Utility applications like slavelog and slavecore use IPC directly.


Module 2 – Forwarding Architecture


Route Processor


The Route Processor (RP)

• The RP’s control path for Line Cards uses IPC via the switch fabric or Mbus

• The switch fabric connection is the main data path for route table distribution

• The Mbus connection enables the RP to download a bootstrap image, collect or load diagnostic information, and perform general maintenance operations


RP Responsibilities

• Running routing protocols

• Builds and distributes the routing tables to Line Cards (i.e. routing table maintenance)

• Provides general maintenance functions (i.e. Booting Line Card processors)


RP Routing Table Maintenance

• Using the RIB the RP maintains a complete forwarding table of its own (RP-FIB)

• Routing updates are forwarded from RP-RIB to each Line Card (LC-FIB)

• Each LC-FIB entry corresponds to an interface which contains a MAC encapsulation string, output interface and MTU


RP Routing Table Maintenance

• FIB distribution is done through reliable IPC updates

• When the routing protocol triggers an update, it is placed into the FIB of the RP then sent to the Line Cards

• Updates are unicast across the fabric to all Line Cards


GRP


Major Components: The GRP

• R5000 CPU (a.k.a. P4)

• Mbus Module

• Tiger ASIC

• CSAR ASIC

• FIA ASIC

• SLI ASIC

• Power Modules


GRP ComponentsPower Units

DRAM

Tiger ASIC

CPU

Fabric Interface ASIC (FIA)

Serial Line Interface ASIC (SLI)

Cisco cell Segment And Reassembly (CSAR)


GRP Component Groups

I/O Sub-system

Fabric

Mbus

Logic


CSAR (ciscoCell Segmentation and Reassembly) ASIC

• Buffer manager ASIC for the GRP (equivalent to Rx and Tx BMA on Engine 0 LCs)

• The CSAR contains two 64k buffers

• Messages are placed in a hold queue if these buffers are full

• An interrupt is sent to the CPU when the buffers are free

• The CSAR contains 32 reassembly areas when receiving ciscoCells from the fabric for unicast and multicast providing 64 areas

• Connects to fabric at OC12


Performance RP 1(PRP-1)


PRP-1 Architecture

CHOPPERXCVR

ASSEMBLER

2M L3Cache

Main(RAM)

Memory

CPUVoyager

(PPC7450)

I/O Bus

SystemController

(Discovery)

XCVR

XCVR

XCVR

XCVR

MBUS2x10/100 DUART PCMCIA NVRAMBootflash BootPROM

FUSILLI

64@133Mhz


Performance Route Processor (PRP)

• The PRP is fully compatible with the GRP at the hardware level

• One of the major differences with the PRP is the use of the V’ger processor, a Motorola PPC processor running at 655MHz

• The future Apollo processor running at 1GHz will replace V’ger

• Connects to fabric at OC48 – requires at least 1CSC and 3 SFCs to operate



• The PPC CPU also supports on-chip 32Kbs of Layer 1 cache and on-chip 256Kb of Layer 2 cache with an external 2MB of Layer 3 cache controller.

• The realized performance improvement is 4 – 5 times that of the current GRP



• Default 512Mb DRAM upgradeable to 2Gb

• 2 10/100 Enet ports

• RJ-45 Console port

• 64Mb Flash Disk as standard


Line card concepts


Line Card Concepts

Components:

• PLIM - Physical Layer

Optics, Framer, SAR, etc.

• Layer 3 Forwarding Engine

IP/MPLS Switching and Services

• Fabric Interface

Transmission

Physical Layer

(Optics)

Physical Layer

(Optics)

Layer 3 Engine

Layer 3 Engine

Fabric InterfaceFabric

Interface

RXRX

TXTX

CPUCPU

To Fabric

To Fabric

From FabricFrom Fabric


• Handle L2 protocol encap/decap

- SONET/SDH framing- ATM cell segmentation/re-assembly- Channelization

• Receives packet off the wire and passes it to the forwarding engine

PLIM – Physical Interfaces


• Runs IOS and maintains CEF tables

• Provides CEF switching services, feature capabilities

• Provides queuing and QoS services (through the RX and TX queue managers)

NOTE – QoS will be covered in detail in the ‘Applications section’

FE - Forwarding Engine


• Provides fabric transmission services

• Two components:

1. FIA – interface between forwarding engine and fabric interface

2. SLI - does 8B/10B encoding and decoding of Cisco cells

FIM - Fabric Interface Module


Line Card Concepts:A Reference Architecture


Line Card Reference Architecture

PLIMForwarding &

Feature Complex

CPU

ToFab queue mgr

ToFab packet memory

FrFab queue mgr

FrFab packet memory

ToFabFabric

Interface

ForwardingLookup Tables

FrFabFabric

Interface


Various routing protocols maintain individual routing databases.

Forwarding Architecture

The routing table is built by using the best available paths from the routing protocols.

From the IP routing table, we pre-resolve recursive routes and build the CEF table (a.k.a. FIB table)

The CEF table is pushed down from the GRP to each linecard via IPC

From the CEF table, HW-based linecards will build their own hardware forwarding tables


Summary

• Multiple levels of routing/forwarding information

• RP provides control plane services

- IP routing protocols- MPLS label exchange protocols

• RP maintains RIB, FIB, LFIB

• LC have a copy of FIB and LFIB

E2/3/4/4+/6 have a HW forwarding FIB and LFIB as well


Engine Architectures


Line Card - Switching Engines & ASICs

• Engine 0 – BMA – 622Mb

• Engine 1 - Salsa/BMA48 – 2.5Gb

• Engine 2 - PSA/TBM/RBM – 2.5Gb

• Engine 3 (aka ISE) – Alpha/Conga/Radar – 2.5Gb

• Engine 4 – RX/MCC/TX – 10Gb

• Engine 4+ - RX+/MCC/TX+ - 10Gb

• Engine 6 – Hermes/Ares/Hera – 20 Gb


Engine 0 Architecture


Engine 0 - Components

• R5000 CPU + L3FE ASIC

• BMA

QoS support with performance hit

• Main Memory

Up to 256MB of DRAM

• Packet Memory

Up to 256MB SDRAM split equally between Rx and Tx



ToFabBMA

ToFabBMA

L3FEL3FE

CPUCPU

FrFabBMA

FrFabBMA

PacketMemoryPacket

Memory

PacketMemoryPacket

Memory

LC

IOS

Me

mo

ryL

C IO

S M

em

ory ToFab

FIAToFab

FIA SLISLI

FrFabFIA

FrFabFIA

SLISLI

PLIM L3 Engine Fabric Interface

RxPOSRx

POS

TxPOSTx

POS

FramerFramerOpticsOpticsX

C

V

R

S


1 port OC12 Engine 0 line card

Optics

Mbus Agent Module

L3FE

SLI

FIA

RxBMARx Packet Memory

TxBMA

Tx Packet Memory

CPU


Engine 0 – OC12 with Features

• CPU-based switching

• Provides OC-12 performance with features

• Extensible/flexible architecture- easy to add more features

• WRED/MDRR in HW with performance hit

• Performance:

No features - ~ 420 kpps

With features - ~ 250 kpps





• R5000 CPU + Salsa ASIC

Salsa = Enhanced Layer 3 Fetch Engine (L3FE)

Hardware IP lookup with software re-write

• BMA48

Performance enhanced BMA

No QoS support

• Main Memory

Up to 256MB of DRAM

• Packet Memory

Up to 256MB SDRAM split equally between Rx and Tx


ToFabBMA48ToFabBMA48

SalsaSalsa

CPUCPU

FrFabBMA48FrFab

BMA48

PacketMemoryPacket

Memory

PacketMemoryPacket

Memory

LC

IOS

Me

mo

ryL

C IO

S M

em

ory ToFab

FIA48ToFabFIA48 SLISLI

FrFabFIA48FrFabFIA48

SLISLI


RxSOPRx

SOP

TxSOPTx

SOP

X

C

V

R

S


OpticsOptics

GigaMACGigaMAC

RxTrans

RxTrans

TxTrans

TxTrans


1 port GigE Engine 1 line card

Rx BMA48

Salsa

Tx BMA48

CPU


Engine 1- Salsa

• Hardware enhancements to IP packet validation and FIB lookup assist

• Verify

• packet is IPv4 packets with no options.

• Identify that packet is PPP/HDLC encapsulated.

• checksum, length, TTL

• Update IP header (TTL, checksum)

• Perform IP lookup and cache FIB pointer for CPU re-write operation


E0/1 - Life of a Packet:Watching the Queues


Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------

4 non-IPC free queues:26626/26626 (buffers specified/carved), 50.90%, 80 byte data size1 101 26726 26625 65535 16184/16184 (buffers specified/carved), 30.94%, 608 byte data size2 26727 42910 16184 65535 7831/7831 (buffers specified/carved), 14.97%, 1568 byte data size3 42911 50741 7831 65535

IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 67 66 100 65535

Raw Queue:31 0 0 0 65535

ToFab Queues:Slot0 0 0 0 65535 1 0 0 0 655352 0 0 0 65535 3 0 0 0 65535 4 0 0 0 65535 5 0 0 0 65535 6 0 0 0 65535 7 0 0 0 65535

Mcast 0 0 0 65535

Packet Arrives on Line Card (tofab)


Move the Buffer onto the Raw Q (tofab)Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------



Raw Queue:31 0 0 1 65535


Mcast 0 0 0 65535





Raw Queue:31 0 0 0 65535


Mcast 0 0 0 65535

FIB Result and ToFab Queuing (tofab)





Raw Queue:31 0 0 0 65535


Mcast 0 0 0 65535

Return the Buffer to the Free Q (tofab)



4 non-IPC free queues:26560/26560 (buffers specified/carved), 50.90%, 80 byte data size1 101 26660 26559 65535 16144/16144 (buffers specified/carved), 30.94%, 608 byte data size2 26661 42804 16144 65535 7811/7811 (buffers specified/carved), 14.97%, 1568 byte data size3 42805 50615 7811 65535 1562/1562 (buffers specified/carved), 2.99%, 4544 byte data size4 50616 52177 1562 65535


Raw Queue:31 0 83 0 65535

Interface Queues:0 0 0 0 65535 1 0 0 0 65535 2 0 0 0 65535 3 0 0 0 65535

Egress Card Receives the Packet (frfab)


Queuing for Transmission (frfab)Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------



Raw Queue:31 0 83 0 65535



Return the Buffer to the Free Q (frfab)Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------



Raw Queue:31 0 83 0 65535





Engine 2 Overview

• First programmable, hardware-based forwarding engine

• Multi-million PPS with some features

• Up to 4Mpps performance (no features)


RBMRBM

PacketMemoryPacket

Memory


PSAPSA

TBMTBM

PacketMemoryPacket

Memory

PSAMemory

PSAMemory

ToFabFIA48ToFabFIA48 SLISLI


SLISLI


RxSOPRx

SOP

TxSOPTx

SOP


C

V

R

S

LC

IOS

Me

mo

ry

SalsaSalsaCPUCPU


1 port OC48 POS Engine 2 line card

TBM

RBM

PSA



• R5000 CPU -> Slow Path

Slow path (CPU computed) CEF tables, ICMPs, IP options, etc…

• PSA (Packet Switched ASIC) -> Fast Path

Microcoded IP/MPLS lookup & feature processing

• RBM/TBM (Receive/Transmit Buffer Manager)

Hardware WRED, MDRR

• Packet Memory

256MB SDRAM can be upgraded to 512MB SDRAM

• PSA Memory

PSA copy of FIB table


RBMRBM

PacketMemoryPacket

Memory

Engine 2 – Rx Packet flow

PSAPSA

TBMTBM

PacketMemoryPacket

Memory

PSAMemory

PSAMemory



SLISLI


RxSOPRx

SOP

TxSOPTx

SOP


C

V

R

S

LC

IOS

Me

mo

ry

SalsaSalsaCPUCPU

•SONET/SDH framer

•Extract packets from SONET/SDH payload

•Pass indication of input interface and packet header to PSA

•Payload passed to RBM

•Packet validation •IP/MPLS lookup

•Feature processing (ACLs, CAR, Netflow, etc...)

•Append buffer header

•Determine loq, oq and freeq for packet

•Tofab queueing

•WRED

•MDRR

•Segment packet into ciscoCell

•add CRC to ciscoCell

•send transmission request to SCA

•8B/10B encoding

•send cells to fabric


Engine 2 – PSA Forwarding

• The Packet Switching ASIC is an IP and TAG forwarding engine

• The ASIC contains a 6 stage pipeline, Pointer and Table Lookup memory

• As packets move through the PSA pipeline, the forwarding decision and feature processing is completed


PSA Architecture

MAC header checking, protocol ID checking, IP header checking, extraction of IP/MPLS address fields

Microcode engine which performs checks on the packet (protocol, length, TTL, IP CHKSUM) and extracts the appropriate address(es) for the main lookup. Some feature processing.

IP/MPLS lookup machine

Adjacency Lookup, Per Adjacency Counters

Fetch PreP PLU TLU Gather

FIB TREE(256K) LEAVES/ADJ/STATS(256K)

PoP

Ext. SSRAM

Microcode engine which applies the results of the PLU/TLU lookup to the packet. Tasks include COS handling, MTU check, special case tests, setup of gather stage, feature processing, etc...

Modifications to packet header (e.g. pushing MPLS Labels). Prepare packet for transmission to RBM

Each stage has a 25 clock budget @100MHz = 250ns, i.e. 4Mpps


RBM: Rx Queue Manager

• The RBM manages the linecard’s receive packet memory buffers and queues

• There are two major types of queues in RBM:

LowQs (16 FreeQs, 1 RAWQs, an IPC FreeQ and spare queues)

2048 unicast Output Queues and 8 multicast queues

• 16 slots per chassis, 16 ports per slot, 8 queues per port = 2048 queues

• One hpr (high priority) queue is allocated per destination slot/port.


TBM: Tx Queue Manager

• The TBM manages the linecard’s transmit packet memory buffers and queues

• Three types of queues:

Non-IPC freeQs, 1 CPU RawQ, IPC FreeQ

128 Output Queues

Multicast RawQ

• 8 CoS queues per output port, 16 ports = 128 queues


RBMRBM

PacketMemoryPacket

Memory

Engine 2 – Tx Packet flow

PSAPSA

TBMTBM

PacketMemoryPacket

Memory

PSAMemory

PSAMemory



SLISLI


RxSOPRx

SOP

TxSOPTx

SOP


C

V

R

S

LC

IOS

Me

mo

ry

SalsaSalsaCPUCPU

•Remove 8B/10B encoding

•Verify and remove CRC from ciscoCell

•send cells to TBM

•Re-assemble packet from ciscoCells

•FrFab queueing

•WRED

•MDRR

•Append L2 header and send packet to PLIM

•Mcast duplication

•Put packets in SONET/SDH payload

•SONET/SDH framer


E2 Feature Support

• Designed to be forwarding ASIC on a “backbone” card, ie does not natively support any features

• Features like ACLs, SNF, BGP PA added later on, but take performance hit

• Most new features require a separate ucode load and are mutually exclusive

• Performance varies with features (eg. ACLs):128 line iACLs – 800kpps

128 line oACLs – 675 kpps

448 line iACLs – 690 kpps

448 line oACLs – 460 kpps

99© 2001, Cisco Systems, Inc. All rights reserved.

Session NumberPresentation_ID

Engine 3 - IP Services Engine (ISE)

100100100© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID

ISE Overview

• Programmable, hardware-based forwarding engine

• Up to 4Mpps performance (with features)

Uses TCAMs for advanced feature processing

• Traffic shaping and advanced QoS support

Flexible mapping of queues


ISE – Architecture

ALPHA

PICANTECPU

CONGA

PacketMemory

FIB TableMemory

X

C

V

R

S


Optics

SPECTRA

RADAR

PacketMemory

LC

IOS

Me

m

ALPHA

SLIFIA

SLIFIA

FUSCILLIGULF

TCAM

TCAM


4xOC12 POS ISE Linecard

TX TCAM

RX TCAM

RX ALPHA

SPECTRA

GULFTX ALPHA

CONGA

FUSCILLI

R7K

PICANTE

RADAR


ISE – Rx Packet flow

ALPHA

PICANTECPU

CONGA

PacketMemory

FIB TableMemory

X

C

V

R

S


Optics

SPECTRA

RADAR

PacketMemory

LC

IOS

Me

m

ALPHA

SLIFIA

SLIFIA

FUSCILLIGULF

TCAM

TCAM

•Packet validation •IP/MPLS lookup

•Feature processing (ACLs, CAR, Netflow, etc...)

•Append buffer header

•Determine loq, oq and freeq for packet

•SONET/SDH framer

•Handle channelization


•Pass indication of input interface and packet header to RX Alpha

•Payload based to Radar

•Tofab queueing

•Input rate shaping

•WRED

•MDRR

•Segment packet into ciscoCell

•add CRC to ciscoCell

•send transmission request to SCA

•8B/10B encoding

•send cells to fabric


ALPHA• Advanced Layer 3 Packet Handling ASIC

• Performs forwarding, classification, policing and accounting

• Two ALPHA chips, one in the receive path, one in the transmit path. This allows features to be implemented in both the ingress (RX) and egress (TX) paths

• 11 pipeline stages 3 micro-code stages for future expandability

• Utilizes TCAMs to perform high-speed feature processing. Each ALPHA has its own TCAM

ISE - ALPHA


11 stages of ALPHA Pipeline

Fetch PreP PLU PCM TLU

Gather

FIB TREELEAVES/ADJ/STATS

PoP

Ext. SSRAM

CAMP3 stages

Ext. CAM + SSRAM

MIP

MAC header checking, protocol ID checking, IP header checking, extraction of IP/MPLS address fields

Microcoded stage which is capable of any general purpose activity on the packet

MTRIE lookup machine

TCAM access for altering PLU results (PBR, MPLS)

Adjacency Lookup, Per Adjacency Counters

Microcoded stage which is capable of any general purpose activity on the packet

CAM Processor – Lookups for xACL (permit/deny), CAR token bucket maintenance, Netflow counters update

Processing packet structure, including stripping the old input encapsulation, stripping old MPLS labels if necessary, pushing new labels and computation of the new IP checksum

Microcoded stage – Performs feature actions, handling exception packets


RADAR: Rx Queue Manager

• The RADAR manages the linecard’s receive packet memory buffers and queues

• There are three major types of queues in RADAR:

LowQs (16 FreeQs, 3 RAWQs, an IPC FreeQ and spare queues)

2048 Input Shape Queues (rate-shaping)

2048 unicast Output Queues (16 unicast high priority queues)

and 8 multicast queues

• One local output-queue is allocated per destination interface

• One hpr (high priority) queue is allocated per destination slot


RADAR: Input Shape Queues

• There are 2048 queues dedicated to ingress traffic shaping each with an independent ‘leaky bucket’ circuit.

• Each flow can be shaped in increments of 64kbps (from 64kbps up to line rate)


RADAR: Rx Queue Manager

• The Rx ALPHA decides which type of queue will be used for each packet.

Rx PacketProcessor

IputShapeQueue

VirtualOutputQueue

SwitchFabric

Interface

RawQueue

LinecardCPU

Rx QueueManager

The Packet Processordecides which patheach packet will take


ISE – TX packet flow

ALPHA

PICANTECPU

CONGA

PacketMemory

FIB TableMemory

X

C

V

R

S


Optics

SPECTRA

RADAR

PacketMemory

LC

IOS

Me

m

ALPHA

SLIFIA

SLIFIA

FUSCILLIGULF

TCAM

TCAM

•Remove 8B/10B encoding

•Verify and remove CRC from ciscoCell

•send cells to TX ALPHA•Adjacency lookup

•Feature processing (ACLs, CAR, MQC, etc...)

•Update output_info field of buffer header with info from adjacency

•Re-assemble packet from ciscoCells

•FrFab queueing

•Output rate shaping

•WRED

•MDRR

•Append L2 header and send packet to PLIM

•Mcast duplication

•Handle channelization

•Put packets in SONET/SDH payload

•SONET/SDH framer


CONGA: Tx Queue Manager

• The CONGA manages the linecard’s transmit packet memory buffers and queues


Non-IPC freeQs, 3 CPU RawQs, IPC FreeQ

2048 Output Queues (2 leaky buckets per queue for rate-shaping)

Multicast RawQ

• Output queues divided equally among output ports

• Support for 512 logical interfaces

• Max Bandwidth shaping per port, Min and Max Bandwidth shaping per queue


CONGA

• Each ‘Shaped Output’ queue has a built in dual-leaky bucket mechanism with a programmable maximum and minimum rate (I.e. a DRR bandwidth guarantee)

• A second level of shaping is available per ‘Port’.

Port 0 ShapedOutput Queues

HP

LP0

LPn

To PLIM

To PLIM

DRR

DRR

Port 511 ShapedOutput Queues

HP

LP0

LPnRepresentsA Shaper



Engine 4+ - 10G Edge Services


RX+RX+

TX+TX+

PacketMemoryPacket

Memory

LookupMTRIE

LookupMTRIE

10GFIA

10GFIA


MCCMCC

PacketMemoryPacket

Memory

PicantePicanteCPUCPU

LC

IOS

Me

mo

ry SerDes

BACKPLANE

PHAD

(OpticalInterface

ASIC)

PHAD

(OpticalInterface

ASIC)

OpticsOptics

Engine4+ “Edge Services”

FramerFramer


Engine 4+ - Components

• R5000 CPU -> Slow Path

Slow path (CPU computed) CEF tables, ICMPs

• RX+ ASIC -> Fast Path

Hardware IP/MPLS lookup inc. Multicast, CAR, ACLs, MPLS-PE

• MCC ASIC

Hardware WRED, MDRR

• Packet Memory

256MB SDRAM can be upgraded to 512MB SDRAM and 1024 in future

• TX+ ASIC -> Fast Path

Hardware outbound traffic shaping, ACLs,


10x1 GigE Engine 4 linecard

TX

10x1GE PLIM

MCC

RXPicante


RX+ - Packet processing ASIC

• Non-programmable high-speed ASIC providing 25 Mpps switching capacity

• Virtual CAM (vCAM) for features

ACLs, CAR and PBR

• Line-rate for 40 byte packets at /32 FIB lookup


MCC – ToFab Queueing ASIC

• Manages receive packet memory and queues


Non-IPC freeQs, 8 CPU RawQs, IPC FreeQ

2048 Unicast VOQs, 8 multicast VOQs

• One high priority queue per destination slot/port


RX+RX+

TX+TX+

PacketSRAMPacketSRAM

LookupMTRIE

LookupMTRIE

10GFIA

10GFIA


MCCMCC

PacketSDRAMPacketSDRAM


LC

IOS

Me

mo

ry SerDes

BACKPLANE

PHAD

(OpticalInterface

ASIC)

PHAD

(OpticalInterface

ASIC)

OpticsOptics

Engine4+ - RX Packet Flow

FramerFramer

• SONET/SDH framer


•Protocol identification

•Verify packet length

•Append PLIM header

•IP unicast and multicast lookup

•MPLS lookup

•CAR/ACL feature processing

•Append buffer header and update loq, oq, and ideal freeq values

•Manage packet buffers

•Perform WRED, MDRR

•Packets segmented into cells

•Make packet transmission request

•Append ciscoCell CRC

• 8B/10B encoding

•Transmit cell over fabric


RX+RX+

TX+TX+


LookupMTRIE

LookupMTRIE

10GFIA

10GFIA


MCCMCC



LC

IOS

Me

mo

ry SerDes

BACKPLANE

PHAD

(OpticalInterface

ASIC)

PHAD

(OpticalInterface

ASIC)

OpticsOptics

Engine4+ - TX Packet Flow

FramerFramer

•PLIM header removed

•Packets segments queued to SONET channels

•Packets sent within SONET payloads (POS)

•Multicast packets duplicated

•Header re-written for output(MAC re-write)

•ACL/CAR performed

•RED/WRED performed

•Packets queued for output(16 ports. 8 queued/port)

•MDRR scheduling and output shaping performed

•Cells re-assembled into packets •CRC checked

•Packet header reconstructed

•Packets scheduled to TX

• SONET/SDH framer


TX+ - TX Queueing ASIC

• Manages transmit packet memory and queues

• Four types of queues:

Non-IPC freeQs, 8 CPU RawQs, IPC FreeQ,

Multicast Raw Queue

128 Unicast OQs, 8 multicast OQs

• Per-destination port LLQ

• Performs output CAR, rate-shaping



Engine 6 – 20Gb Edge Services


HermesHermes

HERAHERA

PacketMemoryPacket

Memory

LookupMTRIE

LookupMTRIE

TFIA

FFIA


AresAres

PacketMemoryPacket

Memory


LC

IOS

Me

mo

ry

EROS

SERDES

BACKPLANE

Zeus

Framer& PHAD

Zeus

Framer& PHAD

OpticsOptics

Engine 6

TCAMTCAM

TCAMTCAMFFIA

TFIA



• RM7000 CPU -> Slow PathSlow path (CPU computed) CEF tables, ICMPs

• Hermes ASIC -> Fast Path – 50Mpps @ 40BytesHardware IP/MPLS lookup inc. Multicast, CAR, ACLs, MPLS-PE, 6-PE, PBR, Loose & Strict uRPF

• Ares ASICHardware WRED, MDRR

• Hera ASIC -> Fast PathHardware outbound traffic shaping, ACLs, SNF, Mcast

• 512Mb Dram Route Memory• 512Mb RLDRAM Packet Memory• TCAM4 ASICs

Attached to Hermes and Hera for feature processing


Engine6 2xOC192 layout

TCAM

Optics

Power

Supply

CPU

memory

CPU

PICANTE

Zeus

Hermes Ares

Eros

Hera

MBUS

TCAM

Optics

Zeus

TCAM

Hermes Ares

Eros

Hera

Power Supply

PICANTE

CPUCPU memory

MBUS


Engine6 8xOC48 layout

SFP Pluggable Optics

Surface mounted RLDRAMs


HermesHermes

HERAHERA

PacketMemoryPacket

Memory

LookupMTRIE

LookupMTRIE

TFIA

FFIA


AresAres

PacketMemoryPacket

Memory


LC

IOS

Me

mo

ry

EROS

SERDES

BACKPLANE

Zeus

Framer& PHAD

Zeus

Framer& PHAD

OpticsOptics

Engine 6

TCAMTCAM

TCAMTCAMFFIA

TFIA

Framer + PHAD integrated•Layer-1 processing alarms ,crc check, APS..•Pkts buffering•Verify packet length•Append PLIM header

•IP/MPLS Lookup•TCAM based feature processing (ACL/CAR/PBR/VRFs)•PKT MOD TTL adj, ToS adj,IP checksum adj.•Append buffer header and update loq, oq etc..

•Queueing ASIC •Manage packet buffers•Perform WRED, MDRR

•TFIA + FFIA ASIC•Packets segmented into cells•Make packet transmission request•Append ciscoCell CRC

•Cells re-assembled into packets CRC checked•Packet header reconstructed•Packets scheduled to TX

•Multicast packets duplication•MAC rewrite for output•TCAM based output feature - ACL/CAR•Output packet queuing•RED/WRED performed•MDRR scheduling •Output traffic shaping

•Layer1 processing•PLIM header removed•Packets segments queued to SONET channels•Packets sent within SONET payloads (POS)


TCAM based feature

• TCAM used to implement key features

• 32000 ingress entries, 32000 egress entries shared between…

ACL

CAR – 32 car rules per port

PBR

VRF Selection

• Security ACLs are not merged


TCAM Basics

• TCAM - Ternary Content Addressable Memory – Match on 0 , 1 , X (don’t care)

• ACL/CAR/PBR/VRFs rules from CLI converted into Value Mask Result (VMR) format to be inserted in TCAM

Value cells – key values

ACL/CAR/PBR/VRFs values

Mask cells = Significant Value Bits to be matched

Result = Value && Mask - Action

Security ACL – permit/deny

CAR - Pointer to CAR buckets

PBR – Adjacency

VRF Selection – VRF Root


QoS flow

ACL iCAR WRED

drop drop dropCOS Queues

WRED oCAR ACL

drop drop dropCOS Queues

MDRR

Shaping

Hermes Ares

Hera

MDRR

tofab

frfab


QoS support

• 2064 tofab queues

16x16x8 = 2048 unicast queues

8 local CPU queues

8 multicast queues

Per priority queue per destination port

• 136 frfab queues

16x8 = 128 unicast queues

8 local CPU queues

Per priority queue per port

131131131© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID 131131131© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID


Engine 4+ - Line Card Family

• OC192 POS

• 4 x OC48 POS

• 1 x 10 GE

• 10 x 1 GE - EOS

• Modular GE

• 2 x OC48 DPT*

• OC192 DPT*

Documents

1 © 2001, Cisco Systems, Inc. All rights reserved. © 2002, Cisco Systems, Inc. All rights reserved. 12K Support Training