Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400

Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved. 1

Внутренняя архитектура IOS-XE: Средства траблшутинга предачи трафика на ASR1k и ISR4400 Oleg Tipisov Customer Support Engineer, Cisco TAC

Apr, 2015. Revision 1.0 Cisco Public

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2

Сегодня на семинаре эксперт Cisco TAC Олег Типисов

расскажет об особенностях аппаратной и программной архитектуры платформ ASR1k и ISR4400/ISR4300. Также будут рассмотрены

диагностические средства IOS-XE, используемые для траблшутинга передачи трафика

Олег Типисов

Инженер центра

технической поддержки

Cisco TAC, Москва


Технические эксперты

Сергей Василенко




Дмитрий Леонтьев




Дата проведения вебинара – 22 апреля 2015г.


• Сегодняшняя презентация включает опросы аудитории

• Пожалуйста, участвуйте!


Скачать презентацию Вы можете по ссылке:

https://supportforums.cisco.com/ru/document/12483586





Уважаемые пользователи мы предлагаем Вам принять

участие в конкурсе после проведения вебкаста,

который так и будет называться «Внутренняя

архитектура IOS-XE: Средства траблшутинга

предачи трафика на ASR1k и ISR4400».

• Первые три победителя получат фирменный куб Cisco-TAC

• Ответы присылайте на [email protected]

• Задание конкурса будет размещено сегодня после проведения вебкаста (14-00мск)

mailto:[email protected]




• Используйте панель Q&A, чтобы задать вопрос

• Наши эксперты ответят на них

Cisco Confidential 8 © 2013 Cisco and/or its affiliates. All rights reserved.


• Hardware and Software Architecture

• Conditional Debugging

• Packet Tracer

• Embedded Packet Capture



RP1 (in slots “r0”&“r1”)

ESP10 (in slots

“f0” & “f1”)

SIP10

SPAs


SP

A C

arr

ier

Ca

rd

SPA SPA

IOCP Marmot

…

Scooby

SPA-SPI

SPI4.2

Route Processor

(active)

RP

Scooby

HT-DP

Route Processor (standby)

RP

Scooby

HT-DP

ESI

SP

A C

arr

ier

Ca

rd

SPA SPA

IOCP Marmot

…

Scooby

SPA-SPI

SPI4.2

ESI

SP

A C

arr

ier

Ca

rd

SPA SPA

IOCP Marmot

…

Scooby

SPA-SPI

SPI4.2

ESI

Forwarding Processor (active)

FECP

HT-DP

Scooby

QFP subsystem Crypto assist

Fwding engine

Scooby

Forwarding Processor (standby)

FECP

HT-DP

Scooby

QFP subsystem Crypto assist

Fwding engine

Scooby

11.5Gbps 11.5Gbps 11.5Gbps

11.5Gbps

11.5Gbps 11.5Gbps 11.5Gbps

11.5Gbps 11.5Gbps

Other (e.g. CPP client IPC)

Punt/Inject/ctl pkts

Network pkts

HT-DP – DMA pkt protocol over HT

State sync pkts

Other pkts


• MCP – Midrange Convergence Platform

Initial name for the ASR1k project, replacement platform for C7200 / C7300 / C10K routers

• ESP (aka FP) – Embedded Services Processor (or Forwarding Processor)

Board that integrates QFP subsystem, hardware crypto engine (Nitrox II in classic ASR1k models), control processor in classic models (FECP), TCAM, interconnect ASICs, DRAM, etc.

• QFP – Quantum Flow Processor (aka CPP - Cisco Packet Processor)

Forwarding engine that integrates PPE matrix, BQS ASIC, packet buffers, etc.

• PPE – Packet Processing Element

Processor core that implements ASR1k datapath


• FECP – Forwarding Engine Control Processor

Control processor for ESP

• RP – Route Processor

Implements control plane and handles legacy protocols

• IOSd – IOS daemon

IOS code running on RP under Linux (linux_iosd_image RP process)

• BQS – Buffering, Queuing and Scheduling ASIC

Data plane QoS ASIC

• SIP (or CC) – SPA Interface Processor or Carrier Card

• SPA – Shared Port Adapter

• IOCP – I/O Control Processor


http://www.cisco.com/cdc_content_elements/flash/netsol/sp/quantum_flow/demo.html

http://www.cisco.com/cdc_content_elements/flash/netsol/sp/quantum_flow/demo.html


show platform hardware slot ?

0 SPA-Inter-Processor slot 0



F0 Embedded-Service-Processor slot 0


P0 Power-Supply slot 0

P1 Power-Supply slot 1

R0 Route-Processor slot 0


show platform hardware qfp ?

active Active instance

standby Standby instance

show platform software ipsec ?



FP Embedded-Service-Processor



RP Route-Processor

show platform software ipsec fp ?

active Active instance

standby Standby instance


• First generation ASR1000 routers: ASR1000 (ESP5, ESP10, ESP20, ESP40; RP1/RP2), ASR1001

asr1000rp1-advipservicesk9.03.13.02.S.154-3.S2-ext.bin


asr1001-universalk9.03.13.02.S.154-3.S2-ext.bin

• Second generation ASR1000 routers: ASR1000 (ESP100, ESP200), ASR1001-X, ASR1002-X

asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin

asr1002x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin


IOS-XE Version

IOS Version

Extended Lifetime Release

Platform

RP

Feature Set


• Virtual router: CSR1000V

csr1000v-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin

• New generation ISR routers: ISR4300 (ISR4351, ISR4331, ISR4321), ISR4400 (ISR4451, ISR4431)

isr4300-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin

isr4400-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin

• Routers for mobile backhaul: ASR900, ASR903, ASR920


IOS-XE Platforms Family

ISR

ISR4400

ISR4300

ASR1K

(1001/1001-X/1002-X/1004/1006/1013)

CPP10/10+

Cavium Nitrox II

Yoda / Luke

Cavium Octeon

CSR

(Ultra)

VMware

XEN

Hyper V

ESP10 & ESP20 – CPP10 ASIC

ESP40 – CPP10+ ASIC

ESP100 & ESP200 – 2x or 4x Yoda ASIC

ASR1002-X – Yoda ASIC

ASR1001-X – Luke ASIC

ISR4400 – Octeon processor

ISR4300 – RP cores

Data path implementation


Embedded Services Processor

Route Processor (RP)

SPA Interface Processor

Control Messaging

Linux Kernel Linux Kernel

Linux Kernel

QFP Client/Driver

Chassis Manager

Forwarding Manager

SPA Driver

SPA Driver

SPA Driver

SPA Driver

IOS

(Standby)

Forwarding Manager

Chassis Manager

IOS

(Active)

IOS-XE Platform Abstraction Layer (PAL)

Chassis Manager

• IOS-XE (BinOS) – Linux OS running multiple processes

• IOS runs as its own Linux process

• IOS-XE design goals:

Modularity

Preemptive scheduling of processes

Fault isolation and containment via memory protection

Software infrastructure designed for high availability

Operational consistency – same look and feel as IOS router

Rapid feature development and built-in development and diagnostic tools


ESP FECP

Interconn.

Crypto assist

RP Chassis Mgr.

Forwarding Mgr.

Chassis Mgr.

Forwarding Mgr.

QFP Client / Driver

Interconn.

Interconn.

SIP

SPA SPA

IOCP

SPA Agg.

…

Interconn.

Kernel (incl. utilities)

Chassis Mgr. SPA driver

SPA driver

SPA driver

SPA driver

IOSd





QFP subsystem

QFP microcode

• Runs Control Plane

• Generates configurations

• Populates and maintains routing tables (RIB, FIB…)

• Implements forwarding plane for all features

• Executes egress QoS in hardware

• Communicates with Forwarding manager on RP

• Provides interface to QFP Client / Driver

• Maintains copy of FIB

• Programs QFP forwarding plane and QFP DRAM

• Statistics collection and communication to RP

• Process scheduling, memory management, interrupts

• Suite of low-level applications (OBFL, debugging...)

• Provides IPC to other system components

• Provides abstraction layer between hardware and IOS

• Manages ESP redundancy

• Maintains copy of FIB and interface list

• Communicates FIB status to active & standby ESP (or

bulk-download state info in case of restart)


• IOSd is a user-level process scheduled by the Linux kernel

• IOSd runs in a protected address space so it is isolated from other components on the RP

• IOSd preserves the run-to-completion scheduler model for IOS processes, but IOSd itself can be preempted by the Linux scheduler

• Internally, IOSd provides an IOS environment controlled by the traditional IOS process scheduler

• IOSd consists of several pthreads:

IOS processes (BGP, OSPF, etc.) run in the main IOS thread

Fastpath IOS thread handles punted packets and IPC messages


• IOSd has no direct access to any hardware

• IOSd interacts with the rest of the system through platform-dependent shims but all of the hardware-specific processing occurs in other modules

• The shims communicate with the other processes running on the RP via IPC messages and via regions of shared memory with per-process access controls

• IOSd has access to an isolated “container” filesystem, which is within the Linux filesystem space. IOSd views this filesystem as the root (“/”) directory and has no means to climb “higher” in the path


• IOSd is responsible for processing of: Locally-addressed packets

Legacy protocol packets

Exception packets (e.g. packets with Router Alert IP option)

Glean packets (e.g. when ARP request needs to be sent)

• IOSd does not execute any code in the context of an interrupt handler or at interrupt level

• When a packet is sent to the RP, the interconnect ASIC generates an interrupt which is handled by a Linux kernel driver

• The driver sends an event to the IOSd punt path handler which is implemented within IOSd as a high priority fastpath thread

• If the IOSd process is blocked waiting for an event, it is marked as runnable and scheduled by the Linux


• So, the punt path handler in IOSd is the replacement for the interrupt handler in IOS

• Packets are received and transmitted by IOS from a virtual ring-based packet interface

show platform software infrastructure lsmpi

...

Lsmpi0 is up, line protocol is up

Hardware is LSMPI

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Unknown, Unknown, media type is unknown media type

...

Input queue: 0/1500/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

22373606 packets input, 0 bytes, 0 no buffer

...

1276902 packets output, 119357659 bytes, 0 underruns

...

Linux Shared Memory Punt Interface

LSMPI a module in Linux kernel to support

zero-copy transfer of packets between the

IOSd and QFP using Linux memory mapping


• If the packet cannot be forwarded in the IOSd fast path, it gets punted in the usual IOS manner to an IOS process for process switching

• Remember that most transit traffic is processed by QFP running its own code and IOSd doesn’t see it

• Although statistics is updated in IOSd via IPC messages, e.g.:

• But statistics for process-switched packets is not correct:

• CEF forwarding runs on QFP and this statistics is always zero:

show interfaces

show interfaces summary

show interfaces stats

show interfaces switching

show ip cef switching statistics

show ip cef switching statistics feature


• In this test we send continuous ping (timeout 0) from telnet session opened to ASR1k (ESP10/RP1)

show platform software status control-processor brief

...

CPU Utilization

Slot CPU User System Nice Idle IRQ SIRQ IOwait

RP0 0 44.24 16.81 0.00 36.93 1.90 0.10 0.00

ESP0 0 2.30 18.40 0.00 79.30 0.00 0.00 0.00

ESP1 0 3.09 17.28 0.00 79.62 0.00 0.00 0.00

SIP0 0 1.70 1.00 0.00 97.30 0.00 0.00 0.00

Total RP CPU utilization


• This is an IOS interface to Linux ‘top’ tool

• It can display per-process CPU utilization for processes running on RP, FECP, IOCP

show platform software process slot r0 monitor cycles 10 interval 5 lines 10

top - 00:06:30 up 10 days, 7:44, 0 users, load average: 0.25, 0.17, 0.06

Tasks: 152 total, 3 running, 149 sleeping, 0 stopped, 0 zombie

Cpu(s): 3.3%us, 3.3%sy, 0.0%ni, 93.2%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st

Mem: 2009376k total, 1874704k used, 134672k free, 144276k buffers

Swap: 0k total, 0k used, 0k free, 1055620k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

3223 root 20 0 979m 552m 208m S 51.6 28.1 370:39.81 linux_iosd-imag

8201 root 15 -5 0 0 0 S 1.9 0.0 3:53.05 lsmpi-xmit

8202 root 15 -5 0 0 0 R 1.9 0.0 4:17.45 lsmpi-rx

This statistics is not correct

show platform software process slot {f0 | f1 | fp active | r0 | r1 | rp active | 0 | 1 | 2} ...

IOSd process


• CPU utilization inside IOSd process (16 + 19.75 + 9.43 = 45)

show proc cpu sorted 1m | ex _0.00%_

CPU utilization for five seconds: 45%/16%; one minute: 32%; five minutes: 16%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

614 28167 141868 198 19.75% 13.89% 6.72% 2 Virtual Exec

114 295382 5653468 52 9.43% 6.20% 3.01% 0 IOSXE-RP Punt Se

15 1101101 6322367 174 0.15% 0.08% 0.08% 0 ARP Input

68 661399 3599770 183 0.07% 0.07% 0.08% 0 IOSD ipc task

Total utilization Fastpath thread utilization. The thread handles

punted packets and IPC messages

Utilization due to processes running

within the main IOS thread

“IOSXE-RP Punt Service Process” is the process

that handles IPv4 punt queue inside IOSd, analyzes

“punt cause” in the punt header and enqueues the

packet into the respective IOS process queue.

We also have “IOSXE-RP Punt IPV6 Service Process”.


Embedded Services Processor

Route Processor (RP)

SPA Interface Processor

Control Messaging

Linux Kernel Linux Kernel

Linux Kernel

QFP Client/Driver

Chassis Manager

Forwarding Manager

SPA Driver

SPA Driver

SPA Driver

SPA Driver

IOS

(Standby)

Forwarding Manager

Chassis Manager

IOS

(Active)

IOS-XE Platform Abstraction Layer (PAL)

Chassis Manager

• RP processes

Chassis Manager (cmand)

Host Manager (hman)

Forwarding Manager (fman_rp)

Interface Manager (imand)

Shell Manager (smand)

Logging Manager (plogd)

• FP processes

Chassis Manager (cman_fp)

Forwarding Manager (fman_fp_image)

Logging Manager (plogd)

QFP Client Control Process (cpp_cp_svr)

QFP Client Service Process (cpp_sp_svr)

QFP Driver Process (cpp_driver)

show platform software process list {rp |

fp} active [sort memory]


• Each software layer has its own diagnostic commands, but most of them are only used by TAC and development team

! IOS layer

{show | debug} crypto ...

! IOSd shim layer

{show | debug} platform software ipsec ...

! FMAN-RP layer

show platform software ipsec rp active ...

! FMAN-FP layer

show platform software ipsec fp active ...

! CPP client layer

{show | debug} platform hardware qfp active feature ipsec ...

! CPP µcode (datapath)

{show | debug} platform hardware qfp active feature ipsec datapath ...

! Crypto hardware (only “statistics” is available on ISR4k routers)

show platform hardware crypto-device ...


• IPSec SA at different software layers

• IOS layer (PI)

show crypto ipsec sa | i interface|ident|esp|spi|flow

interface: Tunnel1

local ident (addr/mask/prot/port): (192.168.1.1/255.255.255.255/47/0)

remote ident (addr/mask/prot/port): (192.168.2.2/255.255.255.255/47/0)

current outbound spi: 0x6721A788(1730258824)

inbound esp sas:

spi: 0x9E6410A3(2657357987)

transform: esp-aes esp-sha-hmac ,

conn id: 2003, flow_id: HW:3, sibling_flags 80004008, crypto map: Tunnel1-head-0

outbound esp sas:

spi: 0x6721A788(1730258824)

transform: esp-aes esp-sha-hmac ,

conn id: 2004, flow_id: HW:4, sibling_flags 80004008, crypto map: Tunnel1-head-0



• FMAN-FP layer (PD)

show platform software ipsec fp active flow id 3

=========== Flow id: 3

mode: transport

direction: inbound

protocol: esp

SPI: 0x9e6410a3

local IP addr: 192.168.1.1

remote IP addr: 192.168.2.2

crypto device id: 0

crypto map id: 1

SPD id: 1

ACE line number: 1

QFP SA handle: 5

IOS XE interface id: 19

interface name: Tunnel1

Crypto SA ctx id: 0x000000002e03bffd

cipher: AES-128

auth: SHA1

...

...

show platform software ipsec fp active flow id 4

=========== Flow id: 4

mode: transport

direction: outbound

protocol: esp

SPI: 0x6721a788

local IP addr: 192.168.1.1

remote IP addr: 192.168.2.2

crypto device id: 0

crypto map id: 1

SPD id: 1

ACE line number: 1

QFP SA handle: 6

IOS XE interface id: 19

interface name: Tunnel1

use path MTU: 1500

Crypto SA ctx id: 0x000000002e03bffc

cipher: AES-128

auth: SHA1

...



• CPP Client layer (PD)

show platform hardware qfp active feature ipsec sa 5

QFP ipsec sa Information

QFP sa id: 5

pal sa id: 3

QFP spd id: 1

QFP sp id: 2

QFP spi: 0x9e6410a3(2147483647)

crypto ctx: 0x000000002e03bffd

flags: 0xc000800 (Details below)

: src:IKE valid:True soft-life-expired:False hard-life-expired:False

: replay-check:True proto:0 mode:0 direction:0

: qos_preclassify:False qos_group:False

: frag_type:BEFORE_ENCRYPT df_bit_type:COPY

: sar_enable:False getvpn_mode:SNDRCV_SA

: doing_translation:False assigned_outside_rport:False

: inline_tagging_enabled:False

...

Inbound IPsec SA, which means that anti-replay check is

important, but fragmentation type (before/after encryption),

or QoS pre-classify is not.



• CPP Client layer (PD)

show platform hardware qfp active feature ipsec sa 6

QFP ipsec sa Information

QFP sa id: 6

pal sa id: 4

QFP spd id: 1

QFP sp id: 2

QFP spi: 0x6721a788(1730258824)

crypto ctx: 0x000000002e03bffc

flags: 0x4240040 (Details below)

: src:IKE valid:Yes soft-life-expired:No hard-life-expired:No

: replay-check:No proto:0 mode:0 direction:1

: qos_preclassify:No qos_group:No

: frag_type:AFTER_ENCRYPT df_bit_type:COPY

: sar_enable:No getvpn_mode:SNDRCV_SA

: doing_translation:No assigned_outside_rport:No

: inline_tagging_enabled:No

...

Outbound IPSec SA, which means that frag_type is important,

but anti-replay check is not. We always fragment after encryption

if “tunnel protection ipsec profile …” is applied to the tunnel,

hence always configure “ip mtu” on mGRE interfaces (for p2p

GRE system can set it automatically as of CSCtq09372 fix).



• ASR1k crypto hardware layer (PD)

show platform software ipsec fp active encryption-processor 0 context 2e03bffd

show platform software ipsec fp active encryption-processor 0 context 2e03bffc


• In XE3.7 several handy macro commands were introduced to make troubleshooting of IPSec control plane easier

show crypto ipsec sa peer 10.48.67.149 platform | i --- show

------------------ show platform software ipsec fp active flow identifier 19

------------------ show platform hardware qfp active feature ipsec sa 7

------------------ show platform software ipsec fp active encryption-processor 0 context 2e03bfed



------------------ show platform software ipsec fp active encryption-processor 0 context 2dc3bfec

show crypto ipsec sa interface tunnel1 platform | i --- show

------------------ show platform software ipsec fp active interface name Tunnel1

------------------ show platform hardware qfp active feature ipsec interface Tunnel1



------------------ show platform software ipsec fp active encryption-processor 0 context 2e03bfdd



------------------ show platform software ipsec fp active encryption-processor 0 context 2e03bfdc

! Use with caution, because the output can be huge in a scaled setup!

show tech-support ipsec platform


• Here we send “show tech” output to FTP server

show tech | redirect ftp://<ip>/<file>.txt

show processes cpu sorted 5sec | ex _0.00%_

CPU utilization for five seconds: 14%/0%; one minute: 7%; five minutes: 2%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

614 16392 127450 128 9.57% 3.99% 0.93% 3 Virtual Exec

612 1132 16114 70 2.59% 1.27% 0.28% 3 FTP Write Proces

613 2056 7633 269 1.21% 0.09% 0.02% 2 Virtual Exec

show platform software process slot r0 monitor cycles 10 interval 5 lines 10

...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

5800 root 20 0 145m 132m 7608 R 54.4 6.7 3:13.29 smand

3263 root 20 0 979m 543m 205m S 21.4 27.7 20:58.75 linux_iosd-imag

2217 root 20 0 47980 20m 5800 S 13.6 1.0 14:21.85 hman


...

CPU Utilization


RP0 0 84.59 15.00 0.00 0.00 0.19 0.19 0.00


• In customer case we observed that IPSec SVTI tunnels may go down on ASR1k (RP1) when “show tech” is copied to external FTP server, if periodic DPD is configured with aggressive 10/3 timers on several hundred spokes and on the ASR

show platform resources slot r0

**State Acronym: H - Healthy, W - Warning, C - Critical

Resource Usage Max Warning Critical State

-----------------------------------------------------------------------

RP0 (ok, active) C

Control Processor 100.00% 100% 90% 95% C

DRAM 1813MB(92%) 1962MB 90% 95% W

...

show processes cpu platform sorted 5sec location r0 | ex _0%_

CPU utilization for five seconds: 99%, one minute: 26%, five minutes: 10%

Pid PPid 5Sec 1Min 5Min Status Size Name

--------------------------------------------------------------------------------

5800 4756 59% 6% 1% R 152535040 smand

3263 2650 13% 10% 4% S 1027596288 linux_iosd-imag

2217 997 4% 1% 1% R 49135616 hman


• ASR1k RP and FECP memory utilization

• Linux memory management is complicated…

• The “free” memory includes “cached” memory which can be reused, so low “free” doesn’t mean that the system memory is low

• Refer to ASR1k Troubleshooting TechNotes and CSCuc40262

http://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-tech-notes-list.html


...

Memory (kB)

Slot Status Total Used (Pct) Free (Pct) Committed (Pct)

RP0 Healthy 2009376 1873508 (93%) 135868 ( 7%) 1553268 (77%)

ESP0 Healthy 2009400 702804 (35%) 1306596 (65%) 490840 (24%)

ESP1 Healthy 2009400 693428 (35%) 1315972 (65%) 491144 (24%)

SIP0 Healthy 471804 318548 (68%) 153256 (32%) 245744 (52%)

The “committed” is the sum of all malloc().

This doesn’t mean that all this memory was

really allocated… “Committed” can be

more than 100%.




















• QFP datapath utilization reflects how many PPE's/threads are busy with packets at a given point of time

• Calculated as an exponentially damped moving average

• Output collected on a very busy BRAS router doing NAT (ESP40)

show platform hardware qfp active datapath utilization

CPP 0: Subdev 0 5 secs 1 min 5 min 60 min

Input: Priority (pps) 939 931 977 806

(bps) 2888288 2953600 3122040 1787376

Non-Priority (pps) 1601727 1606945 1586457 1541474

(bps) 10671107208 10668441928 10514528440 10342623728

Total (pps) 1602666 1607876 1587434 1542280

(bps) 10673995496 10671395528 10517650480 10344411104

Output: Priority (pps) 572 557 551 574

(bps) 380912 360048 353688 376280

Non-Priority (pps) 1550452 1555896 1535883 1490399

(bps) 10149855856 10148858160 9996408704 9819515880

Total (pps) 1551024 1556453 1536434 1490973

(bps) 10150236768 10149218208 9996762392 9819892160

Processing: Load (pct) 58 59 58 56


• QFP memory utilization

• Output collected on ASR1k ESP20 doing NAT (2.3M PAT translations)

show platform hardware qfp active infrastructure exmem statistics

QFP exmem statistics

Type: Name: DRAM, QFP: 0

Total: 1073741824

InUse: 793689088

Free: 280052736

Lowest free water mark: 208302080

Type: Name: IRAM, QFP: 0

Total: 134217728

InUse: 118105088

Free: 16112640


Type: Name: SRAM, QFP: 0

Total: 32768

InUse: 14848

Free: 17920


1GB PPE RLDRAM2 (RDRAM or Resource DRAM)

- NAT sessions

- NetFlow cache

- Firewall sessions / hash tables

- IPSec SA

- QoS marking / policing

128MB instruction RAM

- Used for QFP code (FIA array)

- Can also store data

32KB SRAM

- High speed traffic management functions

- E.g. virtual reassembly


• ASR1k QFP TCAM utilization

• ASR1k BQS resources (queues, etc.) and packet buffers

show platform hardware qfp active tcam resource usage

QFP TCAM Usage Information

...

Total TCAM Cell Usage Information

----------------------------------

Name : TCAM #0 on CPP #0

Total number of regions : 3

Total tcam used cell entries : 104332

Total tcam free cell entries : 944244

Threshold status : below critical limit

show platform hardware qfp active infrastructure bqs status

show platform hardware qfp active bqs 0 packet-buffer utilization

This means that everything is fine

Unavailable on ISR4k routers, because

they use software TCAM and CACE –

Cisco Adaptive Classification Engine

BQS ASIC is unavailable on ISR4k routers. QoS is implemented on a separate Octeon core.

Software QoS uses same control plane code as ASR1k BQS, except the hardware layer (RM).


• ISR4451: single control plane CPU – Intel Crystal Forest Gladden CPU 4C/8T @2.0MHz, universal data plane DDR3 memory

• QFP is emulated on Cavium Octeon 6645 (10 cores, one thread per core, 1 core runs QoS code)


Load Average

Slot Status 1-Min 5-Min 15-Min

RP0 Healthy 0.00 0.00 0.00

Memory (kB)

Slot Status Total Used (Pct) Free (Pct) Committed (Pct)

RP0 Healthy 3970904 3142812 (79%) 828092 (21%) 2384508 (60%)

CPU Utilization


RP0 0 1.80 1.40 0.00 96.30 0.00 0.50 0.00

1 4.80 0.90 0.00 94.29 0.00 0.00 0.00

2 0.20 4.80 0.00 95.00 0.00 0.00 0.00

3 0.80 3.70 0.00 95.49 0.00 0.00 0.00

4 0.70 0.70 0.00 98.59 0.00 0.00 0.00

5 0.20 1.20 0.00 98.59 0.00 0.00 0.00

6 1.60 1.40 0.00 97.00 0.00 0.00 0.00

7 4.09 0.89 0.00 95.00 0.00 0.00 0.00

show platform hardware qfp active

infrastructure exmem statistics

QFP exmem statistics

Type: Name: DRAM, QFP: 0

Total: 2147483648

InUse: 1713403904

Free: 434079744


Type: Name: IRAM, QFP: 0

Total: 0

InUse: 0

Free: 0


Type: Name: SRAM, QFP: 0

Total: 0

InUse: 0

Free: 0



• Integrated view of platform resources – XE3.13

show platform resources slot [f0 | f1 | r0 | r1 | 0 | ...]

**State Acronym: H - Healthy, W - Warning, C - Critical

Resource Usage Max Warning Critical State

--------------------------------------------------------------------------------------------------

RP0 (ok, active) W

Control Processor 6.30% 100% 90% 95% H

DRAM 1797MB(91%) 1962MB 90% 95% W

ESP0(ok, active) H


DRAM 657MB(33%) 1962MB 90% 95% H

QFP H

TCAM 14cells(0%) 131072cells 45% 55% H

DRAM 125263KB(23%) 524288KB 80% 90% H

IRAM 9941KB(7%) 131072KB 80% 90% H

ESP1(ok, standby) H


DRAM 669MB(34%) 1962MB 90% 95% H

QFP H

TCAM 14cells(0%) 131072cells 45% 55% H

DRAM 125263KB(23%) 524288KB 80% 90% H

IRAM 9941KB(7%) 131072KB 80% 90% H

SIP0 H


DRAM 293MB(63%) 460MB 90% 95% H


• New commands for CPU and memory monitoring – XE3.14

• CLI interface to Linux ‘top’ tool – XE3.14

show processes memory platform [sorted] location {rp active | fp active | r0 | r1 | f0 | f1 | 0 | 1

| 2 | ...}

show processes cpu platform [sorted [5sec | 1min | 5min]] location {rp active | fp active | r0 | r1

| f0 | f1 | 0 | 1 | 2 | ...}

show processes cpu platform monitor [cycles <N> [[interval <M>] [lines <K>]]] [location ...]



PPE

ASIC

BQS

ASIC

FECP

R0 R1

GE

EOBC

Serdes Serdes

SPI4.2 SPI

Mux Crypto SPI4.2

SPI4.2

SPI4.2

HT

Packet

Memory

128M

CC0 CC1 CC2 RP0 RP1 FP-stby

TCAM Resource

DRAM

DRAM

Data Path

ESI Links

Control

Path

PPE

ASIC + BQS

ASIC = QFP


• Implements data plane on PPEs

• Feature Invocation Array (FIA) determines feature ordering

show platform hardware qfp active interface if-name GigabitEthernet0/0/1.99

…

Protocol 0 - ipv4_input

FIA handle - CP:0x1091ed50 DP:0x8091f680

IPV4_INPUT_DST_LOOKUP_ISSUE (M)

IPV4_INPUT_ARL_SANITY (M)

IPV4_INPUT_DST_LOOKUP_CONSUME (M)

IPV4_INPUT_FOR_US_MARTIAN (M)

IPV4_INPUT_VFR

IPV4_NAT_INPUT_FIA

IPV4_INPUT_LOOKUP_PROCESS (M)

IPV4_INPUT_IPOPTIONS_PROCESS (M)

IPV4_INPUT_GOTO_OUTPUT_FEATURE (M)

Protocol 1 - ipv4_output

FIA handle - CP:0x1091ed1c DP:0x8091ff00

IPV4_OUTPUT_VFR

IPV4_NAT_OUTPUT_FIA

IPV4_OUTPUT_THREAT_DEFENSE

IPV4_VFR_REFRAG (M)

IPV4_OUTPUT_L2_REWRITE (M)

IPV4_OUTPUT_FRAG (M)

IPV4_OUTPUT_DROP_POLICY (M)

MARMOT_SPA_D_TRANSMIT_PKT

DEF_IF_DROP_FIA (M)

show run int g0/0/1.99

Current configuration : 115 bytes

!

interface GigabitEthernet0/0/1.99

encapsulation dot1Q 99

ip address 1.1.1.1 255.255.255.0

ip nat outside

End


• Feature processing order follows the 12.0S data path implementation

L2/L3 Classify

IPv4 Validation

Netflow

BGP Accounting

NBAR Classify

MQC Classify

LI

Firewall / IDS / Proxy

Security ACL

RPF

MQC Marking

MQC Policing

MAC Accounting

Prec. Accounting

NAT

PBR

WCCP

Server LB

Dialer IDLE Rst

URD

Firewall / CBAC

TCP Intercept

MQC Marking

IP Accounting

RSVP

MQC Policing

MAC Accounting

Prec Accounting

URDIP Frag

Netflow

Firewall / IDS / Proxy

WCCP

NAT

NBAR Classify

BGP Accounting

LI

Crypto

MQC Classify

FW ACL & Pregen Check

Security ACL

WRED

Queuing

F

F

F

F

F

Forwarding

• IP Unicast • Loadbalancing • IP Multicast • MPLS Imposit. • MPLS Dispos. • MPLS Switch. • FRR • AToM Dispos. • MPLSoGRE

IPv6 IPv4 MPLS XConnect L2 Switch


GPM & Packet Distribution / Gather

IPM HT i/f OPM

Pkt Memory

FECP

SERDES SERDES

On chip packet memory

CC0 CC1 CC2 RP0 RP1 FP-Stby

CRYPTO SPI Mux

Recycle

PPEs & HW Assists

PPE ASIC BQS ASIC

FE

…


• Frame is received and classified (‘hi’ / ‘lo’) by either SPA or SIP

• Frames are scheduled based on priority and sent to QFP over ESI ‘hi’ or ‘lo’ priority channel

• Entire L2 frame is received by QFP Input Packet Module (IPM) and stored in Global Packet Memory (GPM)

• A free PPE thread is assigned to process the packet

• Packet remains in on chip memory (GPM) while it is processed by one of the PPEs

• The PPE thread runs through a Feature Chain in software. It can access resources like the HW-assists and TCAM and perform deep packet inspection, e.g. NBAR


• When processed, the PPE thread releases the packet to the Traffic Manager and its own packet buffer for placement into an output queue for scheduling

• The Output Packet Module (OPM) pulls the selected packet for transmission

• The packet is either transmitted out a physical interface

• Or transmitted back to another PPE thread for further processing (Recycle Path)


• From OPM traffic can be sent to a SIP module, punted to RP, sent to crypto co-processor for encryption or decryption or recycled back to QFP

• This command displays default interface queues (QoS can create its own queues)

show platform hardware qfp active infrastructure bqs queue output default all | i Interface

Interface: internal0/0/recycle:0 QFP: 0.0 if_h: 1 Num Queues/Schedules: 0

Interface: internal0/0/rp:0 QFP: 0.0 if_h: 2 Num Queues/Schedules: 2


Interface: internal0/0/crypto:0 QFP: 0.0 if_h: 4 Num Queues/Schedules: 2

Interface: CPP_Null QFP: 0.0 if_h: 5 Num Queues/Schedules: 0

Interface: Null0 QFP: 0.0 if_h: 6 Num Queues/Schedules: 0

Interface: GigabitEthernet0/0/0 QFP: 0.0 if_h: 7 Num Queues/Schedules: 1





Interface: Loopback0 QFP: 0.0 if_h: 12 Num Queues/Schedules: 0

Interface: Tunnel1 QFP: 0.0 if_h: 17 Num Queues/Schedules: 0

Interface: GigabitEthernet0/0/1.75 QFP: 0.0 if_h: 18 Num Queues/Schedules: 0

Interface: Virtual-Template1 QFP: 0.0 if_h: 21 Num Queues/Schedules: 0

Interface: DmvpnSpoke16908304 QFP: 0.0 if_h: 22 Num Queues/Schedules: 0

RP and crypto chip have

two queues: ‘hi’ / ‘lo’.

There are many recycle

queues (see next slides).



• After PPE has finished processing a packet, it is gathered from the GPM and written to a queue in BQS

• The queue may be used to recycle the packet back to the GPM for further processing. E.g. fragmentation or reassembly

show platform hardware qfp active infrastructure bqs queue output recycle summary

Recycle Queue Summary Table (Total Recycle Queues: 73)

ID Name ParentID Prio Bandwidth RateType Mode Limit

=============================================================================================

0x0003 MulticastLeafHigh 0x0002 01 0 00 00 0

0x0004 MulticastLeafLow 0x0002 00 100 01 00 0

0x0005 L2MulticastLeafHigh 0x0002 01 0 00 00 0

0x0006 L2MulticastLeafLow 0x0002 00 100 01 00 0

0x0007 LSMMulticastLeafHigh 0x0002 01 0 00 00 0

0x0008 LSMMulticastLeafLow 0x0002 00 100 01 00 0

0x0009 SBCMMOHLeafHigh 0x0002 01 0 00 00 0

0x000a SBCMMOHLeafLow 0x0002 00 100 01 00 0

0x000b IPFragHi 0x0002 01 0 00 00 0

0x000c IPFragLo 0x0002 00 100 01 00 0

0x000d IPReassemblyHi 0x0002 01 0 00 00 0

0x000e IPReassemblyLo 0x0002 00 100 01 00 0

…


show platform hardware qfp active infrastructure bqs queue output recycle summary

Recycle Queue Summary Table (Total Recycle Queues: 73)

ID Name ParentID Prio Bandwidth RateType Mode Limit

=============================================================================================

…

0x000f IPv6ReassemblyHi 0x0002 01 0 00 00 0

0x0010 IPv6ReassemblyLo 0x0002 00 100 01 00 0

0x0011 IPv4vasi 0x0002 00 100 01 00 0

0x0012 IPv6vasi 0x0002 00 100 01 00 0

…

0x001e MulticastReplicationHigh 0x001d 01 0 00 00 0

0x001f MulticastReplicationLow 0x001d 00 100 01 00 0

…

0x003e ICMPRecycleQ 0x0037 00 100 01 00 0

…

0x0042 FwallRecycleHi 0x0037 01 0 00 00 0

0x0043 FwallRecycleLo 0x0037 00 100 01 00 0

…

0x0047 SSLVPNRecycleQ 0x0037 01 100 01 00 0

0x0048 TcpRecycle 0x0037 01 100 01 00 0

…

0x0057 MetaPkt_Hi 0x0056 01 0 00 00 0

0x0058 MetaPkt_Lo 0x0056 00 100 01 00 0


• Statistics is available for recycle queues

show platform hardware qfp active infrastructure bqs queue output recycle id 12

Recycle Queue Object ID:0xc Name:IPFragLo (Parent Object ID: 0x2)

plevel: 0, bandwidth: 100 , rate_type: 1

queue_mode: 0, queue_limit: 0, num_queues: 1

Queue specifics:

Index 0 (Queue ID:0x11, Name: IPFragLo)

Software Control Info:

(cache) queue id: 0x00000011, wred: 0x88b160f0, qlimit (pkts ): 8192

parent_sid: 0x208, debug_name: IPFragLo

sw_flags: 0x00010001, sw_state: 0x00000c01, port_uidb: 0

orig_min : 0 , min: 0

min_qos : 0 , min_dflt: 0

orig_max : 0 , max: 0

max_qos : 0 , max_dflt: 0

share : 1

plevel : 0, priority: 65535

defer_obj_refcnt: 0

Statistics:

tail drops (bytes): 0 , (packets): 0

total enqs (bytes): 79591976 , (packets): 379948

queue_depth (pkts ): 0

show platform hardware qfp active infrastructure bqs queue output recycle {all | id <number>}

This is a bug CSCut83283.

We increment a counter for each and every packet that

needs to be encrypted on a tunnel interface with tunnel

protection applied, even if the packet is small. This is a

counter issue. Packets are sent to IPFragLo(Hi) recycle

queue only if they need be fragmented.

“all” gives incomplete info – bug CSCub11524



• Mechanism to send a packet from QFP to either RP, or (back to) QFP for further processing

• Why punt to RP? Basically this is where all the packets QFP can’t process go: control plane protocols, traffic to router IP, legacy protocols

• Why punt to (back to) QFP? This is analogous to RP injecting a packet to QFP. For example, ICMP echo request/response. When QFP receives an echo request, it will create the echo reply packet and use the Punt/Inject path to transmit the packet


QFP

LSMPI/

IOS-shim IOS process

QFP

Punt packet to RP

Punt packet back to QFP

1

2

3

1 2 1. Receive pkt from network

2. Packet marked for punting

to RP. Transmit packet out

Packet is processed by PD

LSMPI/IOS-shim and sent

to IOS PI for processing

1. Receive pkt from network


to QFP. Packet is formatted

w/ an inject header and recycled

back to QFP.

3. QFP internal interface FIA processes

packet and packet will be transmitted

out appropriate physical interface.


• Mechanism for RP (or QFP) to transmit packets out of ASR1k. RP will inject packets to QFP for transmission

• Injects from RP: There’s a few flavors. We can break these down into either fully formatted packets (ie: L2+L3+payload) or L3 packets (ie: IP, IPv6, MPLS)

• Injects from QFP? Ditto what we said w/ punt… A feature needs to transmit a new (generated) packet out. The feature uses the CPP inject path to route and transmit the packet


QFP

IOS-shim

IOS

process

QFP

Inject packet from RP

Inject packet from QFP

1

2

3

1 2 1. IOS PI sends packet via IOS-shim

IOS-shim formats the CPP inject

headers

2. Inject infra processes inject header

QFP internal interface FIA processes



1. Receive pkt from network


to QFP. Packet is formatted

w/ an inject header and recycled

back to QFP.

3. Inject infra processes inject header

QFP internal interface FIA processes




• Punt/Inject to/from RP is easy to understand…

• Punt/Inject to/from QFP is complicated…

• Example: Single ICMP Ping to the router IP:

show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_

Per Punt Cause Statistics

Packets Packets

Counter ID Punt Cause Name Received Transmitted

--------------------------------------------------------------------------------------

026 QFP ICMP generated packet 1 1

Per Inject Cause Statistics

Packets Packets

Counter ID Inject Cause Name Received Transmitted

--------------------------------------------------------------------------------------



• Router received 1 echo request and generated 1 reply, but, as you can see, three packets were captured by PACTRAC

show platform packet-trace statistics

Packets Summary

Matched 3

Traced 3

Packets Received

Ingress 2

Inject 1

Count Code Cause

1 9 QFP ICMP generated packet

Packets Processed

Forward 1

Punt 1

Count Code Cause

1 26 QFP ICMP generated packet

Drop 0

Consume 1

show platform packet-trace summary

Pkt Input Output State Reason

0 Gi0/0/1 Gi0/0/1 CONS Packet Consumed

1 Gi0/0/1 internal0/0/recycle:0 PUNT 26 (QFP ICMP generated packet)

2 INJ.9 Gi0/0/1 FWD

0: ICMP Echo Request

1, 2: ICMP Echo Reply


• There are many commands for Punt Path troubleshooting

• Major punt statistics

show platform software infrastructure punt

...

IOSXE-RP Punt packet causes:

1874682 Layer2 control and legacy packets

1918031 ARP request or response packets

57 Reverse ARP request or repsonse packets

64429 For-us data packets

125191 RP<->QFP keepalive packets

2 Glean adjacency packets

7856 Subscriber session control packets

1577645 For-us control packets

268613 IP subnet or broadcast packet packets

FOR_US Control IPv4 protcol stats:

19101 TCP packets

228855 UDP packets

2505 GRE packets

58177 EIGRP packets

1252125 OSPF packets

16882 PIM packets

...


• Aggregated punt statistics for RP0 low and high priority queues

show platform hardware qfp active infrastructure bqs queue out default interface-string internal0/0/rp:0


Queue specifics:

Index 0 (Queue ID:0x86, Name: i2l_if_2_cpp_0_prio0)


(cache) queue id: 0x00000086, wred: 0x88b16862, qlimit (bytes): 6250048

parent_sid: 0x25c, debug_name: i2l_if_2_cpp_0_prio0

...

Statistics:



queue_depth (bytes): 0

Queue specifics:

Index 1 (Queue ID:0x87, Name: i2l_if_2_cpp_0_prio1)


(cache) queue id: 0x00000087, wred: 0x88b16872, qlimit (bytes): 6250048

parent_sid: 0x25c, debug_name: i2l_if_2_cpp_0_prio1

...

Statistics:



queue_depth (bytes): 0


• Per-cause punt/inject statistics

show platform hardware qfp active infrastructure punt statistic type per-cause | ex _0_

Global Per Cause Statistics

Number of punt causes = 106


Packets Packets


--------------------------------------------------------------------------------------

003 Layer2 control and legacy 1877032 1876909

007 ARP request or response 1977106 1920808

008 Reverse ARP request or repsonse 57 57

011 For-us data 64519 64519

021 RP<->QFP keepalive 125351 125351

024 Glean adjacency 2 2


027 Subscriber session control 7867 7866

055 For-us control 1615501 1579662

060 IP subnet or broadcast packet 268677 268677



• Используете ли вы маршрутизаторы, работающие под управлением IOS-XE, и для чего?

Для BGP, как граничный роутер моей AS

Как PE для организации MPLS VPN

Как Internet Gateway для выполнения NAT

Для Broadband Aggregation (BRAS)

В качестве Cisco Unified Border Element (CUBE)

Для организации Site-to-Site VPN

Для организации Remote Access VPN

В качестве Firewall

Для Mobile Backhaul

Использую также, как и маршрутизаторы ISR G2, для решения различных мелких задач

Для обогрева серверной комнаты



• System-wide conditions can be used by Packet Tracer tool for data path troubleshooting and by various features to limit the scope of the debug

• In this presentation we will not talk about feature debugs

• Implemented in XE3.10

• http://www.cisco.com/c/en/us/td/docs/routers/asr1000/troubleshooting/guide/Tblshooting-xe-3s-asr-1000-book.html

http://www.cisco.com/c/en/us/td/docs/routers/asr1000/troubleshooting/guide/Tblshooting-xe-3s-asr-1000-book.html















• Conditional Debug configuration

• Global and interface conditions cannot be enabled simultaneously

• Special interfaces: Internal-RP Dataplane Punt/Inject interface

Internal-Recycle Dataplane Recycle interface

• The “<ipv4-addr[/mask]>” condition matches traffic bi-directionally

• The “access-list <name>” condition is unidirectional

debug platform condition [interface <name>] ipv4 [access-list <name> | <ipv4-addr>[/mask]] {ingress

| egress | both}

debug platform condition [interface <name>] ipv6 [access-list <name> | <ipv6-addr>[/mask]] {ingress

| egress | both}

debug platform condition [interface <name>] mpls [<label-ID>] {ingress | egress | both}

debug platform condition {ingress | egress | both}


• Ingress Conditional Debug in the packet processing path

• Egress Conditional Debug in the packet processing path

show platform hardware qfp active interface if-name <interface-name>

...

Protocol 0 - ipv4_input

FIA handle - CP:0x1091f05c DP:0x80917700

IPV4_INPUT_DST_LOOKUP_ISSUE (M)

IPV4_INPUT_ARL_SANITY (M)

CBUG_INPUT_FIA

DEBUG_COND_INPUT_PKT

...

show platform hardware qfp active interface if-name <interface-name>

...


FIA handle - CP:0x108db890 DP:0x80791c80

CBUG_OUTPUT_FIA

IPV4_OUTPUT_VFR

IPV4_OUTPUT_NAT


IPV4_VFR_REFRAG (M)




DEBUG_COND_OUTPUT_PKT


DEF_IF_DROP_FIA (M)

Conditional Debug also notifies

Packet Tracer on “match”

Packet Tracer packet copy


• This command displays all configured conditions

• “Show debug” includes above output

show platform conditions

Conditional Debug Global State: Start

Conditions Direction

------------------------------------------------------------------------------------|---------

GigabitEthernet0/0/1.75 & IPV4 ACL [145] ingress

GigabitEthernet0/0/1.99 & IPV4 ACL [144] ingress

Feature Condition Type Value

-----------------------|-----------------------|--------------------------------

Feature Type Submode Level

------------|-------------|---------------------------------------------------------|----------

show debug


• Conditions can be removed or cleared

• Next command doesn’t clear conditions, but it stops all debugs including conditional debug

• Next command starts/stops conditional debug

• Without conditions it enables debug for all packets

no debug platform condition <exact command needs to be entered here>

clear platform condition all

debug platform condition {start | stop}

no debug all




• XE3.11 – Drop Tracing support

• XE3.11 – Recycle Enhancements

• XE3.11 – "decode" Option

• XE3.12 – CSCug38748 – PACTRAC: packet-trace summary output should print timestamp in datetime

• XE3.13 – Punt/Inject Tracing

• XE3.13 – VASI support

• http://www.cisco.com/c/en/us/td/docs/routers/asr1000/configuration/guide/chassis/asrswcfg/Packet_Trace.html

http://www.cisco.com/c/en/us/td/docs/routers/asr1000/configuration/guide/chassis/asrswcfg/Packet_Trace.html




• This example provides a quick overview of using Packet Tracer with a simple IPv4 address condition

! Step1: Define a condition

debug platform condition ipv4 address 172.27.1.1/32 ingress

! Step2: Enable Packet Tracer

debug platform packet-trace packet 2048

debug platform packet-trace enable

! Step3: Start Conditional Debugging (this also starts Packet Tracer)

debug platform condition start

! Step4: Display Packet Tracer configuration, accounting and summary data

show platform packet-trace configuration



! Step5: Stop Conditional Debugging (this also stops Packet Tracer)

debug platform condition stop

! Step6: Clear all information collected by Packet Tracer (optional, requires “stop”)

clear platform packet-trace statistics

! Step7: Clear Packet Trace configuration

clear platform packet-trace configuration


• This example illustrates how to use FIA trace to understand where certain features live in the packet processing path

policy-map inner

class Prec5

priority percent 20

class Prec3

bandwidth percent 50

policy-map outer

class class-default

shape average 32000

service-policy inner

interface Tunnel0

nhrp map group TEST service-policy output outer

tunnel source GigabitEthernet0/0/2

tunnel mode gre multipoint

tunnel protection ipsec profile prof1

…


• Conditional Debug

• Packet Tracer

access-list 166 permit ip host 192.168.1.1 host 192.168.2.2

debug platform condition interface tunnel0 ipv4 access-list 166 egress


Conditional Debug Global State: Stop


------------------------------------------------------------------------------------|---------

Tunnel0 & IPV4 ACL [166] egress

debug platform packet-trace packet 256 fia-trace


debug platform condition start


• After sending 100 continuous pings (timeout 0) we see that 35 packets were dropped by QoS

show policy-map multipoint Tunnel0

Interface Tunnel0 <--> 1.1.1.2

Service-policy output: outer

Class-map: class-default (match-any)

166 packets, 106384 bytes

5 minute offered rate 0000 bps, drop rate 0000 bps

Match: any

Queueing

queue limit 64 packets

(queue depth/total drops/no-buffer drops) 0/35/0

...

show platform hardware qfp active statistics drop

-------------------------------------------------------------------------

Global Drop Stats Packets Octets

-------------------------------------------------------------------------

TailDrop 35 37790


• Accounting info (statistics)

• Summary info


Packets Summary

Matched 100

Traced 100

Packets Received

Ingress 100

Inject 0

Packets Processed

Forward 65

Punt 0

Drop 35

Count Code Cause

35 22 TailDrop

Consume 0



0 Gi0/0/0.27 Gi0/0/2 FWD

...

64 Gi0/0/0.27 Gi0/0/2 FWD

65 Gi0/0/0.27 Gi0/0/2 DROP 22 (TailDrop)

...



• Path info for forwarded packet #64 (part 1) show platform packet-trace packet 64

Packet: 64 CBUG ID: 64

Summary

Input : GigabitEthernet0/0/0.27

Output : GigabitEthernet0/0/2

State : FWD

Timestamp

Start : 1398207324379 ns (01/19/2000 04:49:22.995458 UTC)

Stop : 1398207470896 ns (01/19/2000 04:49:22.995604 UTC)

Path Trace

Feature: IPV4

Source : 192.168.1.1

Destination : 192.168.2.2

Protocol : 1 (ICMP)

...

Feature: FIA_TRACE

Entry : 0x8200ed80 - IPV4_OUTPUT_QOS

Lapsed time: 3164 ns

...

Feature: FIA_TRACE

Entry : 0x80128400 - IPV4_OUTPUT_TUNNEL_PROTECTION_ENCRYPT

Lapsed time: 657 ns

Feature: IPSec

Result : IPSEC_RESULT_SA

Action : ENCRYPT

SA Handle : 4

Peer Addr : 1.1.1.2

Local Addr: 1.1.1.1

...

Lapsed time is displayed for each FIA element.

Can be used for datapath profiling!

QoS classification

(output FIA of interface tunnel)

Tunnel protection


We leave tunnel output FIA

at this point and the packet

is sent to crypto engine for

encryption


• Path info for forwarded packet #64 (part 2)

• The packet is received from crypto engine and the processing continues

...

Feature: FIA_TRACE

Entry : 0x80424e18 - IPV4_IPSEC_FEATURE_RETURN

Lapsed time: 497 ns

Feature: FIA_TRACE

Entry : 0x80126c3c - IPV4_TUNNEL_GOTO_OUTPUT


...

Feature: FIA_TRACE

Entry : 0x8062fc68 - IPV4_TUNNEL_ENCAP_GOTO_OUTPUT_FEATURE


...

Feature: FIA_TRACE

Entry : 0x8200e480 - IPV4_OUTPUT_DROP_POLICY


Feature: FIA_TRACE

Entry : 0x82016c80 - MARMOT_SPA_D_TRANSMIT_PKT


We enter egress physical interface

output FIA at this point

Packet is transmitted


• Path info for dropped packet #65 (part 1) show platform packet-trace packet 65


Summary



State : DROP 22 (TailDrop)

Timestamp

Start : 1398207410699 ns (01/19/2000 04:49:22.995544 UTC)

Stop : 1398207589076 ns (01/19/2000 04:49:22.995722 UTC)

Path Trace

Feature: IPV4

Source : 192.168.1.1


Protocol : 1 (ICMP)

...

Feature: FIA_TRACE

Entry : 0x8200ed80 - IPV4_OUTPUT_QOS


...

Feature: FIA_TRACE

Entry : 0x80128400 - IPV4_OUTPUT_TUNNEL_PROTECTION_ENCRYPT

Lapsed time: 977 ns

Feature: IPSec

Result : IPSEC_RESULT_SA

Action : ENCRYPT

SA Handle : 4

Peer Addr : 1.1.1.2

Local Addr: 1.1.1.1

...

Lapsed time is displayed for each FIA element.

Can be used for datapath profiling!

QoS classification


Tunnel protection


We leave tunnel output FIA

at this point and the packet

is sent to crypto engine for

encryption


• Path info for dropped packet #65 (part 2)

• The packet is received from crypto engine and the processing continues, but the packet is dropped by QoS code

...

Feature: FIA_TRACE

Entry : 0x8062fc68 - IPV4_TUNNEL_ENCAP_GOTO_OUTPUT_FEATURE


...

Feature: QOS

Direction : Egress

Action : DROP

Drop Cause : TailDrop

Policy : Tail drop

Pak Priority : FALSE

Priority : FALSE

Queue ID : 145 (0x91)

PAL Queue ID : 1073741829 (0x40000005)

Queue Limit : 64

WRED enabled : FALSE

Inst Queue len: n/a

Avg Queue len : n/a

Feature: FIA_TRACE

Entry : 0x806c1acc - OUTPUT_DROP

Lapsed time: 302 ns

Feature: FIA_TRACE

Entry : 0x8200e480 - IPV4_OUTPUT_DROP_POLICY


We enter egress physical interface

output FIA at this point

Packet is dropped. Important point

here is that it’s dropped after IPSec

encapsulation, which can cause

IPSec anti-replay drops on the

receiver side.


• Packet Tracer relies on the Conditional Debug to determine which packets are interesting. The condition infra provides the ability to filter by protocol, IP address and mask, ACL, interface and direction

• Conditions define what the filters are and when the filters are applied to a packet. For example, “debug platform condition interface g0/0/0 egress” means that a packet will be identified as a match when it reaches the output FIA on interface g0/0/0 so any packet-processing that took place from ingress up to that point is missed

• It is recommended to use ingress conditions for Packet Tracer to get the most complete and meaningful data. Egress conditions can be used, but just be aware of the limitation above


• Packet Trace captures different levels of packet processing detail and provides commands to display the captured data

• Four detail levels:

1) Accounting

2) Packet summary

3) Packet details

4) Packet details with FIA trace and optional packet copy

• Packet details, FIA trace and packet copy are collected per packet when the packet is processed in data path. The detailed information collected is commonly referred to as “Path Data”


• Accounting (or statistics) level is always enabled if Packet Tracer is enabled. Per-packet info is not collected in this mode. Performance impact is low



Packets Summary

Matched 31

Traced 2

Packets Received

Ingress 31

Inject 0

Packets Processed

Forward 0

Punt 31

Count Code Cause

10 3 Layer2 control and legacy

3 7 ARP request or response

7 11 For-us data

9 21 RP<->QFP keepalive

2 27 Subscriber session control

Drop 0

Consume 0

Packets matched by conditional debug

Packets traced:

- limited by the max number of traced packets configured

- or PACTRAC can set additional creteria (e.g. PUNT code #27)

Forward – “ready to go to SIP/SPA”

Punt and drop codes are printed for

punted and dropped packets

Packets consumed by data path code

This command is required for all detail levels


• Per-packet info is collected: input and output interfaces, final packet state, punt/inject/drop codes and tracing start and stop timestamps

• Collecting summary data uses little performance over the normal packet processing

• An example usage may be to isolate which interfaces are dropping traffic so more detailed inspection can be used after applying interface specific conditions

debug platform packet-trace packet <16-8192> [circular] summary-only



0 Gi0/0/0.27 Gi0/0/2 FWD

...

64 Gi0/0/0.27 Gi0/0/2 FWD


...


Punt and drop codes are printed for

punted and dropped packets

What happened with each packet


• Summary information is always collected whenever any per packet data is collected. The summary information is displayed by the “summary” command and also the “per packet” command



0 Gi0/0/0 internal0/0/rp:0 PUNT 27 (Subscriber session control

1 Gi0/0/0 internal0/0/rp:0 PUNT 27 (Subscriber session control

show platform packet-trace packet 0


Summary

Input : GigabitEthernet0/0/0

Output : internal0/0/rp:0

State : PUNT 27 (Subscriber session control

Timestamp

Start : 4994905059758 ns (12/13/2014 19:23:54.523840 UTC)

Stop : 4994905077772 ns (12/13/2014 19:23:54.523858 UTC)

Summary info for specified packet


• Path data may be collected per packet for a limited number of packets and is made up of different types of data as follows:

Common path data (e.g. IP tuple)

Feature specific data (major features only, e.g. NAT, QoS, VPN, ZBF, etc.)

Feature Invocation Array (FIA) trace – if enabled

Packed dump – if enabled

• Capturing per packet data requires the use of QPF DRAM

• Capturing path data has the greatest impact on packet processing capability specifically FIA trace and packet copy

FIA tracing creates many path data entries costing instructions and DRAM writes

Packet copy creates many DRAM read/writes


• The “data-size” option allows the user to specify the size of the path data buffers used to store per feature and FIA-trace data. The default value is currently 2048 and need not be changed

• Using circular mode means that all matching packets are traced until Packet Trace is halted so it has a greater impact on system resources

• Packet copy: “input” - copy the packet when the packet is injected or seen on ingress interface

“output” - copy the packet at the moment of drop, punt or forward

“both” - copy the packet twice

start the copy from l2/l3/l4 header

the default packet size is 64

debug platform packet-trace packet <16-8192> [circular] [data-size <2048-16384>] [fia-trace]

debug platform packet-trace copy packet {input | output | both} [size <16-2048>] {l2 | l3 | l4}


• User config affects µcode performance and QFP DRAM usage based on the type and amount of tracing requested

• Packet Tracer statistics

Always tracked if PACTRAC enabled (“debug platform packet-trace enable”)

Least performance impact

• Per packet summary data

Always collected if per packet enabled (“debug platform packet-trace packet ...”)

Minor performance impact

• Per packet feature path data

Enabled by default when per packet enabled, can be disabled with “summary-only”

Variable performance impact – totally depends on feature mix

• Per packet ingress/egress packet copy

Enabled when per packet and packet copy enabled

Noticeable performance impact


• XE3.11 – Drop Tracing, XE3.13 – Punt/Inject Tracing

• XE3.14 – List of Drop/Punt/Inject codes

• Drop and Punt tracing can be enabled with and without conditions

• When enabled with conditions, the per-packet data is collected for all packets matched, but then collected data is discarded if the packet wasn’t dropped (or punted) – performance impact similar to “circular” mode

• When enabled without conditions, only the drop event is traced – very low performance impact, but information collected is limited

• “debug platform condition start” is still required

debug platform packet-trace {punt | inject | drop} [code <0-65534>]

show platform packet-trace code {drop | punt | inject}


• XE3.11: You can use embedded decoder, but only few protocol dissectors are currently supported (CSCul62487)

• This simple script can help decode a single packet

show platform packet-trace packet {<number> | all} [decode]

#!/usr/bin/perl

foreach (<>) {

s/[^a-fA-F0-9]//g;

print join("", pack("H*", $_));

}

cat packet.txt | hex2der.pl | od -t x1 | text2pcap -o oct - packet.pcap

Create this script, save file as hex2der.pl

Don’t forget to run “chmod 700 ./hex2der.pl”

To add fake Ethernet header run text2pcap with -e 0x0800


• This simple example illustrates the interactions between NAT and output packet copy


Conditional Debug Global State: Start


----------------------------------------------------------------------|---------

GigabitEthernet0/0/0 & IPV4 [10.1.75.2/32] egress


debug platform packet-trace packet 16 fia-trace data-size 2048

debug platform packet-trace copy packet output size 2048 L2

interface GigabitEthernet0/0/0

ip address 10.48.66.159 255.255.254.0

ip nat outside

interface GigabitEthernet0/0/1.75

encapsulation dot1Q 75

ip address 10.1.75.1 255.255.255.0

ip nat inside

We’re going to capture packets on

NAT outside interface on “output”.


• Packet Tracer will start tracing packets as soon as they reach egress interface FIA, but packet copy will happen after NAT when the packets are about to be transmitted to a SIP module

show platform hardware qfp active interface if-name g0/0/0

...


FIA handle - CP:0x108db890 DP:0x80791c80

CBUG_OUTPUT_FIA

IPV4_OUTPUT_VFR

IPV4_OUTPUT_NAT


IPV4_VFR_REFRAG (M)




DEBUG_COND_OUTPUT_PKT


DEF_IF_DROP_FIA (M)

“match” by inside IP,

but “copy” after NAT


show platform packet-trace packet 0 decode


Summary



State : FWD

Timestamp

Start : 461570571226

Stop : 461570727146

Path Trace

Feature: IPV4

Source : 10.1.75.2


Protocol : 1 (ICMP)

Feature: FIA_TRACE

Entry : 0x803550d8 - IPV4_OUTPUT_VFR

Timestamp : 461570576503

Feature: FIA_TRACE

Entry : 0x802a7f40 - IPV4_OUTPUT_NAT

Timestamp : 461570577819

Feature: NAT

Direction : IN to OUT

Action : Translate Source

Old Address : 10.1.75.2 00013

New Address : 10.48.66.159 00002

...


...

Packet Copy Out

0006f62a c4a30021 d89a0600 08004500 0064003d 0000fe01 235c0a30 429f0a30

42010800 33eb0002 00000000 000009f1 406cabcd abcdabcd abcdabcd abcdabcd

abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd

abcdabcd abcdabcd abcdabcd abcdabcd abcd

Ethernet

Destination MAC : 0006f62ac4a3

Source MAC : 0021d89a0600

Type : 0x0800 (IPV4)

IPv4

Version : 4

Header Length : 5

ToS : 0x00

Total Length : 100

Identifier : 0x003d

IP Flags : 0x0

Frag Offset : 0

TTL : 254

Protocol : 1 (ICMP)

Header Checksum : 0x235c

Source Address : 10.48.66.159

Destination Address : 10.48.66.1

ICMP

Type : 8

Code : 0x00

Checksum : 0x33eb

Identifier : 0x0002

Sequence : 0x0000

Translated IP address



• Что по вашему мнению необходимо сделать в первую очередь, чтобы улучшить мнение пользователей о платформах ASR1k и ISR4400/4300?

Выпустить еще больше бессмысленных маркетинговых брошюр

Написать наконец нормальную документацию

Выпустить в Cisco Press пару книжек о них

Повысить надежность работы ПО

Повысить надежность аппаратуры

Отказаться от IOS-XE. Нам ни к чему все эти сложности




• Embedded Packet Capture (EPC) is a powerful troubleshooting and tracing tool, it allows for network administrators to capture data packets flowing through, to, and from a Cisco router

• EPC is a software feature consisting of infrastructure to allow for packet data to be captured at various points. The network administrator may define the capture buffer to save capture and capture filter to customize the capture rules

• http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/epc/configuration/xe-3s/epc-xe-3s-book.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/epc/configuration/xe-3s/epc-xe-3s-book.html













IP cloud

ASR1000

SPA Driver

SPA Driver

SPA Driver

SPA Driver

IOSd

QFP ESP

SIP

RP

Replicate with

classification

Punt

Data

Data

Data


• Configuration

• Supported interfaces

• Up to 8 concurrent sessions (captures)

• More than one interface in each session

• Classification by ACL (only named ACLs supported!), class-map or inline “match”

monitor capture <name> {interface <name> | control-plane} {in | out | both} {access-list <name> |

class-map <name> | match {any | ipv4 | ipv6 | mac} <criteria>} [<options>]

For control-plane:

“in” – Inject

“out” – Punt

monitor capture cap1 interface ?

GigabitEthernet GigabitEthernet IEEE 802.3z

Multilink Multilink-group interface

Port-channel Ethernet Channel of interfaces

Tunnel Tunnel interface


• Capture options

• Defaults:

linear buffer

10MB buffer

40,000pps max

no sampling

entire packets are captured

monitor capture cap1 [buffer size <1-2000 MB>] [circular]

monitor capture cap1 [limit [packets <1-100000>] [duration <sec>] [every <Nth>] [packet-len <64-

9500>] [pps <pps>]]


• Configuration

ip access-list extended A198

permit ip host 192.168.2.1 host 192.168.1.1

monitor capture cap1 interface tunnel 1 in access-list A198

show monitor capture cap1

Status Information for Capture cap1

Target Type:

Interface: Tunnel1, Direction: in

Status : Inactive

Filter Details:

Access-list: A198

Buffer Details:

Buffer Type: LINEAR (default)

Limit Details:

Number of Packets to capture: 0 (no limit)

Packet Capture duration: 0 (no limit)

Packet Size to capture: 0 (no limit)

Packet sampling rate: 0 (no sampling)

show monitor capture cap1 parameter

monitor capture cap1 interface Tunnel1 in

monitor capture cap1 access-list A198


• Capture buffer

monitor capture cap1 start

show monitor capture cap1 buffer

buffer size (KB) : 10240

buffer used (KB) : 128

packets in buf : 5

packets dropped : 0

packets per sec : 113

show monitor capture cap1 buffer brief

-------------------------------------------------------------

# size timestamp source destination protocol

-------------------------------------------------------------

0 114 0.000000 192.168.2.1 -> 192.168.1.1 ICMP

1 114 0.001999 192.168.2.1 -> 192.168.1.1 ICMP

2 114 0.014999 192.168.2.1 -> 192.168.1.1 ICMP

3 114 0.016998 192.168.2.1 -> 192.168.1.1 ICMP

4 114 0.044996 192.168.2.1 -> 192.168.1.1 ICMP


• Capture buffer

show monitor capture cap1 buffer detailed

-------------------------------------------------------------

# size timestamp source destination protocol

-------------------------------------------------------------

0 114 0.000000 192.168.2.1 -> 192.168.1.1 ICMP

0000: 00000000 00000000 00000000 08004500 ..............E.

0010: 006486F5 0000FF01 B050C0A8 0201C0A8 .d.......P......

0020: 01010800 AC410018 00000000 00008404 .....A..........

0030: 4DECABCD ABCDABCD ABCDABCD ABCDABCD M...............

…

show monitor capture cap1 buffer dump

0

0000: 00000000 00000000 00000000 08004500 ..............E.

0010: 006486F5 0000FF01 B050C0A8 0201C0A8 .d.......P......

0020: 01010800 AC410018 00000000 00008404 .....A..........

0030: 4DECABCD ABCDABCD ABCDABCD ABCDABCD M...............

0040: ABCDABCD ABCDABCD ABCDABCD ABCDABCD ................



0070: ABCD


• Other commands

! Stop Capture session

monitor capture cap1 stop

! Export capture buffer

monitor capture cap1 export <URL>

! Clear capture buffer

monitor capture cap1 clear

! Clear configuration

no monitor capture cap1


• EPC per-cause punt policer

show platform hardware qfp active infrastructure punt statistics type per-cause | i Punt

Cause|Packets|Counter ID|075


Packets Packets


075 EPC 5 5

show platform software punt-policer | i ^ 75|pps|Cause

Per Punt-Cause Policer Configuration and Packet Counters

Punt Configured (pps) Conform Packets Dropped Packets

Cause Description Normal High Normal High Normal High

75 EPC 40000 1000 5 0 0 0

conf t

platform punt-policer 75 <new-value> [high]


• http://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-tech-notes-list.html

• http://www.ciscolive.com/global/



















http://www.ciscolive.com/global/




• Standard support releases

18 months lifetime, 3 scheduled rebuilds

3.11S, 3.12S, 3.14S, 3.15S, etc.

• Extended support releases

48 months lifetime, 8 scheduled rebuilds

3.10S, 3.13S, 3.16S, etc.

• http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/product_bulletin_c25-726436.html

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/product_bulletin_c25-726436.html
















• Используйте панель Q&A, чтобы задать вопрос

• Наши эксперты ответят на них


Получить дополнительную информацию, а также задать вопросы эксперту в рамках данной темы Вы можете на странице, доступной по ссылке:

https://supportforums.cisco.com/community/russian/expert-corner

Вы можете получить видеозапись данного семинара и текст сессии Q&A в течении ближайших 5 дней по следующей ссылке

https://supportforums.cisco.com/community/russian/expert-corner/




https://supportforums.cisco.com/community/netpro/ask-the-expert/webcasts





Тема: VoLTE – технологии передачи голоса в LTE сети

в среду, 20 мая, в 12.00 мск

Присоединяйтесь к эксперту Cisco

Владимиру Суконкину

Во время презентации эксперт Cisco Владимир Суконкин

рассмотрим архитектуру голосовых сервисов поверх LTE

сети (VoLTE), а так же технологии для поэтапного перехода

от существующей традиционной 2G/3G сети к VoLTE

архиетектуре.


• http://www.facebook.com/CiscoSupportCommunity

• http://twitter.com/#!/cisco_support

• http://www.youtube.com/user/ciscosupportchannel

• https://plus.google.com/110418616513822966153?prsrc=3#110418616513822966153/posts

• http://itunes.apple.com/us/app/cisco-technical-support/id398104252?mt=8

• https://play.google.com/store/apps/details?id=com.cisco.swtg_android

• http://www.linkedin.com/groups/CSC-Cisco-Support-Community-3210019

• Newsletter Subscription: https://tools.cisco.com/gdrp/coiga/showsurvey.do?surveyCode=589&keyCode=146298_2&PHYSICAL%20FULFILLMENT%20Y/N=NO&SUBSCRIPTION%20CENTER=YES

http://www.facebook.com/CiscoSupportCommunity

http://www.facebook.com/CiscoSupportCommunity

http://twitter.com/#!/cisco_support

http://twitter.com/#!/cisco_support

http://www.youtube.com/user/ciscosupportchannel



https://plus.google.com/110418616513822966153?prsrc=3#110418616513822966153/posts




http://itunes.apple.com/us/app/cisco-technical-support/id398104252?mt=8






https://play.google.com/store/apps/details?id=com.cisco.swtg_android

https://play.google.com/store/apps/details?id=com.cisco.swtg_android

http://www.linkedin.com/groups/CSC-Cisco-Support-Community-3210019










https://tools.cisco.com/gdrp/coiga/showsurvey.do?surveyCode=589&keyCode=146298_2&PHYSICAL FULFILLMENT Y/N=NO&SUBSCRIPTION CENTER=YES





Если вы говорите на Испанском, Португальском или Японском, мы приглашаем вас принять участие в сообществах:

Русский язык:

https://supportforums.cisco.com/community/russian

Испанский язык:

https://supportforums.cisco.com/community/5591/comunidad-de-soporte-de-cisco-en-espanol

Португальский язык:

https://supportforums.cisco.com/community/5141/comunidade-de-suporte-cisco-em-portugues

Японский язык:

http://www.csc-china.com.cn/





































Спасибо за Ваше время

Пожалуйста, участвуйте в опросе

Thank you.

Documents

Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400