Nexus 9000 Update

© 2013 Cisco and/or its affiliates. All rights reserved.

NEXUS 9000

An exciting Introduction to Cisco Nexus 9000 Series Switches

Cisco’s NX-OS Mode

The next generation of data center switching

John Lawrence

Data Center Consulting Systems Engineer

Mobile: 303-886-4507


AgendaNexus 9000

Check us out Online

www.cisco.com/go/aci

• Overall Design Goals

• Nexus 9500 Modular Switch

• Airflow and Power

• Out of Band Management Paths

• Fabric Scale and Connectivity

• Nexus 9300 Fixed Switch

• Cost Effective 40Gb Optics

• Use Cases and DesignsAPIC

http://www.cisco.com/go/aci



HardwareWhat We’re Building

iNXOS

Unified Information Model & API

Application/ Workload Orchestration & Scheduler

Hardware Innovation and 40G optimization

Object Oriented Switch OS

Object Oriented Data Model, Single MIT (mgmt information tree), Policy

Engine

Unified Northbound API

Open Ecosystem

APIC

Cisco Nexus 9000 Series SwitchesHigh-Performance 1/10/40/100 Gbps Switch Family

FLEXIBLE FORM FACTORS CAN ENABLE VARIABLE DATA CENTER DESIGN AND SCALING

Nexus® 9300 Nexus 9500

48 1/10G SFP+ & 12 QSFP+

SC

AL

AB

LE

1/1

0 /

40

/10

0 G

E P

ER

FO

RM

AN

CE

PERFORMANCE PORTS PRICE PROGRAMMABILITYPOWER

FCS Q4

2013

96 1/10G-T & 8 QSFP+FCS Q1

2014

12-port QSFP+ GEMFCS Q1

2014

ACI Ready Leaf Line Card48 1/10G-T & 4 QSFP+

FCS Q1

2014

ACI-ready Leaf line card48 1/10G SFP+ & 4 QSFP+

FCS Q1

2014

Aggregation line card36 40G QSFP+

FCSQ4

2013

C9500 8-SlotFCSQ4

2013


Merchant+ Foundation State of the Art Mechanical Design Object Oriented Programmable

Operating System Next Generation Development and

Verification Methodology Two Modes of Operation

Standalone (NX-OS) Fabric Mode (NX-OS + ACI)

HardwareBuild a Better Switch

NXOS Modeand ACI


NX-OS EnhancementsOptimized for Fabric• Modern: 64-bit Linux 3.4.10 Kernel

• Comprehensive Purpose Built DC Feature Set: L2/L3/VXLAN, ISSU, Patching* (Cold, Hot), Online Diagnostics (GOLD) etc.

• Trimmed base image: (approximately 50%) and single binary image - common image across all platforms for deployed mode

• Fault Containment: Complete process isolation for both features & services

• Resiliency: Restartable user-space network stack & driver, stateful restart of processes – no switch reboot etc.

• Open Management and Programmable Infrastructure: CLI, SNMP, NetConf/XML, OnePK, Open Containers, JSON etc.

NXOS ModeOnly


NX-OS EnhancementsFlexible Software Architecture

Single Image

NXOS image vs. system/kickstart

Same image across Fixed/Modular

Integrated Fault Monitoring, Detection and Recovery

Process crashes

Kernel crashes

Hardware Failures

Solutions – Fixed and Modular The network cannot go down

Stateful Process re-startability

Stateful Switchover

Software Patching (Cold and/or In-Service)

In Service Software Upgrade (ISSU)

64-bit Kernel

Netstack

ManagementInfrastructure

Hardware Drivers

FeatureAPI

API

Hgh Availability

Infrastructure

API

NXOS ModeOnly


Begin Code Test & Qualification Cycle

Target Deployment

Bug Found, Diagnose, Root

Cause

Defect Resolved, integrated into Maint.

Maint. ReleasedRestart Qual Cycle Actual Deployment

6 Months

10 Months

Many customers spend extensive time and effort to test and qualify software prior to deployment. In today’s environments, if a defect is found, effectively root-caused, and integrated, since it is rolled out through a maintenance release, customers would need to restart their qualification cycle, wasting time, and pushing out deployment dates…

NX-OS EnhancementsImage Management – Time to Patch

NXOS ModeOnly


The Nexus9000 Standalone platforms introduces new patching capabilities that allows specific defects to be rolled out in an independent package that can be applied to existing base software binaries. This will help reduce customer code certification times, leading to greater customer satisfaction…

Begin Code Test & Qualification Cycle

Target Deployment

Bug Found, Diagnose, Root

Cause

Defect Resolved, Patch Released

Continue QualifyWith additional tests Actual Deployment

6 Months

7 Months

NX-OS with Patching

NX-OS EnhancementsImage Management – Faster Patching

NXOS ModeOnly


NX-OS EnhancementsImage Patching – Functionality• Upgrade service executable or library in a NX-OS image

• Version and Compatibility control

• Allows Reverting from a Patch

• Integration with server management tools like Chef/Puppet

AdminPuppet Enterprise

Console

ImageServer

PuppetMaster

Devices

NXOS ModeOnly


NX-OS EnhancementsConsistency Checkers

• To effectively scale, a distributed management approach is optimal.• In distributed systems, maintaining consistency between resources is key to

scaling safely.

• Why is my traffic not going through this path... • My routing table looks good .. How do I debug

sys05-eor2# show forwarding ipv6 inconsistency module 22IPV6 Consistency check : table_id(0x80000001) slot(22)Execution time : 239 ms ()No inconsistent adjacencies.Inconsistent routes: 1. slot(22), vrf(default), prefix (2001:65:80::/45), Route missing in FIB Hardware. sys05-eor2#

• Link consistency

• Port Channel consistency

• IPv4, IPv6 route consistency

• VLAN membership consistency

• STP state consistency

NXOS ModeOnly


AgendaNexus 9000

Check us out Online











C97-730019 -01 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13

Hardware – ModularChassis Design – Components

Designed for: Power Efficiency Cooling Efficiency Reliability Future Scale

8 Line Card SlotsMax 3.84 Tbps/Slot duplex

Redundant Supervisor Engines

3 or 6 Fabric Modules (behind fan trays)

3 Fan Trays

Redundant System Controller Cards

No Mid-plane for LC to FM connectivity

3000W AC Power Supplies2+0, 2+1, 2+2 Redundancy

Support up to 8 Power supports

Nexus 9508 Front View Nexus 9508 Rear View

NXOS Modeand ACI


• Maximum three chassis per rack– Assuming 18KW per rack

• Up to 3,456 10G line rate ports per rack

• Up to 864 40G line rate ports per rack

• Designed for at least 2.5x speed increase in next gen ASICs

13 RU

19 in

Hardware – ModularChassis Architecture – Density

17.5 in30 in

Front View

NXOS Modeand ACI


Hardware – ModularChassis – Power Supplies Units (PSU)• Single 20A input at 220V

• Support for range of international cabling options

• 92%+ Efficiency

• Range of PS configurations• Minimum 1 PSU, Maximum 8 PSU

• (2) PSU for fully loaded chassis

• N+1 redundancy

• N+N grid redundancy

• 2x head room for future port densities, bandwidth, and optics

• Power Efficiency• Platinum rated power supplies, 92% power

efficiency across all workloads

• 3.5W per 10 Gbps Port

• 14W per 40 Gbps Port

80 Plus Platinum is equivalent to ClimateSaver/ Green Grid Platinum rating

3000W AC PSU

NXOS Modeand ACI


Hardware – ModularChassis – Supervisor Modules• Redundant half-width

supervisor engine• Sandy Bridge, Quad Core,

1.8GHz

• 16GB Memory

• RAM upgradable to 48GB

• 64 GB SSD (default)

• Common for 4, 8 and 16 slot chassis

• Performance/ Scale Focused

• Range of Management Interfaces

External Clock InputPulse per Second

(Precision Time Protocol)

Management Port

(2) USB Ports

Console Port

NXOS Modeand ACI


Hardware – ModularChassis – System Controllers• Redundant half-width system

controller• Offload supervisor from switch

“control plane” tasks

• Increased System Resiliency

• Increased Scale

• Common for 4, 8 and 16 slot chassis

• Performance/ Scale focused• Dual Core ARM Processor, 1.3GHz

• Central Point of Chassis Control

• Ethernet Out-of-Band Channel (EOBC) switch between Supervisors and Line cards

• Ethernet Protocol Channel (EPC) switch

• 1Gbps switch for Intra-node Data Plane communication (Protocol Packets)

• Manages / Monitors• Power Supplies via SMB (System

Management Bus) • Fan Trays

NXOS Modeand ACI


Exhaust

Hardware8-Slot Modular Chassis Air Flow

• Chassis is complete Front-to-Back Airflow

• Airflow direction is NOT Reversible

• Fan Trays are fully redundant

• Fan Trays must be removed in order to service Fabric Modules

• Designed for speed increase in multiple next gen ASICs

Air Intake

FrontView

RearView

Fan TrayRemoved

Fabric Modules

NXOS Modeand ACI


Hardware – ModularFabric Modules and Fan Trays• Up to 6 Fabric

Modules• Different cost points for

1/10G access and 40G aggregation

• Flexibility for future generation of fabric modules

• Quad Core ARM CPU 1.3 GHz for Supervisor offload

• Hot Swappable

• All Modules Forward Traffic

• Smooth degradation during replacement

• 3 Fan Trays• (3) dual fans per tray

• Dynamic speed control driven by temperature sensors

• Straight Airflow across Line Cards and Fabric Modules

• N+1 Redundancy per Tray

FabricModule

FanTray

NXOS Modeand ACI


Hardware – ModularFabric Module – Overview• 8-Slot Chassis Fabric

Module:• (2) Packet Processors

• (32) 40G Hi-gig2 links per Packet Processor

• (8) 40G Hi-gig2 links assigned to each I/O slot

• Unified Forwarding Table on Packet Processor is programmed to support• 128 LPM (Longest Prefix

Match) routing entries

Packet

Processor

Packet

Processor

32 x 40G

Hi-G

ig232 x 40G

H

i-Gig2

64 x 40G H

i-Gig2

Line Card

Line Card

Line Card

Line Card

Line Card

Line Card

Line Card

Line Card

To Fabric

To Line Cards

Fabric Module

NXOS Modeand ACI


Hardware – Line CardsFabric Module – Data Plane Scaling for 8-Slot Chassis• A Fabric Module in an 8-Slot Chassis can provide up to 320Gbps

to each Line Card slot.

• With 6 Fabric Modules, each Line Card slot can have up to 1.92Tbps forwarding bandwidth in both directions.

PP

Fabric 1

PP

320 Gbps(8x 40Gbps)

PP

Fabric 2

PP

320 Gbps(8x 40Gbps)

PP

Fabric 3

PP

320 Gbps(8x 40Gbps)

PP

Fabric 4

PP

320 Gbps(8x 40Gbps)

PP

Fabric 5

PP

320 Gbps(8x 40Gbps)

PP

Fabric 6

PP

320 Gbps(8x 40Gbps)

Line Card Slot

320 Gbps

640 Gbps

960 Gbps

1.28 Tbps

1.60 Tbps

1.92 Tbps

NXOS Modeand ACI


Hardware – ModularLine Cards – Overview

48 ports 10G SFP+ & 4 ports 40G QSFP+ 48 ports 1/10G-T & 4 ports 40G QSFP+

(non blocking)

36 ports 40G QSFP+ ((1.5:1 oversubscribed)

1/10G Access and 10/40G Aggregation

36 ports 40G QSFP+ (Non Blocking)

40G Aggregation

ACI Access Ready

36 ports 40G QSFP+ (Non Blocking)

40G Fabric Spine

NXOS Mode

ACI Access

ACI Spine

ACI Access Ready


A 36-Port 40G QSFP+ I/O Module needs 6 Fabric Modules to operate at line rate.

Hardware – ModularLine Cards – 40G Aggregation • (36) 40G QSFP ports

• 1.44 Tbps full-duplex fabric connectivity

• L2/L3 line rate performance for all packet sizes

• Distributed multicast replication

• Supported in 4 and 8 slot chassis

• Part Number: N9K-X9636PQ

NXOS ModeOnly


• 48p 1G SFP/10G /SFP+ and 4p 40G QSFP

• 48p 1G/10G Base-T ports and 4p 40G QSFP

• 1280 Gbps full-duplex fabric connectivity

(4) 40G Uplinks

(4) 40G Uplinks

48-Port 1/10G T + 4-Port 40G

48-Port 1/10G F + 4-Port 40G

Hardware – ModularLine Cards – 1/10G Access and 10/40G Aggregation

36-Port 40G

• 36p 40G 1.5:1 oversubscribed option



• Requires 3 Fabric Modules

• Supported in 4, 8, 16 slot Chassis

NXOS Modeand ACI

N9K-X9564PX

N9K-X9564TX

N9K-X9636PQ


A 36-Port 40G QSFP+ I/O Module needs 6 Fabric Modules to operate at line rate.

Hardware – Modular40G Spine Line Card• (36) 40G QSFP ports


• L2/L3 line rate performance for all packet sizes


• Supported in 4, 8 and 16 slot chassis

ACI SpineOnly


Hardware – ModularFabric Extenders – Nexus 2000 Support

Nexus 2248TPNexus 2248TP-E

Nexus B22-HPNexus B22-Dell

Nexus 2232PP-10GNexus 2232TM-10G

Fabric Extender Design Options

Supported Fabric Extenders

NXOS Modeand ACI


Nexus 9000 SeriesFull Line-Rate Throughput Performance

Unprecedented, Full Line-Rate Performance:

Proven with RFC 2544, RFC 2889, RFC 3918 throughput tests Results were taken using a fully loaded Nexus® 9508 switch with 288 40 GE ports:• All ports are line rate at 100% unicast traffic load

• All ports are line rate at 100% multicast traffic load

• Full line rate for all packet sizes (64 to approx. 9216 Bytes)

1- 64-1 1- 128-1 1- 256-1 1- 512-1 1- 1024-1 1- 1280-1 1- 1518-1 1- 9216-1

1.2E+012

1.0E+012

0.0E+012

2.0E+012

4.0E+012

6.0E+012

8.0E+012 TX

RX(fra

mes

)


Nexus 9000 SeriesLow-Latency Performance

Low Latency (Same for Both Unicast and Multicast)

Proven with RFC 2544, RFC 2889, RFC 3918 throughput tests Results were taken using a fully-loaded Nexus® 9508 switch with 288 40 GE ports:

• Unicast latency at 100% traffic load: �̶ 1.6 usec (64-Byte packets)�̶ 3.5 usec (9216-Byte packets)

• Multicast latency at 100% traffic load:�̶ 1.6 usec (64-Byte packets)�̶ 3.5 usec (9216-Byte packets)


AgendaNexus 9000

Check us out Online












Hardware – FixedNexus9300 – Fixed Switch Platform

Nexus 9396PQ• (48) port 10G SFP+ • (12) port 40G QSFP+ • 2-RU

Nexus 93128TX• (96) port 1/10G-T • (8) port 40G QSFP+ • 3-RU

Nexus 9300 – Common• Redundant Fan and Power Supply• Front-to-Back and Back-to-Front airflow• Dual or Quad Gladden 1.6 GHz Core

CPU• 16 GB DRAM• Default 64GB SDD

Nexus ACI Ready Access GEM• (12) port 40G QSFP+• Additional 40MB buffer

• 3.5x of merchant Packet Processor• Full VxLAN Bridging and Routing Capability

NXOS Modeand ACI


Hardware – FixedUplink Options / Power and Cooling

• Redundant (1+1) Dual 650W and 1200W AC PS option• Redundant (2+1) Fan Trays• Hot Swappable• Cold air intake (blue) and Hot air exhaust (red) options

Generic Expansion Module (GEM)

Fan and Power Supply

• 12 port 40G QSFP (FCS)• 6 port 40G QSFP (Post FCS)

Non-Blocking 3:1 Oversubscribed Uplinks

* 80 Plus Platinum is equivalent to Climate Saver/ Green Grid Platinum rating

Nexus 9396PQ Nexus 93128TX

NXOS Modeand ACI


Nexus 9300 Platform Architecture

• 2 RU height

• 48 1 Gb SFP/10 Gbps SFP+ ports

• 12 40 Gbps QSFP ports (on GEM

module)

• 1 100/1000baseT management port

• 1 RS232 console port

• 2 USB 2.0 ports

• Front-to-back and back-to-front

airflow options

• 1+1 redundant power supply options

• 2+1 redundant fans

• No-blocking architecture with line-rate

performance on all ports for all packet

sizes

Cisco Nexus® 9396PX

ConsoleManagement Port

USB Ports

48 1Gbps SFP/10Gbps SFP+ ports

GEM module with 12 40 Gbps QSFP+ ports

Power supply

Power supply

(2+1) fan trays



• 3 RU height

• 96 1/10 Gbps BaseT ports

• 8 40 Gbps QSFP ports (on GEM

module)

• 1 100/1000baseT management port

• 1 RS232 console port

• 2 USB 2.0 ports

• Front-to-back and back-to-front

airflow options

• 1+1 redundant power supply

options

• 2+1 redundant fans

Cisco Nexus® 93128TX

Console management port

USB ports

96 1 GBaseT/10 GBaseT ports

GEM module with 12 40 Gbps QSFP+ ports

(8 active uplinks)

Power supply

Power supply

(2+1) fan trays



• 12-port 40 Gbps QSFP (FCS)

• Additional 40 MB buffer (3.5 times of BCOM NFE)

• Full VXLAN gateway, bridging, and routing capability

• Common for Nexus® 9396 and Nexus 93128 Switches

• Four ports will be disabled when installed in a Cisco® Nexus 93128 Switch.

• A white LED under each QSFP port pair indicates port-pair availability.

• The LED will be on if the port pair is available.Generic Expansion Module

• Redundant (1+1) 650 W and 1200 W AC PS options

• 80-Plus-Platinum-certified power supplies*

• Redundant (2+1) hot-swappable fan trays

• Cold-air intake (blue) and hot-air exhaust (red) options to support front-to-back or back-to-front airflow

Fan and Power Supply* 80 Plus Platinum is equivalent to a Climate Saver or Green Grid Platinum rating


AgendaNexus 9000

Check us out Online












Challenge• 40G Optics are significant portion of CAPEX• 40G Optics require new cabling

Solution• Re-use existing 10G MMF cabling

infrastructure• Re-use patch cables (same LC connector)

Cisco 40G SR-BiDi QSFP • QSFP pluggable, MSA compliant• Dual LC Connector• Support for 100m on OM3 and 125m+ on OM4• Transmit/Receive on 2 wavelengths at 20G each

Available end of CY13 and supported across all Cisco QSFP ports

Hardware – Line CardsOptical Innovation – Removing 40G Barriers

NXOS Modeand ACI


AgendaNexus 9000

Check us out Online











© 2013 Cisco and/or its affiliates. All rights reserved. Nexus 9000 Boot Camp

(X)

Row 1

RacksN5548

N2000

Pod 1

(Y) Rows

(X)

Row Y

RacksN5548

N2000

(X)

Row 1

RacksN5548

N2000

Pod N

(Y) Rows

(X)

Row Y

RacksN5548

N2000

(N) PoDs

Layer 3Layer 2

Layer 3Layer 2

DC Edge

DC Core

DC Aggr.

DC Access DC Access

DC Aggr.

Design and Migration – Design Option 1aTraditional 3-tier with 1/10 GE Server Access

NXOS ModeOnly


N9500

N9500

(X)

Row 1

Racks

N2000

N9500Pod 1

(Y) Rows

(X)

Row Y

Racks

N2000

N9500

N9500

(X)

Row 1

RacksN9300

N2000

N9500Pod N

(Y) Rows

(X)

Row Y

RacksN9300

N2000

(N) PoDs

Layer 3Layer 2

Layer 3Layer 2

DC Edge

DC Core

DC Aggr.

DC Access DC Access

DC Aggr.

Design and Migration – Design Option 1aMigration to Nexus 9000

The same design can be built with Nexus9500 at the Core and Nexus9300 with Fabric Extenders at the edge.

NXOS ModeOnly


Pod 1

(Y) Rows

Pod N

(Y) Rows

(N) PoDs

Layer 3Layer 2

Layer 3Layer 2

DC Edge

DC Core

DC Aggr.

UCS Access DC Access

DC Aggr.

UCS FI

Design and Migration – Design Option 1bTraditional UCS/Blade Server Access

NXOS ModeOnly


N9500

N9500 N9500Pod 1

(Y) Rows

N9500

N9500

N9300

N9500Pod N

(Y) Rows

(N) PoDs

Layer 3Layer 2

Layer 3Layer 2

DC Edge

DC Core

DC Aggr.

UCS AccessBlade Server

Access

DC Aggr.

UCS FI

FEX

N9300

FEX

Design and Migration – Design Option 1bMigration to Nexus 9000

The same design can be built with Nexus9500 at the Core and Nexus9300 with Fabric Extenders at the edge

NXOS ModeOnly


In Summary• The Nexus 9500 Chassis provides scalable performance

with consistent parts across the Nexus9504, 9508, 9516 platforms.

• The platform is future proofed for higher power ASICs.

• The Nexus 9500 has no mid-plane to further improve front-to-back airflow.

• The Merchant+ strategy minimizes the number of ASICs and lowers power consumption while providing value-added network options.

• The QSFP BiDi optics allow 40Gb connectivity over standard OM3/OM4 cable plants.


Technology

Nexus 9000 Update