JetCore Based Chassis Systems - Dich

JetCore™ Based Chassis Systems An Architecture Brief on NetIron, BigIron, and FastIron

Systems

JetCore™ Based Chassis Systems An Architecture Brief

Page 2 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Table of Contents 1. Preface......................................................................................................................... 4 2. Introduction ................................................................................................................. 4 3. Chassis Systems Overview.......................................................................................... 4 4. JetCore Interface Module Architecture ....................................................................... 6

4.1. Physical Ports ..................................................................................................... 10 4.2. Port Group Switching Logic and Switch Fabric ................................................ 10

4.2.1. MAC.......................................................................................................... 10 4.2.2. Packet Classifier........................................................................................ 11 4.2.3. VLAN Multicast Assist Module ............................................................... 13 4.2.4. Transmit Pipeline ...................................................................................... 13 4.2.5. Shared Memory Switch Fabric.................................................................. 13 4.2.6. Buffer Manager ......................................................................................... 14 4.2.7. Shared Memory Buffer Pool ..................................................................... 14 4.2.8. CAM.......................................................................................................... 15 4.2.9. PRAM........................................................................................................ 15 4.2.10. System Management Interface .................................................................. 15

4.3. Module Crosspoint Switch Fabric...................................................................... 15 5. Packet Forwarding..................................................................................................... 16

5.1. Packet Level Walk-Through .............................................................................. 16 5.2. Multiple Destination Output Priority Queues .................................................... 17

6. The Chassis Crosspoint Switch Fabric...................................................................... 18 6.1. Handling Unicast Traffic.................................................................................... 20 6.2. Handling Multicast Traffic................................................................................. 21

7. System Management Architecture ............................................................................ 21 7.1. System Management Interface ........................................................................... 22 7.2. Command Bus .................................................................................................... 22

8. IronWare OS Overview............................................................................................. 22 9. Benefits of the JetCore Architecture ......................................................................... 24 10. Summary .................................................................................................................. 27


Page 3 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Table of Figures Figure 1: Four slot chassis feature summary....................................................................... 5 Figure 2: Eight slot chassis feature summary...................................................................... 5 Figure 3: Fifteen slot chassis feature summary. .................................................................. 6 Figure 4: Logical architecture of an interface module using the 2nd generation chipset,

IronCore™........................................................................................................... 7 Figure 5: Logical architecture of a 16 port interface module using the 3rd generation

chipset, JetCore™................................................................................................ 8 Figure 6: A close up view of the JetCore™ port group switching logic and switch fabric.9 Figure 7: Four priority queues per destination. ................................................................. 18 Figure 8: Chassis crosspoint switch fabric for an 8 slot chassis. ...................................... 20 Figure 9: IronWare OS architecture. ................................................................................. 23


Page 4 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

1. Preface Foundry Networks lately introduced a new line of products based on its new third generation ASICs, known as JetCore™. The purpose of this paper is to introduce the reader to that new architecture as well as to the value added features that come with such an architecture.

2. Introduction The new JetCore chipset represents the third generation of ASICs created by Foundry Networks. Built on 0.18 micron technology, the new chipset consolidates a lot of functions into a compact design. This brings cost benefits to the end user as well as increased reliability – since the number of components is reduced. The new compact design allows for higher port densities not achievable before using prior chipsets. In addition, the new chipset contains new features, allowing more functions to be performed in hardware, which in turn, allows new applications to be deployed in a scalable manner.

3. Chassis Systems Overview Foundry’s chassis systems include three series: • FastIron: aimed at high performance layer-2/layer-3 switching for enterprise

applications. • BigIron: aimed at high performance layer-2/layer-3 switching for data centers as well

as large campus networks. • NetIron: aimed at high performance layer-2/layer-3 and MPLS switching for service

providers, and wide area network backbones. All of the chassis series above are capable of leveraging the new JetCore architecture to deliver more functionality and cost efficiency to the end user.


Page 5 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Chassis systems are offered in three configurations: • 4 slot chassis • 8 slot chassis • 15 slot chassis The diagrams below summarize the general features of those chassis.

Figure 1: Four slot chassis feature summary.

Figure 2: Eight slot chassis feature summary.


Page 6 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

NEBS Level 3 compliant version

available for NetIron

Figure 3: Fifteen slot chassis feature summary.

Foundry chassis switching routers deliver high performance and high port density suitable for several applications, both in the enterprise world as well as the service provider world. The systems employ a truly distributed switching architecture in order to provide wire speed non-blocking packet forwarding with ultra low latency. In order to accomplish that, the systems place the intelligent packet forwarding functions on the interface modules, i.e., all the interface modules incorporate their own switching engines, buffers, queues, and switching fabrics without requiring any upgrades later on. This leaves the chassis’ switch fabric with the simple job of non-blocking forwarding of packets between the individual interface modules.

4. JetCore Interface Module Architecture The “BigIron Architecture Brief” white paper describes the high level architecture of Foundry’s interface modules as per the 2nd generation chipset. The 3rd generation chipset, described here, consolidates all the required functions as well as some new functions into fewer ASICs to provide higher port density, reliability, cost savings, and added functionality.


Page 7 of 27

Figure 4: Logical architecture of an interface module using the 2nd generation chipset, IronCore™.


Page 8 of 27

Module Crosspoint Switch Fabric

8 Gbp

s to

Chas

sis

Cros

spoin

t

8 Gbps from Chassis

Crosspoint

Port Group Switching Logic and Switch

Fabric

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort


Fabric

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort


Fabric

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort


Fabric

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort

Phys

ical P

ort

Sys. Mgmt. Interface

Sys. Mgmt. In terface Sys. Mgmt. In terface Sys. Mgmt. In terface Sys. Mgmt. In terface

Comm

and B

us

Figure 5: Logical architecture of a 16 port interface module using the 3rd generation chipset, JetCore™.


Page 9 of 27

Physical Port

MAC

Input Traffic Policer

Output Traffic Policer

Queue Mgmt.

QoS Mgmt.

Buffer Manager

Shared Memory Switch Fabric

Shared Memory Buffer Pool

Physical Port

MAC

Physical Port

MAC

Physical Port

MAC

Packet Classifier

MCast Assist PRAM

CAM

Packet Classifier

MCast Assist

Packet Classifier

MCast Assist

Packet Classifier

MCast Assist

TX Pipeline

TX Pipeline

TX Pipeline

TX Pipeline

To M

od.

Cros

spoin

t

From Mod. Crosspoint

System Mgmt.

Interface

CMD

Bus

Figure 6: A close up view of the JetCore™ port group switching logic and switch fabric.


Page 10 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Note The diagrams above are functional diagrams for illustrative purposes only. The modules in the diagrams do not necessarily correspond to actual physical ASICs. More than one function may be combined in a single ASIC. It is worth noting that JetCore combined the functions of 7 of the prior IronCore ASICs into one ASIC, yielding higher port density, better cost efficiency, as well as increased reliability. As show in Figure 5, JetCore introduces the concept of a module crosspoint switch fabric, as opposed to IronCore which relied on a shared memory switch fabric that is central to the whole interface module. Groups of ports – 4 for Gigabit Ethernet, 24 for 10/100 Mbps Ethernet – have their own hardware logic and shared memory switch fabric which in turn interfaces with the module crosspoint switch fabric. This concept allows for more scalability and higher port density in the JetCore interface modules due to the way such functional components are implemented in ASICs. For instance, the module crosspoint switch fabric is implemented on a single ASIC referred to as the Backplane Interface Adapter (BIA). The whole port group switching logic and switch fabric – apart from the CAM, PRAM, and shared memory discussed later – are implemented on a single ASIC referred to as the Integrated Gigabit Controller (IGC) for Gigabit Ethernet ports, or the Integrated Port Controller (IPC) for 10/100 Mbps Ethernet ports. The components of a JetCore interface module are described below.

4.1. Physical Ports Physical ports provide connectivity to end stations or other network devices. Foundry’s products offer a full range of auto sensing, half or full-duplex switch and router port options for Ethernet, Fast Ethernet, Gigabit Ethernet, and 10 Gigabit Ethernet connectivity, as well as Packet over SONET and ATM interface options.

4.2. Port Group Switching Logic and Switch Fabric

4.2.1. MAC The Media Access Controller (MAC) provides continuous data flow between the physical port and the packet classifier. The MAC is responsible for encoding and decoding, CRC verification for incoming packets, CRC calculation for outgoing packets, and auto-negotiation logic.


Page 11 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

4.2.2. Packet Classifier A packet classifier operating at the full line rate is responsible for parsing and examining the incoming packets at layers 2, 3, and 4, in order to make a decision on how the packet should be forwarded. The packet classifier interfaces with the CAM against which it checks the packet headers, as well as the PRAM which serves as the repository of information that describe where and how the packet should be forwarded in case a match in the CAM is found. Essentially, the packet classifier is the switching engine responsible for making the packet switching decision, and hence, it is the most important module to Foundry’s distributed architecture. The packet classifier’s processing results in a forwarding identifier (FID) that describes the destination port(s) for the packet processed. As common to high end switching architectures, when the packet classifier is done processing the packet, it attaches an internal Foundry header to the packet. This is an optimized header that describes to the hardware how the packet should be handled from that point onwards. The FID is included as part of that header. The JetCore packet classifier, also, is an enabler for XRMON and sFlow (RFC 3176) traffic monitoring. The packet classifier is capable of randomly selecting packets out of the examined packet stream (1 in N sampling) and flagging them as packets to be copied to the management module – a function that is required for both sFlow and XRMON. A software agent running on the management module, examines those packets, extracts sFlow/XRMON information, and sends it along with a portion of the packet headers to an external collector station. The parameters of the packet random selection algorithm are controlled by the operating system, and are tunable via user configuration. Inside the packet classifier module, packet processing is performed in a pipelined manner among smaller submodules in order to maintain wire speed performance. Perhaps, the most important and innovative submodule is the Programmable Lookup Processor (PLP), which allows the hardware to be flexible in terms of packet parsing in order to accommodate future packet switching features. Below is a brief description of the most important submodules of the packet classifier.

4.2.2.1. Programmable Lookup Processor (PLP) The programmable lookup processor is a submodule within the packet classifier module. The PLP is responsible for parsing the packet headers and extracting information to be checked against the CAM by the CAM lookup handler later on. The PLP is one of the innovative features of the JetCore architecture. It is essentially a very high speed, small RISC microprocessor embedded within the packet classifier. The PLP is capable of parsing packet headers at line rate – according to a simple program


Page 12 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

provided by the operating system – and extracting the information that should be checked against the CAM to its (PLP) registers. Packet switching is always evolving, new protocols are created, new encapsulations are used, even new rules for switching the packet might evolve to address certain customer needs. The PLP represents a protection of investment to both Foundry as well as the end user. Due to its nature, should a new packet parsing/lookup technique be required, i.e., something beyond the standard lookups known today, the PLP’s internal program could be enhanced to allow for that new feature. Such a PLP program enhancement could be accomplished via an operating system upgrade without requiring any change to the existing ASICs/hardware. In that sense, JetCore provides a future proof architecture.

4.2.2.2. CAM Lookup Handler As mentioned above, the PLP uses its registers to export the information that needs to be checked against the CAM to the CAM lookup handler. The CAM lookup handler takes a snapshot of the PLP registers, thus, freeing the PLP to process another packet while the CAM lookup handler is performing its task – part of the pipelined approach mentioned before. The CAM lookup handler takes the snapshot and checks it against the CAM, looking for a match. CAM lookups are usually performed on packet header information like the Ethernet destination and/or source address (DA/SA), destination and/or source IP address, destination and/or source TCP/UDP ports, etc. The result of the lookup is an index into the PRAM, which contains information on how to handle the packet. That index is used by the PRAM lookup handler.

4.2.2.3. PRAM Lookup Handler As part of the pipelined packet processing, the index obtained by the CAM lookup handler is used by the PRAM handler to retrieve information from the PRAM on how and where the packet should be forwarded. While the PRAM lookup handler is performing its job, the PLP and the CAM lookup handler could also be processing other packets, which maintains wire speed performance even when performing complicated lookups. The PRAM lookup results in information about the outgoing port, DA/VLAN/ToS/DSCP replacement, and other packet header modifications required for the completion of the packet switching process.


Page 13 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

4.2.3. VLAN Multicast Assist Module The VLAN multicast assist module works on the transmit side. Normally, Foundry’s hardware architecture is capable of attaching the proper VLAN ID to a unicast/multicast packet before it goes out a given port. In case a multicast packet needs to be sent out a given port multiple times with different VLAN IDs – say to different subnets attached to that port –, the VLAN multicast assist module is responsible for managing such a packet replication functionality in hardware. The module maintains its own per port VLAN multicast counters, as well as its own VLAN multicast map. These data structures are programmable by the operating system, and they allow the module to automatically manage the transmission of the multicast packet multiple times up to the number specified by the VLAN multicast counter, each time with a new VLAN ID as specified by the VLAN multicast map.

4.2.4. Transmit Pipeline The transmit pipeline is the last module before the MAC on the transmit side. It is responsible for inserting the proper VLAN ID into the packet. Under the control of the VLAN multicast assist module, transmission of a multicast packet over multiple VLANs out the same port could be accomplished in hardware without any performance degradation. The module is also responsible for performing any required IP/IPX packet modifications prior to packet transmission. Finally, the transmit pipeline strips the internal Foundry header before handing the packet to the MAC.

4.2.5. Shared Memory Switch Fabric The shared memory switch fabric is responsible for transferring the packets to and from the high speed shared memory. The fabric interfaces with the packet classifier, the VLAN multicast assist module, the buffer manager, and the module crosspoint switch fabric. When the shared memory switch fabric receives the packet from the packet classifier, it transfers the packet to a free location in the shared memory. The fabric then reports the following information to the buffer manager: • Buffer number – where the packet is stored • Forwarding Identifier (FID) • Packet priority • Whether the packet should be copied to the management module


Page 14 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

The buffer manager uses the above information to determine the location of the packet in the shared memory, manage buffer space, determine the destination(s) of the packet, and consequently, which queue(s) it should be placed on. Under the control of the buffer manager, the shared memory switch fabric, also, retrieves packets from the shared memory and sends them to their destination ports, as instructed by the buffer manager.

4.2.6. Buffer Manager The buffer manager is responsible for managing the packet storage space inside the shared memory. It, also, performs the following functions: • Managing per port queues • Managing QoS, and making sure that each packet is assigned to the proper queue

based on its priority. Foundry’s architecture offers 4 queues per port, to allow 4 levels of traffic prioritization, with either strict priority or weighted fair queuing (WFQ) scheduling.

• Traffic policing for inbound packets • Traffic policing for outbound packets When the buffer manager is notified of the buffer number of a new packet in the shared memory, the buffer manager assigns the buffer number to the appropriate queue based on the packet’s destination and priority. When a packet reaches the head of its queue, the buffer manager instructs the shared memory switch fabric as to where to retrieve the packet from the shared memory (using the buffer number), and which physical port to send it to. The most significant feature of the JetCore buffer manager is the ability to perform traffic policing (rate limiting), both inbound and outbound, in hardware. Unlike other architectures, a JetCore based chassis does not need to send the packets to a central module where traffic policing is performed in software. A JetCore based interface module is capable of performing its own traffic policing, in hardware, without any assistance from any other module, and hence, traffic policing could be deployed in a more scalable manner.

4.2.7. Shared Memory Buffer Pool The shared memory buffer pool provides storage for the received packet(s) till it is time to the send the packet out – based on the packet’s priority, and the status of the port queue it was assigned to. Foundry’s interface module architecture relies on a high speed shared memory to implement this functionality, capable of storing and retrieving packets at the sum of the line rates of the physical ports within the port group as well as the connection to the module crosspoint switch fabric, in order to provide a true wire speed non blocking architecture.


Page 15 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

4.2.8. CAM The content addressable memory (CAM) is used by the packet classifier to provide reverse lookups based on the content of the packet. The packet classifier extracts a portion(s) of the packet headers – such as destination IP address, destination MAC address, source and destination IP addresses, etc. – and hands it over to the high speed CAM to perform a lookup. In case a match is found, the CAM returns an index of a location in the PRAM, where information on how to forward the packet is stored.

4.2.9. PRAM The PRAM is a high speed memory that takes the index returned by the CAM lookup and returns to the packet classifier information on how to forward the packet: FID (destination port or ports), MAC address of next hop, VLAN ID that should be attached to the packet as it goes out a given interface, etc. The information retrieved by the PRAM is used to build the internal Foundry header that gets attached to the packet. This information is used by other modules of the hardware to determine how to forward the packet.

4.2.10. System Management Interface The system management interface connects the port group hardware to the command bus. Through the command bus, the management module is able to initialize, control, and update any of the port groups in the chassis. Also, the management interface allows the management module to collect various packet statistics from the port group.

4.3. Module Crosspoint Switch Fabric The module crosspoint switch fabric provides high speed packet forwarding between the different port groups. As mentioned earlier, each port group in JetCore has its own shared memory switch fabric. When a packet is destined for another port on the same interface module but the port is within a different port group, the module crosspoint switch fabric provides the forwarding function. Within the port group logic, once the packet’s buffer number reaches the head of a queue associated with a port in a different port group, the packet is sent to the module crosspoint switch fabric which switches it to the destination port group. The module crosspoint switch fabric, also, serves the purpose of interfacing with the backplane slot. Adjacent to the module crosspoint fabric, on the same BIA chip, is an aggregator that takes care of the necessary word length conversion to interface with the backplane. The fabric offers the bandwidth to serve all the connected port groups, as well


Page 16 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

as the backplane slot simultaneously at full data rate. Non blocking performance is, also, provided through the use of Virtual Output Queuing1 on the fabric to avoid head of line blocking. For multicast packets coming from the backplane slot, should the packet be destined for more than one port group (as indicated by its FID), the fabric provides the necessary replication, one copy per port group. At the port group level, the shared memory switch fabric takes care of any further replication in case more than one port is to receive the multicast. In order to handle the case where a port group might be temporarily overwhelmed with too much traffic, as in oversubscription cases, the module crosspoint switch fabric offers its own buffering. While normally packets are switched on the fly to their destination port group(s), in case a port group is temporarily unable to receive packets, the packet is stored in the buffer, till the port group asserts the Ready signal again. This, obviously, has the impact of decreasing packet loss for unicast. It, also, has the impact of providing expedited non blocking forwarding of multicast. For instance, assume a multicast packet arrives from the backplane targeting ports in port groups 1, 3, 4, and port group 3 is temporarily unable to receive packets. The fabric could still forward the multicast packet to port groups 1 and 4, while the packet gets buffered to be sent later to port group 3. In other words, the forwarding of the multicast packet does would not be blocked (also, blocking other packets behind it) nor delayed due to a member port group being temporarily not ready.

5. Packet Forwarding

5.1. Packet Level Walk-Through As the physical port receives the packet, the MAC performs decoding, as well as CRC verification. The packet is forwarded to the packet classifier that examines the packet, performs CAM and PRAM lookups to determine the forwarding identifier (FID), priority, VLAN, and other information required for the forwarding of this packet. The forwarding information is encoded into a Foundry internal header that is attached to the packet by the packet classifier. In case no match is found in the CAM, the packet is forwarded to the management module for further processing, and to create CAM and PRAM entries that will allow hardware forwarding of subsequent packets targeting the same destination.

1 Refer to section 6 for an overview of Virtual Output Queuing.


Page 17 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

The shared memory switch fabric transfers the packet to the high speed shared memory, and notifies the buffer manager of the packet’s FID, priority, buffer number, and whether it should be copied to the management module. The buffer manager, responsible for managing the priority queues, assigns the buffer number of the packet to the appropriate queue(s) of the destination port(s). When the buffer number gets to the front of its queue, the buffer manager instructs the shared memory switch fabric to retrieve the packet from the shared memory and send it to the destination port. The destination(s) of the packet could be a port on the same module (within the same port group or not), a port on another module, a mirror port, or the management module. The buffer manager then deletes the packet from the shared memory. In the case of a multicast packet, a counter is initialized to a value equal to the number of destinations that should receive the packet. The counter is decremented whenever the packet is transmitted out one of the destination ports. When the last destination port (in the multicast group) sends out the packet, the counter reaches zero, and the buffer manager removes the packet from the shared memory. Multicast replication occurs in a hierarchical manner in order to keep the replication to a minimum at each switch fabric level. Depending on the destination of the multicast packet, forwarding occurs as follows: • If the packet has a destination port(s) within the same port group, one copy of the

packet (per port) is sent to that port(s). • In case the packet has a destination port(s) within another port group(s) on the same

module, only one copy (per port group) is sent to the switch fabric(s) of that other port group(s).

• In case the packet has a destination port(s) on another interface module(s), only one copy (per module) is sent to that module(s).

On the destination port group, the multicast packet is buffered and queued as usual till it reaches the head of its queue. The shared memory switch fabric retrieves the packet and delivers it to the VLAN multicast assist module and transmit pipeline of the destination port. There, VLAN ID attachment is performed, and the internal header is stripped. The MAC receives the packet, performs encoding, CRC calculation and attachment, and transfers the packet to the physical port.

5.2. Multiple Destination Output Priority Queues Foundry’s architecture offers 4 queues per destination to allow up to 4 levels of traffic prioritization. The architecture honors those priority levels, and schedules service for the prioritized traffic flows through the use of: • 4 output queues per physical port within the same port group • 4 output queues per neighboring port group


Page 18 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

• 4 output queues per neighboring interface module The diagram below depicts port group 1 of an interface module in slot 1 of a 4 slot chassis. The diagram shows the queues available for transmission out the local ports, as well as to other modules in the same chassis.

Priority 0Priority 1Priority 2Priority 3Physical

Port


Port


Port


Port

To Port Group 2

To Port Group 3

To Port Group 4

Port Group Switching Logic and Switch Fabric

To Module 2

To Module 3

To Module 4

To Module Crosspoint

Switch Fabric

Priority 0Priority 1Priority 2Priority 3






Figure 7: Four priority queues per destination.

6. The Chassis Crosspoint Switch Fabric The chassis crosspoint switch fabric provides high performance non blocking packet forwarding between the different interface modules within the chassis.


Page 19 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

By its very nature, the cross point switch fabric connects each interface slot to every other slot within the chassis. In other words, it allows for multiple simultaneous flows to be carried between the interface modules without having to arbitrate for a single path, or to be controlled by a single management entity, in order to avoid blocking within the fabric. The switch fabrics of all chassis were designed with ample bandwidth capable of serving all interface module slots simultaneously at full speed without any blocking (offering each interface slot an 8 Gbps feed into it, and 8 Gbps out of it). The use of a traditional input queuing approach or an output queuing approach always results in blocking situations. Rather than relying on any of those techniques, the crosspoint switch fabric queues packets on the input side of the fabric in separate queues based on the target slot, a technique known as Virtual Output Queuing, which eliminates head of line blocking. Head of line blocking typically occurs in input queuing fabrics, with a single queue per input. It occurs when a packet has a clear path to its destination, yet remains in the queue, because a packet in front of it, at the head of the queue was not yet scheduled for delivery to its destination. Hence, a packet at the head of the queue waiting to be served blocks other packets behind it, even though they could have been served during that same time cycle. This caveat significantly reduces the overall throughput of the switch fabric. Even under the most normal traffic conditions, head of line blocking could reduce the throughput of a switch fabric to 59%. Virtual output queuing eliminates head of line blocking, and boosts throughput to 100% of the switch fabric bandwidth. The technique places multiple queues on the input side, each queue corresponding to a given output (destination) – hence, the name. By doing so, a packet at the head of a queue that can not be transferred during a time cycle – say, because its destination is busy – would not block other packets that have clear paths to their destination – since those packets will be in other queues. Perhaps a good analogy that describes this is a highway with multiple exit lanes, each leading to a different exit. Cars taking an exit lane in order to gain access to the associated exit (i.e., queuing for that exit), will not block other cars heading for other exits, nor will they block the cars continuing forward. The diagram below illustrates the chassis crosspoint switch fabric for an 8 slot chassis. Similar architectures are used for the 4 and 15 slot chassis, with different slot numbers, paths, and overall fabric bandwidth.


Page 20 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Figure 8: Chassis crosspoint switch fabric for an 8 slot chassis.

6.1. Handling Unicast Traffic Based on the information contained in the FID and the packet priority, the buffer manager places the buffer number into the appropriate priority queue. The packet sits in the shared memory buffer pool until its buffer number reaches the head of the queue. Once there, the buffer manager instructs the shared memory switch fabric to deliver the packet to the destination module. The shared memory switch fabric delivers the packet to the module crosspoint switch fabric, which in turn, transfers the packet on the fly to the backplane. At the destination module, the packet is transferred on the fly through the module crosspoint switch fabric to the destination port group’s switch fabric. The packet is stored in the port group’s shared memory, and the buffer manager is notified of its buffer number. Based on the packet’s FID and priority, the buffer manager queues the packet.


Page 21 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

When the packet’s buffer number reaches the head of its queue, the buffer manager instructs shared memory switch fabric to deliver the packet to the port, and deletes the packet from the shared memory.

6.2. Handling Multicast Traffic With multicast traffic, the FID identifies all the outgoing ports for the multicast group. If the members of the multicast group exist on multiple interface modules, the buffer manager places the buffer number (only once) into the appropriate output priority queue for each destination module. The same process takes place as previously described for unicast traffic with some minor modifications. The packet sits in the shared memory buffer pool until its buffer number reaches the head of the queue, which causes the buffer manager to instruct the shared memory switch fabric to deliver the packet to its destination module. Only one copy of the packet travels to each destination interface module (not one per destination port), so if multicast members exist on three modules, three copies (one per module) of the packet move through the switch fabric. Only after all modules have received their copy of the packet does the ingress module clear the packet from the shared memory. In other words, a counter based on the number of destinations (local port group ports, destination port groups, and destination modules1) is used. When the counter reaches zero, this means that all destinations have received their copy of the packet and the ingress module removes the packet from its shared memory. At the destination module(s), the packet is transferred to its destination port group(s) through the module crosspoint switch fabric. Should the packet be destined for ports lying within more than one port group, the module crosspoint switch fabric replicates the packet, one copy per destination port group, as mentioned in section 4.3. At the port group level, the packet is stored in the shared memory. Based on the packets FID, the buffer manager places the packet’s buffer number on the appropriate port queue or queues – if the port group has more than one multicast member port. Once the buffer number reaches the head of its queue, the buffer manager instructs the shared memory switch fabric to send the packet to the associated port. A counter is used to make sure that all multicast member ports – within the port group – received copies of the multicast packet. Once the counter reaches zero, the packet is removed from the shared memory.

7. System Management Architecture The system management functions are achieved through the addition of a Management Module to the chassis, which provides the operating environment and feature set available to the user. The management module could generally be plugged in any slot in 1 Refer to section 5.1 for a description of multicast replication in the JetCore architecture.


Page 22 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

the chassis without restrictions. Since the management module has access to the chassis crosspoint switch fabric, versions of the management module(s) are offered that have physical ports on them which increases the port density and versatility of the chassis.

7.1. System Management Interface As shown earlier in Figure 5, each of the components on an interface module has a system management interface that connects it to the command bus. Through the command bus, the management module is able to initialize, control, and update any of the interface modules in the chassis – layer 2/3/4 updates, FIDs, etc. The system management interface, also, provides access to the various packet statistics from the interface module.

7.2. Command Bus A 2.1 Gbps dedicated bus for management connects the management module to all the other interface modules and packet classifiers in the chassis. It carries the information associated with the chassis configuration and control.

8. IronWare OS Overview IronWare is the operating system managing all Foundry chassis products, with different builds optimized for the system they are intended for and the environment they are intended to be run in – enterprise vs. service provider. By doing so, the IronWare OS offers a consistent user interface, and a consistent behavior (from protocol implementations) across all platforms which facilitates the deployment and management of the device, thus, lowering the total cost of ownership. The following is a brief overview of the IronWare OS architecture. The diagram below depicts the major components – from a high level – of the operating system. The diagram should be viewed as a component relationship diagram rather but not as a protocol stack diagram.


Page 23 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Hardware

Kernel/Hardware Drivers

L-2/3/4/MPLS Forwarding Tables

L-2 ProtocolsSTP, RSTP, MRP, VSRP

IP

IP Routing ProtocolsRIP, OSPF, IS-IS, BGP, DVMRP, PIM-DM, PIM-SM

MPLSRSVP/TE, LDP

IPX AppleTalk

System Management

syslog

Web

SNMP

telnet

Console

Chassis Mgmt.

Figure 9: IronWare OS architecture.

Kernel The kernel manages the system resources, and is, also, responsible for providing an interface to the hardware: for initialization, programming, monitoring, etc. Layer-2/3/4/MPLS Forwarding Tables The forwarding tables constitute the forwarding information bases that are going to be used for programming the hardware (the CAM). These include for instance the layer-2 MAC station table (for L-2 switching), the IP forwarding table, IP cache table, the session table (for ACLs). Layer-2 Protocols Several layer-2 protocols are supported: STP, RSTP (802.1w), VSRP, MRP. They determine the active layer-2 topology to be used, and control the logical blocking and unblocking of the ports, as well as the aging or flushing of the MAC station table entries in case of a topology change. IP This is the actual implementation of the TCP/IP protocol stack, which is used by other IP routing protocols.


Page 24 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

IP Routing Protocols These include RIP, OSPF, IS-IS, BGP, DVMRP, PIM-DM, and PIM-SM. Each of these protocols uses services from the IP component as well as the layer-3 forwarding tables. Each protocol maintains its own routing table, and only the best routes out of all such routes are included in the layer-3 forwarding table. MPLS MPLS protocol implementations are included in this area: RSVP/TE, LDP. These protocol modules use services from the IP component – for protocol information transport – and interact with the layer-3 and MPLS forwarding tables to obtain layer-3 topology information and create MPLS label forwarding entries that would be programmed in the hardware. IPX and AppleTalk On some enterprise builds of IronWare these two protocol implementations are provided to allow for multiprotocol routing within the enterprise campus. Chassis Management The chassis management component is responsible for the general monitoring and management of the hardware within the chassis. This includes monitoring the insertion and removal of interface/management modules, initializing those modules, checking software versions and managing software upgrades on those modules, monitoring temperature sensors, etc. It interacts with the system management component to provide hardware monitoring and management functions. System Management System management includes all the software modules that facilitate the management and monitoring of the system. The console, telnet, and the web modules facilitate the management and interactive monitoring of the system. The syslog, and SNMP modules allow for the long term (non-interactive) monitoring of the system. The system management modules use services from the kernel as well as the chassis management component.

9. Benefits of the JetCore Architecture The new breed of chassis based on the JetCore architecture packs more features than its predecessor, IronCore, better cost efficiency, while maintaining Foundry’s commitment to providing high performance solutions to both enterprise as well as service provider infrastructures.


Page 25 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

A Wire Speed Non Blocking Architecture Foundry’s interface modules are built using high bandwidth components – crosspoint fabrics, shared memory, shared memory switch fabric, packet classifiers, etc. – capable of serving traffic from and to all the physical ports on the module without any sacrifice in performance, or any packet being blocked by another packet due contention for a limited bandwidth resource. The chassis crosspoint switching fabric connecting the interface modules is built with enough bandwidth to serve all the interface module slots within the chassis at their full data rate, full duplex 8 Gbps (8 Gbps into the slot, 8 Gbps out the slot). The use of Virtual Output Queuing eliminates head of line blocking making 100% of the switch fabric bandwidth truly available for packet transfer. The combination of wire speed performance, and a non blocking architecture makes it easier for the network architect to provide bandwidth guarantees for the application flows that are going to use the network. A non blocking architecture, also, avoids packet loss, and makes packet delay and jitter more predictable. Ultra Low Latency By their very design, the Foundry chassis products provide ultra low latency. The interface module design as well as the chassis crosspoint switch fabric design allow for a very low switch/router latency of around 6 microseconds – typically measured for 64 octet packets. Low latency is a key attribute of high performance network based computing, and is also important for delay sensitive applications – voice over IP, video conferencing, etc. – especially in large scale networks where device delay and jitter could significantly add up. Packet Prioritization and Coloring JetCore provides the ability to prioritize packets based on either 802.1p bits, or ToS/DSCP bits. A choice of either strict prioritization, or weighted fair queuing (with configurable weights) makes it possible to tune the switch/router traffic management to the needs of the applications using the infrastructure. In addition, JetCore allows for rewriting ToS/DSCP, or 802.1p priority bits in hardware. This feature allows for applying stricter policies to traffic management within the network. Enterprise network managers could promote/demote priorities of traffic flows based on their current needs, and the available network resources. Service providers could use this feature at the network edge in order to offer different service levels to their customers. Four Levels of Prioritization As in IronCore, JetCore provides 4 levels of prioritization by using 4 queues per port in the chassis, which makes it suitable even for the highly QoS demanding deployments.


Page 26 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

Many other platforms existing today might provide only 2 or 3 levels of prioritization within their architecture. Hardware Based Traffic Policing JetCore is capable of performing traffic policing via a built-in credit (token) based system that assigns a traffic flow a certain number of credits to consume within a given interval (1 ms), with the possibility of carrying over unused credits from a previous interval(s). Unlike other systems, traffic policing could be performed at the interface level without any assistance from the management module, or any other policy module. Performing this function in hardware, in such a distributed manner, allows for the scalable deployment of traffic policing without any sacrifice in performance, since traffic is never sent to a central module for inspection. Jumbo Frame Support JetCore introduces jumbo frame support up to 14,336 octet frames. In high speed computing applications – supercomputing, large clusters – where large amounts of data are exchanged, and processor interrupts are costly, Jumbo frames provide much higher performance. With Jumbo frames, the application data throughput increases, and the processors do not have to be interrupted as often as with the smaller normal sized frames. Most practical applications of Jumbo frames today use a frame size up to 9 k octets. Larger, More Efficient Ternary CAM JetCore uses ternary CAMs, which allows for the creation of CAM entries having Don’t Care bits. This results in a more efficient utilization of CAM space. Combined with larger CAM sizes than its predecessor, IronCore, the JetCore architecture offers even better scalability and higher performance. Wire Speed Policy Based Routing & ACLs The new architecture provides line rate policy based routing and filtering in hardware, without requiring that the first packet of the flow goes to the management module. This makes the architecture suitable for environments requiring routing and access policies, yet have very bursty applications. Scalable Traffic Monitoring via sFlow & XRMON The random packet sampling and monitoring capability embedded in the JetCore ASICs makes scalable traffic monitoring, traffic pattern analysis, and capacity planning more viable than before. Other approaches to solving this problem proved to be not so scalable, unfeasible, and typically caused some considerable performance degradation. By providing the capability in the ASICs, JetCore provides a solution that is scalable even up to 10 Gigabit Ethernet speeds, cost efficient, and has no performance penalties, while other solutions still have to face those challenges.


Page 27 of 27

FOU

ND

RY

NE

TW

OR

KS

WH

ITE

PA

PER

10. Summary Foundry’s new JetCore based chassis switches and routers provide a very high performance solution, exhibiting wire speed non blocking packet forwarding, 4 levels of traffic prioritization, while maintaining cost efficiency. The chassis offer the industry’s highest Gigabit Ethernet port density, up to 232 in a single 15 slot chassis (only 17 rack units), with layer-2/3 switching capabilities. Combined with JetCore’s new value added features as hardware based traffic policing, sFlow/XRMON traffic monitoring, hardware based VLAN multicast, larger ternary CAMs, the new chassis provide a very attractive solution for both enterprise (LAN/Campus) networks and service provider networks (MAN/WAN).

Ahmed Abdelhalim Product Marketing Engineer Foundry Networks, Inc. Headquarters 2100 Gold Street P.O. Box 649100 San Jose, CA 95164-9100 U.S. and Canada Toll-free: 1 (888) TURBOLAN Direct Telephone: +1 408 586-1700 Fax: +1 408 586-1900 Web: http://www.foundrynet.com © 2003 Foundry Networks, Inc. All Rights Reserved. Although Foundry has attempted to provide accurate information in these materials, Foundry assumes no legal responsibility for the accuracy or completeness of the information. More specific information is available on request from Foundry. Please note that Foundry's product information does not constitute or contain any guarantee, warranty or legally binding representation, unless expressly identified as such in a duly signed writing.

Documents

JetCore Based Chassis Systems - Dich