SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) – Networking RCEs RCE Training Workshop Matt Weaver, [email protected]

SLAC Particle Physics & Astrophysics

The Cluster Interconnect The Cluster Interconnect Module (CIM)Module (CIM)

– Networking RCEs– Networking RCEs

RCE Training WorkshopMatt Weaver,

[email protected] June, 2009


The Cluster Interconnect Module (CIM) 2

OutlineOutline

• ATCA Networks– Zone 2 Base + Fabric– Redundancy traded for multiplicity/bandwidth

• CIM Hardware Design– Switch ASIC Properties– Switch mapping to Zone 2 and Zone 3– Rear Transition Module (RTM)– Switch Host Processor

• CIM Software– Development environment– Protocols

• CIM Usage– ATCA Hub slot mapping to RCEs– Front Panel Features


ATCA NetworksATCA Networks



Zone 2 Base + FabricZone 2 Base + Fabric

• CIM serves as a hub in the dual-star networks

• CIM Base supports– 1G serial ethernet (SGMII) for RCE– 10/100/1000 BaseT ethernet for commercial

• CIM Fabric supports– 10G ethernet (4-lanes XAUI)

Dual-Star



Redundancy Traded for Redundancy Traded for Multiplicity/BandwidthMultiplicity/Bandwidth

• Dual Star topology introduced to provide redundancy for maximizing uptime in telecom environments

• Data acquisition environment can make use of 2x bandwidth from each node

• Requires extra connectivity between hubs and outside the shelf

Dual-Star


CIM Hardware DesignCIM Hardware Design



Switch ASIC PropertiesSwitch ASIC Properties

• The current generation Cluster Interconnect Fabric is based upon the Fulcrum Micro FM2224 ASIC– 24-port Ethernet Layer 2 switch– Each port 1Gb / 2.5Gb / 10Gb capable

– Fully provisioned, non-blocking, shared memory– Low-latency cut-through operation (200ns)– Pause frame flow control– 1.5W per channel– Hardware MAC address lookup, static & dynamic– Supports trapping of link protocol messages

Compatible with RCE MGT serial 1 (2.5)Gb



Switch Mapping to Zone 2 and Zone Switch Mapping to Zone 2 and Zone 33

FM2224

Fabricnetwork

1-GE

10-GE XFP

XFP

XFP

XFP

XFP

XFP

XFP

XFP

XFP

Front Board RTM

10-GE1-GE

10-GE XFP

10-GE

(external network)

(external nextork)

(external network)

(external network)

FM2224

Basenetwork

( Dual Star Backplane Channels )

( Neighbor Hub Interconnect )

( Neighbor Hub Interconnect )

13

13

5

4

13

3

2



Rear Transition Module (RTM)Rear Transition Module (RTM)

• Bulk cabling to external network is in the rear• Technology choice is largely independent of front-

board design– CX4 electrical connections

• Inexpensive, Passive• short range (~15m)

– XFP optical transceivers• Pluggable transceivers• Short range or long range ($250-$1000)• ~1W / channel

• Total external bandwidth limited by panel real estate– 8 XFPs or 8 CX4– High density optical technology becoming available

(20-40)



Switch Host ProcessorSwitch Host Processor

• One host processor for managing both network switches– Implemented in Xilinx Virtex-4 Embedded PPC405 (a

la RCE)– 128MB DRAM– 32MB flash memory file system– FM2224 management bus interface– Logical CPU port with DMA Engine on each network– XFP and BaseT PHY management bus masters– IPM Controller for ATCA electronic-keying


CIM SoftwareCIM Software



Development EnvironmentDevelopment Environment

• GNU C++ cross-platform programming• RTEMS 4.9• Same as RCE

– Similar hardware– Inherits as much core software and tools as possible

• Most likely developments will be to implement new management protocols or enhance monitoring



Management ProtocolsManagement Protocols

• IGMP snooping (multicast control) implemented– Limit forwarding of multicast data to only interested

ports– Lessens burden on resource limited NICs

• IPMI electronic-keying implemented– Interfaces with ATCA Shelf Manager to achieve

safe integration with commercial board Fabric channels

• Large network management protocols not yet implemented– Desirable for large sets of switches { RSTP, LACP }– Not difficult, just hasn’t been a development priority



User ConfigurationUser Configuration

• DHCP Vendor Encapsulated Options– IP address assignment of both network host ports– Switch channel configurations (in absence of Shelf

Mgmt)– VLAN port assignments– Static MAC address assignments

• Terminal -> Flash memory– Manual changes applied via the shell– Accumulated configuration is written to flash

memory


CIM UsageCIM Usage



ATCA Hub Slot Mapping to RCEsATCA Hub Slot Mapping to RCEs



Front Panel Features – Rotary Switch and Reset Front Panel Features – Rotary Switch and Reset ButtonButton

• 16-position rotary switch– Configuration

selection– Read following reset

• Hard reset button



Front Panel Features – Auxiliary Front Panel Features – Auxiliary PortsPorts

• XFP and RJ45 external connectivity to each network

• Convenience for test-stand setups

FabricNetwork

BaseNetwork

RJ45

XFP



Rear Panel PortsRear Panel Ports

• High throughput external connectivity• 5 x10Gb (Fabric), 3 x10Gb (Base)

FabricP3-7

BaseP3-5

Example XFP RTM



Front Panel Features – Multi Function DisplayFront Panel Features – Multi Function Display

• CIM – following reset• DHCP – attempting DHCP• B … - Base channels status• F … - Fabric channels status

enabled

link up

enabled

link up

Fabric orBase Zone 2

Front PanelZone 3

Slo

t 1/

2

Slo

t 3

Slo

t 4

Slo

t 5

Slo

t 14

P1

P2 P3

P5

P6

P7

P4

H-A

H-C

H-B



SummarySummary

• CIM is designed to make full use of ATCA provided bandwidth– Dual 10Gb and dual 1Gb channels to each slot– Several external 10Gb channels (6/4)– Fully provisioned packet switching fabric

• CIM Software– Same development environment as RCE– Customizable for large scale aggregation

Documents

SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) – Networking RCEs RCE Training Workshop Matt Weaver, [email protected]