WWhhiittee PPaappeerr CARRIER CLASS GIGABIT ETHERNET IN TRANSIT APPLYING ENABLING NETWORKING TECHNOLOGY TO LIGHT RAIL. A CASE STUDY FOR THE NEW PHOENIX ARIZONA LIGHT RAIL PROJECT.

Prepared by: Karl Witbeck, PMP Parsons Brinckerhoff 101 N 1st Ave, Suite 1300 Phoenix, AZ 85003 Ph: 602-495-8236 Email: witbeck@pbworld.com

Glen Tuner Atrica Inc. 1436 Cambridge Pointe Drive Hixson TN 37343 Ph: 423-667-6548 Email: glen_turner@atrica.com

For the: AREMA C & S Enabling Operations

Technology Conference May 22, 2007 Calgary, Alberta, Canada

TABLE OF CONTENTS

1. ABSTRACT ............................................................................................................2. PHOENIX COMMUNICATIONS DESIGN BACKGROUND................................... 3. DESIGN EVOLUTION ............................................................................................ 4. CARRIER CLASS ETHERNET BACKGROUND................................................... 5. CARRIER CLASS ETHERNET TECHNOLOGY.................................................... 6. CONCLUSION........................................................................................................

TABLE OF TABLES Table 1. Network Technology Comparison ................................................................

TABLE OF FIGURES Figure 1 Single Ring Topology ..................................................................................... Figure 2. Quad dual collapsed rings @ 1 Gbps each ................................................. Figure 3. Typical Station Node..................................................................................... Figure 4. Connections are established via VLAN tunnels and MPLS LSPs. ...........

1. ABSTRACT

The networking industry is transitioning from legacy networks such as ATM, SONET,

Frame Relay, RPR Ethernet and others to integrated Internet Protocol (IP) based

networks. As an industry, Transit agencies typically are slow to embrace new

technology, preferring to rely on technologies field proven over many years to

understand and mitigate all potential risks.

However, communications networks are evolving at a rapid pace requiring all industries,

including railroad and transit to adopt a more flexible and timely approach to evaluating

and implementing new communications technologies to provide new and enhanced

services that are increasingly being requested by Transit customers.

The 20 mile $1.43 Billion new starts Light Rail project is no exception to this trend with

the communications systems design evolving over several years along with the many

other design elements of the project. Initially, well established legacy SONET

technology was assumed to be the safest approach and would leverage economies of

scale. However, as the applications evolved from analog to digital for CCTV, Telephone,

SCADA, PA/VMS and others the need to re-think the network technology selection

became apparent which resulted in the selection of an emerging technology called

Carrier Class Gigabit Ethernet.

This white paper is a case study of this technology as implemented for the light rail

project in the Phoenix, Arizona metro area. The advantages of the technology to the

project and the applications that were “enabled” by this technology are discussed.

Transit organizations considering upgrading existing networking systems or construction

of new infrastructure should consider and evaluate the technology reviewed in this white

paper.

2. PHOENIX COMMUNICATIONS DESIGN BACKGROUND

The preliminary design for the project was completed in late 2003 based on standard

SONET OC-48 technology with the following design criteria.

• Single-mode fiber optic Dual Collapsed Ring with a SONET node at each passenger station (27 stations nodes), the control center and maintenance facility

• Sub 50ms self healing Node protection

• High system reliability (99.9995% Availability)

• Scalable for future expansion without fork-lift upgrade

• Re-configurable in the field without replacement or factory changes

• Mixed applications/configurations on one network (voice/data/video)

• Field proven with verifiable history

• Must meet environmental requirements (+122 deg. F/ +50 C)

The original topology developed for the SONET based design used a single dual

collapsed ring (72 fiber cable) at the OC-48 (2.488 Gbps) rate with fixed bandwidth (via

hardware) dropped and inserted at each node as needed. A collapsed ring is the use of

a single cable with fibers terminated every other node in a leap-frog arrangement to

create a functional ring without the use of two physical cables and routes. The following

figure illustrates this topology.

Figure 1 Single Ring Topology

The preliminary design also assumed the following applications would be supported by

the network:

• Analog CCTV system at passenger stations, park & ride lots and maintenance facility.

• Analog PBX Telephone system

• SCADA system for traction power and train control (non-IP based).

• Analog Public Address system at each station and maintenance facility.

• Variable Message Signs at each station

• Ticket Vending Machines at each station

• Analog voice radio system for operations and maintenance.

Following these efforts the IEEE 802.17 Resilient Packet Ring (RPR) was released

September 2004 and was determined that it would be advantageous to consider this

option. Subsequently a bid was awarded with the Gigabit Ethernet RPR option in early

2005. However this technology would be short lived and be superseded with another

technology called Carrier Class Ethernet in mid 2006 developed by industry and the

Ethernet Forum. Additionally, one of the key 802.17 RPR vendors (Luminous) went

bankrupt in 2006 in part do to this technology shift.

3. DESIGN EVOLUTION

To make use of this new technology, the original topology needed to be reconfigured for

optimal bandwidth utilization, availability and utilization of available single mode fiber.

3.1 Fiber Optic Network Topology

The available network speeds for this technology at the time (mid 2006) the analysis was

performed was 1 Gigabit (GB) and was determined to be adequate. Since that time

higher speeds have become available (up to 10 GB). To deliver the applications needed

it was concluded that four fully protected rings at 1 GB speed each was needed as

illustrated in the figure below. Note that if full protection is not required at a node, then

the total of both set of fiber rings (1GB each), or 2 GB could be utilized. All station and

facility nodes were evenly distributed between each ring to balance the loading to the

extent possible.

Figure 2. Quad dual collapsed rings @ 1 Gbps each

The main driver which necessitates the adoption of faster and more efficient networking

technology is of course the applications desired or required by the end users and

ultimately the service provider organization. In the case of the phoenix light rail project,

the following applications were enabled by the utilization of a Carrier Class Gigabit

Ethernet Network.

• Digital CCTV – Real time streaming video (MPEG4 CODEC) and client/server base camera viewing and control.

• IP Based Telephony – Server based call management and voice mail

• IP Based SCADA – Remote Switches, PLC’s and terminal servers

• Digital PA System –Hardened PC at station controls .wav file messages

• All Digital Variable Message System – Hardened PC at station controls synchronized text messages to passengers.

• Ticket Vending and credit/debit validation over network

• IP Based Voice Radio Console

It should be noted also that the CCTV and IP telephony markets are rapidly evolving to

drive the expectations of users.

Beyond these applications, more advanced CCTV applications such as abandoned

package recognition and other security related applications are being considered for the

future. The network topology has the ability to expand or be reconfigured as the need

arises.

3.2 Node Configuration

A key advantage of delivering native Ethernet to the edge node is the ability to aggregate

all equipment and applications across one network. Through the connection oriented

services, each application can have it’s own dedicated bandwidth and priorities to

prevent any conflicts between them. More will be said about this discussed later in this

report. Once the fiber optic node topology was established, connections to the node by

each device were reconfigured to the standard RJ-45 Ethernet port. From each node a

100 MB Ethernet connection was allocated to a standard aggregation Ethernet switch at

each station, providing a standard 10/100 MB LAN at each station. On this station LAN,

standard network components could be utilized, leveraging economies of scale and

facilitating compatibility and standardization. The figure below illustrates a typical station

node configuration.

Figure 3. Typical Station Node

3.3 Network Technologies

We compared the three networking technologies most applicable to the project and

developed the following table to help identify and analyze the features and attributes that

were important and matched the design criteria. To the best of our knowledge, this the

first application of this technology in light rail in North America.

The key takeaways from this table are:

• CCE reduces the limitations traditionally associated with legacy technologies and

SONET specifically. From an owners perspective this is key to any long term

technology strategy.

• CCE provides a specific SLA (or QOS) to the edge of the network as needed for

each application.

• CCE provides a flexible and adaptable platform, mitigating short term

obsolescence.

The table below is a summary of this comparisons used to help make the decision to

adopt this technology.

Table 1. Network Technology Comparison

FEATURES

SONET802.17 RPR

Carrier Class

EthernetPROTECTIONSub 50 ms Link

Sub 50 ms Node N/ASCALABILITYNO VLAN Limitation LIMITED NO

Services Mapped to MPLS Label Switch Path NO NO

Integrated Optics

Flexible Remote Service Creation LIMITED LIMITEDService Level Agreement (SLA) CapabilitiesConnection Oriented Services *NO NO

Guaranteed End-to-End SLA (Quality of Service) NO NO

Integrated Network Management to User Port NO NO

End-to-end CIR & EIR CIR Only No CIRIntegration CapabilitiesTDM Traffic

Voice applications (Analog and VoIP) LIMITED

Data applications (Real time Video, database etc) LIMITED LIMITED

ILEC SupportRELIABILITYField Proven NO

99.999% or better Reliability (Availability)

Meet Environmental RequirementsManagementSNMP Capable

Realtime/Historical SLA metrics Reporting NO NOService EnablementPoint to Point (Ethernet private line)

Multipoint (Ethernet LAN Service) NO

Point to Multipoint (Multicast) NO NOTechnology ViabilityIndustry acceptance-product availability NO

TECHNOLOGY

*NOTE: SONET supports connection oriented circuits for point to point circuits only.

The use of Layer 2 switches on the edge to facilitate required applications negates any

SONET benefits.

4. CARRIER CLASS ETHERNET BACKGROUND

Networking Technology for public or private networks is rapidly changing as lower cost

transport technology for businesses, agencies and other organizations is introduced to

the industry. These services increasingly utilize packet-based networks, allowing

enterprises to connect to multiple sites, organizations, agencies and the Internet. In

addition, these new data networking technologies make it possible to provide enhanced

voice and data services via a common network. As organizations add new voice and

data services to their infrastructure, the demand on network resources continues to rise

exponentially. While legacy transport technologies including T1/E1, DS-3, OC-3/STM-1,

OC12/STM-4, frame relay, and ATM have been proven adequate in the past for delivery

of enterprise applications, they are not designed for transporting new and increasingly

complex data applications and services. These issues perhaps can best be explained by

examining how voice and data traffic traverses a network. In a typical TDM network such

as SONET, voice traffic must maintain a constant bit rate, i.e., a sustained, dedicated

slice of bandwidth needed to maintain a call. In contrast, data traffic is bursty,

characterized by a variable bit rate. When mapping data traffic into a voice model, a

network device must allocate the maximum amount of bandwidth corresponding to the

peak variable bit rate of the data transported. Since data rarely bursts to its maximum

capacity, this results in enormous bandwidth inefficiencies that lower the maximum

available capacity.

Recognizing the need to solve these challenges, the industry is constantly exploring

alternative architectures for delivering high bandwidth Ethernet services to the edge of

the network. However, such systems historically have lacked the scalability for mass

deployment, 50 millisecond protection, TDM support, and guaranteed Service Level

Agreements (SLAs). Nor do they provide the carrier class service provisioning and

management systems necessary for offering Ethernet services on a large scale.

As an alternative, organizations have attempted to utilize their established

SONET/Synchronous Digital Hierarchy (SDH) networks to deliver Ethernet services.

While this approach can be cost effective for introduction of point-to-point Ethernet

services, the resulting architecture does not scale well for deployment of bandwidth

intensive applications. In addition, Ethernet services deployed using this method cannot

be managed to the high degree of control necessary to maintenance cost structures.

Another approach involves the deployment of Ethernet services using a SONET/SDH

architecture coupled with Layer 2 aggregation switches. However, this method lacks the

ability to provide ironclad end-to-end service level guarantees and adds multiple

elements and complexity to the network. Not only does a combined SONET/Ethernet

system have a high capital expense, the costs for deploying, provisioning, and

maintaining Ethernet services across multiple platforms is a challenging proposition.

Plus, this lacks carrier class reliability and protection, and a scalable Layer 2 Ethernet

switching infrastructure necessary to provide bandwidth efficient Ethernet services.

Given these trends, a new networking technology called Carrier Class Ethernet was

developed. The following sections describe the technology in more detail.

5. CARRIER CLASS ETHERNET TECHNOLOGY

Carrier Class Ethernet (CCE) optical technology differs from traditional Ethernet

delivered over Local Area Networks (LANs). By contrast, CCE supports carrier-grade

features such as:

• Class of service (COS)

• Quality of service (QoS)

• Security

• Scalability from 1 to 100 Gbps

• SONET/SDH-like network protection with sub 50 ms restoration.

This technology eliminates the SONET/SDH and/or ATM layers between connections,

enabling organizations to simplify and minimize maintenance and operational costs.

Generally, delivery of Optical Ethernet services over a wide area network requires

deployment of components in the following three sectors of the network:

1. Access: Cost effective edge access equipment for terminating and aggregating

end point services should be deployed in the access segment and should be

able to terminate multiple services each with a measured service level

agreement (SLA).

2. Core: Scalable core network device(s) should be used in the core network for

aggregating traffic from the edge and providing carrier class services. This

equipment should also be able to route traffic to other core network devices.

3. Services: A carrier-class service management system capable of provisioning,

monitoring, and managing Ethernet services is necessary for successful

deployment of any Optical Ethernet network. The Ethernet service management

system should be integrated with the organizations existing network

management systems.

CCE provides technology for all of the above sectors. It provides a high-bandwidth

network architecture that allows scalability, flexibility, and efficiency as required to satisfy

growing bandwidth demands, decrease equipment and operations costs, and deliver

new services. It also includes low-cost, carrier optimized platforms that combine the

latest in packet switching technologies, traffic engineering enhancements, and optical

technologies suitable for delivery of services.

CCE equipment provides flexible bandwidth from 1 to 100 Gbps and multiple interfaces

to support legacy services, such as SONET and TDM traffic. Functionality is managed by

network management software providing rapid service provisioning, comprehensive

network, performance, and fault management, and integration with existing applications.

5.1 Features and Benefits

• Multiple Interfaces: CCE equipment provides many benefits, including

reductions in capital and on-going operational costs and also provides a variety of

interfaces. CCE equipment supports flexible service interfaces in a modular form

factor. This enables the flexibility to provide 10/100 Ethernet, Gigabit Ethernet, or

TDM (E1/T1, E3/DS3, OC-n/STM-n) interfaces at the edge or the core of the

network.

• Multiple Services Per Port. CCE equipment enhances service flexibility with the

ability to provide multiple service connections per port, with a committed SLA

corresponding to each service. In addition, subscriber traffic can be mapped to a

service based on multiple criteria, including port mapping, destination or source

IP address, DSCP/Pbit or VLAN ID.

• Flexible Physical Topologies. Provides for nearly any physical topology,

including:

o Point to point

o Dual homed point-to-point links

o 1G access ring

o 10G access rings

o Dual homed access rings.

• High-Density. CCE systems provide 1G, 10G, and optional Dense Wavelength

Division Multiplexing (DWDM) interfaces, enabling the platform to serve areas of

very high density if needed. The architecture supports up to thirty-two 10 Gigabit

Ethernet DWDM functionality. A switching capacity of 320 to 600 Gbps also is

supported. The system supports more than 500,000 connections in a single MAN

and 64,000 can be terminated on an individual device.

• Flexible Core Network Topologies. The core architecture works with multiple

network topologies that today’s organizations must support to continue

operations. These include mesh, partial mesh, or ring networks. In addition, a

single core node can support multiple rings, meshes, or partial meshes, enabling

scalable growth.

• Simplified Architecture. CCE utilizes a minimum number of high-density

devices each optimized for a specific role within the network, thus reducing

operational expenses. This is in contrast to other standard SONET/DWDM

systems, which require a far greater number of nodes and equipment, and much

more complicated and lengthy configuration.

• Integrated Optical Functionality For Increased Scalability. Integrated optical

capability in the core network architecture enables organizations to add 10 Gb

DWDM segments as required. Additional integrated 1310nm/1550nm

multiplexing enables integration SONET/SDH networks with N x 10 Gigabit

Ethernet transport networks and provides full optical network capabilities,

including amplification and dispersion compensation.

5.2 CCE Technology Services

Any system designed for delivery of Optical Ethernet services that is classified as carrier-

class includes six key features as follows and each are discussed in more detail below:

1. Guaranteed SLAs

2. Optical integration

3. SONET/SDH support

4. Circuit emulation over Ethernet

5. Carrier-class protection

6. Network management

5.2.1 Guaranteed SLAs

CCE technology provides SLA control via a connection-oriented approach with strict

adherence to SLA parameters. Connections are created using Virtual LAN (VLAN) and

MultiProtocol Label Switching (MPLS) technologies for tunnel definition. The system

creates a VLAN tunnel and then sends traffic to the core network. At the core, the tunnel

is mapped to an MPLS Label Switch Path (LSP). After leaving the core, the packet is

converted back into a VLAN tunnel and sent to the egress device as illustrated in the

figure below. Connections are established via VLAN tunnels and MPLS LSPs.

Karl Witbeck Page 16 of 22 Glen Turner

Figure 4. Connections are established via VLAN tunnels and MPLS LSPs.

All devices/application traffic is placed within a tunnel, each conforming to the following

SLA parameters:

• Delivery guarantees [using Committed Information Rate (CIR)/Excess Information Rate (EIR) values provisioned in 1 Mbps increments]

• Delay and jitter tolerance

• Protection levels

Delivery Guarantees — During the provisioning process, each connection is assigned

CIR and EIR values conforming to a subscriber’s SLA and associated applications. For

example, an SLA for a subscriber who frequently uses mission critical applications can

be assigned a higher CIR value and a lower EIR value to ensure guaranteed traffic

delivery. Alternatively, best effort services can be provisioned through a CIR value of 0

and a non-zero EIR value. These parameters form a traffic contract that is strictly

monitored and controlled throughout the session to ensure QoS.

Delay and Jitter Tolerance —CCE includes an advanced queuing mechanism that

controls the amount of delay and jitter. This enables network operators to establish

priorities for delay and jitter sensitive applications. In addition, the system can query an

end-to-end connection to determine the delay across the network.

5.2.2 SONET/SDH Support

CCE technology enables organizations with existing SONET/SDH platforms to deploy

Optical Ethernet technology, allowing them to leverage their existing equipment

investment. In addition, this eliminates the need for SONETSDH ring upgrades, greatly

reducing capital expenses for future improvements, while enabling deployment of new

services.

Various methods are provided for SONET/SDH integration such as:

• Coexistence with legacy SONET platforms

• SONET/SDH circuit emulation

Coexistence with legacy SONET platforms — For locations with limited fiber access,

CCE enables organizations to deploy optical Ethernet services that can coexist with

legacy SONET platforms. This is accomplished by using a simple optical splitter. Optical

splitters add entire C-band wavelengths to an existing 1310nm SONET/SDH ring,

enabling delivery of Ethernet data services without sacrificing the performance,

operation, or manageability of an existing SONET network or its components.

SONET/SDH Circuit Emulation Services (CES) — CCE offers advanced SONET/SDH

circuit emulation as needed such as; Packet over SONET/SDH (POS) and CES. CES

provides the ability to emulate SONET/SDH over Ethernet networks by converting

SONET/SDH streams into Ethernet frames and then translating frames back to

SONET/SDH at the destination node. POS technology compresses a large number of

Ethernet frames into Synchronous Transport Signal (STS) frames, with minimal

overhead, for efficient transport over optical networks. CCE QoS mechanisms provide

guaranteed delivery, jitter specifications, and delay that SONET/SDH requires for both

CES and POS solutions.

5.2.3 Circuit Emulation Over Ethernet

CCE TDM CES technology enables organizations with Ethernet-based infrastructure to

offer TDM services with the service quality of SONET/SDH. CES leverages advanced

QoS mechanisms, protection schemes, and VLAN/MPLS based connection capabilities,

enabling transport of synchronous traffic over asynchronous transport systems. This

technology works by packing multiple TDM streams into an Ethernet packet and then

reassembling Ethernet packets into the original TDM frames while maintaining clock

synchronization.

CCE TDM CES can also provide the same capabilities of a SONET/SDH network,

including:

• Synchronous data transport

• Guaranteed bandwidth

• Data multiplexing

• Control channels

• Sub 50 ms protection

• Clock recovery

The TDM CES allows a synchronous clock to be transported via Ethernet frames,

providing:

• Clock accuracy to address all Telco synchronization requirements, including

Stratum 2, Stratum 3, and Stratum 3E

• Compensation for clock changes on detection of dropped packets

• Hierarchical clock dissemination (e.g. source clock is DS-3 and destination is T1)

5.2.4 Protection

CCE provides sub-50 ms protection, a necessary ingredient for providing carrier class

service. CCE not only provides protection similar to that of SONET/SDH, it supports

multiple grades of restoration, from no protection to sub-50 ms restoration. This

protection scheme utilizes MPLS and/or VLAN tagging to create backup tunnels that

guarantee data delivery in the event of a primary path failure. A hardware mechanism is

used to detect link and node outages.

Protection services operate independent of transport, enablingnetworks to utilize various

transmission technologies, interconnected to create a heterogeneous network

infrastructure. Because CCE bases its protection model on industry standard tagging

mechanisms, these services operate over various transport technologies, even those

that do not have their own protection capabilities.

5.3 Network Management System

Network Management System (NMS) software enables full control of an CCE equipment,

allowing organizations to create, maintain, and report on Ethernet services via a single

platform. Traditionally, such functionality has only been possible using multiple network

management platforms for access, core, and DWDM systems. The software supports

carrier-class management capabilities across all CCE devices, including fault

management, configuration management (devices and configuration), performance

management, security management, and accounting support.

• Modular Architecture – CCE modular architecture enables all management

elements to coexist on a single platform or be separated on dedicated servers.

The primary elements of the NMS software include, Backend server, Application

server, Database server, and Upper layer API/OSS server. The NMS is based

on a redundant, scalable, modular, and multi-layered architecture that allows

management of hundreds of network elements. Additionally, the architecture

enables management and SLA assurance of hundreds of thousands of services

and connections. The system maintains real-time and historic statistics.

• Rapid service provisioning – The NMS enables fast and accurate provisioning

and support of Ethernet services and associated SLA parameters via a simple to

use point and click interface. Once a service has been provisioned, a full suite of

management applications support on-going operations. The NMS also maintains

a detailed inventory of all connections, network resources, network elements, and

network capacity to assist in the on-going network engineering and capacity

planning.

• QoS management – CCE allows bandwidth provisioning based on CIR and EIR,

enabling network operators to optimize ring traffic and provision more services.

Best effort, guaranteed with no burst, and guaranteed with burst ability service

levels can be supported on a single ring. This model can also be applied to TLS,

providing the ability to assign different guarantees to different locations in a

multisite network.

6. CONCLUSION

For the Phoenix Light Rail project, Carrier Class Ethernet was the right technology at the

right time at an affordable price that met or exceeded our design requirements.

As other transit organization’s strive to become increasingly efficient and increase the

quality, quantity, and types of services offered to it’s customers; new network transport

technology is needed that help meet the needs of increased rider ship and service level

expectations. To this end, CCE is an excellent combination of enabling technology,

architecture, and service features. CCE, with its carrier-class, high-density, and scalable

features, is an enabling technology worth considering, enabling organizations to reduce

capital, operating and maintenance expenses, simplify operations, and provide quality

network based services to an ever demanding public.

We recommend that all Transit agencies, railroads and other organizations include into

their technology strategy planning to consider Carrier Class Ethernet for their

infrastructure improvement projects.

FOR MORE INFORMATION CONTACT:

KARL WITBECK – 602-495-8236, kwitbeck@valleymetro.org,

http://www.valleymetro.org/METRO_light_rail/

GLEN TURNER - 423-667-6548 glen_turner@atrica.com

www.atrica.com