Test Report - EANTC · Some Carrier Ethernet transport technologies implement advanced traffic management beyond traffic prioritization. A test of Differentiated Services Traffic

Test ReportMulti-Vendor Carrier Ethernet

Interoperability Event

Madrid, September 2006

Carrier Ethernet World Congress 2006 Public Interoperability Event

Editor’s NoteSince 1995, our multi-vendorinteroperability events havealways been technologyfocused: ATM, MPLS, or Voiceover IP.

In contrast, Carrier Ethernethas proven to be an interdisci-plinary challenge — exercis-ing our knowledge in diversephysical, transport, and appli-cation layer technologies.

For this event, we have created a full-fledged MPLSbackbone, tested 10Gigabit Ethernet optics betweencore router vendors, verified interoperability ofvarious new IEEE standards for Ethernet resilienceand OAM, and provided mock users with multicastIPTV content.

Since our last interoperability test at CEWC 2005,Carrier Ethernet marketing success has beenoutstanding. Last year we saw a few showstoppersfor Carrier Ethernet deployment: lack of implementa-tion of resilience and OAM mechanisms and lack ofend-to-end provisioning and fault management. It isa very positive surprise to witness many improve-ments in most of these areas:

• Six vendors sucessfully tested multi-vendorinteroperability of Ethernet OAM "Dying Gasp"(802.3ah) and continuity checks (802.1ag draft)for the first time ever, worldwide.

The existence of pre-standards implementationsshows that vendors are eager to deliver OAMand fault detection solutions in their Carrier Ether-net products.

• 13 implementations passed the Ethernet LinkAggregation (802.3ad) tests, proving that nativeEthernet link resilience is widely supported bynow. Carriers urged us to address native Ethernetbased node resilience in future events.

• MPLS, a widespread deployed foundation forlarge Carrier Ethernet networks, is a mature andreliable solution. We did not see any issues what-soever for basic Ethernet services with sevenvendors. Only small interoperability issuesremained in the MPLS OAM space (LSP ping).

• Support for advanced traffic engineering in MPLSis slowly growing (the standards are ofunequalled complexity); three vendors masteredour very challenging DiffServ-TE tests.

We built a state-of-the-art backbone, aggregationand edge network of massive scale and diversity to

support Carrier Ethernet services. The preparationstook three months and the hot-staging event keptover 40 engineers from 16 vendors busy for eighttesting days. Over 50 devices, 300 cables, 2.5metric tons of devices were installed and untoldamounts of coffee consumed …

The interoperability event was very successful andthe experience gained is priceless. We hope youfind the results useful.

IntroductionOur interoperability events enclose two targets. Forthe participating vendors the event is a uniqueopportunity to test implementations against othervendors. Vendors gather information about thenature of any problem and in many cases receivenew code updates from developers before the eventis over. Interoperable vendors use the results to gainfootprint in new markets and display their level oftechnological readiness. A second target is provid-ing carriers and service providers with a realisticpicture of available technical solutions, potentialdeployment issues and best practices networkdesign. In recent years carriers’ involvement in ourinteroperability events has grown.

A resulting effect of the interoperability event is feed-back to technical committees and standard bodiesregarding the readiness of standards for real worlddeployment.

Figure 1: Hot-staging at EANTC, Berlin

The test plan was developed by EANTC betweenJune and August 2006. The test areas were finalizedafter extensive review with participating vendorsand service providers.

Based on EANTC’s experience in organizing andexecuting multi-vendor interoperability events aneight days, closed doors, hot-staging event wasconducted in EANTC’s lab in Berlin, Germany. Theresults are summarized in this white paper.

Carsten RossenhoevelManaging Director

2


Participants and Devices Carrier Involvement

Germany’s T-Systems and Colt Telecom reviewed thetest plan and provided detailed comments. Serviceprovider participation ensured that the test areaswere realistic and matched their requirements.

In addition, three T-Systems engineers participatedfor the duration of the whole EANTC hot-stagingevent. T-Systems engineers were responsible forcarrying out the service performance tests and the10Gigabit Ethernet physical layer tests.

Network DesignThe interoperability test network resembled a next-generation end-to-end carrier infrastructure designedfor residential IPTV, Triple Play, and businessservices.

Given a total of 16 participating vendors, wecreated a multi-vendor network with many interoper-ability test opportunities. This amount of vendorswould be quite unusual in a real service providernetwork; carriers often use heterogeneous CarrierEthernet networks from a few vendors — typicallytwo vendors for the core, aggregation and CPEs.

All services in the test network were based on»Carrier Ethernet« as specified by the Metro EthernetForum (MEF) with technical standards as defined bythe IEEE 802 committee and ITU.

The MEF is agnostic to the underlying network tech-nology used to deliver Ethernet services. Thecreation and support of such services is, however,an essential component of the interoperability testevent. Participating vendors were required to facili-tate the establishment of Ethernet services startingfrom their customer facing ports (User-Network Inter-face, UNI) over a diverse number of access technol-ogies.

Figure 2 illustrates the five logical parts of thenetwork that were the focus of the tests:

Actelis Networks ML 130ML 628

ADVA OpticalNetworking

FSP 150CCfFSP 150M

Alcatel 7750 SR17750 SR75620 SAM1850 TSS-40

Ciena DN7100

Circadiant Systems OST-ST

Corrigent Systems CM-21 PTSCM-106V PTS

IXIA 1600T

HuaweiTechnologies

ME60S8500

Juniper Networks M320 Multi Service Edge RouterM10i Multi Service Edge Router

Lucent Technologies Metropolis AMUMetropolis ADMMultiService MuxEthernet Router 15202Ethernet Router 15800Ethernet Router 15102

Nortel MERS 8600

RAD DataCommunications

ETX-202RICi-8E1RICi-E1IPMUX-14Egate-100

Redback Networks SmartEdge 400

Stratex Networks Eclipse Microwave Link

Telco Systems,a BATM Company

T-Metro-100T-Metro-200T-Marc-250T-Marc-254

Tellabs 6325 Edge Node6315 Metro Ethernet Node

Document Structure

Participants ➔ Page 3

Network Design ➔ Page 3Test Areas ➔ Page 4Interoperability Results ➔ Page 6Topology Diagram ➔ Page 13Problem Summary ➔ Page 14References ➔ Page 15

3


1. Backbone. Core infrastructure technologiessuitable for transport Ethernet services andservice attributes were demonstrated here. Weevaluated the support for private line and trans-parent LAN services together with QoS andTraffic Engineering, OAM, and multicast.During the preparation phase, the technologiesfor the backbone were selected. Participatingvendors suggested to use MPLS and ProviderBackbone Bridges. Both technologies wereselected; multi-vendor interoperability tests in thebackbone were carried out over MPLS technol-ogy only.

2. Aggregation. Diverse technical solutions toimplement aggregation and access networks forCarrier Ethernet services were demonstrated inthis area. The tests evaluated both user-networkinterface (UNI) interoperability at network cloudboundaries and technology-specific multi-vendorinteroperability. Specifically, we evaluatedVLAN-based Ethernet transport solutions, Ethernetprovider bridges, provider backbone bridges,Ethernet over resilient packet rings (RPR), Ethernetover SDH rings, and Ethernet-based wirelessrings.

3. Ethernet Access. The wide range of accesstechnologies and media for extending the reachof Ethernet services has become the focus of theMetro Ethernet Forum Objective 3 — EthernetAccess. This year, for the first time, EANTC

developed a suite of tests to assess the ability ofa variety of access devices (Aggregation andCPE) to seamlessly extend both business and resi-dential "triple-play" Ethernet services overdiverse access media including fiber and voicegrade copper infrastructure.

4. Network Services Area. Triple play applica-tions such as VoIP and IPTV (IP-Multicast based)servers were positioned in this area.

5. Residential Customers. CPEs targeted at resi-dential users were connected to the edge of thenetwork. A diverse number of access technolo-gies such as Ethernet in the first mile, Ethernetover copper and Ethernet over E1 bundles weretested and demonstrated. Residential serviceswere to be received from applications residing inthe network services area.

6. Business Customers. MEF defined E-LANservices were to be configured for usage bymock business customers. Business services suchas Voice over IP (VoIP) and Internet access wouldreceive different levels of classification and willbe treated in the backbone according to differentSLA levels.

Test Areas and Test PlanThe number of test areas in our interoperabilityevents is being expanded every year: A growingnumber of vendors apply the Carrier Ethernet stan-

dards for different types ofproducts and applications.This year, our tests rangedfrom physical layer (10GigEoptical performance) up tothe application layer (IPTVquality), from data plane(link aggregation) andcontrol plane (MPLS) to themanagement plane (OAM).

This is very promising fromthe user point of view.Carrier Ethernet is becom-ing a complete networkprotocol family for end-to-end networks.

In detail, our tests coveredthe following areas. Custo-mer services were emulatedusing point-to-point EthernetPrivate Lines (EPL) and busi-ness-oriented Ethernet Trans-parent LAN services at the

NetworkServices

Area

Aggregation

Busi

nes

s/Res

iden

tialA

pplic

ations

Backbone

Dem

arc

ation/M

anaged

CPEs

Aggregation

Services

Customers

Backbone (P) Router

Backbone (PE) Router

Aggregation Switch

Demarcation / Managed CPE

Business VPN

Set-top Box

VoIP Phone10Gigabit EthernetGigabit EthernetFast Ethernet

areas using varioustechnologies

Backbone and Aggregation

Figure 2: Schematic Network Design

4


periphery of the network. At the service providerarea Network Operations Center (NOC) functional-ity was tested and residential services such as IPTVwere demonstrated while at the Carrier EthernetNetwork several technologies were simultaneouslyused to depict the richness in possible ways of real-izing MEF user services.

• E-Line and E-LAN support: The Metro Ether-net Forum (MEF) defines two fundamental Ether-net service types. E-Line is defined as point-to-point ethernet virtual connection while E-LAN isdefined as multipoint-to-multipoint virtual connec-tion. These service types are used to create Ether-net services.

The Ethernet Private Line (EPL) service is specifiedusing an E-Line service type and is created froma point-to-point Ethernet Virtual Connection (EVC)connecting two dedicated User-Network Inter-faces (UNIs). Similarly Ethernet Virtual PrivateLine is specified using an E-Line service type,however, it does not require UNI exclusivity anda lower level of transparency is expected fromthe service provider.

• Quality of Service. To properly cope withcongestion in the carrier network and to maintainservice-level agreements, mechanisms to priori-tize traffic classes are required. This test areawas defined with three traffic classes in the coreof the network: one for voice; video; and Internettraffic, and verified the different per-service quali-ties facilitated by the traffic classes.

• Traffic Management. Some Carrier Ethernettransport technologies implement advancedtraffic management beyond traffic prioritization.A test of Differentiated Services Traffic Engineer-ing (DiffServ-TE) in the MPLS cloud was sched-uled to verify class-based routing, preemption,and prioritization of paths for E-LANs and E-Linesbased on administrative settings.

Operation, Administration and Maintenance (OAM)support is required to operate a Carrier Ethernetnetwork efficiently. Recently, OAM protocols both forEthernet physical layer and for VLAN-based connec-tivity have been defined. Our interoperability testswere of these procedures were, to the best of ourknowledge, conducted for the first time ever.

• Ethernet Physical Layer OAM. The IEEEStandard 802.3ah-2004 is commonly referred toas Ethernet in the First Mile (EFM). The documentdefines services and protocols that are meant tobe »used between stations in a subscriber accessnetwork«. Given the time constraints for the hot-

staging event a single function defined in theIEEE 802.3ah was outlined for testing: EthernetDying Gasp. Dying Gasp is a bit defined in EFMOAMPDUs that informs a neighboring Ethernetstation that its peer station suffered a local unre-coverable failure condition. The vendors partici-pating in this test generated Dying Gaspmessages when the station was rebooted fromthe management interface; when the station wasshutdown from the management interface and inseveral cases when the power was disconnected(using a a battery or a capacitor to enable thestation to send a Dying Gasp message beforeshutting down).

This mechanism enables service providers toextend their OAM domain all the way to thecustomer premise and by monitoring first mileOAM messages to offer the customers a highlevel service quality and proactive customercare.

• Connectivity Fault Management. The IEEEstandard 802.1ag "Connectivity Fault Manage-ment" is still in draft form, however, severalvendors expressed interest is testing their imple-mentations against other vendors. ConnectivityFault Management (CFM) defines capabilities todetecting, verifying and isolating connectivityfailures in virtual bridges networks. The standardprovides both providers and customers with highlevel of visibility into the status of their Ethernetnetwork. The protocol is meant to be used acrossEthernet domains and does not require physicalconnectivity between the stations.

For the purpose of the interoperability event oneaspect of the CFM protocol was tested: Continu-ity Check (CC). CC messages are used forconnectivity failure detection and are a usefulcomplement to the EFM Dying Gasp message.Each station belonging to a maintenance domaincan be configured to periodically send continuitycheck messages to other stations within its main-tenance domain. When the station fails toreceive three consecutive CC messages orreceives a CC message with incorrect informa-tion (Such as interval or maintenance domain ID)the peer is considered faulty.

• Service Performance. The MEF defines threeperformance attributes for Ethernet Private Lines:Frame Delay Performance, Frame Delay Varia-tion Performance, and Frame Loss Ratio Perfor-mance. The test plan called for service perfor-mance verification from end to end.

• Ethernet Link Aggregation. Carrier Ethernetinfrastructures use link aggregation primarily for

5


link resiliency of Fast Ethernet and Gigabit Ether-net connections, by grouping of several inter-faces into a single logical unit. If traffic across thelink aggregation group (LAG) is kept at half theavailable bandwidth (by provisioning for exam-ple) the LAG can guarantee that when a memberof the group fails, the rest of the LAG memberswill carry the traffic load without permanent loss.

• TDM Support. Legacy TDM applicationsupport is important in a next-generation packetnetwork. We verified whether circuit emulationsolutions worked through the backbone. TDMtraffic has high requirements on latency, jitter,and packet loss, so connectivity was establishedusing services with the highest quality of serviceavailable.

• Bandwidth Profiles at the UNI. Serviceproviders often use traffic classification andbandwidth limitation enforcement (Committedand Excess Information Rate) for different appli-cations at the user-network interface (UNI) facingthe customer. These functions were verifiedaccording to MEF standards per Ethernet VirtualConnection (EVC).

• 10 Gigabit Ethernet Physical LayerPerformance. In order to confirm vendorcompliance to the 10GigE (IEEE 802.3ae) speci-fication, a test of physical layer impairments andsimulated real world conditions was scheduled.

The effects of optical noise and signal degrada-tion over long fiber links and aging optical trans-ceivers were observed in relation to core networkrerouting and traffic engineering functions.

Interoperability Test ResultsAfter eight intense days of hot-stage testing wecollected a substantial amount of results. The resultsprovide a detailed insight into the current status ofCarrier Ethernet solutions. The following sectionssummarize our findings.

Results: Ethernet E-Line / E-LANServices

As in a real Carrier Ethernet network, point-to-pointand multipoint-to-multipoint services were created inthe test environment.

Point-to-point Ethernet private line (EPL) services werecreated between mock customer pairs. The connec-tions were created according to MEF6 which definesEPL as a dedicated port for a single user on eachend of the Ethernet Virtual Connection (EVC). We setup a total of 18 end-to-end virtual connectionsbetween all the participating managed CPE anddemarcation devices. Each of the EVCs traversed thebackbone, so EVC support of the aggregation andcore network elements was tested at the same time.

In the aggregation network, EVCs were realized astwo-port VLANs. They were delivered to theemulated customer premise equipment (CPE) withouttags. The Ethernet Provider Bridges standard (IEEE802.1ad; »Q-in-Q«) was not employed due tolimited time. In the MPLS backbone, EVCs weretransported as Ethernet pseudowires — see MPLSsection below for more details.

In addition, we configured E-LAN services with multi-ple network access locations. Basic E-LAN interoper-ability was reached between all CPE vendors includ-ing across all participating access technologies.

There were no technical interoperability issues withE-Lines; however, manual provisioning of each indi-vidual E-Line on every node is a time consumingprocess. A common edge-to-edge provisioning plat-form based on standardized configuration mecha-nisms would have sped testing in this multi-vendornetwork and would have mostly eliminated configu-ration issues.

Most configuration problems were associated withEthernet auto-negotiation, Maximum TransmissionUnit (MTU) mismatch, and VLAN tag mismatch(where a vendor is supposed, but fails, to pop aVLAN tag, so that the next hop accidentally stacksmultiple tags).

6


Results: MPLS Backbone Services

Figure 3: MPLS RSVP-TE SignalingConnections

Routing and Signaling. After all core routershad been installed, the MPLS backbone was quicklyestablished. RSVP-TE was used for transport signal-ing, OSPF-TE as the routing protocol. Most MPLSdevices participating in this test were used both as»PE« (Provider Edge) and »P« (Provider) routers,forwarding and terminating label-switched paths.

There were no connectivity issues for basic signalingand routing interoperability between Alcatel 1850TSS, Alcatel 7750, Ciena DN7100, Huawei ME60,Juniper M10i, Juniper M320, Lucent 15800, Lucent15202, and Redback SmartEdge 400.

MPLS Point-to-Point And MultipointServices. Ethernet pseudowires were preconfig-ured for all E-Lines required in the end-to-endnetwork. There were no interoperability limitationsother than configuration issues in the beginning. Allpseudowires were successfully established.

Three Virtual Private LAN Services (VPLS) domainswere created in the MPLS backbone for multipointtriple play services — video, voice, and Internetaccess. These domains were interconnected with theE-LAN services in the aggregation and edge areas.

We used two different IETF-specified protocols forVPLS service creation: LDP and BGP. Alcatel,Huawei, Lucent, Redback, and Telco Systems partici-pated in the LDP-VPLS service. A hierarchicaldomain was created with H-VPLS to improve networkscalability and to demonstrate the different roles ofthe systems under test. The Alcatel 1850 TSS,Alcatel 7750 SR1, Huawei ME60, Lucent 15102,and Telco Systems T-Metro participated as Multi-Tenant Units (MTUs) while Alcatel 7750 SR7, Lucent15800, Lucent 15202, and RedbackSmartEdge 400 took the roles of VPLS PE devices.

Figure 4: MPLS Pseudowires

In addition, we established a BGP-based VPLSdomain for the same three services. Huawei ME60and Juniper M10i participated in these services.Note that BGP Router Reflectors can be used toestablish a hierarchy in order to scale the controlplane for BGP-VPLS (this has not been tested due tolimited time).

Figure 5: VPLS Domains

All VPLS services were quickly configured and full-mesh connections were established, proving thatVPLS is a well-understood and widely interoperableindustry solution by now. The only fundamental issueis that BGP-VPLS and LDP-VPLS are incompatiblebecause of the different signaling protocols used. Asa temporary solution, we established a singlepseudowire per broadcast domain interconnectingthe LDP- and BGP-VPLS. The Alcatel 7750 SR7(Video domain) and Redback SmartEdge 400(Voice domain) provided the pseudowire intercon-

HuaweiME60

RedbackSmartEdge 400

Alcatel7750 SR7

Lucent15800

Lucent15202

JuniperM10i

JuniperM10i

CienaDN7100

Alcatel7750 SR1

Alcatel1850 TSS

HuaweiME60


JuniperM10i

Alcatel1850

CienaDN7100

MPLS Router Ethernet Pseudowire

Lucent15800

Alcatel7750 SR7

Alcatel7750 SR1

Alcatel7750 SR7

Alcatel7750 SR1

Lucent15202

HuaweiME60


Alcatel7750 SR7

Lucent15800 Lucent

15202

JuniperM10i

JuniperM10i

Alcatel1850 TSS

BGP

Lucent15102

Telco SystemsT-Metro

Alcatel7750 SR7

Alcatel7750 SR1

VPLS

LDPVPLS


HuaweiME60

Alcatel7750 SR1


H-VPLSMTUs

7


nection on the LDP side and the Juniper M10i on theBGP side. The interconnection worked well.MPLSConnectivity Verification

LSP ping tests were carried out in the MPLS back-bone with Alcatel 7750 SR1 and SR7, Ciena DN7100, Huawei ME60, Juniper M10i, Juniper M320,Lucent 15202, Lucent 15800, RedbackSmartEdge 400 and Telco Systems T-Metro. Seefigure 6 for a detailed logical test topology.

We discovered a total of three interoperabilityissues; this calculates to a 88 % success rate and isa great improvement over the last MPLS LSP pingtests we carried out a year ago.

Test System Attachment. The Ixia 1600T wasconnected with Gigabit Ethernet to every provideredge router and full-mesh connectivity within theVPLS domain was successfully verified.

Figure 6: MPLS Ping

Results: Quality of Service and TrafficEngineering

Ciena DN7100, Juniper M10i, Juniper M320 andHuawei ME60 participated in the DiffServ-TE testsand successfully established DiffServ-TE enabledtunnels.

DiffServ-TE is complex to implement and suffers frommany options. Its complexity led to a misunderstand-ing in the implementation of OSPF-TE details in onecase. The fact that the IETF specified three differentallocation models did not help interoperability either— all network elements should implement the sameallocation model if DiffServ-TE is to be used. Unfortu-nately some participating vendors implemented

Maximum Allocation Model (MAM) while othersused the Russian Dolls Model (RDM). Juniper was theonly vendor supporting both models in the sameimplementation.

One tunnel was configured per TE-class for eachdirection and each link, resulting in a total of eighttunnels (four per direction) between each pair of PErouters. As a result of the network configuration,tunnels of different priority had to compete for theallocated bandwidth and used different pathsthrough the backbone. We verified that the pathselection worked properly.

Results: Aggregation and AccessServices

As expected, all devices in the multi-vendor VLAN-based (IEEE 802.1q) aggregation network interwor-ked smoothly. There were only few configurationissues related to Ethernet auto-negotiation (webelieve it should always be disabled betweennetwork elements).

In the CEWC 2005 event, multi-home connection ofVLAN clouds using the Spanning Tree protocol wasa major source of headache. This year we acceptedthat Spanning Tree is not a carrier-grade solutionand connected all equipment to the MPLS backbonein a star topology.

A substantial number of carrier-class aggregationtechnologies interworked with VLAN-based infra-structures in the test network (in no specific order):

Carrier Ethernet Services over RPR overSDH. Corrigent used a packet-optimized opticaltransport solution using standards-based ResilientPacket Ring (RPR) technology (IEEE 802.17),

MPLS Router Successful LSP Ping

Alcatel

Juniper

Alcatel

M320

7750 SR-7

7750 SR-1


CienaDN7100

RedbackSE400

HuaweiME60

Lucent15800Lucent

15202

JuniperM10i

Cla

ss T

ype

Traffi

c Ty

pe

Traffi

c En

g.

class

es

Setu

p/H

old

Pri

ori

ty Bandw

idth

Const

rain

t(M

bit/s

)

Tunnel

BW

Res

v Req

ues

t(M

bit/s

)

MAM RDM

CT0 EFa

a. Voice traffic

TE0 0/0 450 900 10

TE1 1/1 50

TE2 2/2 100

CT1 AF1b

b. Video traffic

TE3 3/3 450 450 200

8


combining standard MPLS with RPR to provide abandwidth-efficient universal transport solution.

The CM-106V PTS and CM-21 PTS providedconnectivity through the MPLS core for Ethernetservices supporting multiple applications. The Corri-gent Packet Transport Solution participated in theservice performance tests.

Ethernet over SDH rings. Lucent ADM andTellabs 6325 demonstrated Gigabit Ethernet overSDH using GFP (Generic Framing Procedure, ITU-TG.7041) and LCAS (Link Capacity AdjustmentScheme ITU-T G.7042). LCAS allows dynamicchange of bandwidth if, for instance, impaired SDHlinks cause some virtual containers (VCs) to fail.

This was tested in an STM-16 ring between LucentADM, Lucent AMU, and Tellabs 6325 switches. Forextra resiliency some of the VCs were SNCprotected over an STM-4 link.

T-MPLS rings. T-MPLS based ring protection wasused between the two Alcatel 1850 TSS participat-ing in the test. The solution is a pre-standard imple-mentation based on a draft ITU-T recommendation.T-MPLS ring protection further extends resiliencecapabilities. The Alcatel 1850 TSS participated ininteroperability tests both as an Aggregation Switchand as a PE Router.

For the same purpose of increasing the network resil-ience, Telco Systems employed a ring solution basedon H-VPLS and MPLS rerouting protocols.

Resilient Wireless Packet Ring. StratexNetworks took part in the EANTC Interoperabilitytest with their Wireless Ethernet Product — Eclipse.Three Eclipse 18 GHz Microwave Radios weredeployed as an access ring with a capacity of155 Mbit/s carrying native Ethernet traffic.

Each node was configured to operate first as anAggregation device and then as a CPE. Wirelessring protection was configured using StratexNetworks’ Resilient Wireless Packet Ring Protocol(RWPR). The Eclipse Ring participated in link aggre-gation and service performance tests.

Ethernet in the First Mile Over Copper. Abonded group of eight pairs of voice-grade copperwas connected between an Actelis ML 130 CarrierEthernet over Copper aggregation switch and anActelis ML 600 Ethernet Access Device (EAD), tocreate a 45 Mbit/s Ethernet service to residentialsubscribers.

We verified that the Ethernet over Copper servicewas able to support the high performance, lowlatency service requirements for successfully deliver-

ing the IPTV video stream to a set-top box connectedto the EAD and the RAD IP-MUX Circuit Emulationover Ethernet device which was also connected tothe ML 130.

Results: OAM Ethernet in the First Mile

Five vendors participated in the first public multi-vendor IEEE 802.3ah test ever. ADVA, Huawei,RAD, Telco Systems and Tellabs provided severalimplementations of physical layer OAM.

According to the standard, devices were assignedactive or passive roles. Typically, the aggregationdevice will serve as an active station, initiating theOAM discovery process, sending variable requestsand loopback control OAMPDUs. The CPE device isset to passive mode, enabling it to participate in thediscovery mechanism and to send event notifica-tions.

Figure 7: OAM Physical Layer Tests

We were able to complete the discovery processbetween all but one paired devices in the test. Inseveral of the test pairs, both stations had to beconfigured in active mode and in one instance bothstations were able to exchange OAMPDUs onlywhen both were set to enhanced mode (to supportorganization specific OAMPDUs).»Dying Gasp«messages were sent by CPEs upstream to the aggre-gation network upon an electrical power loss. (Typi-cally, only CPEs are expected to generate thesemessages.) Once OAMPDUs were exchangedbetween the two stations, the CPE was shutdownand the aggregation device was expected to informthe operator about the neighbor’s condition. Alldevices receiving dying gasp messages notified theuser by way of log messages, command line notifi-cation or SNMP traps.

Telco SystemsT-Marc

ADVATellabs6315

TellabsFSP 150M


HuaweiS8500

6325

HuaweiME60 ADVA

FSP 150CCf

RADETX-202

Neighbor Detection Dying Gasp Messages

CPEs

Access

9


The level of basic OAM 802.3ah support was prom-ising given the fact that the standard is still young.We look forward to seeing more complete 802.3ahimplementations in the future.

Results: VLAN-Based OAMConnectivity Messages

In addition to physical layer OAM we evaluatedEthernet-based connectivity messages as defined inIEEE 802.1ag. Nortel, RAD, Telco Systems, andTellabs participated in these tests. It is important tomention that 802.1ag is still in the draft status.

The challenge in testing CFM proved to be in themany configuration parameters and options definedin the draft standard. In addition the exact version ofthe draft and the corresponding ITU document(Y.1731) added complexity to the testing.

Several vendors implementations offered predefinedconfigured parameters that were not tunable. Forexample, a simple predefined Continuity Check(CC) message interval parameter set to 100 ms inone vendor’s implementation did not allow testingagainst another vendor who had a predefined CCmessage interval of one second. This issue and afew further early implementation problems wereresolved during the hot-stage event. In addition, RADdemonstrated 802.1ag-based performancemeasurements based on the MEF defined parame-ters for Delay, Delay Variations and Frame Loss. Wewere impressed with the level and commitment to thesuccess of the 802.1ag related tests.

Figure 8: OAM Connectivity Tests

Lucent ADM provided proprietary pre-standard802.1ag OAM used for round trip delay measure-ments, Ethernet loopbacks and continuity checkwithin the Lucent MSPP domain.

Results: Service Performance

With support from T-Systems, we carried out end-to-end service performance tests over the completenetwork and all technologies deployed.

The measurements were carried out in accordancewith MEF specifications. An Ixia 1600T load gener-ator was used to emulate bi-directional customertraffic on point-to-point Ethernet Private Lines (EPLs).Packet sizes of 64, 512, and 1500 Bytes (withoutEthernet headers) were used to verify service perfor-mance for small and large packets and to ensurethere were no MTU size issues.

»Service performance« is unrelated to backbone ordevice performance. In this interoperability test,different transport technologies limited the end-to-end bandwidth. We measured all end-to-end rela-tionships with a fixed bandwidth of 5 Mbit/s perdirection.

The table below summarizes the results of 17 Ether-net Virtual Circuits (EVCs). Each of the end-to-endflows traversed at least four, in average sevenswitches.

The minimum latency across the backbone wasoutstanding (0.04 ms), but even the maximumlatency at Gigabit Ethernet edges was far belowexpectations (0.74 ms). This result shows that multi-vendor Carrier Ethernet backbones and aggregationnetworks can keep up in service quality with anyother legacy network technologies.

At the edge, the majority of results were similar tothe core. The maximum delay of a few EVCs wasinfluenced by physical layer multiplexing technolo-gies (Ethernet over n x E1) or by radio networkaccess latency. However, even over an E1 link themaximum latency was 6.80 ms which is still muchbelow xDSL latency.

Continuity Check Messages

RADETX-202


RADRICi-8E1

Nortel8600

Tellabs6325

Telco SystemsT-Marc

PacketSize(Bytes)

PacketLoss(%)

End-To-End Latency

Min

(ms)

Max

Gig

Eth

(ms)

Max

Low

BW

(ms)

64 0.0 0.04 0.16 1.00

512 0.0 0.13 0.63 1.94

1500 0.0 0.32 0.74 6.80

10


Results: Bandwidth Profiles

The Metro Ethernet Forum defines Bandwidth Profileas the characterization of the lengths and arrivaltimes for ingress service frames at the User NetworkInterface (UNI). In the Ethernet Services AttributesPhase 1 technical specification (MEF10) threemodels are defined for the level of granularity atwhich UNI bandwidth profiles are applied.

For the purpose of this test we chose one of themodels -- Bandwidth Profile per EVC. In this modeleach EVC across the UNI is applied with a differentBandwidth Profile. For service providers, this level ofgranularity allows each customer to receive a differ-ent level of service while all customer connectionstraverse the same access device.

The goal of the test was to show that participatingvendors can apply different Bandwidth Profiles perEVC. In our case two EVCs were configured acrosseach capable UNI, each with a different bandwidthprofile.

Vendors were instructed to configure two parametersfor the Bandwidth Profile — Committed InformationRate (CIR) and Excess Information Rate (EIR). Oncethe Bandwidth Profiles were configured and appliedto the EVCs we generated traffic across both EVCsand measured the loss rate. We expected loss at theaggressive Bandwidth Profile and zero loss at thesecond Bandwidth Profile.

All participants that could apply a per-EVC Band-width Profile passed the test. Some of the partici-pants could not apply a per-EVC bandwidth profile,however, these vendors’ products are aimed atdifferent network topologies or market segments, allof which would enforce bandwidth profiles at differ-ent areas of the network.

The parameters to be configured for BandwidthProfile have also proven to be difficult to translatefrom the MEF technical specifications to variousvendors’ implementations. The concept of EIR has, inmore than one case, been consumed by the aggre-

gation of Committed Information Rate and PeakInformation Rate (the difference between the tworesulting in EIR), lacking an implicit definition of EIR.

Results: Link Aggregation

We tested link aggregation according to IEEE802.3ad for the purpose of link resiliency.

Link Aggregation Groups (LAG) containing pairs ofGigabit Ethernet interfaces were defined betweenparticipating vendors. Once the pairs establishedconnectivity, we used the Ixia 1600T traffic genera-tor to send traffic at 25% of the aggregated linkcapacity. Redundancy switchovers were forced bydisabling each of the links in turns.

All participating devices passed the test in general:Alcatel 1850 TSS, Alcatel 7750 SR1 and7750 SR7, Corrigent CM-106V PTS,Huawei ME60, Juniper M320, Lucent 15102,Lucent 15202, Lucent 15800, Lucent ADM,RAD ETX-202, Redback SmartEdge 400, StratexNetworks Eclipse, and Telco Systems T-Metro.

We noticed differing switchover times betweenvendors on physical link disconnection. Packetswere lost for intervals between 0.5 ms and 99 mswhile the ingress switch was calculating the newlink. The rerouting time is related to the hash algo-rithm that controls traffic distribution to each of thelinks. These aspects should be investigated further.

In one case, Ethernet auto-negotiation was miscon-figured over the link aggregation group: It wasenabled on one side and disabled on the other. Thissituation resulted in a rerouting time of 372 ms, untilboth ends had (correctly) completed the timer-basedsemi-autonegotiation. Again we recommend todisable Ethernet auto-negotiation on network links.Standards committees should consider making this adefault.

Traffic Generator Traffic Generator

NetworkCore

CPE Aggregation

Provider Edge Provider EdgeProvider

CPEAggregation

EVC With Bandwidth Profile EVC Without Second Bandwidth Profile

Figure 10: Bandwidth Profile Reference Model

11


Results: TDM over Ethernet

TDM over Ethernet connection (as per MEF 8) wascreated between the RAD IPMUX-14 and TelcoSystems T-Marc. An Ethernet pseudowire with highpriority (Class Type 0, see MPLS) was created toforward encapsulated E1 frames. The connectionworked properly.

Results: Impact of Physical LayerImpairments on 10GigE Systems

In order to confirm vendor compliance to the IEEE802.3ae specification a Circadiant Systems OpticalStandards Tester (OST-ST) was connected in-line intothe core network. This provided the facility to createphysical layer impairments and simulate real worldconditions in the captive lab environment.

Stressed eye conditions were applied, see thefigures below.

Figure 11: 10GigE Optical Eye(Stressed but Acceptable)

At the same time we ran packet throughput and lossmeasurements. Margin testing was also made byincreasing stress beyond the specification require-ment to the point where packet loss occurred, andultimately rerouting took place (see figure 12).

Figure 12: 10GigE Stressed Optical Eye(Resulting in Packet Loss)

All tested 10 Gigabit Ethernet transmitters andreceivers were confirmed to exceed the requirementsof the IEEE 10GigE standard. For example, thereceiver of the link under test allowed for an extrabudget of 3 dB for the vertical eye closure penalty(VECP) before packet loss occurred, as shown infigure 13.

Figure 13: Frame Loss versus Vertical EyeClosure Penalty (VECP)

Frame Loss vs VECP

1

10

100

1000

10000

100000

1000000

10000000

1.129 1.82 2.68 3.4 4.7 6.97 8.3 9.6

VECP (dB)

Lost Frames

IEEE 802.3ae Threshold 2.7dB

12


Final Integrated Test Network

Backbone (P) Router

Backbone (PE) Router

Aggregation Switch

Demarcation / Managed CPE

Traffic Generator

Set-top Box

10Gigabit EthernetGigabit Ethernet

E1

Aggregation AreaBackbone/Provider Edge

CPE Area

Alc

ate

l

Tella

bs

6325

Tella

bs

Junip

er

Red

back

Adva

Adva

Huaw

ei

Nort

el

Nort

el8600

Stra

tex

Luce

nt

Luce

nt

STM

-16

Telc

oSy

stem

sRA

D

Stra

tex

Corr

igen

t

Corr

igen

t

Telc

oSy

stem

sT-

Mar

c-25

0

Telc

oSy

stem

s

Adva

Nort

el

Corr

igen

t

RA

D

RA

DET

X-2

02

RA

D

RA

D

RA

DRIC

i-E1

T-M

arc

-25

4

CM

-106V

PTS

CM

-106V

PTS

RIC

i-8E1

RA

D

T-M

etro

-100

RA

DET

X-2

02

Luce

nt

15

102

Smart

Edge

400

6315

S85

00

Egate

-100

Telc

oSy

stem

sT-

Marc

-25

4

Act

elis

OC-

192

MPLS

-RPR

FSP

15

0CCf

RW

PR

Alc

ate

l

Nort

el8600

Act

elis

ML1

30

MTU

MTU

IXIA

16

00

T

IXIA

16

00

T

IXIA

16

00

T

IXIA

16

00

T

RA

DET

X-2

02

SDH

Rin

g

M10i

775

0SR

1

Eclip

se

Eclip

seSt

rate

x

Eclip

se

CM

-21

PTS

Telc

oSy

stem

sT-

Marc

-25

0

OP1

55

1

FSP

15

0M

775

0 S

R1

FSP

15

0CCf

AM

UM

L628

Act

elis

ML6

28

8600

Adva

FSP

15

0M

Alc

ate

l

T-M

PLS

Rin

g

185

0TS

S

Alc

ate

l185

0TS

S

MTU

ETX

-202

Alc

ate

l775

0SR

7

Luce

nt

Tella

bs

AD

M

6325

Telc

oSy

stem

sT-

Met

ro-2

00

Telc

oSy

stem

sT-

Met

ro-2

00

Luce

nt

15

800

Huaw

eiM

E60

Junip

erM

320

Alc

ate

l775

0SR

7

Junip

er

Cie

na

DN

7100

M10i

Cir

cadia

nt

OST

-ST

Luce

nt

15

202

IPM

UX

-14

MTU

Video Headend

VoIP PhoneFast Ethernet

Other

8600

AM

U

Telc

o

T-M

etro

-200

Syst

ems

MTU

H-V

PLS

Rin

g

13


Problem Summary

ProblemArea

Description TemporarySolution, if any

Recommendation

MPLS DiffServ-TE Order of RSVP-TE objects in the RESVmessage matter for some implementa-tions

Tolerate differentorder

Clarify RFC2205 with regards to»recommended« object order

Compatibility MAM / RDM not clearlydefined

— IETF should state in how far MAMand RDM can be used together

Vendors implement different allocationmodels

— IETF should invalidate all but oneallocation model

MPLS Ping Router Alert flag not taken into accountwhen sending replies

Fix code —

Ethernet LinkAggregation

Aggregated link connections convergevery slowly (> 500 ms) when auto-negoti-ation is involved

Disable auto-negoti-ation on all networklinks

IEEE should recommend todisable auto-negotiation for GigE/ 10GigE connections perdefault

Not all aggregated links are used in abalanced way

— Implement multiple hash algo-rithms for different network appli-cations

FaultManagement802.1ag

Maintenance domain name format wasincompatible and not configurable. IEEEprefers format #2; ITU supports format#32.

Implement a configu-ration option

Resolve preferred formatsbetween IEEE and ITU

Continuity check (CC) transmit intervalwas different and not configurable

— Add more flexibility to implemen-tations of CC parameter control

OAM FaultManagement802.3ah

Several stations could only act as Activerole

— Implement passive role for allCPE equipment

Some stations could not generate DyingGasp messages

— Add support for Dying Gasp forall CPE equipment as this is oneof the main reasons to implement802.3ah

BandwidthProfiles

Parameter units defined wrong (CIR =2 Mbit/s requires EIR ≥ 2 Mbit/s)

Implement EIR according to MEF:As additional bandwidth (inexcess of CIR)

Small EIR/CIR values not supported(128 kbit/s)

— MEF Ethernet Service Attributesspec requires support

Ingress/per-VLAN policing not supported —

E-LAN Number of E-LANs per uplink streamlimited to 1 (one)

— MEF should define minimumperformance recommendationsas part of the certificationprogram

14


AcknowledgmentsWe would like to thank Ralf-Peter Braun, Thomas Kessler, Manuel Paul, and Sabine Szuppa from T-Systems forthe extensive support during the hot-staging event, guidance during test plan development and network design.

This document has been edited by Gabriele Schrenk, Carsten Rossenhoevel, Jambi Ganbar, andSergej Kaelberer (EANTC).

References• Ethernet Services Attributes Phase 1 (MEF 10)

• Requirements and Framework for Ethernet Service Protection in Metro Ethernet Networks (MEF 2)

• Metro Ethernet Network Architecture Framework - Part 1: Generic Framework (MEF 4)

• Ethernet Service Definitions (MEF 6)

• Abstract test suite for Ethernet Services at the UNI (MEF 9)

• Abstract Test Suite for Traffic Management Phase 1 (MEF 14)

• Media Access Control (MAC) Bridges (IEEE 802.1D)

• Media Access Control (MAC) Bridges Amendment 2:Rapid Reconfiguration (IEEE 802.1W)

• Virtual Bridged Local Area Networks (IEEE 802.1Q)

• Link Aggregation Control Protocol (IEEE 802.3ad)

• 10Gb/s Ethernet Task Force (IEEE P802.3ae)

• Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical LayerSpecifications. Amendment: Media Access Control Parameters, Physical Layers, and Management (IEEE Std802.3ah-2004)

• Virtual Bridged Local Area Networks - Connectivity Fault Management (IEEE 802.1ag/D7)

• OAM functions and mechanisms for Ethernet based networks (ITU-T Y.1731)

• Traffic Engineering (TE) Extensions to OSPF Version 2 (RFC 3630)

• RSVP-TE: Extensions to RSVP for LSP Tunnels (RFC 3209)

• Pseudowire Setup and Maintenance Using the Label Distribution Protocol (LDP) (RFC 4447)

• BFD For MPLS LSPs (draft-ietf-bfd-mpls-03.txt)

• Bidirectional Forwarding Detection (BFD) (draft-ietf-bfd-base-05.txt)

• Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures (RFC 4379)

• Protocol Extensions for Support of Diffserv-aware MPLS Traffic Engineering (RFC 4124)

• Requirements for support of Diff-Serv-aware MPLS Traffic Engineering (RFC 3564)

• Requirements for Traffic Engineering Over MPLS (RFC 2702)

• Maximum Allocation Bandwidth Constraints Model for Diffserv-aware MPLS Traffic Engineering (RFC 4125)

• Russian Dolls Bandwidth Constraints Model for Diffserv-aware MPLS Traffic Engineering (RFC 4127)

15

EANTC AGEuropean Advanced Networking Test Center

IIR Conferences

Einsteinufer 1710587 Berlin, GermanyTel: +49 30 3180595-0Fax: +49 30 [email protected]

29 Bressenden PlaceLondon SW1E 5DR, UKTel: +44 (0) 20 7017 7483Fax: +44 (0) 20 7017 [email protected]

This report is copyright © 2006 EANTC AG. While every reasonable effort has been made to ensure accuracy andcompleteness of this publication, the authors assume no responsibility for the use of any information contained herein.

All brand names and logos mentioned here are registered trademarks of their respective companies in the UnitedStates and other countries.

Documents

Test Report - EANTC · Some Carrier Ethernet transport technologies implement advanced traffic management beyond traffic prioritization. A test of Differentiated Services Traffic