1

Problems in Quality Assurance of 100GbE CFP Modules and Validation of Tests

1. Introduction The number of IP networks worldwide is expected to grow exponentially due to the spread of cloud computing, increasing speeds of LTE mobile terminals, and the rapid spread of high-definition IP TV services. To meet the massive increase in IP traffic, R&D into commercialization of high-speed 100 Gigabit Ethernet technology (100 GbE) is gaining pace and these networks will soon replace today’s 10 GbE networks. Attention is focused on CFP(1) modules as one key component in early commercialization of 100 GbE. As defined by MSA(2), a CFP module meeting the IEEE P802.3ba standards uses 100GBASE-LR4 and 100GBASE-ER4 interfaces, and rapid adoption of CFP modules is expected by carriers and equipment vendors focusing on 100 GbE. However, achieving high-capacity 100G transmissions is dependent on several technical issues. Among these, the biggest issue is assuring the quality of the transmission path loss and crosstalk between signals on a high- density printed circuit board (PCB) mounting the CAUI(3) interface using a total of 40 connectors with twenty 10-lane connectors at the Tx side and twenty 10-lane connectors at the Tx side for connecting the CFP module to equipment for a 10-Gbps differential signal. Circuit design technologies for solving these issues, mounting technologies and related evaluation tests are all important in shortening development times, preventing problems during field compatibility tests, and increasing production line yields. This article explains the CFP module QA themes, the evaluation tests and actual measured data.

(Note 1) CFP: Abbreviation for 100G Form-factor Pluggable (C is Roman numeral 100)

(Note 2) MSA: Abbreviation for Multi Source Agreement

(Note 3) CAUI: Abbreviation for 100G Attachment Unit Interface (C is Roman numeral 100)

2. 100-GbE CAUI Measurement Issues This chapter explains the measurement issues and evaluation tests for the CAUI interface between 100 GbE CFP modules and devices. Figure 1 shows the configuration of the PCS(4) (CAUI Lane) system, which is the interface between the CFP module and device. The interface in the CFP module for connecting with the device side is defined as the PMA(5). Since the 10:4 SERDES(6) used by the 100 GbE CFP module has functions for converting the lane number and bit rate, it is called the gearbox; it is a highly integrated device for converting a 10-Gbps x 10-lane signal into a 25.78-Gbps x 4-lane signal. The CAUI interface between the CFP module and the device transfers 10-lane, duplex differential signals via the connector.

Figure 1: System Configuration (Simplex Only)

Application Note

MP1800A Signal Quality Analyzer

2

The issue considered most important here is the SERDES function for assuring error-free transfer of the signal with minimum crosstalk between lanes and minimum loss in the transmission path under high-density mounting conditions. Because it is not possible to monitor the waveform after transmission path loss is calibrated at the equalizer circuit of the SERDES input section, one issue is the degree to which the waveform is recovered; a second issue is that it is difficult to evaluate whether or not the margin can be assured quantitatively just by adjusting the optimum point using a combination of the Tx side device emphasis function and the Rx side device equalizer function. In addition, the tolerance of the clock recovery circuit after the equalizer circuit is also an important element. With the clock recovery circuit, reproduced clock signals follow the input data when the input data signal jitter is within the loop bandwidth. Conversely, since the jitter of the recovered clock signal is suppressed outside the loop bandwidth, the smaller data signal margin in the next retiming circuit may induce bit errors. The CFP SERDES has a built-in FIFO circuit to buffer variations occurring in the transmission path at the stages after the clock recovery circuit and it is necessary to quantify the jitter tolerance included in data captured at the FIFO circuit. The jitter tests standardized in the IEEE P802.3ba CAUI Electrical parameter measurement methods specifys how to perform the jitter tolerance test using a PRBS31 pattern with the Tx side emphasis function set to off. Figure 2 shows the IEEE P802.3ba mask and jitter specifications at the CAUI Rx side.

Figure 2: CAUI Receiver Mask Standards (IEEE P802.3ba)

Figure 3 shows examples of CAUI signal crosstalk . For easier understanding, this measurement compares the PRBS31 and PRBS7 patterns when the measured lane is using a 1010 pattern and the adjacent-lane signal is off. The waveform data on the oscilloscope clearly shows the crosstalk noise imposed on the 1010 signal when the PRBS31 signal is transferred to the lane adjacent to the measured lane. In addition, when the PRBS7 signal is generated in the adjacent lane, crosstalk clearly causes components of the PRBS7 spectrum to appear near the 1010 signal carrier of the measured lane.

Adjacent lane Signal Off Adjacent Lane PRBS31 Signal Spectrum near Carrier with Adjacent Lane PRBS7 Signal

Figure 3: CAUI Crosstalk Examples

These results suggest that there is a possibility of a decrease in the margin of the downstream SERDES due to the increase in noise caused by crosstalk between 10-lane CAUI signals in the same manner as when the

3

actual frame signal is passed. The jitter tolerance test is standardized as shown in Figure 4 using a test setup as close as possible to the real circumstances considering this type of crosstalk, transmission-path loss and signal variation; quantitative margin measurements based on these standards might be considered a possible shortcut at the early R&D stage to stable mass production and assured connection compatibility.

Figure 4: CAUI Sinusoidal Jitter Mask Standards (IEEE P802.3ba)

(Note 4) PCS: Physical Coding Sublayer

(Note 5) PMA: Physical Medium Attachment

(Note 6) SERDES: SERializer/DESerializer (serial-parallel conversion circuit)

3. 100 GbE CFP Module Measurement This chapter describes optical loopback tests of CFP modules using the Anritsu 100 GbE analyzer now in development and CFP module tests using the Anritsu MP1800A Bit Error Rate Tester (BERT). 3.1. CFP module operation evaluation using 100 GbE analyzer (in development) Figure 5 shows the results of CFP loopback measurements made using the 100 GbE Analyzer. This 100 GbE analyzer supports the emphasis function on the CAUI signal Tx side and outputs the Tx waveform in compliance with the IEEE P802.3ba CAUI Transmitter Eye Mask standards. Moreover, the equalizer function at the Rx side supports the relevant CFP module interfaces. This measurement confirms the absence of BIP errors in all PCSL(7) 20 lanes when an actual 100-GbE MAC frame is sent using this function as well as confirms operation up to PRBS15 in the No Frame condition. (Accurate measurement at PRBS23 and beyond is impossible because of problems at the CFP module side.) As a result, it is not possible to quantify to what degree the margin at the CAUI side of the actual CFP module can be assured at testing using just this analyzer. (Note 7) PCSL: Physical Coding Sub-layer Lane

Figure 5: 100 GbE Analyzer CFP Loopback Measurement Result (100 GbE MAC Frame)

4

3.2. CFP module CAUI tests with Bit Error Rate Tester CAUI tests were performed with the MP1800A BERT using the configuration shown in Figure 6 to evaluate the performance details of the CFP module.

Figure 6: CAUI Test Configuration with BERT

Skew is caused by a variety of factors such as differences in the propagation delay of the transmission path and the propagation delay for each wavelength, as well as PCB pattern lengths, cable lengths, IC propagation delay times, etc. Due to this skew, sometimes, the PRBS pattern sent from the PPG as the CAUI signal cannot be recovered exactly by the Rx ED. As a result, the CAUI 10G x 10-lane separated by the SERDES (4:10 DEMUX) in the CFP module will not have the same pattern as the sent PRBS pattern. Figure 7 shows the CAUI Tx-side bit sequence and Figure 8 shows Rx-side bit sequence when skew occurs via the CFP. As shown in Figure 8, due to skew, the Rx-side CAUI signal is not returned as the sent known signal.

2-0 2-1 2-2 2-3 2-4 2-5 2-6 2-7 2-8 2-9 2-10 2-11 2-12 2-13 2-14 2-15 2-16 2-17 2-18 2-191-0 1-1 1-2 1-3 1-4 1-5 1-6 1-7 1-8 1-9 1-10 1-11 1-12 1-13 1-14 1-15 1-16 1-17 1-18 1-19

VL 0-0 0-1 0-2 0-3 0-4 0-5 0-6 0-7 0-8 0-9 0-10 0-11 0-12 0-13 0-14 0-15 0-16 0-17 0-18 0-19

2-1 2-3 2-5 2-7 2-9 2-11 2-13 2-15 2-17 2-192-0 2-2 2-4 2-6 2-8 2-10 2-12 2-14 2-16 2-181-1 1-3 1-5 1-7 1-9 1-11 1-13 1-15 1-17 1-191-0 1-2 1-4 1-6 1-8 1-10 1-12 1-14 1-16 1-180-1 0-3 0-5 0-7 0-9 0-11 0-13 0-15 0-17 0-19

CAUI 0-0 0-2 0-4 0-6 0-8 0-10 0-12 0-14 0-16 0-18

1-13 1-15 1-17 1-191-5 1-7 1-9 1-111-16 1-18 1-1 1-31-8 1-10 1-12 1-141-0 1-2 1-4 1-60-13 0-15 0-17 0-190-5 0-7 0-9 0-110-16 0-18 0-1 0-30-8 0-10 0-12 0-14

Transmission lanes 0-0 0-2 0-4 0-6

PMA

PCSL

Figure 7: Tx-side Bit Sequence

5

0-0 0-12 0-2 0-3 X 0-5 0-6 0-7 0-8 0-1 0-10 0-11 0-4 0-13 0-14 0-15 0-16 0-9 0-18 0-19VL 1-0 1-12 1-2 1-3 0-17 1-5 1-6 1-7 1-8 1-1 1-10 1-11 1-4 1-13 1-14 1-15 1-16 1-9 1-18 1-19

0-0 0-2 X 0-6 0-8 0-10 0-4 0-14 0-16 0-180-12 0-3 0-5 0-7 0-1 0-11 0-13 0-15 0-9 0-191-0 1-2 0-17 1-6 1-8 1-10 1-4 1-14 1-16 1-18

CAUI 1-12 1-3 1-5 1-7 1-1 1-11 1-13 1-15 1-9 1-19

0-0 0-2 X 0-60-8 0-10 0-4 0-140-16 0-18 0-12 0-30-5 0-7 0-1 0-110-13 0-15 0-9 0-191-0 1-2 0-17 1-61-8 1-10 1-4 1-141-16 1-18 1-12 1-31-5 1-7 1-1 1-11

Transmission lanes 1-13 1-15 1-9 1-19

PCSL

Figure 8: Rx-side Bit Sequence with Skew

Anritsu recommends following two methods to avoid this problem.

1. Rx measurements with 1:2DEMUX and ED 2. Synchronized ED Quick auto correction measurements

In method 1, measurement uses the PRBS recovered at PCSL with the PRBS pattern DEMUXed to 1:2. Measurement at PRBS31 specified in the standards is supported using the MU18204xA 25G DEMUX and MU181040A Error Detector. In method 2, the CAUI lane can be measured directly using the ED. This method uses the Quick Sync function built into the ED. This Quick Sync function captures the data pattern input to the ED into built-in memory and measures the bit errors by comparing the input data with this reference data. The ED-side can measure up to PRBS23 using a 128-Mbit data pattern. Figure 9 shows the CAUI lane measurement results for two sample CFP modules from company A using the measurement system in Figure 6. CFP Sample 1 Company A CFP Sample 2 Company A Company A CFP Sample 2 Jitter Tolerance

0.01

0.1

1

10

10 100 1,000 10,000 100,000

Jitter Frequency [kHz]

Jitte

r Am

plitu

de[U

I]

Tx Lane Error Ratio

0 Error Free

0 Error Free

1 Error Free

1 Error Free

2 Error Free

2 Error Free

3 Error Free

3 Error Free

4 Error Free

4 Error Free

5 Error Free

5 Error Free

6 Error Free

6 Error Free

7 1.71E-09

7 3.10E-09

8 Error Free

8 Error Free

9 Error Free

9 Error Free

Tx Lane Error Ratio

0 Error Free

0 Error Free

1 Error Free

1 Error Free

2 Error Free

2 Error Free

3 Error Free

3 Error Free

4 Error Free

4 Error Free

5 Error Free

5 Error Free

6 Error Free

6 Error Free

7 Error Free

7 Error Free

8 Error Free

8 Error Free

9 Error Free

9 Error Free

PRBS15 Measure Results PRBS9 Measure Results The above measurement is not accurate due toloss of CDR Lock at the CAUI input.

0.05

5

Jitter Tolerance Measurement at PRBS7

Specification

Result

Figure 9: CAUI Lane Measurement Results with BERT

From Figure 9, we can see that an error occurs in lane 7 at the Tx side when the first sample CFP module of company A is measured at PRBS15. (The error location lane can be confirmed using the PPG error insertion

function.) Moreover, because the results show a deterioration of the error rate up to about 1E-7 when the skew of the data phase changes between lane 7 where the error occurs and the adjacent error, it seems likely that the lane margin is inadequate due to the effects of crosstalk. In the measurement of sample 2, although the results are error free up to PRBS9, accurate measurement is impossible due to periodic Sync Loss at the Rx side at PRBS11 and above detected by the ED. This seems to be caused by periodic loss of the CDR Lock due to the inadequate CDR margin at the SERDES CAUI input, causing drift in the Rx-side measurement lane as a result of the SERDES reset time. In addition, the CAUI jitter tolerance results at PRBS7 confirm the inadequate performance of the SERDES device in this measured CFP module. At measurement with the BERT, although it is possible to perform the required worst-case tests of the physical layer, including the CAUI input sensitivity, skew tolerance, crosstalk tolerance, jitter tolerance, etc., the large scale of the measurement system when using a 10-lane system is a serious disadvantage. In addition, module vendors also require tests passing the actual frame signal. At Anritsu, we recommend the setup shown on Figure 10 as the minimum configuration capable of testing both the PCS layer and physical layer using an analyzer and BERT. In this setup, the CAUI signals for nine lanes are generated from the analyzer as the Aggressor signal, and the signal for the other remaining lane is generated from the BERT as the Victim signal. This configuration supports both the skew test (settings changed in 2-mUI steps) using the phase shift function of the BERT as well as tolerance measurements using the jitter generation function, helping to keep down equipment costs by using the smallest possible single configuration with a general-purpose BERT.

Figure 10: Recommended CFP Measurement Setup with 100 GbE Analyzer and BERT

5. Summary This article explains 100 GbE CAUI evaluations and problems using actual measurement data as well as the required tests and test systems from the standpoint of a measuring instrument manufacturer. Anritsu is continuing to develop even better measurement solutions to assure the highest possible product quality for its customers.

References http://grouper.ieee.org/groups/802/3/hssg/index.html IEEE802.3ba HSSG Tutorial_1107.pdf IEEE802.3ba 40/100G Architecture and Interfaces proposal IEEE802.3ba BaselineSummary_0508 IEEE P802.3ba /D2.3

(Prototype)

http://www.anritsu.com

5-1-1 Onna, Atsugi-shi, Kanagawa, 243-8555Phone : +81 46 223-1111

No. MP1800A-CAUI_Test-E-F-1-(1.00) Printed in Japan 2010-2 PRS