Data Serialization, Transmission and Deserialization at 3.2 Gbps and Beyond

M.Matveev

Rice UniversityHouston, TX 77005 USA

April 3, 2006

4th CMS Workshop on Detectors and Electronics for the SLHC. April 3-4, 2006. Perugia, Italy

Outline

• Architectures of serializers (SER) anddeserializers (DES)

• Our experience at Rice University withSERDES @ 1.6Gbps and 3.2Gbps

• Programmable and 10Gbps SERDES

SERDES Architectures

• Discrete SERDES Low- and Mid- Rate SERDES (F < 160MHz)

- Parallel Clock SERDES- Embedded Clock (Start-Stop) Bits SERDES- SERDES with 8B/10B encoding

A detailed overview of these architectures is available in the articleDave Lewis. SerDes Architectures and Applicationshttp://www.national.com/appinfo/lvds/files/designcon2004_serdes.pdf#page=2

High Rate SERDES (F = 622/644MHz)

• Programmable SERDES FPGA (Xilinx, Altera, Lattice Semiconductor)

Parallel Clock SERDES

1

7

••MUX

7:117 6

LVDS Data Streams

14

••8

21

••15

MUX7:1

MUX7:1

PLL

•• 5 4 3 2• •

•• 14 13 12 11 810 9• •

• • 21 20 19 18 151617• •

LVDS Clock

• 21:3, 28:4, 48:8 multiplexing• National Semiconductor DS90CR2xx and DS90CR4xx Channel Links• Up to 133MHz clock • Latency (SER + DES) ~4 clock periods

Embedded Clock Bits SERDES

MUX

1

2 LVDS Data Stream3

16

•••

C0 C1

C0 C1 1 2 ••• C0 C116•••

• Two clock bits are embedded into the data stream every cycle, framingthe start and end of each serialized word

• Automatic receiver lock to random data • Several 10-, 16-, 18-bit devices available from National Semiconductor(DS92LVxx and SCAN 92xx families), Maxim MAX9205/9206/9207/9208

• Up to 80MHz parallel data rate• Latency (SER + DES) ~3 clock periods

8B/10B SERDES

8B/10Bencoder

Parallel to

Serial

ClockMultiplier

Serializer

TxCLK

Serial to

Parallel

10B/8Bdecoder

and clock recovery

Deserializer

RxCLK

Media

• 8B/10B encoding/decoding allows to maintain a proper number of “0” and “1” in the data stream for DC balance and:

• provide a reliable clock recovery• determine symbol and packet boundaries• detect transmission errors

• Used by Gigabit and 10G Ethernet, Fibre Channel, ATM, Infiniband transmission interfaces

Texas Instruments TLK 8B/10B SERDES

TLK1501

TLK2501

TLK3101125 156.25

75 125

30 75

F, MHz40 80 120 160

Assumptions and Requirements

Assumptions on Serial Data Links• Typical distance of optical links is < 100 m at the LHC experiments• Typical frequencies of parallel data are 40/80MHz at the LHC and will

likely increase to 160MHz at the SLHC• In most cases the source and target of data transmission links are the FPGA

Requirements • Low cost• Small size• Low power consumption• Radiation tolerance in many cases

Preferences • Low latency• Easy and quick synchronization• Simple control interface• Built-in Self Test (BIST) capability

Why 3.2 Gbps?

Serialization/Deserialization at 1.6 Gbps and below is (generally)straightforward

• 1.6 Gbps = 80MHz x (16 + 4) bits = 40MHz x (32 + 8) bits• Commercial and custom components are available (Texas Instruments

TLK1501/TLK2501, CERN GOL, Agilent HDMP-1022/1024, 1032/1034, Cypress HOTLink, National Semiconductor DS92LV16…)

• Widely used at the LHC (CMS EMU Trigger, LHCb muon trigger…)

Serialization/Deserialization at 3.2 Gbps is more challenging • 3.2 Gbps = 160MHz x (16 + 4) bits • Components need to be evaluated (most commercial parts are targeted

to Gigabit Ethernet, SONET OC-48, Fibre Channel applications whichutilize reference clocks of 106MHz, 125MHz, 155.52MHz, 156.25MHz)

CSC EMU/Trigger Electronics at CMS

80MhzLVDS

80Mhz GTLP80Mhz GTLP

40MhzLVDS

1.6Gbps

Cathode StripChambers(CSC)

1.6 Gbps CSC Trigger Optical Links

MPC

MPC

SP

TITLK2501

FinisarFTRJ8519

16-bit @ 80Mhzfrom FPGA

Muon Port Card

XilinxXCV

FPGA

FinisarFTRJ8519

TITLK2501

16-bit @ 80Mhzto FPGA

Sector Processor

XilinxXC2VFPGA

MPC

MPC

MPC

x 12180 optical links

• 50/125 um multimode fiber• 100 m long

• 2x5 SFF optical transceiver• 1.6 Gbps simplex transmission

• 16-bit SERDES• 8B/10B encoding• PRBS capability

Comparison of suitable SERDES Devices

85~10025Price, $ (small quantity)

Yes

64-pin VQFP

0.45 W

2.5

PRBS 27-1

Simple

3 consecutive IDLE cycles *

<45 ns @ 156MHz

Yes

125 - 156.25

16 LVTTL

Texas Instruments TLK3101

??Synchronization method

195-pin BGA256-pin BGAPackage

??

2.1 W

2.5

256-byte “counter”

Complex

< 63 ns @ 156MHz

Yes

30 – 39.6; 60-78.125; 120-156.25; 240-312.5

8 LVTTL*4 channels

Vitesse VSC7226-01

ModestControl interface

??

1.2 W

1.8

No

> 100 ns @ 160MHz

No

78-84.4;155-169

16 LVDS/LVPECL

Silicon Laboratories Si5100

Built-in Self Test

Latency (SER+DES)

Transmitter Reference Clock Frequency, MHz

Availability

Power consumption, (typ)

Power supply, V

8B/10B Encoder/Decoder

Number of parallel I/O

* As defined in IEEE 802.3 standard

SFP Plugguble Optical Transceivers

20-pin connector CageOptical transceiver

• Multi Source Agreement (MSA) compliant Small Form Factor Pluggable (SFP) footprint

• Up to 4.25 Gbps bidirectional links• 850 nm oxide VCSEL laser transmitter• Duplex LC connector• Up to 150 m on 50/125 um MMF @ 4.25 Gbps • Metal enclosure for lower EMI• 20-pin surface mounted connector• Several vendors, typical price < $100 per module

3.2 Gbps Optical Link

TITLK3101

TITLK3101

FinisarFTRJ8524

16-bit @ 160Mhzto/from FPGA

16-bit @ 160Mhzto/from FPGA

FinisarFTRJ8524

• 50/125 um multimode fibers

• 20-pin SFF Pluggable optical transceiver• Up to 4.25Gbps duplex transmission

• 16-bit SERDES• 8B/10B encoding• PRBS capability

~$80 ~$25 ~$80

Xilinx XC2V250

FPGA

Xilinx XC2V250

FPGA

Evaluation Board

233 x 160 mm

FTRJ8524

FTRJ8524

TLK3101

TLK3101

XilinxXC2V250

FPGA

TTCrqMezzanine

board

VME

Link Control Interface

XC2V250 FPGA

FIFO_A1

FIFO_B1

FIFO_A2

FIFO_B2

CONTROL

TXDTLK3101

TLK3101

RXD, RX_ER, RX_DV

TXD

RXD, RX_ER, RX_DV

RX_CLK

RX_CLK

CSR

TX_CLK

TX_CLK

LINK 116

18

LINK 216

18

• 160Mhz from QPLL on TTCrq mezzanine• On-board oscillator• External source

Optical Link

Top view

Bottom view

Total PCB space required for the link (SERDES + optical module): ~16 cm2 (2.5 sq in)

Test Configurations

TRRC

TRRCLink 1

Link 2

Board 1

TRRC

TRRCLink 1

Link 2

Board 1

TRRC

TRRCLink 1

Link 2100m 5.0m

100m

Board 1

5.0m

TRRC

TRRC

Link 2

Link 1

Board 2

100m100m100m

100m1 2

3

Test Configuration 3

Results of Measurements

< 10-14 (PRBS), <10-9 (Data)**

155.52;161.132

Epson EG-2101-CA oscillator(peak-to-peak jitter ~ 25 ps)

~ 10-9 (PRBS)40…160Xilinx XC2V250 Digital Clock Manager (peak-to-peak jitter ~ 250 ps)

~ 10-10 (PRBS)160.3147Cardinal Components CPPHV7-A7BC programmable oscillator (jitter ~ 80 ps)

~10-14 (PRBS)*, <10-9 (Data)**

160.3147QPLL ASIC on TTCrq mezzanine board (peak-to-peak jitter ~ 50 ps)

Bit Error Rate (BER)

Frequency, MHz

Clock Source

* 3 errors observed, ~2.3x1014 bits transmitted ** Random patterns transmitted from FIFO_A to FIFO_B

Power Supply and Grounding

• TLK3101 device is powered from Vdd=2.5V (2.3V < Vdd < 2.7V)Optoboard has an adjustable voltage regulator for +2.5V powerDoes the Bit Error Rate depend on power supply voltage? The error free PRBS data transmission over 100 m optical fibers was observed for 2.42V < Vdd < 2.65V

• The TLK3101 is housed in 64-pin VQFP PowerPAD package, which has an exposed die pad on the bottom of the device. It is important to solder this pad to thermal land connected to PCB ground for better thermal performance and device grounding (see TLK3101 specification).

125

80.00

190

125.00

230

155.52

240235Measured supply current for the TLK3101 @ Vdd = 2.5V, mA

161.1328160.3147Reference clock frequency, MHz

• Total power consumption for the TLK3101 + FTRJ8524 @ 160MHz is235mA x 2.5V + 150mA x 3.3V = 1.1W

Conditions: PRBS pattern transmitted and received

Latency MeasurementsFrom the TLK3101 specification:

So the total link (transmitter + receiver) latency may vary 110..145 bit times, or34..45 nanoseconds @ 160Mhz

Our measurements conducted at room temperature and 2.5V power for six TLK3101devices indicated that the total link latency is between 40 and 43 ns @ 160Mhz.

While the exact value is different at each link initialization in increments of the serialbit clock (310 ps), it has never varied more than the 3 nanoseconds total.

Radiation Tolerance

• Our 1.6Gbps link based on TLK2501 (same family as TLK3101)and Finisar FTRJ-8519 (850 nm VCSEL laser transmitter) was tested under irradiation, see report at the LECC2001http://lhc-electronics-workshop.web.cern.ch/LHC-electronics-workshop/2001/

posters/matveev2.pdf

Two TLK2501 devices were exposed up to ~270 kRad TID and produced 12 and 19 data errors respectively while transmitting the RRBS data through 100 m of fiber at 80Mhz. Similar results were reported inLHCb-2003-008 Technical Note.

While no errors were observed during the exposure of two Finisar FTRJ-8519 optical modules, both devices failed permanently at ~70 kRad.

Serial Transmission over Copper Cable

SFP solutions available from Molex

Cable directly terminated to module

HSSDC2 Copper SFP moduleand cable assembly• Industry standardHSSDC2 plug interface

• Passive or active transceiver

Optoboard with Molex 73929 Copper Cable

Molex 73929-0003 SFP Copper Patch Cable• Amphenol Spectra Strip Skewclear Cable• 5 m long, passive• 2 shielded parallel pairs• Skew (within pair) < 7.14 ps/m• Attenuation 0.57dB/m @ 1562.5 MHz (3.12Gbps)

Error free transmission (PRBS patterns @ 3.2Gbps) within 48 hours. BER < 10-14

Eye patters for 7 m and 17 m Skewclear cables (PRBS @ 3.1875Gbps) are available at http://www.t11.org/ftp/t11/pub/fc/10gfc/01-283v0.pdf. Look good.

10 Gigabit Industry Standards

SONET/SDH

Ethernet

Fibre Channel

1 2 3 4 5 6 7 8 9 10 11 Gbps

164.35 or 657.4210.5187510 Gigabit Fibre Channel155.52 or 622.089.9532 10GBase-W161.13 or 644.5310.312510GBase-RIEEE 802.3ae

10 Gigabit Ethernet

167.33 or 669.3310.709Optical Transport Network (OTN) G.709

155.52 or 622.089.9532Synchronous Optical Network SONET OC-192 (US, Canada) / Synchronous Digital Hierarchy SDH STM-64 (rest of the world)

Reference Clock Frequency, MHz

Serial Bit Rate, Gbps

Standard

10 Gbps SERDES Devices

??

~150

301-pin BGA

1.2 W

1.8/3.3

No

Modest

??

No

155/161/166/167; 622/644/666/669

16-bit LVDS

Broadcom BCM8152 SERDES

SimpleModestModestControl Interface

68-pin QFN244-pin BGA255-pin BGAPackage

~80>100175..210Price, $ (small quantity)

??

1.1 W

1.2&1.8/2.5/3.3

Yes

??

No

155/161;622/644

16-bit LVDS

AMCC S19235/S19237

SERDES

??

1.15 W

1.8/3.3

Yes

??

No

622-644

16-bit LVDS

Vitesse VSC8479 SERDES

27-1 PRBSBuilt-in Self Test

??Latency (SER+DES)

155/161; 622/644

Transmitter Reference Clock Frequency, MHz

YesAvailability (lead time)

1.15/1.5 WPower consumption, typ

3.3Power supply, V

NoEncoder/Decoder

16-bit LVDSNumber of parallel I/O

Maxim MAX3952 SERMAX3953 DES

Programmable SERDES

• RocketIO Transceivers available in Xilinx Virtex-2 Pro & Virtex-4- Full-Duplex SERDES

50Mhz - 156.25 MHz (600 - 3.125 Gbps) for Virtex-2 Pro 62.5Mhz - 425 MHz (2.488 - 6.25 Gbps) for selected Virtex-2 Pro X 106Mhz - 644MHz (622 - 10.3125 Gbps) for Virtex-4

- Up to 24 channels- 8-, 16-, 32-bit selectable FPGA interface- 8B/10B encoder and decoder with bypassing option on each channel- 50Ω/75Ω on-chip selectable transmit and receive terminators

• FPGA with embedded SERDES are available from other vendors as well- Altera Stratix II GX (6.375 Gbps)- Lattice Semiconductor ORLI10G (10Gbps/12.5Gbps)

• In 2003 Xilinx and Lattice announced a 10Gbps families of SERDES devices(RocketPHY and XPIO110GXS), but later both have been discontinued

Latency of SERDES in Xilinx FPGA

• Virtex-2 Pro Transmit latency (from TXDATA to TxP/TxN pins)

- 14..17 clock periods (with CRC)- 8..11 clock periods (without CRC)

Receive latency (delay from RxP/RxN to RXDATA pins)- 25..42 clock periods

• Virtex-4Transmit latency (from TXDATA to TxP/TxN pins) - min 6 clock periods (without 8B/10B encoding)- min 8 clock periods (with 8B/10B encoding)Receive latency (delay from RxP/RxN to RXDATA pins)- min 8 clock periods (without 8B/10B decoding)- min 9 clock periods (with 8B/10B decoding)

Reference Clock Requirements

2.5-3.125Gbps: <40ps1.06-2.5Gbps: <50ps< 1.06Gbps: <120ps

45..55

-100…+100

50 - 156.25

Xilinx Virtex-2 Pro

RocketIO

30..7040..6045..55Reference Clock duty cycle, %

-100…+100-100…+100-350…+350Reference Clock frequency tolerance, ppm

> 6.25Gbps: ?? UI;2.5-6.25Gbps: ?? UI;< 2.5Gbps: ?? UI

106 - 644

XilinxVirtex-4

RocketIO

??< 40 psReference Clock total jitter, peak-to-peak

155/161; 622/644

125 - 156.25Reference Clock frequency range, MHz

Maxim MAX3952/MAX3953

TI TLK3101

Examples of Oscillators for SERDES

55..215; Arrow40..50; Digikey30..40; DigikeyPrice, $ and availability

155 - 85053.125 - 50062.5 – 400Output frequency, MHz

3.3 +/- 0.33; 5.0 +/- 0.5

2.5 +/- 0.1253.3 +/- 0.15Power supply, V

-50 .. +50;-50 .. +50; -100 .. +100

-50 .. +50; -100 .. +100

Frequency tolerance, ppm

45 .. 5545 .. 5545 .. 55Output clock duty cycle, %

Recommended by Xilinx for Virtex-4 RocketIO

Used on evaluation board

Comments

< 5 ps25 ps (typ)25 ps (typ)Output clock total jitter, peak-to-peak

LVPECL

Epson 2101CA

ECL, PECL, LVPECL

LVPECL or LVDS

Output level

Vectron VS500Epson 2121CA

10 Gbps XFP Optical Transceiver

• XFP Multi Source Agreement (MSA), see http://www.xfpmsa.org 10+ vendors (Finisar, Intel, Picolight, Infineon, Tyco…)

• Protocol Agnostic (“any application, any rate”: OC-192, G.709, 10G Fibre Channel, 10G Ethernet)

• 850 nm laser (300 m), 1310 nm (up to 10 km)

850 nm VCSEL XFP & SFP Optical Modules

LC typeOptical Connector

< 82 m (500MHz*km bandwidth)

< 300 m (2000MHz*km bandwidth)

< 150 m (4.25 Gbps; 500MHz*km bandwidth)

< 270 m (4.25 Gbps; 2000MHz*km bandwidth)

Optical Link Distances on 50/125 um MMF

30-pin20-pin Electrical Connector78 x 18 x 8.556 x 13 x 8Dimensions, mm

+1.8; +3.3; +5.0+3.3Power supply, V

~400< 100Average Cost, $< 2.0< 0.6Power Dissipation, W

9.95 – 10.75Up to 4.2Bit Rate, Gbps

XFPSFPParameter

10 Gbps Over Copper Cable

• XFP form factor• Data transmission up to 15..30 m • Differential 100 Ohm twin-ax or 50 Ohm low loss coax cables

Example of the 10Gbps XFP copper transceiver from Vitesse

R&D on Programmable SERDES

Virtex-4XC4VFX20-11X FPGA

RocketIO

RocketIO

Optical XFP Module

• 30-pin XFP Pluggable transceiver

RocketIO

RocketIO

Virtex-4XC4VFX20-11X FPGA

Optical XFP Module

RocketIO

RocketIO

Virtex-4XC4VFX20-11X FPGA

Copper XFP Module

RocketIO

RocketIO

Virtex-4XC4VFX20-11X FPGA

Copper XFP Module

• Spectra Strip Skewclear Cable• HSSDC2 Connector

• Commercial or custom• With or without equalization

Conclusion

Based on our experience with three identical evaluation optoboards (6 links total) in laboratory conditions at room temperature we may conclude:

• The link built on TI TLK3101 SERDES can provide a reliable data transmission at 160Mhz over up to 100 m long multimode optical fiber(with Finisar FTRJ-8524 optical transceiver) and over 5 m of Spectra Strip Skewclear copper cable- low jitter reference clock for the transmitter is required- Virtex-2 FPGA works well at 160 MHz as a data source and target

• The board can be used for evaluation of several pluggable optical and copper modules and TI multigigabit transceivers TLK1501/TLK2501/TLK3101 with various clock options as well as for pre-selection of these devices

• We plan to continue R&D on evaluation of:- Programmable SERDES in Xilinx FPGA- 3.2Gbps…10Gbps components (SERDES, optical and copper transceivers) - High speed parallel and serial backplane technologies (GTLP, LVDS)