Jim Grahl, University of Minnesota ECAL HEP seminar, U. Minnesota, April 29 th 20031 Optical Data Links in CMS ECAL CMS ECAL architecture and needs Optical

1Jim Grahl, University of Minnesota

ECALECAL

HEP seminar, U. Minnesota, April 29th 2003

Optical Data Links in CMS ECAL

• CMS ECAL architecture and needs• Optical Data Links in HEP Experiments• ECAL Data Link system description• Data Link components• Data Link system• Opto-Hybrid assembly, Quality Assurance• Schedule


ECALECAL


The CMS Detector

Silicon Tracker

ECAL

HCAL

Solenoid

Muon ChambersReturn Yoke

Barrel Preshower +Endcap


ECALECAL


CMS ECAL front-end architecture

ECAL has ~77,000 lead tungstate crystals arranged in supercrystals of 25.

Front-end electronics of each supercrystal send the data off-detector via the optical links. The architecture and the needed data rate require ≥600 Mb/s with modularity:

for Data: 1 link / supercrystal

for Trigger: 1 link / supercrystal (barrel)

5 links/supercrystal (endcap)

Total data + trigger: ~9000 links

Supercrystal DataTrigger sums

APD/VPTPreamp

ADC

Front-end board


ECALECAL


Optical Links and HEP

Generic architecture of an optical link:

Tx Rxfiber

“Optical Layer”

Ser

Deser

“First Overlay”

MUX DEMUX

“Framing”

e/o, transport, o/e

encoding/decoding, usually performed in same chip as serialization/deserialization

Multiplexing multiple “conversations” in each link. Not usually relevant for HEP since systems are designed to run each link continuously at maximum data rate.

data in

data out


ECALECAL


Optical Layer considerations - 1

Tx considerations:

Edge-Emitting: linear response (relevant if you design an analog link)

VCSEL: said to be more rad-resistant due to smaller active area, but practical experience is that fractional change of behavior per unit irradiation is typically similar. VCSEL is cheaper to produce and test on wafer. Was once thought that they would entirely replace Edge-Emitting. But benefit is diluted since packaging dominates the cost of both.

In any case: device chosen must be consistent with the system speed and environmental specifications (often unique to HEP; more on this later).

Tx: In HEP, usually either Edge-Emitting Laser Diode or VCSEL.

Rx: Typically photodiode or photodiode array (typically of 12), followed by digital amplifying ASIC.

Tx Rxfiber

“Optical Layer”

e/o, transport, o/e

Rx considerations:

Speed, wavelength, saturation and sensitivity limits must be consistent with system specifications.

VCSEL diagram

Edge-Emittingdiagram

test on wafer


ECALECAL


Optical Layer considerations - 2

Optical fiber considerations:

Two principle kinds: Single Mode (SM) and Multi Mode (MM). Differences in impurities in the silica and differences in core and cladding diameters give different spectral transmission characteristics.

SM: lower attenuation, is the favored choice for telecom (longhaul) distances (10 km and greater). Uses smaller core to allow only one mode to propagate, making connector alignment more critical (and expensive).

MM: favored for “datacom” (LAN) distances (~5 to 10 km). In principle cheaper than SM, but fiber itself may be more expensive since market is smaller (datacoms did not take off).

HEP distances are short (~100m), so SM/MM choice is not critical. Usually free to select the cheapest solution satisfying environmental specifications. Qualification is essential:

Early ECAL qualification of MM fiber of various suppliers: Good fiber loses <50% transmission after 1013 p/cm2. Bad fiber turns black.

P. Denes


ECALECAL


First Overlay considerations

Serializer/deserializer considerations:

Encoding: Two principle protocols are CIMT (“G-Link”, Agilent proprietary) and 8-bit/10-bit. Both share the following characteristics:

• introduce an overhead of two bits per 8 bits of data• are usually employed continuously transmitting, either data or idle pattern• aid in maintaining synchronization of transmitter and receiver• allow some error checking• stabilize 0 and 1 levels by keeping number of bits of each the same

Principle protocol differences: G-Link is considered more robust in terms of maintaining and recovering synchronization. But it is proprietary. At the moment the fastest CIMT deserializer runs up to 1.4 Gb/s. In general there is no guarantee that a company employing proprietary technology will continue developing new products with it, nor any guarantee that it will not discontinue existing ones.

Speed and bits serialized: Typical scheme at LHC experiments is one (two) 16-bit words serialized at LHC clock rate of 40 MHz implying optical link speed of 800 (1600) Mb/s of which 80% is data and 20% is overhead.

Tx Rxfiber

Ser

Deser

“First Overlay”

encoding/decoding, usually performed in same chip as serialization/deserialization


ECALECAL


General considerations

In designing an optical link for a HEP experiment:

Typically start with a shopping list of needs.

From that define specifications and select/develop technology.

End up with a laundry list of “issues”.

The typically O($1M) procurement for a HEP experiment is relatively small for any company large enough to collaborate in the task. Development teams in laser transmitters are smaller than one might think, typically 10-30 people for a large company. => The experiment will incur significant development and qualification effort and costs.

HEP experiments typically have many specifications which are unusual in industry. The closer the transmitter is to the center of the detector, the more likely this is to be true. Every need has its consequence(s):

radiation resistance, low power (heat), temperature cycle, non-magnetic materials

extensive qualification of several products (variation of behavior between different products can be large)

limited spacecustom laser package, with consequences for hermeticity and mounting techniques

long (3-10 yrs) development time

possibility that a product will be discontinued

100% reliability of ~10k links over 10 years

sleepless nights


ECALECAL


Optical Data Link System Diagram

12Rx moduleRx module

12

1GOHGOH1212

1

96

DistributeDistributed Patch d Patch PanelPanel

Back-end Back-end Patch Patch PanelPanel

Off DetectorOff DetectorFront EndFront End Pigtail Pigtail

fiberfiber

RuggedizeRuggedized ribbond ribbon

Dense Dense multi-multi-ribbon ribbon cablecable

GOGOLL

Laser Laser diodediode In-Line In-Line

Patch Patch PanelPanel

CMCMSS

PIN photo-PIN photo-diode diode arrayarray

Digital Digital amp. amp. ASICASIC

1212

Edge-Emitting Laser Diode:ST Microelectronics

Digital OpticalReceiver: NGK

GOL ser. ASIC:CERN MIC

• GOL Opto-Hybrid (GOH): ECAL designs, prototypes, qualifies, defines manufacturing specifications, procures the manufacture, tests samples during production.• 12-channel NGK Rx: Off-the-shelf component: ECAL qualifies, procures, tests samples during production.• Fiber, connectors and adaptors: ECAL uses solutions already developed and procured for CMS Tracker.


ECALECAL


Components: Transmitter - 1

The transmitter of the data link is the GOH. Its principle components:

Some specifications of the GOH:• Receives LV, clock, control and 16-bit parallel digital input at 40 Mb/s from FE board via connector.• Transmits serialized optical output at 1310 nm, 800 Mb/s, either protocol, via pigtail fiber.• Output signal power ~-6dBm, depending on bias levels chosen. (0 dBm = 1 mW)

GOHGOH

• Designed by CERN Microelectronics group.• Implemented in 0.25μm CMOS technology employing radiation-tolerant layout practices.• Speeds of 0.8 and 1.6 Gb/s. ECAL will run the GOH at 0.8 Gb/s. • Two protocols (G-Link and 8b/10b). ECAL will use 8b/10b for data and G-Link for trigger.

• The Gigabit Optical Link (GOL) serializer and laser driver ASIC

• The laser diode, made by ST Microelectronics, based on a Mitsubishi die

• Custom-made for CMS Tracker (linear response), suitable also for digital use in ECAL (rise time consistent with 800 Mb/s operation).• Die wafer lots are radiation-qualified (gammas and neutrons) before assembly into laser diodes.• Pigtail fiber is attached and calibration data are taken by ST.


ECALECAL


Components: Transmitter - 2

Status of the GOH:

• The design is complete, prototypes exist.

• Functions well within the full data link (more on this later).

• Driven by GOH evaluation board (a modified GOL eval board), gives a clean eye diagram at 800 Mb/s.


ECALECAL


Components: Fiber and Connectors

Fiber and connectors are adopted from the CMS Tracker system

• All specifications consistent with use in ECAL as well (e.g. temperature, rad-hardness, attenuation, safety).

• ECAL benefits from commercial work, qualification and acceptance test setups already a part of Tracker program.

• ECAL’s contribution is the in-kind to staffing of acceptance tests.

12

12

1

96

MU-sMU adapter (Sumitomo)

MFS adapter (Diamond)

MFS-MPO multi-ribbon cable (Ericsson, Diamond)

sMU-MFS fanout (Ericsson, Sumitomo, Diamond)


ECALECAL


Components: Receiver

12-Channel digital NGK Rx

• COTS device; ECAL tasks are to qualify and procure.

• Specifications are consistent for use within the full data link, e.g.:

• wavelength: 1310 nm• speed: up to 1.25 Gb/s• saturation: -5 dBm• sensitivity: -18 dBm

• Functions well within the link (see following slides).

Rx moduleRx module

1212

PIN photo-PIN photo-diode diode arrayarray

Digital Digital amp. amp. ASICASIC

1212


ECALECAL


Data Link System Tests - 1

• Early system tests involved:• GOH driven by GOH evaluation board• Rx on Rx evaluation board• Deser (G-Link or 8b/10b) on Deser eval board

• counted Deser word errors (frame errors)…

Rx Err

Rx Out

ScopeClock Generator

Counter

Attenuator

GOH Evaluation Board

Rx Evaluation Board

Deser Evaluation Board

• …in this case, to study effect of crosstalk in the Rx• Conclusions:

• Link functions well, BER < 10-13 should be obtainable• Rx is well within sensitivity spec (-18 dBm)• Crosstalk costs about 2 dB of the optical power budget

~2 dB

Channel under study

Noise data generated on neighboring channels


ECALECAL


Data Link System Tests - 2

High = Des “Data Valid”

(Four) Trigger-generating pulses

High = Des “Rx Err”

• Four triggers generate four bursts of data.

• Each trigger involves the absence of one Control clock cycle.

• Valid data seen by deserializer, except occasionally soon after the missing clock cycle(s). It is expected that this will be resolved when QPLL (CERN MIC) is integrated into Front-end and Off-detector boards.

• The good news is that FE + Data Link works!

• First test of the Data Link with the prototype FPGA version of the Front-End board (mid-Feb 03):


ECALECAL


Optical Power Budget

Launched from GOH -6 dBm---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----------------------------------------------------------------------------------------------------

Laser diode efficiency 1 dB

Fiber and connector interfaces 3 dB

Rx crosstalk 2 dB---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----------------------------------------------------------------------------------------------------

Effective signal power at Rx -12 dBm

Rx sensitivity limit -18 dBm*

*Rx sensitivity spec is “-19 dBm typical, -18 dBm minimum”. We have so far observed better than –20 dBm.

=> At least 6 dB of margin in the power budget


ECALECAL


GOH Assembly

• Manufacturing Specifications document for the GOH is complete. We plan to submit the RFQ as soon as possible to have the maximum amount of time to test the first 100 (Engineering Run).

• GOH assembly involves a number of steps:

Wire-bond protective cover attachment

Laser diode attachment and wire-bonding

Fiber clamp attachment

GOL BGA mounting

PCB manufacture, passivecomponent mounting


ECALECAL


GOH QA and Link System Tests - 1

Construction of first 100 GOH (Engineering Run) will begin when all of these are ready:

• RFQ completed and manufacturer selected• Laser diodes are available• Tested GOL’s are available

We plan thorough tests to verify functionality and reliability.

• The tool for these tests will be our Bit Error Rate Test system. Based on PC, CMS “GIII DAQ” cards and extensive Altera programming.

• Principle is the same as for the frame error tests: plot BER vs. optical power, make a quantitative measure of a stress on the system in terms of a “penalty” on the optical power budget.

• Advantages over frame errors are that it is bit-by-bit, independent of protocol.

• BERT system has been working since beginning of the year. Presently runs at 100 Mb/s: BER of <10-13 in ~1 day.

• Present effort is to automate it to perform many measurements with minimal human intervention.


ECALECAL


GOH QA and Link System Tests - 2

• The functionality tests we plan include:

Temperature sensitivity: GOH, Rx, deserializer.

Sensitivity to LV fluctuations/noise.

Fine-tuning of the eye diagram via GOH compensation network components.

Integration of the Link into the full ECAL electronics chain.

• The reliability tests planned for the GOH include:

Vibration

Mechanical shock

Thermal cycling (-20°C to +40°C)

Pass/fail eye diagram test

Irradiation

Accelerated aging

Long-term stability

GOHGOH

• The goal of the tests is to uncover and resolve any design or manufacturing reliability issues that may exist, and to be able to sign off on the full Link system design by the October ESR.

Manufacturer

Data Links group


ECALECAL


ScheduleECAL Data Links Schedule 2003 2004 2005JG / 02 Apr 03 Apr May Jun Jul Aug Sep Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05

1. Test Beam GOH Assembly

2. GOH Assembly Procurement

3. Rx Procurement

4. GOH Engineering Run (first 100)Assembly and Testing by ManufacturerTesting by Data Links group

5. Milestone: October ECAL ESR

6. Full ECAL Electronics ChainSystem TestBeam Test

7. GOH Pilot Run (next 500)Assembly and Testing by ManufacturerTesting by Data Links group

8. Production for ECAL Barrel + PreshowerGOH and Fiber ProductionAcceptance testingInstallation

9. Production for ECAL EndcapGOH and Fiber ProductionAcceptance testingInstallation

Schedule concerns: May 03: If laser diodes or tested GOL are late, time for testing of Eng Run is reduced. Sep 03: If GOH design modification is necessary, little time to test it before the Pilot Run.

Qualification is crucialbut is the only “compressible” task.


ECALECAL


Summary

• Development of an optical link for a HEP experiment is a fascinating project, invariably involves a lot of co-development with industry.

• ECAL Data Link system design and project have made a lot of progress in a short time. Starting with good components helps.

Documents

Jim Grahl, University of Minnesota ECAL HEP seminar, U. Minnesota, April 29 th 20031 Optical Data Links in CMS ECAL CMS ECAL architecture and needs Optical