1

Beam Secondary Shower Acquisition System: Igloo2 GBT Implementation Status

GBT on Igloo2 MeetingJose Luis Sirvent

PhD. Student06/05/2014

2

0. Introduction1.1 Wire Scanner principle • Invasive method for beam

transverse profile measurement.

• Carbon wire interaction (30um) generates shower of secondary particles.

• Transversal profile:– Wire position (X axis)– Secondary rain (Y axis)

• System compromises:– Wire blow-up (heat)– Losses produced– Mechanical stresses (Bellows)– Calibration procedures– Vibrations

• Types:– Rotating Fast– Rotating Short/Long– Linear

• Total Scanners: 31

• Usage in a daily basis at CERN

Accelerator Type Quantity

SPS Rotating 6

SPS Linear 4

3

ResolverX Axis: Optical position sensor

Y Axis: Diamond Detector

0. Introduction1.2 BWS Prototype (Mechanical Aspects)

• Scan at least as fast as the existing system (20 m/s needed to avoid wire damage)

• Absolute accuracy of beam width determination of about 5 um (~5%)• Reduction of play in mechanical system

All elements mounted on same axis• High accuracy angular position sensor

Optical position sensor (Encoder)• Overcome bellow limitations

Locate all moveable parts in the vacuum• Minimize fork and wire deformations:

Acceleration profile optimized for low vibrationsMechanical design for minimum shaft and forks deformation

• Unified design for integration in the different accelerators (PS, SPS & LHC) MotorMagnetic StopperStepped Vacuum

Barrel

Carbon Wire

4

0. Introduction1.3 Complete test assembly for Proof of concept evaluation

Igloo2 Development board with GBTx firmware emulation

QIE10 Test BoardNeeded to be developed

Motivation:The final proof-of-concept can be evaluated by using this assembly. The Igloo2 in this case could be configured to act purely as a GBTx asic, this way the system could be suitable to work with Igloo2 or GBTx in case of change for final board.

Tasks:1. The system has to be configured to work in a complete assembly by using the knowledge from the previous tasks.2. The set-up should be done in a way to make possible Igloo2-> GBTx migration.3. Final tests and evaluation has to be done in with the assembly for system demonstration4. At this point a decision could be done regarding the FPGA usage or the development of a compact board with GBTx.

QIE10A

Power

Connector

QIE10B

SMA

SMA

pCVDPassiveSplitting System

RF Resistive network

Cividec Module

Coax Cable 50 ohm(1-10m)

VFC BoardOr

Arria V Dev. Kit

MonitoringPC

5

0. Introduction1.4 Development of a compact board for BWS pCVD diamond detector readout

VME FMC Carrier Board (VFC)Back-End VME Crate

Compact Front-End BoardNeeded to be developed

Motivation:Finally the last part of the project would be the development and testing of a very simple and compact board containing the Rad-Hard or Rad-Tol components.

Tasks:1. The QIE10 development board previously done has to be extended house the FPGA (or GBTx) and VTRx in the same board.2. Extensive testing has to be done in this board to guarantee that the performance reached in the development kits is repeated in this

version.3. Radiation studies are also important to characterize the complete radiation hardness of the board and identify failures.4. This system could be installed in paralel with an operational BWS PMT & Scintillator system to crossvalidate performances.

QIE10A

Power

QIE10B

SMA

SMA

pCVD

RF Resistive network

Cividec Module

Igloo2Or

GBTx

VTRx

FMC

Conn

ecto

r

NEW VFCVME FMC Carrier Board

FPGAArria V

Back

pla

ne V

ME6

4 Co

nnec

tor

BOBRVME Board

Back

pla

ne V

ME6

4 Co

nnec

tor

Clk_bunchClk_TurnClk_Events

SFP+

SFP+

SFP+

SFP+

SFP+

SFP+

Mem

ory

Mem

ory

TTC

Ethernet

PassiveSplitting System

Fibre OpticTTCDataControl

6

1.GBT-FPGASome information and resourcesGBT-Protocol implementation on FPGAS for GBTx communication & Emulation:• Public SharePoint: https://espace.cern.ch/GBT-Project/GBT-FPGA/default.aspx• Public SVN Releases (old): https://svnweb.cern.ch/cern/wsvn/gbt_fpga• Mailing List: gbt-fpga-users@cern.ch• Contacts for support: sophie.baron@cern.ch , manoel.barros.marin@cern.ch

Last news:• PH-ESE Repository (last updates): https://svn.cern.ch/reps/ph-ese/be/gbt_fpga• Indico Follow-up (GBT-FPGA): https://indico.cern.ch/event/283075/• Support & code available for (Dev. Kits):

– Xilinx Virtex 5 / 6 / 7 & Kinex 7 …– Altera Cyclone V & Statrix V…– Microsemi: Smatrfusion2, Igloo2 ??? At least not yet… but makes sense (CMS, MOPOS, us…)

Two Versions:Standard (STD) Data Readout (DAQ) Low and Deterministic latency (LATOP) FE control & Time, Trigger and control (TTC)

First Release 2011

First Release March 2014

7

1.GBT-FPGAThe Basic concept (Modular design)

GBT-FPGA Firmware Starter Kit for Altera and Xilinx devices. Sophie Baron 2010. CERN/PH/ESE. GBT-FPTA Starter kit user guide V0.1* Vendor Specific Modules (IP Cores)

8

1.GBT-FPGAThe Basic concept (Clocking Scheme and TTC clock recovery)

GBT-FPGA Comments on deterministic latency and recommendations to handle optimization schemes. Sophie Baron, PH/ESE. 2011

9

1.GBT-FPGACurrent Status (Latency optimization)

GBT-FPGA One unified core for multiple users. Manoel Barros Marin, PH/ESE/BE Students-Fellows seminar (05/02/2014).

10

2. Igloo2 GBT-FPGAThe code and the work I’m carrying out

• For the moment I’ll migrate with the old STD version and only GBT frame (simplest), once done, tested and well understood, I’ll move to LATOP version.

11

2. Igloo2 GBT-FPGA STD: Implementation statusSubstitute Xilinx IP’s by Microsemi IP’s (Some are straighforward)

12

2. Igloo2 GBT-FPGA STD: Implementation statusSubstitute Xilinx IP’s by Microsemi IP’s (Others.. not)• Transceiver with EPCS @ 4.8GBPS:

- A lot of configuration registers- Big amount of documentation- Different implemented protocols (not needed)- Power-Up Initialization needed (HPMS)- Synchronization issues- Needed standalone testing and verification

13

3. GBT-FPGA Overview in Igloo2:(Clock Management as in Virtex 6)

Tx_CLK (240MHz)

Rx_CLK (240MHz)

TX_Data_P (4.8Gbps)TX_Data_N (4.8Gbps)

Rx_Word (19 bits)@ 240 MHz

TX_Frame_CLK (40MHz)

Data_In (83 bits)@40Mhz

TX_Word_CLK (240MHz)

RefCLK1_P (120MHz)RefCLK1_N (120MHz)

RX_Data_P (4.8Gbps)RX_Data_N (4.8Gbps)

GBT_TXScramblerEncoderGearbox

Tx_Word (19 bits)@ 240MHz

GBT_MGT

SERDES_0Vendor Specific

IP

TX_PLL

GBT_RXGearboxDecoder

Descrambler

Data_Out (83 bits)@ 40Mhz

GBT_BANK (Very simplified view)

RX_Frame_CLK (40MHz)

RX_Word_CLK (240MHz)

RX_PLL

SERDES_INIT_MASTERAPB_BUS (PLL)

14

3. GBT-FPGA Overview in Igloo2:3.1 The MGT block (Schematic view) Written from scratch

According to Documentation:

• When using SERDES the HPMS should drive the APB bus for Configuration.

• 1 PLL Needed • No full control over SERDES• Manual configuration is possible

• SERDES Registers should be properly initialized when using EPCS

• When using EPCS the SERDES should be in PMA driven mode (manual bit lock steps).

• Microsemi Recommends Default configurations with minor changes:

• 1.25 / 2.5 GBPS• Lack of documentation for custom

modes

• Libero is constantly being updated and bugs repared…

15

3. GBT-FPGA Overview in Igloo2:3.1 The MGT block (Schematic view) Written from scratch

CLK_TX_WORD

CLK_RX_WORD

APB_BUS

Recommendations by Microsemi:• Shadow_FIFOs with CLKint• Clock Constrains in TX & RX CLKs• Place FIFOS as close as possible from SERDES• Drive the 4 lanes reset with the same signal!!

• Otherwise does not work!!

16

• Done through the Igloo2_APB_MASTER:– Access to the SERDES registers for configuration and status check.

3. GBT-FPGA Overview in Igloo2:3.1 The MGT block : SERDES Initialization & Control

Interface with the APB Bus protocolVery simple Read_Write Operations

Decides which actions to execute, registers to configure and work-flow

17

• Taking a look at Igloo2_SERDES_APB_MANAGER:

3. GBT-FPGA Overview in Igloo2:3.1 The MGT block : SERDES Initialization & Control

Registers Initialization

IDLE

Initialization

Init_Done

RX_RESET

CDR_FREQ_LOCK

CDR_PHASE_COARSE

CDR_PHASE_FINE

RefreshSerdes Status

READSERDES_TEST_OUT

Update Outputs

RESET

RX_RESET Refresh_Status

Register = Register + 1

All Registers initialized

Maintaining its setup time

SERDES_CLK = REF_CLK1EPCS Mode @ 4.8GbpsPMA Driven Mode (Manual bit lock)F = 2M = 1N = 20Lanes Impedance 50ohm

18

• Taking a look at Igloo2_SERDES_APB_MANAGER:

3. GBT-FPGA Overview in Igloo2:3.1 The MGT block : SERDES Initialization & Control

Registers Initialization

IDLE

Initialization

Init_Done

RX_RESET

CDR_FREQ_LOCK

CDR_PHASE_COARSE

CDR_PHASE_FINE

RefreshSerdes Status

READSERDES_TEST_OUT

Update Outputs

RESET

RX_RESET Refresh_Status

Register = Register + 1

All Registers initialized

Maintaining its setup time

SERDES_CLK = REF_CLK1EPCS Mode @ 4.8GbpsPMA Driven Mode (Manual bit lock)F = 2M = 1N = 20Lanes Impedance 50ohm

Caution:

I need to continue working here!!

For the moment the SERDES doesn’t lock to the bit Stream

• Initialization sequence, RX_Reset operation?

19

• Changed the Xilinx IP by the Microsemi’s:– Core: Two-Port Large SRAM– Adapt code for ports names in:

• TX: gbt_tx_gearbox_std_dpram.vhdl• RX: gbt_rx_gearbox_std_dpram.vhdl

3. GBT-FPGA Overview in Igloo2:3.2 The GBT_TX & GBT_RX modifications done:

Igloo2_txdpram_core Igloo2_rxdpram_core

20

3. GBT-FPGA Overview in Igloo2:3.3 Surrounding Infrastructure:

From GBT-FPGA examples• Pattern Generator: Generates Fixed or counts• GBT Bank Reset manager: Resets all in order• Error Detector: To check transmission • Pattern Match: For delay determination

From me• SERDES_INIT_REGISTERS• PLLs for 240 to 40Mhz: Word to Frame CLK• UART communication:

• For debugging on the PC• Check transmission status• Perform modules control :

• Resets, Pattern, Serdes Lane…

From GBT_Bank:• Configuration files

• gbt_bank_package• gbt_bank_user_setup• vendor_specific_gbt_bank_package

21

3. GBT-FPGA Overview in Igloo2:3.4 Communication with PC: Through Igloo2 UART Core

Microsemi Provides Drivers for PC to emulate Serial port through USB:Makes life easier when accessing to the USB when programming an application.

Microsemi Provides Core for FPGA to communicate with the UART (the board has a UART to USB interface)Only needed to develop a “translator” module that interfaces the CORE with the Fabric signals you are interested to control

22

4.Tests done so far in Igloo2: 4.1 Testing GBT_TX & GBT_RX modified modules

• It was needed to verify that the TX & RX modules work well with the modifications done.• Dual Port Rams Xilinx IPs substituted by Microsemi Ips (GBT-FPGA STD Version)• Design unconstrained up to now. (Improvements are specked)

84 Bits @ 40MHz 20 Bits @ 240MHz 84 Bits @ 40MHz

GBT Frame:

Data visible to GBT-TX and from GBT-RX84 bits @ 40MHz

23

4.Tests done so far in Igloo2:4.1 Testing GBT_TX & GBT_RX modified modules

• It was needed to verify that the TX & RX modules work well with the modifications done.• Dual Port Rams Xilinx IPs substituted by Microsemi Ips (GBT-FPGA STD Version)• Design unconstrained up to now. (Improvements are spected)

84 Bits @ 40MHz 20 Bits @ 240MHz 84 Bits @ 40MHz

24

• TX & RX Frame CLK: Comes from the same FRAME_CLK (40MHz)• TX & RX Word CLK: Comes from the same WORD_CLK (240 MHz)• Both clocks are artificially injected in the TestBench.• The TX & RX frames are well recovered with a delay ~ 320ns.

4.Tests done so far in Igloo2:4.1 Testing GBT_TX & GBT_RX modified modules

25

Static frame well recovered!!

4.Tests done so far in Igloo2:4.2 Let’s connect all together GBT_TX, GBT_MGT & GBT_RX

Simulation with static Frame: 0x0000BABEAC1DADCDCFFFF

84 Bits @ 40MHz 20 Bits @ 240MHz 84 Bits @ 40MHz20 Bits @ 240MHz1 Bit @ 4.8GHz

26

Dynamic frame well detected!!We see the TX & RX Flags for delay determination

Delay ~ 314 ns(Non deterministic, STD Version)

Simulation with dynamic Frame: Segmented counter

84 Bits @ 40MHz 20 Bits @ 240MHz 84 Bits @ 40MHz20 Bits @ 240MHz1 Bit @ 4.8GHz

4.Tests done so far in Igloo2:4.2 Let’s connect all together GBT_TX, GBT_MGT & GBT_RX

27

4.Tests done so far in Igloo2:4.3 Making our own delay measurements

By using:• Segmented count for transmission• Pattern detected flags for reference• Serdes Lane1 (looped on the board)

Delay :• 320 ns (Fits the simulations)• Non deterministic (Each start-up is different)

Test Conditions:• Design Unconstrained• Synplify parameters as default• Only SERDES manual initialization

28Slide from Manoel Barros Marin

130.3 ns

4.Tests done so far in Igloo2:4.3 Comparing data with specked results from LATOP version

29

4.Tests done so far in Igloo2:4.4 Interfacing and controlling transmission with PC

Observations and status:• For the moment when 1 FPGA is looped works well with Lane1 as well

as with Lane2 (SMA cables)• Not yet working with 2 FPGA (Synchronization issues?)• The bit lock is not yet reached in any of the configurations.

• Needed to study carefully synthesis constraints and optimizations:• The GBT_RX part seems to have some issues here…

30

• GBT Implementation Status about to be ready on Igloo2.• This is not official & non supported modified GBT-FPGA code, based on STD

version. • We aim as well to recover the TTC clock from the optical lines with the SERDES

CDR circuit. • We have to specify the clocking scheme of our Front-End

– Local Oscillator VS Dedicated clock line @ 40Mhz Beam synchronous.

• Still many things to understand/improve• Any contribution/collaboration is more than welcome. • The code needs to be cleaned and structured properly.

5. Summary:

31

Manual APB Master & ROMFor SERDES Configuration and Initialization: It works!! (Post-synthesis sim)

Tx & RX @

4.8Gbps

Now we have the Two M2G010 PLL’s available for GBT-Bank !!

Igloo2 SERDES Testing (Tx part)Different Speeds & Configurations

EPCS @ 1.25GbpsPre-Configured

EPCS @ 2.5GbpsPre-Configured

EPCS @ 4.8Gbps!!Custom Parameters

**EPCS : External Physical Coding Sublayer

Scope not for eye diagram determination BW 1Ghz, used just for reference 2* Signal Freq = Bit rateTransmission pattern “10101010101010101010”

Igloo2 SERDES Testing (EPCS-4.8Gbps) Looping the lines, TX & RX simulation and start-up sequence

Igloo2 SERDES Testing (TX & RX @ EPCS 4.8GBPS)Working with the Dev. Board and the means it provides

Parallel Input Data (20 bits):“1111 0000 1111 1111 XXXX”

Parallel Received Data (20 bits):“11110 00011 11111 1XXXX”

FramePos 3 2 1 0

FramePos 3 2 1 0

Recovered 11111 111XX XX111 10000

Rx Tx

SERDES

DataTx_0DataTx_1 …DataTx_19

DataRx_0DataRx_1 …DataRx_19