ABCD DAQ Timothy Phung. Topics 1.Introduction 2.Status of Software 3.Overall plan of the software 4.Hardware Setup 5.Threshold Scan 6.Details of how Threshold

ABCD DAQ

Timothy Phung

Topics1. Introduction

2. Status of Software

3. Overall plan of the software

4. Hardware Setup

5. Threshold Scan

6. Details of how Threshold Scan works

7. Analysis of Data

8. Known problems or issues

9. Comparison of old and new system

10. How we should split up the software development

11. Further things to look into

12. Lessons Learned so far

Introduction

• We are working on a DAQ to read out several modules on a stave

• The DAQ uses a digital I/O board from National Instruments (the pxi 6561) which sits in a pxi chassis

• We use labview to program the board to run these tests

• Labview is also used for processing and analysis of data

PXI 6561Triggers are on the back

RSTI

We send clock and commands at the DDC connector. Data is also acquired at the DDC connector

The interface to the computer is through the PXI bus and through a MXI-4 connection

Generation and acquisition memory are independent

PXI 6561 cont.•There are 16 channels of LVDS input/output

•We will use 1 channel for output (commands)

•The other 15 channels will be later used for acquiring data from modules

•Clocks are also sent from the 6561 to the chips using the onboard clock

Hardware SetupPXI chassis with MXI –4 and the 6561 board

Power from SC2001 support card and the SCT LV

Break out panel from NI

Panel Connections

Clock goes to chip

Generation channel and Acquisition Channels

Modules on a Stave readout case

•Several channels of data coming out

•One channel for generating data distributed to all modules

•One clock signal coming in to all modules

Data

Clock and commands

ABCD chip

•We readout and decode the bit streams of the ABCD chip to find the occupancy of the 128 channels. Based on this we can make measurements of gain, noise, etc.,

•ABCD chip has an amplifier, shaper, and comparator at the front end.

•We simulate the particle tracks by using the calibration circuit

Threshold Scan

>

Comparator

V_threshold

V_in

• If V_in >= V_threshold, the comparator gives a true value

• If V_in < V_threshold, the comparator gives a false value

• As V_threshold increases and V_in stays constant, then less channels will respond true

V_threshold

occupancy

An important point is the 50% point VT_50

Threshold Scan cont.

V_threshold

occupancy

•Increasing charge

•As charge injected increases, the Vt_50 value increases

Charge injected

Vt_50

•Gain measurement

•We fit a 2nd order polynomial to the vt-50 values as a function of charge injected.

•The derivative of the curve at each charge point gives us the gain at that point

Status of SoftwareTest

configure hardware scripts

acquisition of data

data processing analysis reports

1 IV Curve2 Stream Delay3 Hard Reset4 Redundancy Test5 Full Bypass Test6 Pipeline Test7 Strobe Delay x x x x x8 3 pt gain x x x x x9 Trim range x x x x

10 Response Curve x x x x x11 Noise Occupancy12 Time Walk13 Threshold Scan x x x x x

Currently working on Noise occupancy

Overall Plan of Software

Configure pxi 6561 hardware clocks, channels, resource names, etc

Configure generation session, write waveform and scripts

Acquisition of Data in a loop

Parsing of Raw Data Bit streams from ABCD chip

Done in parallel

File .xls or .bin file of raw data occupancy as a function of scan parameter and channel number

Each test follows the same basic structure

Overall Plan of Software cont.• Raw data files in .xls

or .bin format are read in and analysis is performed

• We plan to use the reporting feature in labview

Analysis

Plots,

Histograms on Front Panel

ReportsRaw Data Files

Example Program: Threshold Scan

Hardware configure

Writing waveforms and scripts

Data Processing

Acquire Data

Queue structure allows for execution on multiple threads so both processes can occur in parallel

Configure HardwareConfigure resource and channelsThe stream delay is for now placed with a default value of 0.5 * 25ns = 12.5ns

The clock is at 40 MHz

I export the on board clock to chip through the DDC connector

Configuration of Triggers

• Triggers are sent along the RSTI lines at the back of the 6561.

• There are 4 triggers• Triggers are setup to do multirecord acqusition

1. Ready to Start Event

2. Ready to Advance Event

3. Start Trigger

4. Advance Trigger

Scripts• Part of the threshold scan script

• Wait for ready to start event

• Generate configuration“X”“Y” where X = cal line and Y = threshold value rounded to decimal integer

• Generate trigger marker0(160) is the command sequence with the level 1 trigger. The marker event starts the multirecord acquisition.

• Repeat for (N-1) where N = number of (triggers/4)

• Wait for scripttrigger 1 to get a ready to advance event

• Generate trigger marker1(160) to advance to the next record.

script myScript1

Wait until ScriptTrigger0

Generate config050

Generate trigger marker0(160)

Repeat 999



end repeat

Generate config150


Repeat 1000



end repeat

Generate config250


Repeat 1000



end repeat

Generate config350


Repeat 1000

Multirecord Acquisition

• For a threshold scan, we acquire for 4000 records with 3500 samples on each record to roughly account for the maximum data packet length that comes at each threshold step

• There is one trigger from a level 1 command to start the multirecord acqusition and subsequent level 1 commands are used to advance to the next record. These are exported as marker events in the script.

Multirecord Acquisition cont.,We setup multirecord acquisition and then fetch records inside of a loop and then place them into a queue. Another loop parses out data after pulling it out of the queue.

Processing of data•Processing of data occurs by pulling data out of the queue and into a subroutine to do the parsing.

•The parsing is done by using bit wise operations in labview

•For now the parsing is able to keep up with the fetching with data, as the queue size stays zero throughout the entire scan.

•For the case of many modules, the parsing may take longer than the fetching of data and the queue may build up.

•We should look into faster and more efficient bit stream parsing algorithms

Results from this system1000.00

0.00

500.00

400

50

75

100

125

150

175

200

225

250

275

300

325

350

375

Channel Number8000 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750

graph

•Threshold scan from 50 to 400 mV for 1.5 fC injected charge

•Above plots show the occupancy as a function of channel number and threshold

•The occupancy at 50mV

•S-Curve for a channel

Results continued

• A strobe delay plot

• Occupancy as a function of the channel number and strobe delay setting

1000.00

0.00

500.00

65

0

5

10

15

20

25

30

35

40

45

50

55

60

Channel Number8000 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750

graph

Analysis of Data: Finding the Vt_50 point

• We have two cases, rising edges and falling edges

• Currently, I am using the “sparsification threshold measurement” approach to find the vt_50 values

• The vt_50 values are given below:

• Plots to the right are for threshold scan and strobe delay

size step

occupancy) d(normalize

1

150

150

V

N

N

VVV

VVV

total

latchi

N

iiowthresholdl

fallingT

N

iiowthresholdl

risingT

Analysis cont.

• Other analysis routines are just statistical caculations or fits to curves.

• I use the routines from labview to perform these

Speed, Optimization

• The threshold scan for the case of 50 mV to 450 mV with 1.5fC of injected charge is much slower than the old system.

• On a fast dual processor Xeon computer, it is slower by a factor of 2.

• Old system just takes 1 min to do a threshold scan, it takes the new system 2 minutes to do this on a fast computer

Speed Bottlenecks

• Using the profile vis feature it has been identified that the transfer of data from the pxi 6561 to the host pc and the parsing of data are the most time consuming parts of the program

Differences between old and new system

•Old system parses data in hardware, we parse in software

•Also we wait a fixed amount of time between triggers, old system decreases time between triggers as occupancy decreases

MUSTARD VME readout card

Relative Rates of Fetching Data and Parsing Data

Fetching of data stays the same throughout, since we fetch the same amount of data always

Rate

Time or increasing threshold or time as test progresses

Data Parsing Rate decreases because the parsing of data occurs in a while loop that stops when the trailer bit stream is found <1000,0000,0000,000>. As we move the threshold out this is found sooner since there is less occupancy.

Time Between Triggers Compared to old system

Time Between Triggers


Old sct test daq the time decreases since there is hardware parsing of data. When it finds the end of the bit stream for one event, it instructs through vme for another trigger to be sent next

New system: Time between triggers is the same throughout, it has to wait for 3500 clock cycles and a ready to advance trigger to start on the next trigger

Suggested Solution

Rate


Data Parsing Rate

Current System Fetching Rate

Make Fetching Rate decrease by taking data packet size from previous scan and using it as the record size of next scan

Suggested Solution cont.

Configuration of record sizes and data parsing

Boolean T/F

Trailer Bit stream not found.

Taking Data packet length from previous scan and feed it back as the record size of the next scan

Are results consistent between two systems?

The values for the gain, input noise, vt50 values, and the extrapolated offset are consistent

Continue comparison

Style of vis

• It will would aid the software development and debugging if we could have a consistent style of the vis throughout

• For example we should have a color code for the vi icons

• Also consistent names for all the vis• This will help when we have many people

working on the software

A Proposed Color code

• Blue = Threshold Scan• Yellow = 3pt gain• Purple = Strobe Delay• Etc. • All related vis

analysis,data processing should be the same background color as well

Documentation

• Right now most vis are documented inside of themselves

• We should also have some outside documentation

• Vi to the right is for 3 pt. gain

User Interface• It has not been decided

and should be discussed what the user interface of the control program, reports, plots, histograms should look like

• A full response curve

Version Control Software

• NI recommends to now use perforce or Microsoft visual source safe

• I have been using the built in source control, but it has limitations, you cannot rearrange vis into files after you place them into the source code control

• NI has discontinued its built in source code control in newer versions of labview

• Version control software will definitely help if more people are going to work on this project

How we should split up softwareTest


acquisition of data




Mother vi to control everythingUnfinished tests and reports

There are lots of additional things to work on

• Integration with software to control power (Code has already been written by Evan Wulf to control the SCT LV and HV modules)

• A configuration file format

• A fast parsing algorithm for the case of many modules in parallel

• Planning of clusters to pass data around instead of passing 20 controls or indicators to subvis

• We still also need to cleanup the code, document it, and optimize it as well

Things to look into further

• Embedded controller on the pxi chassis to do the parsing• FPGA module from NI to do the parsing• Streaming the data from an entire test to the disk and

parsing out streamed data• The speed of the system for the case of many modules on a

stave• Whether building the program into an executable and

turning off all debugging increases speed a lot• What the maximum data packet size for the ABCD chip is. I

assumed a value of 3500 samples for 4 chips using a rough estimate.

Brief Overview on Streaming vs. Multirecord

• Multiple record acquisition allows for setting up complex trigger sequences on the pxi 6561.

• In streaming you just have one trigger to start and another to end.

• The idea behind streaming is to read in data from the 6561 to the host pc at faster rates than it comes into the 6561 from the device under test.

• For us this means we need to read in data at faster than 80 MB/s for 2 bytes * 40 MHz.

• Streaming is highly dependent upon the computer system.

• I did not look into using streaming in detail yet, because we are more interested in getting a working system for now.

Streaming cont.

• Streaming data is very memory intensive on the computer. • If we operate a test for 1 min we will have 4800MB or 4.8GB of data on the computer.

80MB/s * 60s.• Also, we have to write out the entire script before we are beginning and makes sure that

we wait sufficient time between all trigger sequences before sending the next trigger sequence.

• There is a lot of data to parse, we have to sift through a lot of data after it is streamed in. A lot of it is useless data

• Streaming requires less overhead than multirecord• It will be hard to implement such tests as noise occupancy with streaming, where we have

to decode some data before we send the next commands. For tests where the decoding of data and the next set of commands are not dependent on each other this may be a possible alternative.

• One should also use the benchmark vi before using streaming. Faster pcs usually are able to stream better. It is also dependent upon what other things are on the pci bus, since devices on the pci bus all share the same resources.

Water Analogy of Multirecord and Streaming

Data from the ABCD chip is like a bucket of dirty water

Multirecord Streaming

6561 storage container

6561 storage container

Host pc Host pc

•With multirecord acquisition it is like you turn on and off (based on a trigger) the faucet of dirty water coming in to the 6561 with a filter in front.

•Refined Data comes into the 6561 and host pc.

•Less parsing to do in the pc

•With streaming, you turn on the faucet once, and leave it on with no filter in front, all the dirty water comes in and must be filtered in the pc.

•This is much faster, but the parsing at the end may be slower.

•Also need a large hard drive or else host pc will overflow.

Filter

Fil

ter

Lar

ge F

ilte

r

Green is the dirty water, blue is cleaner, light blue is the cleanest

Lessons learned so far

• Placing data into queues to process in a producer/consumer loop increases performance slightly by putting the two processes on separate threads

• There is some overhead into going into subvis. For example if a subvi resides in a loop, it is better to place the loop in the subvi. In cases where there is not possible, turn the execution priority of the subvi to subroutine to minimize the calling time to the subvi

• Making the vis reentrant also speeds up execution a little because multiple data sets can enter into the same subvi and execute along different threads.

• The stream delay setting on the PXI –6561 does not allow for setting the stream delay to be 0.2 to 0.3 * 25ns or 0.7 to 0.8 * 25ns

See NI Application Note 168 Labview Performance and Memory Management for more details on optimization

Other useful references from NI

• Application Note 199 LabVIEW™ and Hyper-Threading

• Application Note 114 Using LabVIEW to Create Multithreaded VIs for Maximum Performance and Reliability

• LabVIEW Development guidelines describes application of software engineering principles to development of large labview applications

• Lots of references on NI website as well.

An iterative software development lifecycle

• Developing the software in iterations looks like a good idea at this point.

• We are on iteration 1 at this point1. Version 0.1 (iteration 1) get a complete working system

going without worrying about all the small details and optimize later, solution is not the most elegant at this point. We are more or less learning about how building this application in labview works, how to communicate with the abcd chips to perform this test, and how to analyze data

2. (iteration 2) we will start to optimize the application and make the solution look cleaner and more modular

Working Session Results

• Suggestions after meeting with NI engineer and our own discussion

• How software will be split up• Things we can do to make the development

uniform

Suggestions• We spoke to an engineer from NI (Matt

Thompson).• We also came up with a list of ideas talking

amongst ourselves about how to increase performance.

Two main suggestions categories• Immediate suggestions • Extra Hardware suggestions

Immediate suggestions

1. Prioritize loops2. Make the record size smaller and smaller

(presented earlier) for threshold scans.3. Streaming• Prioritizing loops means making the code that is

used to fetch data from the pxi 6561 run at a higher priority than other parts of the code.

• Prioritizing loops can be quickly and easily implemented.

Immediate suggestions (cont.)

• Adjusting the record size appears also to be able to give large performance gains.

• This has to be implemented in the code, and we must code it so that it can recover from the case where the data packet size does not monotonically decrease.

• Matt said that streaming of data would probably not improve performance much

Extra Hardware suggestions

1. Faster Host Computer

2. Embedded controller w/ Windows

3. Embedded controller w/ RTOS

4. PCIe MXI Remote Controller

5. FPGA Card from NI

6. New NI Products

• PXIe NIHSDIO card

• HSDIO card with a FPGA to do onboard signal processing

Extra Hardware Suggestions cont.

• Faster Host Computer definitely helps.• Matt said that an embedded controller w/ Windows

won’t help much. The MXI connections make the host pc think that there are more pci slots, so basically the pxi chassis and the host pc can all be thought to be a single entity.

• An embedded controller w/ Real time operating systems may be faster because the computer is dedicated to doing one task. It doesn’t run the windows background tasks.


• PCIe MXI express connection might be faster (BNL already has this setup). We have a PCI connection at LBL.

• FPGA card from NI (This can be very fast and also scalable for the case of more than 15 modules). It can have up to 160 digital I/O lines and operate up to 40 MHz(this is frequency that are clock is operating at). One problem is that it uses TTL inputs. We would need LVDS/TTL converters. We also would have to look how commands sent to the chip would work.


• New NI Products• They have a pci express nihsdio card (it operates up

to 50 MHz), but uses ttl inputs• Matt said that he will look into other products as

well. • They are going to later have a hsdio card with a

fpga on board to do onboard signal processing.

Software Split-upTest


acquisition of data




I will continue on analog tests

Hucheng will work on digital tests

Andrew said they are beginning their setup and will mimic the setup at BNL.

Cont.

• Analysis is a major component. This can take a long time to get all the analysis that was performed in Peter Phillips code.

• We do not feel that we are ready to discuss user interface, and plots at this point.

• Hucheng will also look into version control with Perforce.

Agreements

• We should all use a consistent version of labview. The latest version 8.20 was recommended because this will optimize the application more for multithreaded applications executing on multicore processors.

• Labview 8.20 also integrates object oriented programming (this may make our code cleaner and more modular). We should look into this further.

Documents

ABCD DAQ Timothy Phung. Topics 1.Introduction 2.Status of Software 3.Overall plan of the software 4.Hardware Setup 5.Threshold Scan 6.Details of how Threshold