Sept 2006Pixel Calibration: what have we learned from the test-stand project? 1 Sept 2006Pixel Calibration: what have we learned from the test-stand project? 2 Designing a program to calibrate the Pixel Detector is an effort that features several commonalities with the pre-assembly tests (the FT test-stand project). I will first discuss how the test-stand code (the Renaissance) has been designed and implemented (few details on hardware, little more on software). Some more time will be spent discussing technicalities such as DAC setting optimization, fitting strategies, monitoring, handling of large number of histograms, off-line versus fully-interactive approach (pros and cons) Then I will show how we used it to calibrate the detector for the test-beam (a relatively small effort but gave us some insights for the final problem). Interaction with the DB will also be addressed. I will then try to summarize what we think we have learned from the implementation and use of the test-stand code that might be relevant to the full-scale detector-calibration problem. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 3 About 700 plaquettes to test Each ROC must be tested before mounting on a panel and retested after (lots of tests, possibly several times) Users are not necessarily experts of the ROC (need a sophisticated GUI to allow both generic and expert user to perform standard as well as specific tests) Time constraints are important: production flow requires a steady state of 6 plaquettes to be tested per day Data persistency: both histograms and derived parameters stored in a DB. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 4 To keep up with schedule, the system had to be able to fully test (at least twice) six plaquettes per day (shipped in from Purdue Univ., where a set of independent tests had already been performed). The test-stand had to be fast (more than one station must be operational during shifts with several operators in parallel). For each ROC in a plaquette there is a list of 21 test to perform, and a plaquette is tested twice (before assembly on a panel and after: the system must be able to deal with bare plaquettes as well as plaquettes mounted on a panel). Many important acceptance criteria are based on parameters which are the results of fits (algorithm speed, robustness and goodness of fit are definitely and issue). Given the amount of tests to perform, a potentially large number of unexpected pathologies can emerge; operators must therefore be able to spot these at an early stage, eventually take action and restart (full interactivity by means of a GUI is absolutely essential). Persistency of results is crucial: the data are fed into the DB and ROOT files are archived with all the histograms (~ histograms per plaquette on average). Sept 2006Pixel Calibration: what have we learned from the test-stand project? 5 Hardware For I/O we adopted the PCI protocol (lots of expertise inherited from BTeV) Off-the-shelf standard, portable, easy to use and cheap (allows multiple stations at low cost) PMC (mezzanine) PTA PSI readout chips TBM ADC PCI Microprogramming of the FPGA on board of the PMC allows implementation of fast PCROC communication, an important contribution to the speed requirement. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 6 Four test-stations currently SiDet Humidity/temperature control centralized through a single PC via GPBI interface. Info is broadcast via TCP/IP socket to all test-stations PC Sept 2006Pixel Calibration: what have we learned from the test-stand project? 7 Software To satisfy the requirements set forward for the test-stand, we developed an ad-hoc program, called The Renaissance. Specialized class to deal with FPGA commands: key for high-speed tests The test-stand is fully interactive, multi-threaded, and asynchronous (a lengthy test can be aborted and restarted at any time). The GUI integrates a control panel (to issue commands) with a histogram browser (to allow users rapid checks of the test progress) The test-stand performs 21 different tests per ROC handling a total of about histograms (2x5 plaquette). Crucial is an appropriate memory-handling strategy: a suitably partitioned directory tree fully contained in memory does the job (requires PC with substantial memory, at least 2Gb: tradeoff between I/O and CPU). Particular care devoted to fit strategy (also critical for speed and convergence) Sept 2006Pixel Calibration: what have we learned from the test-stand project? 8 This is the list of all test automatically performed by the test-stand Snapshot of the CalDelay test result: determines the optimal value of CalDelay DAC register for charge injection for any given Thr value. This is determined automatically and used by any subsequent test whose response depends upon charge injection. 1.I-V Curve 2.Vana 3.Phase Adjust 4.Optimize Ultra-Blacks 5.Last-DAC 6.CalDelay (Threshold/Timing) 7.Crazy cells 8.Optimize Analog Levels 9.Decoding 10.Optimize VHoldDelay 11.Pixel Readout 12.Mask Bit 13.Gain Curves (linearity) 14.Bump Bond Test A 15.Bump Bond Test D 16.Light Test 17.Threshold vs VCal 18.S-Curves (Threshold dispersion) 19.V-Trim 20.Trim Bit Linearity 21.Trimming Sept 2006Pixel Calibration: what have we learned from the test-stand project? 9 Calibrating a pixel (roughly) corresponds to linearize the gain curve. A gain curve is obtained by injecting charge and reading back the pixel ADC: an interpolation of this curve (with a suitably chosen parametrization) allows to derive the correction function needed to interpret data. This is less trivial than it may sound: the shape of the response curve significantly depends on the values of a few DAC settings that must be optimized for each ROC individually (Vana, Vsf, VOffsetOP, VHldDel, VIbias_PH and VOffsetRO). These settings control the dynamic range of the response curve, the saturation point and the linearity in the region of small signals (significant correlations) Prior to fit the gain curves, it is therefore necessary to execute a procedure that automatically tunes these DAC settings to optimize their value. This algorithm must necessarily be fast, accurate, reliable and easy to monitor. Only when these settings are finally optimized a fit would produce a meaningful calibration curve for each pixel cell. Injection (Low + High range) mV ADC (counts) Non-optimized DAC settings Sept 2006Pixel Calibration: what have we learned from the test-stand project? 10 A viable strategy to automatically match the ADC dynamic range and linearize the first part of the gain curve involves several iterations of DAC registers settings, data collection and fits. Set relevant DAC settings (Vsf, VHldDel,) Collect gain curves Fit gain curves with straight line in restricted range to check for deviation from linearity Adjust Vsf until the 2 distribution of the linear part of the gain curve peaks around 1 ( 2 normalized to NDOF) Fit saturation part with straight line and fill histogram with Pedestal and Saturation Find peaks of Pedestal an Saturation plots Adjust relevant DAC settings to bring Pedestal and Saturation peaks to match and maximize the ADC dynamic range. This could be an optimal and complete strategy, but the amount of time required for the whole detector might be excessive: a once-for-all pre-tuning (at C) could ease the problem. Each time a calibration is needed these DAC settings could then be reused without retuning and only gain-curve data-taking plus fit would be necessary. Possible approach and steps needed Sept 2006Pixel Calibration: what have we learned from the test-stand project? 11 Vana: 150 Vsf : 128 An example of the effect of fine-tuning: changing Vsf has an effect on VHldDel. Adjustment algorithms for ROC analog DAC settings (Sarah Dambach) Vana: 150 Vsf : 145 Interesting studies of this and other correlations among DAC settings have been performed by colleagues at ETH in Zurich T: C Tuning Vsf has a marked effect on the linearity of the first part of the gain curve: since most of the DAC settings have effects which are not independent of each other, setting up a completely automatic procedure might prove challenging (particularly so if one has to take into account peculiar pathologies that can unexpectedly distort the foreseen distributions) Sept 2006Pixel Calibration: what have we learned from the test-stand project? 12 Implementation of the DAC setting optimization for gain curves in the Renaissance test-stand: a snapshot of the distributions of intercepts for the gain curve for a complete plaquette (1x5) Intercepts Sept 2006Pixel Calibration: what have we learned from the test-stand project? 13 Same thing for the slopes... Sept 2006Pixel Calibration: what have we learned from the test-stand project? 14 The current implementation of the test-stand has already implemented a simple version of an automatic DAC-settings optimization algorithm (does not require external intervention from an operator, at least for some DAC registers). Issues: could an automatic algorithm be fast enough for a full-scale calibration of the whole detector? how do we define a gain-curve as satisfactory (linearity, all pedestals above zero, optimal match with ADC dynamic range)? Excessively refined requirements make the automatic procedure more prone to convergence failures (and slower to execute). Automatic early detection and notification of abnormal pathologies is also a critical issue An interesting possibility is to use the optimized DAC settings (on a per-ROC basis) which are currently being stored in the database during the Fermilab SiDet assembly procedure by the Renaissance test-stand (these settings would then be considered the reference values). When a calibration is needed, the gain curve will always be derived from a run where each ROC has been initialized with these settings. Another possibility is to perform a once-for-all run to optimize these settings, store them in the DB and use them each time a calibration is actually needed; the time required will only include charge-injection, data-read, fit and persistency of the results. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 15 Once the DAC settings have been suitably optimized, a fit can finally interpolate the missing measurement points. In order to achieve a very fast convergence, it is extremely important to have starting points for the free fit parameters as close as possible to their true final value (fewer iterations are then needed to reach a good convergence, see plots on the right). To this extent a set of optimized DAC settings is once more important since all curves could then be made very similar and a single common set of initial values for the fit parameters could be used. Initial values of fit parameters Injection ADC [counts] Low range High range Sept 2006Pixel Calibration: what have we learned from the test-stand project? 16 The final step towards calibrating a pixel involves inverting the fit function in order to create a look-up table to be used to quickly convert an ADC pulse height into absolute charge. This is really fast and easy: only concern is whether the resulting tables are more efficiently stored as DB entries or highly specialized binary files (pre-caching from DB tables into binary files prior to use for analysis is another viable option). Another issue is whether to store look-up tables (64 bins per pixel cell) or just the fit parameters (5 parameters per pixel cell): a trade-off between memory and CPU requirements. Injection ADC [counts] Assumed: 1 unit V cal = 60 e - 1 M.I.P. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 17 Given the large number of channels involved (~66 M), an efficient organization of the data flow is of paramount importance to achieve the desired result in a finite time. DAC settings must be optimized on a per-ROC basis DAC settings must be optimized on a per-ROC basis. 1. values are set (I/O towards ROC) 2. histograms filled (I/O from ROC and access to large amounts of histograms in memory) 3. distributions analyzed (fits involved: on-line processing and possible feed-back to step 1) 4. results saved (histograms made persistent as archived root files: pointers to those files are saved in the DB for distribution retrieval and calibration) this is iterated until optimization is achieved (dynamic-range and linearity brought within specs given a set of estimators to quantify distance from desired result) Gain curves must be interpolated and look-up tables created Gain curves must be interpolated and look-up tables created (the true calibration) - fits are performed: critical is I/O to acquire data and time required per fit (convergence must be closely monitored) - inverse curves are computed: off-line analysis can then either use pre-computed look-up tables (fast but requires huge amounts of memory), or use the function coefficient (less memory but somewhat more CPU cycles needed) In any case development of an efficient and highly specialized class to handle I/O is certainly an important item of concern Sept 2006Pixel Calibration: what have we learned from the test-stand project? 18 We developed such a class for the purpose of the test-stand Provides methods to transfer large number of histograms from disk to memory (and vice-versa): up to 100k histograms takes about 20 seconds. A root file is a persistent image of the test-stand code memory at any given time: allows histograms to be saved at any time, retrieved and reused (refilled after reset) Each time a file is saved, new histograms can be added (flexibility, no predefined fixed structure) Allows to complete data-collection at different times (no need to always restart from scratch) This is important for long and complex procedures which may need to be interrupted. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 19 Other issues of concern are: Since a calibration already requires substantial amounts of resources and time, it is probably worthwhile to use some additional time to perform a thorough check of the detector (to this extent it would be good to partition the program in separate, independent components to maximize flexibility): - analog levels calibration - address decoding - trimming What is the best channels granularity for an efficient calibration run (memory concern)? - Per ROC: 52x80=4160 histograms to collect and fit - Per plaquette: from a minimum of 2x4160=8320 to a maximum of 10x4160= Per panel: from a minimum of 21x4160=87360 to a maximum of 24x4160= Per blade: about histograms (requires a lot of RAM) Direct access to a DB could be problematic (I/O, network, ): pre-caching of needed quantities from the DB into specialized binary files could speed up the procedure. Sept 2006Pixel Calibration: what have we learned from the test-stand project? 20 Telescope of 6 BTeV pixel detectors (50 m 400 m) 2 Y-measurement planes => Y-resolution ~ 6.24 m 4 X-measurement planes => X-resolution ~ 4.75 m CMS pixel detector in the center (100 m 150 m) Triggers to CMS detector are provided by two upstream scintillators CMS detector irradiated to a dose of 3*10 13 p/cm 2 (200 MeV) (~2Mrad). Sept 2006Pixel Calibration: what have we learned from the test-stand project? 21 b)b) Gain curve of a pixel cell Uncorrected data Single hit (no sharing) V cal ADC counts a)a) Peak ~ 23 ke - ~ 10 ke - # e - M.I.P. peak (isolated hits) # e - Peak ~ 22 ke - ~ 16 ke - M.I.P. peak (2 adjacent hits) Plot c shows the calibration-corrected pulse-height of all pixel cells in the detector: only isolated cells which are pointed at by a reconstructed track from the telescope (cluster size = 1) are considered. b)b) c)c) d)d) Plot d shows the corrected pulse-height when the telescope track points to a cluster of hits of size 2 (adjacent) Sept 2006Pixel Calibration: what have we learned from the test-stand project? 22 Beam spot (only scintillator triggers) Beam spot (all detectors required in an event) Plaquette 8*10 14 p/cm 2 (200 MeV) (~45Mrad = 4.5 years in LHC). Amazing detectors: even a highly irradiated plaquette provides an image of the beam spot quite similar to a good-ol nuclear emulsion! Even more impressive: a reconstructed telescope track crosses the CMS detector and a very energetic ray is emitted! Sept 2006Pixel Calibration: what have we learned from the test-stand project? 23 The test-stand project has been a valuable sand-box to play and investigate many issues relevant to the detector calibration problem. Gave us some ideas about the design of an algorithm to tune and optimize DAC settings - would it be feasible to have this procedure fully automatic for the whole detector? - at what level should experts validate the calibration constants? What should they look at? - redo each time a calibration is needed or once-a-year is sufficient? - just do a calibration when needed or take the chance for a full check of the detector? Already done calibration for test-beam purposes (not automatic, small-scale effort but many lessons learned) Developed many procedures (classes) to handle large amounts of histograms and perform fits efficiently Data transfer strategy to/from DB should be evaluated (on-demand direct access vs pre-caching into specialized binary files)