Start-Up of FEST Niko Neufeld, CERN-PH Tuesday, Jan. 27 th 2009 FEST Start-Up - Niko Neufeld 1

FEST Start-Up - Niko Neufeld 1

Start-Up of FEST

Niko Neufeld, CERN-PH

Tuesday, Jan. 27th 2009


Acknowledgements

• FEST is an endeavor of the whole collaboration, in particular, the online, offline and physics (HLT) teams, so better not to thank anybody in particular: nobody mentioned == nobody forgotten


What is the Full Experiment System Test

• Run simulated data through the whole Online system and the whole Offline chain because:• we had no data in 2008...

• we should be ready to process data hope to get several kHz with the first collisions

1 hour is equivalent to ~ 7 million events…

• Where do we start?

Inject events at into the High Level Trigger (the farm), as if they were coming from TEL/UKL1s

Run the HLT, to exercise control/configuration, trigger configuration, propagation of conditions and alignment constants, express stream, etc…

• Several FEST sessions: 26/01 – 30/01; 02/03 – 06/03


Replacing a 68 Million detector

…by a lot of simulated events and one powerful computer


Design Principles• FEST should not only exercise the Online and

Offline systems, the trigger, the Monitoring, Quality Checking and the prompt (Online) reconstruction

• FEST should help answer operational questions: e.g. “What is the best way to update alignment / calibration constants?”

• FEST should be operationally as close as possible to the “real thing” (like Pepsi Cola not like Virgin):• Same run-control, use the TFC to set trigger rate, self-

regulate rate• This allows re-using FEST infrastructure later for dry-

run tests of various components of the system


(Extra) Input

• New software for DAQ and control of injector

• Some adaptations (network, TFC)• 100 Million min. bias events (@ 5 TeV,

neg. polarity, VeLo closed) ~ 240 input files


The FEST menu of the dayGALA WEDDING MENU(Minimum of 75 people)

RECEPTION Host Beverage PackageIncludes 5 Hours Open Bar Service

HOT & COLD HORS D'OEVURE TRAYED AND SERVED Hearts of Palm Strudel with Remoulade SauceRagout of Marinated Grilled Vegetables on Toasted BaguetteMini Beef Filets with Horseradish Cream on CrostiniTamarind Glazed Lamb KebobsCumin Coconut Skewered ChickenGinger Tuna on Rice Cracker with Wasabi Aioli

DINNER Domestic House Wine with Dinner (1 bottle of red wine, 1 bottle of white wine per table) P.E.I. Gold & Sweet Potato Soup with Smoked Bacon and Coriander CressAntipasto Platter with Grilled Peppers, Spiced OlivesArtichokes, Air Dried Beef and Asagio CheeseAgnolotti with Quattro FormaggioRoasted Garlic and Basil Infused Cream sauceIndividual Filet of Beef with Assorted Baby VegetablesRosemary and Pancetta Potato RostiBlack Truffle Sauce(All meals served with Assorted Rolls and Butter)Baby Greens with Sundried Tomato and Oregano VinaigretteGrilled Shrimp and Scallop Brochette with Citrus Butter Sauce

Bailey's Parfait with Burnt Almond BrittleBourbon Chocolate Sauce

Coffee or Tea Midnight Coffee Station

From the Niagara Falls Hilton

Commission the injector

Pass data through the HLT on a sufficient number of servers (8 – 12)Start with a low rate, periodic trigger of ~ 100 Hz Check the event building and writing of data to local storageEstablish the maximum trigger rate at which the injector works (self-regulating!)Fix all problems, but don’t spend time to increase performance above 100 HzDo not yet send to the Offline

Commission the main HLT feature: Express line, plus other

Get the express stream working: second stream in storage, events properly routedVerify rejection on the normal stream?Verify the output data. They should contain an HLT result bank


FEST tests 2)Online Monitoring

Monitoring of the HLT: Counters, rates, rejectionMonitoring of sub-detectors with ‘normal’ eventsAre plots / histograms filled?Are they useful to identify SD problems?This requires SD experts around -> Advertise at the Commissioning meeting

Reconstruction farm (“prompt reconstruction”)

Sample events and run BrunelMonitor the performance: Rate, data flow.Monitor output, histograms?

Send to Offline

Start with the same low rate: A file every 10 minutes at 100 HzVerify file-transfers, update of LFC entries and Bookkeeping DatabaseIdentify (additional) tools needed to get good control of the systemHandle the Express line.


Fest tests 3)

Data quality checksHow to give the green light for processing? Tools?Processing of the express line, alignment and calibration checks Verify DQ procedures. How do they feed back?Run Brunel + streaming (Offline)Setup a workflowsDistribute the processing of the files, setup the streamingTest first at low rate, with runs of a few files...Test at higher rate (last but not least)Once things are OK at low rate, try to run at 1 kHz for some time


Some FEST facts

• Software versions:• Online v4r19, Moore v6r0, Brunel v34r1

• Events:• 100 Million min. Bias @ 5 TeV, neg

polarity

•


FEST operation

• Like a global commissioning week:• Online & Offline teams present

(physically or remotely)• Subdetector experts for Monitoring and

Data Quality• Control-Room packed

• Unlike a GCW: FEST interferes only mildly with sub-detector work: other partitions can run in parallel


First FEST results

Screenshot from Presenter with Calo data (4 million min bias)


Conclusion• FEST is the last in a time-honored tradition

of data-flow & processing exercises: RTTC (mostly Online), CCRC (mostly Offline), FEST (Online, Offline & HLT)

• By far the most ambitions & most useful of these

• It is very close to the “real thing”: it needs a good amount of effort and work, but it will make us ready for the “real thing” in July!

• FEST’s biggest success will be if after FEST we never have to go back to it!

Documents

Start-Up of FEST Niko Neufeld, CERN-PH Tuesday, Jan. 27 th 2009 FEST Start-Up - Niko Neufeld 1