IFC 10/2011 Ray Culbertson1

CDF Offline Status and Plans

Ray Culbertson for the Offline Group

Ray Culbertson, co-leaderAidan Robson, co-leader

Elena Gerchtein, Assoc Head for ProductionStephan Lammel, Assoc. Head for Services

ProductionElena Gerchtein

CalibrationWillis Sukamoto (L)Dan Goldincalibrators

ProductionElena Gerchtein (L)operators

Ntuplingoperators

Monte CarloCostas Vellidis (L)Oksana TadevosyanLiqun LiMC Reps

ServicesStephan Lammel (Mike Kirby)

Grid/DHJoe Boyd (L,CAF)Eric Wicklund (L,DH)Mike Wang(Robert Illingworth)(Marc Mengel)(REX)(Site Coordinators)

DatabasesEric Wicklund (L)(Barry Blumenfeld)(Dennis Box)(DBAs and admins)

Code ManagementJim Bellinger (L)Lynn Garren(Donatella Torretta)

InfrastructureStephan Lammel (L)(CD)

EndingStarting

9/28/2011

Remote SitesAidan Robson (L)Ray Culbertson (L)Site Coordinators

EndingStarting

9/28/2011

CalibratorsSVX Align Dominik HornSVX Timo Aaltonen COT Kevin Burkett dE/dx Keith MateraBeamlines Roberto Carosi TOF Jesus Manuel Vizan GarciaPES Halley BrownPEM Willis Sakumoto,CHA Fabio HappacherCEM Larry NodulmanCal Timing Adam Aurisano,CP2/CCR Azeddine Kasmi PASS Tom Riddick

OperatorsOlga TerlygaOksana TadevosyanZhenbin WuJon WilsonAristotle Calamba

MC RepsHDG Shalhout ShalhoutTOP Dave MietlickiEWK Maria D'ErricoBOT Hideki Miyaki EXO John StrologasQCD Erik Jens Brucken

Site CoordinatorsFermigrid Steve TimmLCG/CNAF Donatella Lucchesi Silvia Amerio Matteo BaucePACAF Masakazu Kurata Yuji Takeuchi  Suen Hou Tsan  HsiehKISTI Seo-Young Noh Beob Kyun Kim Heejun Yoon Christophe BonnaudMIT Max Goncharov

SAM ShiftersIvan VilaGiovanni Piacentino Stefano Giagu Barry BlumenfeldPeter BusseyThomas Kuhr Alberto Ruiz Aidan Robson 

Ntuple maintainersTopntuple Hyunsu LeeBStntuple Michael Morello Angelo Di CantoStntuple Ray Culbertson

IFC 10/2011 Ray Culbertson4

Major Systems Overview

enstore(tape)

productionservers

dCache diskpool Monte Carloservers

ILPProjectDesktops

cdfgrid(5500 slots)

namgridKISTI/MIT/GP/CMS

Off-site farmsEURGRID/PACCAF/

Raw Data

IFC 10/2011 Ray Culbertson5

► Enstore tape system: 16 LTO3 drive, 26 LTO4 drives ○ Delivering typically 20TB per day, 50TB peak► dCache, main disk cache, ~400TB, with tape backend ○ Delivering typically 50TB per day, 150TB peak

► overall very smooth operations last year!

Data Handling

50TB/day >

< one year >

IFC 10/2011 Ray Culbertson6

► dCache major upgrade 300TB → ~800TB in May ○ smooth transition on this major project – great success!

○ appears to have relieved large backlogs - a significant difference!

Data Handling

capacityupgrade

IFC 10/2011 Ray Culbertson7

► Planning needs ○ no new tape drives added last year ○ loads expected to be manageable (thanks to dCache upgrade) ○ new tape storage robot arrived in July, plenty of space ○ purged ~5% in unused datasets

► Tape generation migration ○ from LTO-4 (0.8TB) to T10K (5TB) ○ testing recently signed off ○ 6 T10K drives in FY11, 4 more in FY12 ○ start migrating raw data this year, the bulk of our 9PB in FY12, 13

Data Handling

IFC 10/2011 Ray Culbertson8

CdfGrid

► 5500 slots ► Smooth operations! ► last maintenance replacements purchased in FY11, start decreasing in FY13

► Loads: - heavy but manageable Jan-Jul

40K >

5K >

< 1 year >

IFC 10/2011 Ray Culbertson9

NamGrid

1000 >

< 1 year >

► A portal to offsite farms running OSG, ~20% of CdfGrid ○ achieved regular access to GP and CMS farms! ○ last fall achieved solid integration of the KISTI site! ○ accessing SAM cache and CdfCode ○ regularly runs MC

► MIT site continues to bevery reliable!

►moderate loads

IFC 10/2011 Ray Culbertson10

► Italian colleages reorganized the CNAF and LCGCAF farms ○

Eurogrid

CNAFhead-node

LCGTier1 Farm at CNAF

VO Frontend

GlideIn Factory

glideinWMS

Site CountryCNAF-T1 Italy

INFN-Padova ItalyINFN-Bari Italy

INFN-Legnaro ItalyINFN-Roma1 ItalyINFN-Roma2 ItalyINFN-Catania Italy

INFN-Pisa ItalyFZK-LCG2 GermanyIN2P3-CC France

IFAE SpainPIC Spain

IFC 10/2011 Ray Culbertson11

► the glidein layer prevents LCGCAF from swallowing jobs this makes a huge difference!!► data transfer are also faster due to general network improvement► A huge success! Users are voting for it!► European CDF resources which were languishing are now used!

Eurogrid

< 3 months >

1.5K >

IFC 10/2011 Ray Culbertson12

Diskpool

► 325TB of “persistent” dCache assigned to physics groups► mostly smooth operations► Alexei Varganov, our diskpool expert, has left for a new job► Physics groups have decided the diskpool is less critical now, and we can backup data and live with the greater uncertainty►backup process is very tedious – thanks to production group for their persistence – it will be done soon!

GB statustop 63 85% backed up, progressinghdg 33 done, almost signed offewk 31 down to 3 usersbnt 26 almost doneexo 17 doneqcd 16 investigating

A lot of work:~2000 datasets!

IFC 10/2011 Ray Culbertson13

Code Management

► Last winter developed large file (2 to 8GB) support ○ contributes to improved tape access speed done and deployed!

► New major project: develop legacy code releases ○ incorporate accumulated patches ○ modernize all support packages ○ finally migrate to root version 5! ○ improve infrastructure

first test release is out now, hope to be done this fall!

IFC 10/2011 Ray Culbertson14

Production Operations

► smooth operations over the year

►New data Production ○ 3 billion events, 450 TB

► Ntupling, 3 flavors, data and MC ○ 6 billion events data/MC, 300TB

► Monte Carlo operations ○ Generated 890M events, 150TB last year - continued strong demand!

► Reprocessing…

IFC 10/2011 Ray Culbertson15

Reprocessing

► About half of our data has non-optimal Si clustering, reduces tagging 5-10% (Periods 18-28 out of 38)► Rerun production and ntupling to recover efficiency

we met the Higgs group request for Mar 11 delivery of their data!!

► now only a tiny bit left to do…

IFC 10/2011 Ray Culbertson16

Two More Projects

Two more projects targeted for B physics…

► BStntuple ○ re-ntuple all the B physics data streams ○ add covariance matrix for tracks - allows arbitrary vertexing choices at the ntuple level ○ enables many new analyses, is flexible for the future, and will replace several custom ntuples ► Generic B MC ○ generate, simulate, produce, ntuple bb Monte Carlo ○ has been a wanted for a long, now becoming more urgent ○ details and targeted dataset size are under design

IFC 10/2011 Ray Culbertson17

The Final Push

planning to get the last data out to physics fast !!

P37 prd

P38 calibP38 prd

P38 ntp

P37 ntp

s-scan s-scan

diskpool upload

BStn reprocessing

reprocessing

Sep Oct Nov Dec

Special low energy data!

IFC 10/2011 Ray Culbertson18

In the Long Term

► FY12: cdfgrid, dCache, services continue as-is► FY13: cdfgrid, dCache reduced by size of production needs► Continue with full functionality, reduced capacity for 5y - farm and interactive CPU - access to all data - production and ntupling capability - full MC simulation, with all generators► Continuing past 5y ○ concepts under discussion ○ how will LHC discoveries overlap Tevatron data? ○ will require funding and attracting experts

IFC 10/2011 Ray Culbertson19

The Bottom Line

► CDF Offline had another very successful year! ○ smooth operations, manageable loads ○ clearing off diskpool ○ preparing future releases ○ delivered reproduced data

► As we dive into the new era ○ finishing strong … and fast! ○ long term still requires work, resources Thanks for your IFC contribution, it is crucial!