José M. Hernández

CIEMAT

Grid Computing in the

Experiment at LHC

Jornada de usuarios de Infraestructuras Grid

19-20 January 2012, CIEMAT, Madrid

José Hernández

The CMS Experiment at the LHC

20 January 2012Grid Computing in CMS 2

The Large Hadron Collider

p-p collisions, 7 TeV, 40 MHz

The Compact Muon Solenoid

Precision measurements

Search for new phenomena

José Hernández

LHC: a challenge for computing

The Large Hadron Collider at CERN is the largest scientific instrument on the planet

Unprecedented data handling scale 40 MHz event rate (~1 GHz collision rate) → 100TB/s → online filtering

to ~300 Hz (~300 MB/s) → ~3 PB/year (107 secs data taking/year)

Need large computing power to process data Complex events

Many interesting signals << Hz

Thousands of scientists around the world access and analyze the data

Need computing infrastructure able to store, move around the globe, process, simulate and analyze data at the Petabyte scale [O(10) PB/year]

3

José Hernández

The LHC Computing Grid

4

LCG: 300+ centers, 50+ countries, ~100k CPUs, ~ 100PB disk/tape, 10k users

The LHC Computing Grid provides the distributed computing infrastructure

Computing resources (CPU, storage, networking)

Computing services (data and job management, monitoring, etc)

Integrated to provide a single LHC computing service

Using Grid technologies

Transparent and reliable access to heterogeneous computing resources geographically distributed via internet

High capacity wide area networking

José Hernández

The CMS Computing Model

Distributed computing model for data storage, processing and analysis

Grid technologies (Worldwide LHC Computing Grid, WLCG)

Tiered architecture of computing resources

~20 Petabytes of data (real and simulated) every year

About 200k jobs (data processing, simulation production and analysis) per day

José Hernández

WLCG network infrastructure

20 January 2012Grid Computing in CMS 6

T0-T1 and T1-T1 interconnected via LHCOPN (10 Gpbs links)

T1-T2 and T2-T2 using general research networks Dedicated network infrastructure

(LHCONE) being deployed

José Hernández

Grid services in WLCG

Middleware providers: gLite/EMI, OSG, ARC

Global services: data transfers and job management, authentication / authorization, information system

Compute (gateway, local batch system, WNs) and storage (gridftp servers, disk servers, mass storage system) elements at the sites

Experiment specific services

20 January 2012Grid Computing in CMS 7

José Hernández

CMS Data and Workload Management

Experiment-specific DMWM services on top of basic Grid services Pilot-based WMS

Data bookkeeping, location and transfer systems

Data pre-located

Jobs go to data

Experiment software pre-installed at sites20 January 2012Grid Computing in CMS 8

Production System

(WMAgent)

Analysis System (CRAB)

Data Bookkeeping

& location system (DBS)

Data Transfer System

(PhEDEx)

gLite WMS

File Transfer System

CE

CE

SE

SE

Local batch

system

Mass storage system

CMS Services Grid Services Sites

Operators

Users

Pilot-based WMS

José Hernández

CMS Grid Operations - Jobs Large scale data processing & analysis

~50k used slots, 300k jobs/day

Plots correspond Aug 2011 – Jan 2012

20 January 2012Grid Computing in CMS 9

José Hernández

Spanish contribution to CMS Computing Resources

10

Spain contributes with ~ 5% of the CMS computing resources

PIC Tier-1

~1/2 average Tier-1

3000 cores, 4 PB disk, 6 PB tape

IFCA Tier-2

~ 2/3 average Tier-2 (~3% T2 resources)

1000 CPUs, 600 TB disk

CIEMAT Tier-2

~ 2/3 average Tier-2 (~3% T2 resources)

1000 cores, 600 TB disk

José Hernández

Contribution from Spanish sites

20 January 2012Grid Computing in CMS 11

~5 % of total CPU delivered for CMS

CPU delivered Feb 2011 – Jan 2012

José Hernández

CMS Grid Operations - Data

Large scale data replication 1-2 GB/s throughput CMS-wide

~1 PB/week data transfers

Full mesh 50+ sites T0 T1 T1 T2 T2

20 January 2012Grid Computing in CMS 12

1 GB/s

Production transfers

debug transfers

1 GB/s

José Hernández

Site monitoring/readiness

20 January 2012Grid Computing in CMS 13

José Hernández

Lessons learnt

20 January 2012Grid Computing in CMS 14

Porting the production and analysis applications to the Grid was easy Package job wrapper and user libraries into input sandbox

Experiment software pre-installed at the sites

Job wrapper sets up environment, runs the job, stages out output

When running at large scale in WLCG, additional services are needed Job and data management services on top of Grid services

Data bookkeeping and location

Monitoring

José Hernández

Lessons learnt

20 January 2012Grid Computing in CMS 15

Monitoring is essential Multi-layer complex system (experiment, Grid, site layers)

Monitor workflows, services, sites

Experiment services should be robust Deal with (inherent) Grid unreliability

Be prepared for retries, cool-off

Pilot-based WMS gLite BDII and WMS not reliable enough

Smaller overhead, verify node environment, global priorities, etc Isolating users from the Grid; Grid operations team

Lots of manpower needed to operate the system Central operations team (~20 FTE)

Contacts at sites (50+)

José Hernández

Future developments

20 January 2012Grid Computing in CMS 16

Dynamic data placement/deletions Most of the pre-located data not really accessed much

Investigating automatic replication of hot data, deletion of cold data

Replicate data when accessed by jobs and cache locally

Remote data access Jobs go to free slots and access data remotely

CMS has improved a lot read performance over WAN

At the moment only used as fail-over and overflow

Service to asynchronously copy user data Remote stage out from WN is a bad idea

Multi-core processing More efficient use of multi-core nodes, savings in RAM, many less

jobs to handle

José Hernández

Future developments

20 January 2012Grid Computing in CMS 17

Virtualization of WNs/Cloud computing Decouple node OS and application environment using VMs or chroot

Allow use of opportunistic resources

CERN VMFS for experiment software

José Hernández

Summary

20 January 2012Grid Computing in CMS 18

CMS has been very successful in using the LHC Computing Grid at large scale

Lot of work to make the system efficient, reliable and scalable

Some developments in the pipeline to make CMS distributed computing more dynamic and transparent