1 PhD meeting 1 Frameworks for GRID and Interactive Parallel Analysis Marco Meoni - ALICE Offline, CERN EPFL – Lausanne – 28/02/2008

11

PhD meeting 1PhD meeting 1

Frameworks for GRID and Frameworks for GRID and

Interactive Parallel AnalysisInteractive Parallel Analysis

Marco Meoni - ALICE Offline, CERN

EPFL – Lausanne – 28/02/2008

22

OutlineOutline

Introduction to CERNIntroduction to CERN• ALICE experiment at LHCALICE experiment at LHC• Grid overview and projectsGrid overview and projects

Frameworks in useFrameworks in use• AliEn, the ALICE's GridAliEn, the ALICE's Grid• ROOT & AliROOT, the basic environmentsROOT & AliROOT, the basic environments

Parallel computingParallel computing• Interactive parallel analysis with PROOFInteractive parallel analysis with PROOF

PhD meeting 1PhD meeting 1• Current activities at CERNCurrent activities at CERN• Open issuesOpen issues

33

Part I:Part I:

Introduction to CERNIntroduction to CERN

44

ALICEALICE

An experiment dedicated to the study of nucleus-An experiment dedicated to the study of nucleus-nucleus collisions at LHC…nucleus collisions at LHC…• Heat and compress matter...Heat and compress matter...

...to a temperature about 100,000 times the temperature in ...to a temperature about 100,000 times the temperature in the centre of the Sun andthe centre of the Sun and

...to densities such that all matter contained in the Kheops ...to densities such that all matter contained in the Kheops pyramid would fit in a pinheadpyramid would fit in a pinhead

• Why would you do that?Why would you do that? To recreate conditions of density and pressure bringing us To recreate conditions of density and pressure bringing us

back to only a few µs after the Big Bangback to only a few µs after the Big Bang In a system we can study in the Lab!In a system we can study in the Lab!

55

The “Standard Model”The “Standard Model”

• The theory which describes the bricks of the universe and the forces through which they interact:

12 elementaryconstituents

4 interactions

graviton photon W, Z gluon

66

Big Bang …Big Bang …Until 10-6 secondsafter the birth of the Universe, matter is colored: quarks and gluons move freely.

As soon as the universe has cooled down to about 1012 K, matter becomes “colourless”: quarks and gluons are locked into hadrons

77

ALICE@LHCALICE@LHC

LHC

ALICE

88

Accelerating nucleiAccelerating nuclei Nuclei (all electrons are Nuclei (all electrons are

stripped off) are accelerated stripped off) are accelerated by an electric fieldby an electric field

The trajectories are bent by The trajectories are bent by dipolar magnetic fieldsdipolar magnetic fields

The flux is focalised by The flux is focalised by quadrupole magnetic fieldsquadrupole magnetic fields

99

LHC: world championLHC: world champion• 27 km circumference• 40 m underground• Cryogeny at 1.9 K

Accelerates p @ 7×10Accelerates p @ 7×101212 eV & ions @ 2,76×10 eV & ions @ 2,76×101212 eV eV A collision generates up to 0,2×10A collision generates up to 0,2×10-3-3 Joules, T=1,000×10 Joules, T=1,000×1099 K K ~10~1088 ions cross 10 ions cross 1088 ions 10 ions 1066 times every second times every second 10k collisions every second, out of which 1% produce 10k collisions every second, out of which 1% produce

”extraordinary” events ”extraordinary” events

1010

level 0 - special hardware8 kHz (160 GB/sec)

level 1 - embedded processors

level 2 - PCs

(4 GB/sec)

(2.5 GB/sec)

(1.25 GB/sec)

data recording &

offline analysis

Total weight 10,000tDiameter 16.00mLength 25mMagnetic field 0.4Tesla

Weight 2,000tLength 17.3m

Weight 53,000tLength 270.4m

1111

Concorde(15 Km)

Balloon(30 Km)

CD stack with oneyear of LHC data!(~ 20 Km)

Mt. Blanc(4.8 Km)

LHC dataLHC data From 1 to 12MB per collision From 1 to 12MB per collision from 0.1 to from 0.1 to

1.2 GB/s1.2 GB/s 10101010 collisions registered every year collisions registered every year ~10 Petabytes (10~10 Petabytes (101515B) per yearB) per year LHC data correspond to 20 millions CD per LHC data correspond to 20 millions CD per

year!year! Computing power equivalent to 100.000 of Computing power equivalent to 100.000 of

today’s PC (today’s PC (>>5000 5000 PC with double CPUPC with double CPU)) Space equivalent to 400.000 large PC disks Space equivalent to 400.000 large PC disks

(>(>15 PB on tapes and disks)15 PB on tapes and disks) Far from being enough!Far from being enough!

1212

Computing at LHCComputing at LHC SimulationSimulation

• Compute what detector should seeCompute what detector should see

ReconstructionReconstruction• Transform signals from detector into physics property Transform signals from detector into physics property

(energy, charge, particle id)(energy, charge, particle id)

AnalysisAnalysis• Apply complex algorithm Apply complex algorithm

to extract Physicsto extract Physics

1313

The Grid projectThe Grid projectThe Grid: connected computing centres and “middleware” programs as “glue” among resources

Researches do their actvity independently from geographic location, interacting with colleagues and exchanging data

Scientific tools and experiments provide huge amount of data

1414

Tim Berners Lee

A famous precedentA famous precedent In 1989-90 Tim Berners-Lee In 1989-90 Tim Berners-Lee

invents a system to exchange invents a system to exchange images and information images and information through Internet that calls through Internet that calls World Wide WebWorld Wide Web

The rest is well knownThe rest is well known Economist has justified the Economist has justified the

award for 2004 innovation to award for 2004 innovation to T.Berners-Lee saying “… it T.Berners-Lee saying “… it has changed forever the way has changed forever the way to exchange information”to exchange information”

1515

Middleware Middleware The Grid uses advanced software, called middleware, to The Grid uses advanced software, called middleware, to

connect resources and applications togetherconnect resources and applications together MiddlewareMiddleware

• Finds the best place to execute applicationsFinds the best place to execute applications• Optimize the usage of resourcesOptimize the usage of resources• Organizes efficient access to dataOrganizes efficient access to data• Authenticates on the sitesAuthenticates on the sites• Executes applications and monitors Executes applications and monitors

their executiontheir execution• Handle error conditionsHandle error conditions• Transfer results to usersTransfer results to users

1616

Science Science computing intensivecomputing intensive

Science: digital and huge amount of dataScience: digital and huge amount of data Nanotechnology – design of new materials at Nanotechnology – design of new materials at

molecolar scalemolecolar scale Simulation of complex systemsSimulation of complex systems

• Wheather forecastWheather forecast• Rivers floodingRivers flooding• EarthquakesEarthquakes

Mapping of genomeMapping of genome High precision sensorsHigh precision sensors

1717

GRIDsGRIDsTeraGRID: Simulation of cellular structuresTeraGRID: Simulation of cellular structures The simulation of The simulation of 10M 10M atoms will allow to model the atoms will allow to model the

function, the movement of the structure and the function, the movement of the structure and the interation at cellular level to design medicines and interation at cellular level to design medicines and understand diseasesunderstand diseases

Baker, N., Sept, D., Joseph, S., Holst, M., and McCammon

Digital Radiology Digital Radiology Hospital digital dataHospital digital data

• Digital storage and manipolation has a clinical value Digital storage and manipolation has a clinical value • Mammography X-rays, MRI, CAT scans, EndoscopiesMammography X-rays, MRI, CAT scans, Endoscopies• Huge money savingHuge money saving• 7 TBs per hospital per year7 TBs per hospital per year

2k hospitals x 7 TB/y = 14 PBs/y2k hospitals x 7 TB/y = 14 PBs/y

(Hollebeek, U. Pennsylvania)

1818

The Grid at CERNThe Grid at CERN LCGLCG

• Grid implementation to allow experiments at LHC to Grid implementation to allow experiments at LHC to gather and analyse datagather and analyse data

• Many thousands computers at tens Institutes connected Many thousands computers at tens Institutes connected as a global computing resourceas a global computing resource

• Based on advanced software projects developed in Based on advanced software projects developed in Europe and USAEurope and USA

CERN Openlab for Data Grid applicationsCERN Openlab for Data Grid applications• Collaboration between CERN and industrial partners to Collaboration between CERN and industrial partners to

study and develop data-intensive solutions to be used by study and develop data-intensive solutions to be used by the worldwide community of scientists working at LHCthe worldwide community of scientists working at LHC

• Partners:Partners: CERNCERN ENTERASYSENTERASYS HPHP IBMIBM INTELINTEL

EGEEEGEE

1919

EGEE EGEE AimAim

• Successor of EDG (2001-2003)Successor of EDG (2001-2003)• Choerent, robust, safe infrastructure of Grid ServicesChoerent, robust, safe infrastructure of Grid Services• Development and maintenance of the middlewareDevelopment and maintenance of the middleware• EGEE I: funded by EU with ~32M€ in 2004-2006EGEE I: funded by EU with ~32M€ in 2004-2006• EGEE II: 2007-2008EGEE II: 2007-2008

StructureStructure• 240 institutions, 45 countries240 institutions, 45 countries• 41 000 CPUs 41 000 CPUs • ~30 000 jobs/day~30 000 jobs/day• 5PB disks + MSS5PB disks + MSS• 20 20 VOVOs (applications) from 11 domainss (applications) from 11 domains• Thousands scientistsThousands scientists

gLitegLite• framework for building GRID applicationsframework for building GRID applications

http://www.eu-egee.org/

2020

Part II:Part II:

Frameworks in use: Frameworks in use: AliEn, ROOT, AliRooTAliEn, ROOT, AliRooT

2121

http://glite.web.cern.ch/glite/alien/http://glite.web.cern.ch/glite/alien/

“The architecure of AliEn provided a blueprint and a starting point for developing the gLite

architecture”

2222

ALICE's Grid - AliEnALICE's Grid - AliEn ALICE Environment is a Grid framework developed ALICE Environment is a Grid framework developed

by ALICE, by ALICE, used in production for >6 yearsused in production for >6 years Based on Web Services and standard protocolsBased on Web Services and standard protocols Built around open source code, less than 5% is Built around open source code, less than 5% is

native code (mainly PERL)native code (mainly PERL) So far, So far, > 1,000,000> 1,000,000 ALICE jobs have been run under ALICE jobs have been run under

AliEn control worldwideAliEn control worldwide

2323

AliEn AliEn PullPull Protocol Protocol One of the major differences between ALiEn and One of the major differences between ALiEn and

LCG grids is that AliEn RB uses the LCG grids is that AliEn RB uses the pullpull rather than rather than pushpush model model

Classic model:Classic model:

ALiEn model:ALiEn model:

user server

ResourceBroker

user server

ResourceBroker

job

list

2424

Site

ALICE central services

Job submission in AliEnJob submission in AliEn

Job 1Job 1 lfn1lfn1, lfn2,, lfn2, lfn3 lfn3, , lfn4lfn4

Job 2Job 2 lfn1, lfn1, lfn2lfn2, lfn3,, lfn3, lfn4lfn4

Job 3Job 3 lfn1lfn1, , lfn2lfn2, , lfn3lfn3

Job 1.1Job 1.1 lfn1lfn1


Job 1.3Job 1.3 lfn3lfn3, , lfn4lfn4

Job 2.1Job 2.1 lfn1, lfn3lfn1, lfn3




Optimizer

ComputingAgent

RB

CE WN

Env OK?

Die with grac

e

Execs agent

Sends job agent to site

Yes No

Close SE’s & SoftwareMatchmaking

Receives work-load

Asks work-load

Retrieves workload

Sends job result

Updates TQ

Submits job (JDL) UserALICE Job Catalogue

Submitsjob agent

Registers output

lfn lfn guid guid {se’s}{se’s}





ALICE File Catalogue

packman

2525

Job workflowJob workflow

2626

Grid data challenge - PDC’06Grid data challenge - PDC’06 The longest running Data Challenge in ALICEThe longest running Data Challenge in ALICE

• A comprehensive test of the ALICE Computing modelA comprehensive test of the ALICE Computing model• Running for 9 months non-stop: close to data taking regime of operationRunning for 9 months non-stop: close to data taking regime of operation• Participating: 55 computing centres on 4 continents: 6 Tier 1s, 49 T2sParticipating: 55 computing centres on 4 continents: 6 Tier 1s, 49 T2s• 7MSI2k • hours 7MSI2k • hours 1500 CPUs running continuously 1500 CPUs running continuously

• 685K Grid jobs total• 40M evts, 0.5PB

generated, reconstructed and stored

• User analysis ongoing

43% T1s57% T2s

T1 sites:

CNAF, CCIN2P3, GridKa, RAL, SARA

FTS tests T0->T1 Sep-DecFTS tests T0->T1 Sep-Dec Design goal 300MB/s reached Design goal 300MB/s reached

but not maintainedbut not maintained

2727

AliEn@AliEn@ CERNCERN

• ALICEALICE• NA48NA48• NA49NA49• ATLASATLAS• COMPASSCOMPASS

RHIC (BNL)RHIC (BNL)• PHENIXPHENIX

EUEU• MammoGRIDMammoGRID

INFNINFN• GPCALMAGPCALMA

2828

• An OO framework to efficiently handle/analyse large amounts of data• Classes for:

• I/O• Containers: lists, arrays, ...• Physics classes (curve fitting, function evaluation) • Histogramming 1/2/3 dim, minimization• 2D, 3D visualization• Interface classes: Operating system, networking, SQL

• I/O• Specialised storage methods to access specific attributes of the objects, without having to touch the bulk of the data• Object stored in a binary machine-independent format (or XML)

ROOTROOT

2929

ROOT (2)ROOT (2)• C++

• Builtin CINT for fast prototyping• Compiler if more performance is needed• Open system that can be dynamically extended by linking external libraries• RTTI (Runtime Type Information)

• Used by most of today's HEP exp.• Users in the ROOT user database

• 60% from HEP• 30% from other scientific fields• 10% commercial

3030

The main software areasThe main software areas

GRID

middleware

RDBMS

run/file

catalogs

Object

persistencyv

2-d, 3-d

graphics

GUI

Toolkits

Math Libs

Statistics

Detector

Geometry

Event

Generators

Dectector

Simulation

Histograming

Fitting

Event Models

Folders

Event Display

Ntuple

analysis

Interpreters

DAQ

Online

System

services

ETC...

3131

XrootdXrootd

Today's experiments produce PB of dataToday's experiments produce PB of data• Analysis needs access to this dataAnalysis needs access to this data

xrootd is a serving programxrootd is a serving program• Requirements: reliable and fast, scalability, Requirements: reliable and fast, scalability,

fault tolerantfault tolerant Developed at SLACDeveloped at SLAC

3232

xrootd architecturexrootd architecture

Client

Redirector(Head Node)

Data Servers

open file X

A

B

C

go to C

open file X

Who has file X?

I have

Cluster

Client sees all servers as xrootd data servers

2nd open X

go to C

RedirectorsCache filelocation

3333

AliRoot: LayoutAliRoot: Layout

ROOT

CINT HIST GRAPH TREES CONT IO MATH …

ALIEN

GRID ITS TPC TRD TOF PHOS EMCAL HMPID MUON

FMD

PMD

T0

VZERO

ZDC

ACORDESTRUCT

STEERAliSimulation

AliReconstructionESD/AOD classes

G3 G4 Fluka

Virtual MC

HIJING

PYTHIA6

DPMJET

ISAJET

PDF

EVGEN

Analysis

HLT

RAW Monit

HBTAN JETAN

3434

AliRoot: Execution FlowAliRoot: Execution Flow

InitializationEvent

GenerationParticle

TransportHits

SummableDigits

Event Merging

(optional)

Digits/Raw digits

Clusters

Tracking PIDESDAOD

Analysis

Simulation

SimulationReconstruction

Reconstruction

3535

Analysis: summaryAnalysis: summary Core of ALICE computing model is AliRoot (ROOT)Core of ALICE computing model is AliRoot (ROOT)

Couple AliEn with ROOT for Grid-based analysisCouple AliEn with ROOT for Grid-based analysis

• Use PROOF – Parallel ROOT Facility. Transparent to usersUse PROOF – Parallel ROOT Facility. Transparent to users

4-tier architecture: 4-tier architecture:

• ROOT client sessionROOT client session

• API server (AliEn+PROOF)API server (AliEn+PROOF)

• Site PROOF master serversSite PROOF master servers

• PROOF slave serversPROOF slave servers

ALICE Computing ModelALICE Computing Model

Generationof calibrationparameters

RAW

Calibration

Disk bufferT0

CERN T0

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressor



are needed to see this picture.






First passReco

Tape T0

To tape To G

rid F

C

Alien FC

CAF

analysis

WNPROOFXROOTD

WNPROOFXROOTD

WNPROOFXROOTD

WNPROOFXROOTD

WNPROOFXROOTD

WNPROOFXROOTD





To T

1s

T1sAnalysisT2s

3737

Part III:Part III:

Parallel Computing Parallel Computing with PROOFwith PROOF

3838

PROOFPROOF

PParallel arallel ROOROOT T FFacilityacility Interactive parallel analysis on a local clusterInteractive parallel analysis on a local cluster

• Parallel processing of (local) data (trivial parallelism)Parallel processing of (local) data (trivial parallelism)• Fast FeedbackFast Feedback• Output handling with direct visualizationOutput handling with direct visualization• NotNot a batch system a batch system

PROOF itself is not related to GridPROOF itself is not related to Grid• Can access Grid filesCan access Grid files

The usage of PROOF is transparentThe usage of PROOF is transparent• The same code can be run locally and in a PROOF The same code can be run locally and in a PROOF

system (certain rules have to be followed)system (certain rules have to be followed) PROOF is part of ROOTPROOF is part of ROOT

3939

root

Remote PROOF Cluster

Data

root

root

root

Client – Local PC

ana.C

stdout/result

node1

node2

node3

node4

ana.C

root

PROOF SchemaPROOF Schema

Data

Proof masterProof slave

Result

Data

Result

Data

Result

Result

4040

Event based (trivial) ParallelismEvent based (trivial) Parallelism

4141

TerminologyTerminology ClientClient

• Your machine running a ROOT session that is connected Your machine running a ROOT session that is connected to a PROOF masterto a PROOF master

MasterMaster• PROOF machine coordinating work between slavesPROOF machine coordinating work between slaves

Slave/WorkerSlave/Worker• PROOF machine that processes dataPROOF machine that processes data

QueryQuery• A job submitted from the client to the PROOF system.A job submitted from the client to the PROOF system.

A query consists of a selector and a chainA query consists of a selector and a chain SelectorSelector

• A class containing the analysis codeA class containing the analysis code ChainChain

• A list of files (trees) to process (more details later)A list of files (trees) to process (more details later)

4242

once on your client

once on each slave

for each tree

for each event

Classes derived from a Selector can run locally, in Classes derived from a Selector can run locally, in PROOF and in AliEnPROOF and in AliEn

• "Constructor""Constructor"

• CreateOutputObjects()CreateOutputObjects()

• ConnectInputData()ConnectInputData()

• Exec()Exec()

• Terminate()Terminate()

SelectorSelector

4343

Class TTreeClass TTree A tree is a container for data storageA tree is a container for data storage It consists of several It consists of several branchesbranches

• These can be in one or several filesThese can be in one or several files• Branches are stored contiguously (split Branches are stored contiguously (split

mode)mode)• When reading a tree, certain branches When reading a tree, certain branches

can be switched off can be switched off speed up of speed up of analysis when not all data is neededanalysis when not all data is needed

Set of helper functions to visualize Set of helper functions to visualize contentcontent

CompressedCompressed

Tree

Bra

nc

h

Bra

nc

h

Bra

nc

h

pointpoint

xx

yy

zz

x x x x x x x x x x

y y y y y y y y y y

z z z z z z z z z z

Branches File

4444

TChainTChain

A chain is a list of trees (in several files)A chain is a list of trees (in several files) Normal TTree functions can be usedNormal TTree functions can be used

• Draw(...), Scan(...)Draw(...), Scan(...)

these iterate over all elements of the chainthese iterate over all elements of the chain

Chain

Tree1 (File1)

Tree2 (File2)

Tree3 (File3)

Tree4 (File3)

Tree5 (File4)

4545

MergingMerging The analysis runs on several slaves, The analysis runs on several slaves,

therefore partial results have to be therefore partial results have to be mergedmerged

Objects are identified by nameObjects are identified by name Standard merging implementation Standard merging implementation

for histograms availablefor histograms available Other classes need to implement Other classes need to implement

Merge(TCollection*)Merge(TCollection*) When no merging function is When no merging function is

available all the individual objects available all the individual objects are returnedare returned

Result fromSlave 1

Result fromSlave 2

Final result

Merge()

4646

Chain

Tree1 (File1)

Tree2 (File2)

Tree3 (File3)

Tree4 (File3)

Tree5 (File4)

Workflow SummaryWorkflow SummaryAnalysis

(AliAnalysisTask)Input

proof

proof

proof

4747

Workflow SummaryWorkflow SummaryAnalysis

(AliAnalysisTask)

proof

proof

proof

Output

Output

Output MergedOutput

4848

PROOF & xrootdPROOF & xrootd

• PROOF slaves/master are started by xrootd and use its communication layer

• Master uses file location information from xrootd redirector to decide which slave processes which events

• Data Access Strategies– Slaves process local data first– If no (more) local data is available, remote data is

processed

4949

Data distribution – Pull model

5050

PackagesPackages

PAR files: PAR files: PPROOF ROOF ARARchive. Like Java jarchive. Like Java jar• Gzipped tar fileGzipped tar file• PROOF-INF directoryPROOF-INF directory

BUILD.sh, building the package, executed per slaveBUILD.sh, building the package, executed per slave SETUP.C, set environment, load libraries, executed SETUP.C, set environment, load libraries, executed

per slave per slave

API to manage and activate packagesAPI to manage and activate packages• UploadPackage("package")UploadPackage("package")• EnablePackage("package")EnablePackage("package")

5151

CERN Analysis FacilityCERN Analysis Facility

The The CCERN ERN AAnalysis nalysis FFacility (CAF) will run PROOF acility (CAF) will run PROOF for ALICEfor ALICE• Prompt analysis of pp dataPrompt analysis of pp data• Pilot analysis of PbPb dataPilot analysis of PbPb data• Calibration & AlignmentCalibration & Alignment

Available to the whole collaboration but the Available to the whole collaboration but the number of users will be limited for efficiency number of users will be limited for efficiency reasonsreasons

Design goalsDesign goals• 500 CPUs500 CPUs• 100 TB of selected data locally available100 TB of selected data locally available

5252

Evaluation of PROOFEvaluation of PROOF

Test setup since beginning 2007Test setup since beginning 2007• 40 machines, 2 CPUs each, 200 GB disk40 machines, 2 CPUs each, 200 GB disk

TestsTests• Usability testsUsability tests• Simple speedup plotSimple speedup plot• Evaluation of different query typesEvaluation of different query types• Evaluation of the system when running a Evaluation of the system when running a

combination of query typescombination of query types Goal: Realistic simulation of users using Goal: Realistic simulation of users using

the systemthe system

5454

Progress dialogProgress dialog

Query statistics

Abort query andview resultsup to now

Abort query anddiscard results

Show logfiles

Show processing rate

5555

ExamplesExamples

5656

The future for Physics AnalysisThe future for Physics Analysis

Each node has a PROOF slave

Each site has a PROOF master server

Uses pull model.• The slaves ask the master for work packets• Slower slaves get smaller work packets

ClientAPI

APIServer

AliEnFC….

List of sites with

data

5757

Part IV:Part IV:

PhD meeting 1PhD meeting 1

5858

• New Grid computing center based on Intel Mac• Based at CERN• Installing AliEn services, afs, nfs, torque, certificates• New machinery coming soon from ARTS project won by ALICE Offline

in 2007 (ISREC@EPFL was awarded in 2006)

• Massive raw data reconstruction on the Grid

• Dec'07 cosmics (~100TB raw chunks)

• Feb'08 cosmics (~60TB so far)

• PROOF• Tutorial

• CPU fairshare and Disk quota

• EMMIT08

Current Activities at CERNCurrent Activities at CERN

5959

• PROOF Benchmarking in CAF

• General SpeedUp (Amdahl's Law)

• Relative SpeedUp for different• types of query• number of users• number of slaves

• PROOF/Grid

• Fairshare of a single site (memory dump)

• Interactive Distributed Parallel Analysis (demo)

Open IssuesOpen Issues

Documents

1 PhD meeting 1 Frameworks for GRID and Interactive Parallel Analysis Marco Meoni - ALICE Offline, CERN EPFL – Lausanne – 28/02/2008