GangaTasks Automating of job submission wit Athena/AthenaMC in Ganga hepcg workshop, siegen,...

GangaTasksGangaTasksAutomating of job Automating of job submission wit submission wit Athena/AthenaMC in GangaAthena/AthenaMC in Ganga

hepcg workshop, siegen, 13.12.2007hepcg workshop, siegen, 13.12.2007

johannes ebke, johannes elmsheuser, johannes ebke, johannes elmsheuser, tariq tariq mahmoudmahmoud

tariq mahmoud hepcg workshop, siegen, 13122007

automating of job submission in Athena/AthenaMC

outline

I. introduction, motivation

II. concept of tasksIII. some illustration IV. conclusion

automating of job submission in Athena/AthenaMC

I. motivation, mc

simul:5

recon:10

1

1 3 5 97

2 4 6 108

1

evgen:50

7 9

tariq mahmoud hepcg workshop, siegen, 13122007

3 5

3

6

3 5

63

3

6

5

6

3

3

3

need automate job

(re)submission

automating of job submission in Athena/AthenaMC

I. motivation, mc

tariq mahmoud hepcg workshop, siegen, 13122007

1 3 5 972 4 6 108

1

1 9

7

7

7 1 3 5 972 4 6 108

1

1 7 9

36

3 5

633

5

56

566

6

1 3 5 972 4 6 108

1

1 7

36

3 5

63

1 3 5 972 4 6 108

1

1 7 9

3 5

3 5

53

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63756

1 3 5 972 4 6 108

1

9

7

7

3

5

55

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63

automating of job submission in Athena/AthenaMC

I. motivation, mc

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63756

1 3 5 972 4 6 108

1

1 9

7

7

3

5

55

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63

1 3 5 972 4 6 108

1

1 9

7

7

3

5

5

6 7

4

4

4

4need automate job

(re)submission

need job chaining

(interconnection)

tariq mahmoud hepcg workshop, siegen, 13122007

automating of job submission in Athena/AthenaMC

I. motivation, analysis

complete dataset

tariq mahmoud hepcg workshop, siegen, 13122007

A,B,C: f1,f2,f3, …,fn

A,B,C f1,f2,f3A,B,C f4,f5,f6

A,B,C f7,f8,f9A,B,C f7,f8,f9

A,B,C f4,f5,f6

A,B,C f7,f8,f9A,B,C fn-2,fn-1,fn

incomplete datasetA: f1,f2,f3B: f1,f5,f6,f7,f8,f9C: f1,f2,f7,f8,f9,f10

A,C f1,f2B f5,f6

A f3

B,C f7,f8,f9C f10

automating of job submission in Athena/AthenaMC II. concept of tasks

taskfew parameters: task parameters: float, name, number of events etc.

process parameters: datasets, process name, geometry etc.

input/output specifications Athena version

AbstractJobgenerated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, …

AbstractJobAbstractJob

jobjob

job Ganga jobs

tariq mahmoud hepcg workshop, siegen, 13122007

automating of job submission in Athena/AthenaMC II. concept of tasks: GangaTasks

MCTask

MCJobMCJobMCJob

AnaTask

AnaJob

Task

AnaJobAnaJob

repository/tools.data

TaskList(GangaObject)

GPI.tasks

tariq mahmoud hepcg workshop, siegen, 13122007

automating of job submission in Athena/AthenaMC III. illustration, resubmission

tariq mahmoud hepcg workshop, siegen, 13122007

automate job (re)submission

need job chaining

(interconnection)

automating of job submission in Athena/AthenaMC

III. illustration: chaining task stepsmc.total_events = 200;

mc.events_per_job={"evgen": 50, "simul": 5, "recon":10}

1 3 5 7 9

1 3 5 7 9

1 3 5 7 9

1 3 5 7 9

1 3 5 972 4 6 108

1 3 5 972 4 6 108

1 3 5 972 4 6 108

1 3 5 972 4 6 108

2 3 41 evgen:i-0

simul:i-j

recon:i-k

tariq mahmoud hepcg workshop, siegen, 13122007

mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10

AbstractJob.prerequisites()

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC III. illustration: interconnections,

prerequisites

automate job (re)submission

job chaining (interconnection)

the user sets few parameters … and gets

RESULTS

automating of job submission in Athena/AthenaMC

IV. conclusion

GangaTask package: automates job management (small scale production system)works with Ganga Athena and Ganga AthenaMC

(available from Ganga443 onwards) works at cernwiki with tutorial: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasksapplicable to any job submission-software on the grid

To do: 1. implementing site-test-code 2. making use of GangaRobot3. act on user suggestionstariq mahmoud hepcg workshop, siegen, 13122007

Configure once – run anywhere