ATLAS Goals and Status
Jim Shank
US LHC OSG Technology Roadmap
May 4-5th, 2005
2
Outline
High level USATLAS goals
Milestone Summary and Goals
Identified Grid3 Shortcomings
Workload Management Related
Distributed Data Management Related
VO Service Related
3
US ATLAS High Level Requirements
Ability to transfer large datasets between CERN, BNL, and Tier2 Centers
Ability to store and manage access to large datasets at BNL
Ability to produce Monte Carlo at Tier2 and other sites and deliver to
BNL
Ability to store, serve, catalog, manage and discover ATLAS wide
datasets by users and ATLAS file transfer services
Ability to create and submit user analysis jobs to data locations
Ability to access opportunistically non-dedicated ATLAS resources
To deliver all these capabilities at a scale commensurate with available
resources (CPU, storage, network, and human) and number of users
4
ATLAS Computing Timeline
2003 • POOL/SEAL release (done)
• ATLAS release 7 (with POOL persistency) (done)
• LCG-1 deployment (done)
• ATLAS complete Geant4 validation (done)
• ATLAS release 8 (done)
• DC2 Phase 1: simulation production (done)
• DC2 Phase 2: intensive reconstruction (the real challenge!) LATE!
• Combined test beams (barrel wedge) (done)
• Computing Model paper (done)
• Computing Memorandum of Understanding (in progress)
• ATLAS Computing TDR and LCG TDR (starting)
• DC3: produce data for PRR and test LCG-n
• Physics Readiness Report
• Start cosmic ray run• GO!
2004
2005
2006
2007
NOW
Rome Physics Workshop
5
Grid Pressure
• Grid3/DC2 very valuable exercise
• Lots of pressure on me now to develop backup plans
if the “grid middleware” does not come through
with required functionality on our timescale.
• Since manpower is short, this could mean pulling manpower out of
our OSG effort.
6
Schedule & Organization
Grid Tools & Services
2.3.4.1 Grid Service Infrastructure
2.3.4.2 Grid Workload Management
2.3.4.3 Grid Data Management
2.3.4.4 Grid Integration & Validation
2.3.4.5 Grid User Support
2005
2006
2007
NO
W
CSC
PRR
CRR
Start
TDR
DC2
ATLAS milestones
7
Milestone Summary (I)
2.3.4.1 Grid Service Infrastructure
May 2005 ATLAS software management service upgrade
June 2005 Deployment of OSG 0.2 (expected increments after this)
June 2005 ATLAS-wide monitoring and accounting service
June 2005 ATLAS site certification service
June 2005 LCG interoperability (OSG LCG)
July 2005 SRM/dCache deployed on Tier2 Centers
Sep 2005 LCG interoperability services for SC05
8
Milestone Summary (II)
2.3.4.2 Grid Workload Management
June 2005: Defined metrics for submit host scalability met
June 2005: Capone recovery possible without job loss
July 2005: Capone2 WMS for ADA
Aug 2005: Pre-production Capone2 delivered to Integration team
Integration with SRM
Job scheduling
Provision of Grid WMS for ADA
Integration with DDMS
Sep 2005: Validated Capone2 + DDMS
Oct 2005: Capone2 WMS with LCG Interoperability components
9
Milestone Summary (III)
2.3.4.3 Grid Data Management
April 2005: Interfaces and functionality for OSG-based DDM
June 2005: Integrate with OSG-based storage services with benchmarks
Aug 2005: Interfaces to the ATLAS OSG workload management system (Capone)
Nov 2005: Implementing storage authorization and policies
10
Milestone Summary (IV)
2.3.4.4 Grid Integration & Validation
March 2005: Deployment of ATLAS on OSG ITB (Integration Testbed)
June 2005: Pre-production Capone2 deployed and validated OSG ITB
July 2005: Distributed analysis service implemented with WMS
August 2005: Integrated DDM+WMS service challenges on OSG
Sept 2005: CSC (formerly DC3) full functionality for production service
Oct 2005: Large scale distributed analysis challenge on OSG
Nov 2005: OSG-LCG Interoperability exercises with ATLAS
Dec 2005: CSC full functionality pre-production validation
11
ATLAS DC2 Overview of Grids as of 2005-02-24 18:11:30
Grid submitted pending running finished failed efficiency
Grid3 36 3 814 153028 46943 77 %
NorduGrid 29 130 1105 114264 70349 62 %
LCG 60 528 610 145692 242247 38 %
TOTAL 125 661 2529 412984 359539 53 %
Capone + Grid3 Performance
Capone submitted & managed ATLAS jobs on Grid3 > 150K
In 2004, 1.2M CPU-hours
Grid3 sites with more than 1000 successful DC2 jobs: 20
Capone instances > 1000 jobs: 13
12
ATLAS DC2 and Rome Production on Grid3
<Capone Jobs/day> = 350
Max # jobs/day = 1020
13
Most Painful Shortcomings
“Unprotected” Grid services
GT2 Gram and GridFTP vulnerable to multiple users and VOs
Frequent manual intervention of site administrators
No reliable file transfer service or other data management
services such as space management
No policy-based authorization infrastructure
No distinction between production and individual grid users
Lack of a reliable information service
Overall robustness and reliability (on client and server) poor
15
ATLAS Production System
LCGNorduGrid Grid3 LSF
LCGexe
NGexe
G3exe
Legacyexe
super super super super
prodDB(CERN)
datamanagement
RLS RLS RLS
jabber soap soap jabber
Don Quijote “DQ”
Windmill
Lexor
AMI(Metadata)
CaponeDulcinea
LCGNorduGrid Grid3 LSF
LCGexe
NGexe
G3exe
Legacyexe
super super super super
prodDB(CERN)
datamanagement
RLS RLS RLS
jabber soap soap jabber
Don Quijote “DQ”
Windmill
Lexor
AMI(Metadata)
CaponeDulcinea
Requirements for
VO and
Core Services
DDMS
WMS
16
Grid Data Management
ATLAS Distributed Data Management System (DDMS)
US ATLAS GTS role in this project: Provide input to design: expected interfaces, functionality,
and scalability performance and metrics, based on experience with Grid3 and DC2 and compatibility with OSG services
Integrate with OSG-based storage services Benchmark OSG implementation choices with ATLAS standards Specification and development of as-needed OSG specific
components required for integration with the overall ATLAS system Introduce new middleware services as they mature Interfaces to OSG workload management system (Capone) Implementing storage authorization and policies for role-based
usage (reservation, expiration, cleanup, connection to VOMS, etc) consistent with ATLAS data management tools and services.
17
DDMS Issues: File Transfers
Dataset discovery and distribution for both
production and analysis services
AOD distribution from CERN to Tier1
Monte Carlo produced at Tier2 delivered to Tier1
Support role-based priorities for transfers and space
21
Reliable FT in DDMS
MySQL Backend Services for agents and clients Schedules transfers
later, monitor and account resources Security and policy issues affecting core
infrastructure Authentication and Authorization infrastructure needs to
be in place Priorities for production managers & end-users need to
be settable Roles to set group and user quotas, permissions
DQ evolving
M. Branco
LC
G D
eplo
ymen
t Gro
up
22
LCG-required SRM functions
SRM v1.1 insufficient – mainly lack of pinning SRM v3 not required – and timescale too late Require Volatile, Permanent space; Durable not practical Global space reservation: reserve, release, update
(mandatory LHCb, useful ATLAS,ALICE). Compactspace NN Permissions on directories mandatory
Prefer based on roles and not DN (SRM integrated with VOMS desirable but timescale?)
Directory functions (except mv) should be implemented asap
Pin/unpin high priority srmGetProtocols useful but not mandatory Abort, suspend, resume request : all low priority Relative paths in SURL important for ATLAS, LHCb, not for
ALICE
CMS input/comments not included yet
DDMS INTERFACING TO STORAGE
23
Managing Persistent Services
General summary: larger sites and ATLAS-controlled sites
should let us run services like LRC, space management,
replication.
Other sites can be handled by being part of some multi-site
domain managed by one of our sites -- ie their persistent
service needs are covered by services running at our sites,
and site-local actions like space management happen via
submitted jobs (working with the remote service and therefore
requiring some remote connectivity or gateway proxying)
rather than local persistent services.
24
Scalable Remote Data Access
ATLAS reconstruction and analysis jobs require
acces to remote database servers at CERN, BNL,
and elsewhere
Presents additional traffic on network especially for
large sites
Suggest use of local mechanisms, such was web
proxy caches, to minimize this impact
25
Workload Management
LCG Core services contributions focused on LCG-RB, etc.
Here address scalability and stability problems experienced on
Grid3
Defined metrics; achieve by job batching or by DAG-appends
5000 active jobs from a single submit host
Submission rate: 30 jobs/minute
>90% job efficiency jobs accepted
All of US production managed by 1 person
Job state persistency mechanism
Capone recovery possible without job loss
26
Other Considerations
Require reliable information system
Integration with Managed Storage
Several resources in OSG will be SRM/dCache, which does not support 3rd party
transfers (used heavily in Capone)
NeST as an option for space management
Job scheduling
Static at present; change to matchmaking based on data location and job queue
depth & policy
Integration with ATLAS data management system (evolving)
Current system works directly with file-based RLS
New system will interface with new ATLAS dataset model; POOL file catalog
interface
27
Grid Interoperability
Interoperability with LCG A number of proof of principle exercises completed Progress of GLUE schema with LCG Publication of USATLAS resources (compute, storage) to
LCG index service (BDII) Use of the LCG developed General Information Provider Submit ATLAS job from LCG to OSG (via an LCG Resource Broker) Submit from OSG site to LCG services directly via Condor-G Demonstration challenge under discussion with LCG in OSG Interoperability
Interoperability with TeraGrid Initial discussions begun in OSG Interoperability Activity F. Luehring (US ATLAS) co-Chair Initial issues identified: authorization, allocations, platform dependencies
28
Summary
We’ve been busy almost continuously with production since July 2004.
Uncovered many shortcomings in Grid3 core infrastructure and our own
services (eg. WMS scalability, no DDMS).
Expect to see more ATLAS services and agents on gatekeeper nodes,
especially for DDMS
Need VO-scoped management of resources by the WMS and DDMS until
middleware service supply these capabilities
To be compatible with ATLAS requirements and OSG principles, we see
resources partitioned (hosted dedicated VO & guest VO resources and general
OSG opportunistic) to achieve reliability and robustness.
Time is of the essence. A grid component, fully meeting our specs, delivered
late is the same as a failed component. We will have already implemented a
work around.
appendix
30
ProdDB
Elements of the execution environment used to run ATLAS DC2 on Grid3. The dashed (red) box indicates processes executing on the Capone submit host. Yellow boxes indicate VDT components.
Condor-G
Condor-G
schedd
GridMgr
CEgsiftp WN
SE
Chimera
RLS
Windmill
Pegasus
VDC
DonQuijote
Mon-Servers
MonALISA
gram
Grid3 Sites
Capone
jobsch
GridCat
MDS
Grid3 System Architecture
Caponeinside
31
Capone Workload Manager
Message interface Web Service & Jabber
Translation layer Windmill (Prodsys) schema
Distributed Analysis (ADA) -TBD
Process execution Capone Finite State Machine
Process eXecution Engine SBIR (FiveSight)
Processes Grid interface
Stub: local shell script testing
Data Management -TBD
The Capone Architecture, showing three common component layers above selectable modules which interact with local and remote services.
Message protocols
JabberWeb Service
Translation
ADAWindmill
Process Execution
PXECPE-FSM
Stu
b
Grid
(GC
E
Clien
t)
Data
Man
ag
emen
t
32