US Grid Efforts Lee Lueking D0 Remote Analysis Workshop February 12, 2002

US Grid Efforts

Lee LuekingLee Lueking

D0 Remote Analysis WorkshopD0 Remote Analysis Workshop

February 12, 2002February 12, 2002

February 12, 2002 Lee Lueking - D0 RACE 2

All of these projects are working towards All of these projects are working towards the common goal of providing transparent the common goal of providing transparent access to the massively distributed access to the massively distributed computing infrastructure that is needed to computing infrastructure that is needed to meet the challenges of modern experiments meet the challenges of modern experiments … … (From the EU DataTAG proposal)(From the EU DataTAG proposal)


Grid Projects Timeline

Q3 00

Q4 00

Q4 01

Q3 01

Q2 01

Q1 01

Q1 02

GriPhyN: $11.9M+$1.6M

PPDG:$9.5M

iVDGL:$13.65M

EU DataGrid: $9.3M

EU DataTAG:4M Euros

GridPP:


PPDGParticle Physics Data Grid

““Vertical Integration” of Grid middleware components into HENP Vertical Integration” of Grid middleware components into HENP experiments’ ongoing workexperiments’ ongoing work

An unparalleled laboratory for “experimental computer science”An unparalleled laboratory for “experimental computer science”

iVDGLInternational Virtual Data Grid Laboratory

A place to conduct Data Grid tests “at scale”A place to conduct Data Grid tests “at scale” A mechanism to create common Grid infrastructureA mechanism to create common Grid infrastructure A facility to perform production exercises for LHC experimentsA facility to perform production exercises for LHC experiments


PPDG Funded through DOE MICS and HENP programsFunded through DOE MICS and HENP programs PPDG will develop, acquire and deliver vitally needed PPDG will develop, acquire and deliver vitally needed

Grid-enabled tools for data-intensive requirements of Grid-enabled tools for data-intensive requirements of particle and nuclear physics.particle and nuclear physics.

PPDG is a collaboration of computer scientists with a PPDG is a collaboration of computer scientists with a strong record in distributed computing and Grid strong record in distributed computing and Grid technology, and physicists with leading roles in the technology, and physicists with leading roles in the software and network infrastructures for major high-software and network infrastructures for major high-energy and nuclear experiments.energy and nuclear experiments.

Our goals and plans are ultimately guided by the Our goals and plans are ultimately guided by the immediate, medium-term and longer-term needs and immediate, medium-term and longer-term needs and perspectives of the physics experiments.perspectives of the physics experiments.


GriPhyN: Grid Physics Network

Virtual data technologies.Virtual data technologies. Advances are required in information Advances are required in information models and in new methods of cataloging, characterizing, models and in new methods of cataloging, characterizing, validating, and archiving software components to implement validating, and archiving software components to implement virtual data manipulations virtual data manipulations

Policy-driven request planning and scheduling of networked Policy-driven request planning and scheduling of networked data and computational resources.data and computational resources. We require mechanisms for We require mechanisms for representing and enforcing both local and global policy representing and enforcing both local and global policy constraints and new policy-aware resource discovery constraints and new policy-aware resource discovery techniques. techniques.

Management of transactions and task-execution across Management of transactions and task-execution across national-scale and worldwide virtual organizations.national-scale and worldwide virtual organizations. New New mechanisms are needed to meet user requirements for mechanisms are needed to meet user requirements for performance, reliability, and cost. Agent computing will be performance, reliability, and cost. Agent computing will be important to permit the grid to balance user requirements and important to permit the grid to balance user requirements and grid throughput, with fault tolerance. grid throughput, with fault tolerance.


iVDGL:International Virtual Data Grid Laboratory

The iVDGL will provide a global computing resource for several The iVDGL will provide a global computing resource for several leading international experiments in physics and astronomy, leading international experiments in physics and astronomy,

Global services and centralized monitoring, management, and Global services and centralized monitoring, management, and support functions functions will be coordinated by the Grid support functions functions will be coordinated by the Grid Operations Center (GOC) located at Indiana University, with Operations Center (GOC) located at Indiana University, with technical effort provided by GOC staff, iVDGL site staff, and the technical effort provided by GOC staff, iVDGL site staff, and the CS support teams. CS support teams.

The GOC will operate iVDGL as a NOC manages a network, The GOC will operate iVDGL as a NOC manages a network, providing a single, dedicated point of contact for iVDGL status, providing a single, dedicated point of contact for iVDGL status, configuration, and management, and addressing overall configuration, and management, and addressing overall robustness issues.robustness issues.


iVDGL:International Virtual Data Grid Laboratory

Management of the Management of the iVDGLiVDGL will be integrated with that of the will be integrated with that of the GriPhyNGriPhyN Project, funded by NSF in September 2000 for $11.9M. Project, funded by NSF in September 2000 for $11.9M.

GriPhyNGriPhyN and and Particle Physics Data GridParticle Physics Data Grid will provide the basic will provide the basic R&D and software toolkits needed for the laboratory. R&D and software toolkits needed for the laboratory.

The The European Union DataGridEuropean Union DataGrid is also a major participant and is also a major participant and will contribute basic technologies and tools. will contribute basic technologies and tools.

The iVDGL will be based on the open Grid infrastructure The iVDGL will be based on the open Grid infrastructure provided by the provided by the GlobusGlobus Toolkit and will also build on other Toolkit and will also build on other technologies such as technologies such as CondorCondor resource management tools. resource management tools.


So What’s the Difference?PPDGPPDG iVDGLiVDGL

FundingFunding US DOE approvedUS DOE approved

1/1/3/3/3 $M, 99 – 031/1/3/3/3 $M, 99 – 03

US NSF proposedUS NSF proposed

3/3/3/3/3 $M, 02 – 063/3/3/3/3 $M, 02 – 06

Computer Computer ScienceScience

Globus (Foster), Condor Globus (Foster), Condor (Livny), SDM (Shoshani), (Livny), SDM (Shoshani), SRB (Moore)SRB (Moore)

Globus (Foster, Kesselman), Globus (Foster, Kesselman), Condor (Livny)Condor (Livny)

PhysicsPhysics BaBar, Dzero, STAR, JLAB, BaBar, Dzero, STAR, JLAB, ATLAS, CMSATLAS, CMS

ATLAS, CMS, LIGO,ATLAS, CMS, LIGO,

SDSS, NVOSDSS, NVO

National National LaboratoriesLaboratories

BNL, Fermilab, JLAB, SLAC, BNL, Fermilab, JLAB, SLAC, ANL, LBNLANL, LBNL

ANL,BNL, Fermilab (all ANL,BNL, Fermilab (all unfunded collaborators)unfunded collaborators)

UniversitiesUniversities Caltech, SDSS, UCSD, Caltech, SDSS, UCSD, WisconsinWisconsin

Florida, Chicago, Caltech, UCSD, Indiana, Boston, Wisconsin at Florida, Chicago, Caltech, UCSD, Indiana, Boston, Wisconsin at Milwaukee, Pennsylvania State, Johns Hopkins, Wisconsin at Milwaukee, Pennsylvania State, Johns Hopkins, Wisconsin at Madison, Northwestern, USC, UT Brownsville, Hampton, Salish Madison, Northwestern, USC, UT Brownsville, Hampton, Salish Kootenai CollegeKootenai College

HardwareHardware NoneNone ~20% of funding~20% of funding(Tier-2 Centers)(Tier-2 Centers)

NetworkNetwork No funding requestedNo funding requested No funding requestedNo funding requestedDataTAG complementaryDataTAG complementary


PPDG Collaborators


PPDG Computer Science GroupsCondor – develop, implement, deploy, and evaluate mechanisms and policies

that support High Throughput Computing on large collections of computing resources with distributed ownership.

http://www.cs.wisc.edu/condor/

Globus - developing fundamental technologies needed to build persistent environments that enable software applications to integrate instruments, displays, computational and information resources that are managed by diverse organizations in widespread locations

http://www.globus.org/

SDM - Scientific Data Management Research Group – optimized and standardized access to storage systems

http://gizmo.lbl.gov/DM.html

Storage Resource Broker - client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and cataloging/accessing replicated data sets.

http://www.npaci.edu/DICE/SRB/index.html


PPDG Project Activities

CMS GDMP: Grid Data Mirroring ProjectCMS GDMP: Grid Data Mirroring Project D0 Job ManagementD0 Job Management CMS-MOP: CMS-MOP: MoMonte Carlo Distributed nte Carlo Distributed PProductionroduction STAR-DDM: Uses HRM, (Hierarchical Resource STAR-DDM: Uses HRM, (Hierarchical Resource

Manager)Manager) JLAB-SRB:Storage Resource Broker, ReplicationJLAB-SRB:Storage Resource Broker, Replication ATLAS MAGDA: distributed data manager ATLAS MAGDA: distributed data manager

(ATLAS-Globus)(ATLAS-Globus)


PPDG Cross-cut Activities

SC2001SC2001 Certificate /Registration AuthorityCertificate /Registration Authority Collaboration with IEPM, Network Collaboration with IEPM, Network

Performance MonitoringPerformance Monitoring


Super Computing 2001 in Denver



Common Services Job Description LanguageJob Description Language Scheduling and Management of Processing and Data Scheduling and Management of Processing and Data

Placement ActivitiesPlacement Activities Monitoring and Status ReportingMonitoring and Status Reporting Storage Resource ManagementStorage Resource Management Reliable Replica Management ServicesReliable Replica Management Services File Transfer ServicesFile Transfer Services Collect and Cocument Current Experimental PracticesCollect and Cocument Current Experimental Practices R & D, EvaluationR & D, Evaluation Authentication, Authorization, and SecurityAuthentication, Authorization, and Security End-to-End Applications and TestbedsEnd-to-End Applications and Testbeds


Delivery of End-to-End Applications& Integrated Production Systems

to allow thousands of physicists to share data & computing resources for scientific processing and analyses

Operators & Users

Resources: Computers, Storage, Networks

PPDG Focus:

- Robust Data Replication

- Intelligent Job Placement and Scheduling

- Management of Storage Resources

- Monitoring and Information of Global Services

Relies on Grid infrastructure:- Security & Policy- High Speed Data Transfer- Network management


Project Activities,End-to-End Applications

and Cross-Cut Pilots

Project Activities are focused Experiment – Computer Science Collaborative developments.

Replicated data sets for science analysis – BaBar, CMS, STARDistributed Monte Carlo production services – ATLAS, D0, CMSCommon storage management and interfaces – STAR, JLAB

End-to-End Applications used in Experiment data handling systems to give real-world requirements, testing and feedback.

Error reporting and responseFault tolerant integration of complex components

Cross-Cut Pilots for common services and policies Certificate Authority policy and authenticationFile transfer standards and protocolsResource Monitoring – networks, computers, storage.


PPDG activities as part of the Global Grid Community

Coordination with other Grid Projects in our field:GriPhyN – Grid for Physics NetworkEuropean DataGridStorage Resource Management collaboratoryHENP Data Grid Coordination Committee

Participation in Experiment and Grid deployments in our field:ATLAS, BaBar, CMS, D0, Star, JLAB experiment data handling systemsiVDGL/DataTAG – International Virtual Data Grid LaboratoryUse DTF computational facilities?

Active in Standards Committees:Internet2 HENP Working Group Global Grid Forum


PPDG and GridPP Projects

Use of Standard Middleware to Promote InteroperabilityUse of Standard Middleware to Promote Interoperability Move to Globus infrastructure: GSI, GridFTPMove to Globus infrastructure: GSI, GridFTP Use of Condor as a supported system for job Use of Condor as a supported system for job

submissionsubmission Publish availability of resources and file catalogPublish availability of resources and file catalog

Additional Grid Functionality for Job Specification, Additional Grid Functionality for Job Specification, Submission, and TrackingSubmission, and Tracking Use Condor for migration and check pointingUse Condor for migration and check pointing Enhanced job specification language and servicesEnhanced job specification language and services

Enhanced Monitoring and Diagnostic CapabilitiesEnhanced Monitoring and Diagnostic Capabilities Fabric ManagementFabric Management


PPDG Management and CoordinationPIs

Livny, Newman, Mount

Steering CommitteeRuth Pordes, Chair

Doug Olson, Physics Deputy ChairMiron Livny, Computer Science Deputy Chair

Computer Science Group RepresentativesPhysics Experiment Representatives

PIs (ex officio)

STAR SDM BaBar SRB JLAB ATLAS Globus CMS Condor DZero

Executive Team (>1.0 FTE on PPDG)Steering Committee Chair

Steering Committee Physics and CS Deputy Chairs


iVDGL

International Virtual-Data Grid LaboratoryInternational Virtual-Data Grid Laboratory A global Grid laboratory with participation from US, EU, Asia, A global Grid laboratory with participation from US, EU, Asia,

etc.etc. A place to conduct Data Grid tests “at scale”A place to conduct Data Grid tests “at scale” A mechanism to create common Grid infrastructureA mechanism to create common Grid infrastructure A facility to perform production exercises for LHC experimentsA facility to perform production exercises for LHC experiments A laboratory for other disciplines to perform Data Grid testsA laboratory for other disciplines to perform Data Grid tests

“We propose to create, operate and evaluate, over asustained period of time, an international researchlaboratory for data-intensive science.”

From NSF proposal, 2001


iVDGL Summary Information

Principal components (as seen by USA)Principal components (as seen by USA) Tier1 sites (laboratories)Tier1 sites (laboratories) Tier2 sites (universities and other institutes)Tier2 sites (universities and other institutes) Selected Tier3 sites (universities)Selected Tier3 sites (universities) Fast networks: US, Europe, transatlanticFast networks: US, Europe, transatlantic International Grid Operations Center (iGOC)International Grid Operations Center (iGOC) Computer Science support teamsComputer Science support teams Coordination, managementCoordination, management

Proposed international partnersProposed international partners Initially US, EU, Japan, AustraliaInitially US, EU, Japan, Australia Other world regions laterOther world regions later Discussions w/ Russia, China, Pakistan, India, South AmericaDiscussions w/ Russia, China, Pakistan, India, South America

Complementary EU project: Complementary EU project: DataTAGDataTAG Transatlantic network from CERN to STAR-TAP (+ people)Transatlantic network from CERN to STAR-TAP (+ people) Initially 2.5 Gb/sInitially 2.5 Gb/s


US Proposal to NSF US proposal approved by NSF Sept. 25, 2001US proposal approved by NSF Sept. 25, 2001

““Part 2” of GriPhyN projectPart 2” of GriPhyN project Much more application oriented than first GriPhyN proposalMuch more application oriented than first GriPhyN proposal $15M, 5 years @ $3M per year (huge constraint)$15M, 5 years @ $3M per year (huge constraint) CMS + ATLAS + LIGO + SDSS/NVO + Computer ScienceCMS + ATLAS + LIGO + SDSS/NVO + Computer Science

Scope of US proposalScope of US proposal Deploy Grid laboratory with international partnersDeploy Grid laboratory with international partners Acquire Tier2 hardware, Tier2 support personnelAcquire Tier2 hardware, Tier2 support personnel Integrate of Grid software into applicationsIntegrate of Grid software into applications CS support teams (+ 6 UK Fellows) to harden toolsCS support teams (+ 6 UK Fellows) to harden tools Establish International Grid Operations Center (iGOC)Establish International Grid Operations Center (iGOC) Deploy hardware at 3 minority institutions (Tier3)Deploy hardware at 3 minority institutions (Tier3)


US iVDGL Proposal Participants

T2/Software

CS support

T3/Outreach

T1/Labs

U FloridaU Florida CMSCMS CaltechCaltech CMS, LIGOCMS, LIGO UC San DiegoUC San Diego CMS, CSCMS, CS Indiana UIndiana U ATLAS, ATLAS, iGOCiGOC Boston UBoston U ATLASATLAS U Wisconsin, MilwaukeeU Wisconsin, Milwaukee LIGOLIGO Penn StatePenn State LIGOLIGO Johns HopkinsJohns Hopkins SDSS, NVOSDSS, NVO U ChicagoU Chicago CSCS U Southern CaliforniaU Southern California CSCS U Wisconsin, MadisonU Wisconsin, Madison CSCS Salish KootenaiSalish Kootenai Outreach, LIGOOutreach, LIGO Hampton UHampton U Outreach, ATLASOutreach, ATLAS U Texas, BrownsvilleU Texas, Brownsville Outreach, LIGOOutreach, LIGO FermilabFermilab CMS, SDSS, NVOCMS, SDSS, NVO BrookhavenBrookhaven ATLASATLAS Argonne LabArgonne Lab ATLAS, CSATLAS, CS


iVDGL Partners

National partnersNational partners PPDG (Particle Physics Data Grid )PPDG (Particle Physics Data Grid ) DTF: Distributed Terascale FacilityDTF: Distributed Terascale Facility CAL-IT2 (new California Grid initiative)CAL-IT2 (new California Grid initiative)

Current international partnersCurrent international partners EU-DataGridEU-DataGrid UK PPARC funding agencyUK PPARC funding agency UK Core e-Science Program UK Core e-Science Program 6 UK Fellowships 6 UK Fellowships INFN (Italy)INFN (Italy) 2 Japanese institutes2 Japanese institutes 1 Australian institute (APAC)1 Australian institute (APAC)


iVDGL Map Circa 2002-2003

Tier0/1 facility

Tier2 facility

10 Gbps link

2.5 Gbps link

622 Mbps link

Other link

Tier3 facility


iVDGL Requirements

Realistic scaleRealistic scale In number, diversity, distribution, network connectivityIn number, diversity, distribution, network connectivity

Delegated management and local autonomyDelegated management and local autonomy Management needed to operate as large, single facilityManagement needed to operate as large, single facility Autonomy needed for sites and experimentsAutonomy needed for sites and experiments

Support large-scale experimentationSupport large-scale experimentation To provide useful information for building real Data GridsTo provide useful information for building real Data Grids

Robust operationRobust operation For long running applications in complex environmentFor long running applications in complex environment

Instrumentation and monitoringInstrumentation and monitoring Required for an experimental facilityRequired for an experimental facility

Integration with international “cyberinfrastructure”Integration with international “cyberinfrastructure” ExtensibilityExtensibility


Approach

Define a laboratory architectureDefine a laboratory architecture Define expected laboratory functionsDefine expected laboratory functions Build in scalability, extensibility, reproducibilityBuild in scalability, extensibility, reproducibility Define instrumentation, monitoringDefine instrumentation, monitoring Establish CS support teams (develop/harden tools, support users)Establish CS support teams (develop/harden tools, support users) Define working relationship, coordination with partnersDefine working relationship, coordination with partners

Create and operate global-scale laboratoryCreate and operate global-scale laboratory Deploy hardware, software, personnel at Tier2, Tier3 sitesDeploy hardware, software, personnel at Tier2, Tier3 sites Establish iGOC, single point of contact for monitoring, support, …Establish iGOC, single point of contact for monitoring, support, … Help international partners establish sitesHelp international partners establish sites

Evaluate and improve iVDGL through experimentationEvaluate and improve iVDGL through experimentation CS support teams will work with experimentsCS support teams will work with experiments Extend results to partnersExtend results to partners

Engage underrepresented groupsEngage underrepresented groups Integrate minority institutions as Tier3 sitesIntegrate minority institutions as Tier3 sites


iVDGL as a Laboratory

Grid ExercisesGrid Exercises ““Easy”, intra-experiment tests first (10-30%, national, Easy”, intra-experiment tests first (10-30%, national,

transatlantic)transatlantic) ““Harder” wide-scale tests later (30-100% of all resources)Harder” wide-scale tests later (30-100% of all resources) CMS is already conducting transcontinental simulation CMS is already conducting transcontinental simulation

productionsproductions Operation as a facilityOperation as a facility

Common software, central installation to ensure Common software, central installation to ensure compatibilitycompatibility

CS teams to “harden” tools, support applicationsCS teams to “harden” tools, support applications iGOC to monitor performance, handle problemsiGOC to monitor performance, handle problems


Emphasize Simple Operation

““Local” control of resources vitally importantLocal” control of resources vitally important (Site level or national level)(Site level or national level) Experiments, politics demand itExperiments, politics demand it

Operate mostly as a “partitioned” testbedOperate mostly as a “partitioned” testbed (Experiment, nation, etc.)(Experiment, nation, etc.) Avoids excessive coordinationAvoids excessive coordination Allows software tests in different partitionsAllows software tests in different partitions

Hierarchy of operation must be definedHierarchy of operation must be defined E.g., (1) National + experiment, (2) inter-expt., (3) E.g., (1) National + experiment, (2) inter-expt., (3)

global testsglobal tests


Other Disciplines

Use by other disciplinesUse by other disciplines Expected to be at the 10% levelExpected to be at the 10% level Other HENP experimentsOther HENP experiments Virtual Observatory (VO) community in Europe/USVirtual Observatory (VO) community in Europe/US Gravity wave community in Gravity wave community in

Europe/US/Australia/JapanEurope/US/Australia/Japan Earthquake engineeringEarthquake engineering BioinformaticsBioinformatics Our CS colleagues (wide scale tests)Our CS colleagues (wide scale tests)


US iVDGL Management and Coordination

Project DirectorsAvery, Foster

Project Coordination GroupProject Coordinator

Project DirectorsCoordinators of Systems Integration, Education/Outreach

Physics Experiment RepresentativesUniversity Research Center or Group Representatives

PACI Representatives

iVDGL Design and Deployment

Integration with Applications

University Research Centers / Groups

International Grid Operations Center

Collaboration Board (Advisory)

External Advisory Board


Conclusion

Grid involvement offers many challenges and Grid involvement offers many challenges and opportunities.opportunities.

PPDG, and iVDGL are complementary in their PPDG, and iVDGL are complementary in their approach and deliverables.approach and deliverables.

These efforts, along with our European partners These efforts, along with our European partners will provide exciting new ways to share data and will provide exciting new ways to share data and computing resources in the future.computing resources in the future.

Acknowledgements: Richard Mount (SLAC) and Acknowledgements: Richard Mount (SLAC) and Paul Avery (University of Florida), Ruth Pordes Paul Avery (University of Florida), Ruth Pordes (FNAL). (FNAL).

Documents

US Grid Efforts Lee Lueking D0 Remote Analysis Workshop February 12, 2002