Upload
berniece-hamilton
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Dynamically Creating Big Data Centers for the LHC
Frank Würthwein
Professor of PhysicsUniversity of California San Diego
September 25th, 2013
Frank Wurthwein - ISC Big Data 2
Outline
• The Science• Software & Computing Challenges• Present Solutions• Future Solutions
September 25th 2013
The Science
Frank Wurthwein - ISC Big Data 4
~67% of energy is “dark energy”
~29% of matter is “dark matter”
All of what we know makes upOnly about 4% of the universe.
We have some ideas but no proof of what this is!
We got no clue what this is.
The Universe is a strange place!
September 25th 2013
Frank Wurthwein - ISC Big Data 5
To study Dark Matter we need to
create it in the laboratory
September 25th 2013
Mont Blanc
Lake Geneva
ALICE
ATLAS
LHCb
CMS
Frank Wurthwein - ISC Big Data 7
“Big bang” in the laboratory
• We gain insight by colliding particles at the highest energies possible to measure:– Production rates– Masses & lifetimes– Decay rates
• From this we derive the “spectroscopy” as well as the “dynamics” of elementary particles.
• Progress is made by going to higher energies and brighter beams.
September 25th 2013
Frank Wurthwein - ISC Big Data 8
Explore Nature over 15 Orders of magnitudePerfect agreement between Theory & Experiment
Dark Matter expectedsomewhere below this line.
September 25th 2013
Frank Wurthwein - ISC Big Data 9
And for the Sci-Fi Buffs …Imagine our 3D world to beconfined to a 3D surface ina 4D universe.
Imagine this surface to be curved such that the 4th Ddistance is short for locationslight years away in 3D.
Imagine space travel bytunneling through the 4th D.
The LHC is searching for evidence of a 4th dimension of space.
September 25th 2013
Frank Wurthwein - ISC Big Data 10
Recap so far …
• The beams cross in the ATLAS and CMS detectors at a rate of 20MHz
• Each crossing contains ~10 collisions• We are looking for rare events that are
expected to occur in roughly 1/10000000000000 collisions, or less.
September 25th 2013
Software & ComputingChallenges
The CMS Experiment
Frank Wurthwein - ISC Big Data 13
The CMS Experiment• 80 Million electronic channels
x 4 bytesx 40MHz-----------------------~ 10 Petabytes/sec of informationx 1/1000 zero-suppressionx 1/100,000 online event filtering------------------------~ 100-1000 Megabytes/sec raw data to tape1 to 10 Petabytes of raw data per yearwritten to tape, not counting simulations.
• 2000 Scientists (1200 Ph.D. in physics)– ~ 180 Institutions– ~ 40 countries
• 12,500 tons, 21m long, 16m diameter
September 25th 2013
Frank Wurthwein - ISC Big Data 14
Active Scientists in CMS
September 25th 2013
5-40% of the scientific members are actively doing large scale data analysis in
any given week.
~1/4 of the collaboration,scientists and engineers,
contributed to the common source code of ~3.6M C++ SLOC.
Frank Wurthwein - ISC Big Data 15
Evolution of LHC Science Program
150Hz 1000Hz 10000HzEvent Rate written to tape
September 25th 2013
Frank Wurthwein - ISC Big Data 16
The Challenge
How do we organize the processing of 10’s to 1000’s of Petabytes of data by a globally distributed community
of scientists, and do so with manageable “change costs” for the next 20 years ?
Guiding Principles for SolutionsChose technical solutions that allow
computing resources as distributed as human resources.Support distributed ownership and control,
within a global single sign-on security context.Design for heterogeneity and adaptability.
September 25th 2013
Present Solutions
Frank Wurthwein - ISC Big Data 18September 25th 2013
Federation of National Infrastructures. In the U.S.A.: Open Science Grid
Frank Wurthwein - ISC Big Data 19September 25th 2013
Among the top 500 supercomputers there are only two that are bigger when
measured by power consumption.
Frank Wurthwein - ISC Big Data 20
Tier-3 Centers
• Locally controlled resources not pledged to any of the 4 collaborations.– Large clusters at major research Universities that are time
shared.– Small clusters inside departments and individual research
groups.
• Requires global sign-on system to be open for dynamically adding resources.– Easy to support APIs– Easy to work around unsupported APIs
September 25th 2013
Frank Wurthwein - ISC Big Data 21
Me -- My friends -- The grid/cloud
O(104) Users
O(102-3) Sites
O(101-2) VOs
Thin client
Thin “Grid API”
Thick VOMiddleware& Support
Me
My friends
The anonymousGrid or Cloud
Domain science specific Common to all sciencesand industry
September 25th 2013
Frank Wurthwein - ISC Big Data 22
“My Friends” Services
• Dynamic Resource provisioning• Workload management
– schedule resource, establish runtime environment, execute workload, handle results, clean up
• Data distribution and access– Input, output, and relevant metadata
• File catalogue
September 25th 2013
Frank Wurthwein - ISC Big Data 23
Optimize Data Structure for Partial Reads
September 25th 2013
Frank Wurthwein - ISC Big Data 24
Fraction of a file that is read
September 25th 2013
# of
file
s re
ad
For vast majority of files, less than 20% of the file is read.
20%
Average 20-35%Median 3-7%
(depending on type of file)
Overflow bin
Future Solutions
Frank Wurthwein - ISC Big Data 26
From present to future• Initially, we operated a largely static system.
– Data was placed quasi-static before it can be analyzed.– Analysis centers have contractual agreements with the collaboration.– All reconstruction is done at centers with custodial archives.
• Increasingly, we have too much data to afford this.– Dynamic data placement
• Data is placed at T2s based on job backlog in global queues.– WAN access: ”Any Data, Anytime, Anywhere”
• Jobs are started on the same continent as the data instead of the same cluster attached to the data.
– Dynamic creation of data processing centers• Tier-1 hardware bought to satisfy steady state needs instead of peak needs.• Primary processing as data comes off the detector => steady state• Annual Reprocessing of accumulated data => peak needs
September 25th 2013
Frank Wurthwein - ISC Big Data 27
Any Data, Anytime, Anywhere
September 25th 2013
Global redirection system to unify all CMS data into one globally accessible namespace.
Is made possible by paying careful attention to IO layerto avoid inefficiencies due to IO related latencies.
Frank Wurthwein - ISC Big Data 28
Vision going forward
Implemented vision for 1st time in Spring 2013using Gordon Supercomputer at SDSC.
September 25th 2013
Frank Wurthwein - ISC Big Data 29September 25th 2013
30
CMS “My Friends” Stack• CMSSW release environment
– NFS exported from Gordon IO nodes– Future: CernVM-FS via Squid caches
• J. Blomer et al.; 2012 J. Phys.: Conf. Ser. 396 052013
• Security Context (CA certs, CRLs) via OSG worker node client• CMS calibration data access via FroNTier
• B. Blumenfeld et al; 2008 J. Phys.: Conf. Ser. 119 072007– Squid caches installed on Gordon IO nodes
• glideinWMS• I. Sfiligoi et al.; doi:10.1109/CSIE.2009.950
– Implements “late binding” provisioning of CPU and job scheduling– Submits pilots to Gordon via BOSCO (GSI-SSH)
• WMAgent to manage CMS workloads• PhEDEx data transfer management
– Uses SRM and gridftp
September 25th 2013 Frank Wurthwein - ISC Big Data
Job
enviro
nm
ent
Data an
d Jo
bh
and
ling
31
CMS “My Friends” Stack• CMSSW release environment
– NFS exported from Gordon IO nodes– Future: CernVM-FS via Squid caches
• J. Blomer et al.; 2012 J. Phys.: Conf. Ser. 396 052013
• Security Context (CA certs, CRLs) via OSG worker node client• CMS calibration data access via FroNTier
• B. Blumenfeld et al; 2008 J. Phys.: Conf. Ser. 119 072007– Squid caches installed on Gordon IO nodes
• glideinWMS• I. Sfiligoi et al.; doi:10.1109/CSIE.2009.950
– Implements “late binding” provisioning of CPU and job scheduling– Submits pilots to Gordon via BOSCO (GSI-SSH)
• WMAgent to manage CMS workloads• PhEDEx data transfer management
– Uses SRM and gridftp
September 25th 2013 Frank Wurthwein - ISC Big Data
Job
enviro
nm
ent
Data an
d Jo
bh
and
ling
This is clearly mighty complex !!!
So let’s focus only on the parts that are specific to incorporating
Gordon as a dynamic data processing center.
Frank Wurthwein - ISC Big Data 32September 25th 2013
Items in red were deployed/modified to incorporate Gordon
BO
SC
O
Minor mod of PhEDEx config file
Deploy SquidExport CMSSW
& WN client
Frank Wurthwein - ISC Big Data 33
Gordon Results
• Work completed in February/March 2013 as a result of a “lunch conversation” between SDSC & US-CMS management– Dynamically responding to an opportunity
• 400 Million RAW events processed– 125 TB in and ~150 TB out– ~2 Million core hours of processing
• Extremely useful for both science results as well as proof of principle in software & computing.
September 25th 2013
Frank Wurthwein - ISC Big Data 34
Summary & Conclusions
• Guided by the principles:– Support distributed ownership and control in a
global single sign-on security context.– Design for heterogeneity and adaptability
• The LHC experiments very successfully developed and implemented a set of new concepts to deal with BigData.
September 25th 2013
Frank Wurthwein - ISC Big Data 35
Outlook• The LHC experiments had to largely invent an
island of BigData technologies with limited interactions with industry and other domain sciences.
• Is it worth building bridges to other islands ?– IO stack and HDF5 ?– MapReduce ?– What else ?
• Is there a mainland emerging that is not just another island ?
September 25th 2013