Upload
eurocloud
View
1.379
Download
0
Tags:
Embed Size (px)
Citation preview
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES CERN's Cloud Computing infrastructure
CERN's Cloud Computing Infrastructure
Tony Cass, Sebastien Goasguen, Belmiro Moreira, Ewan Roche, Ulrich Schwickerath, Romain Wartel
Cloudview conference, Porto, 2010
See also related presentations: HEPIX spring and autumn meeting 2009, 2010 Virtualization vision, Grid Deployment Board (GDB) 9/9/2009 Batch virtualization at CERN, EGEE09 conference, Barcelona
CERNs Cloud computing infrastructure - a status report - 2
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Outline and disclaimer
An introduction to CERN
Why virtualization and cloud computing ?
Virtualization of batch resources at CERN
Building blocks and current status
Image management systems: ISF and ONE
Status of the project and first numbers
Disclaimer: We are still in the testing and evaluation phase. No final decision has been taken yet on what we are going to use in the future.
All given numbers and figures are preliminary
CERNs Cloud computing infrastructure - a status report - 3
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES
European Organization for Nuclear Research
The world’s largest particle physics laboratory
Located on Swiss/French border
Funded/staffed by 20 member states in 1954
With many contributors in the USA
Birth place of World Wide Web
Made popular by the movie “Angels and Demons”
Flag ship accelerator LHC
http://www.cern.ch
Introduction to CERN
CERNs Cloud computing infrastructure - a status report - 4
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Introduction to CERN: LHC and the experiments
LHCb Alice
ATLAS CMS
Circumference of LHC: 26 659 mMagnets : 9300Temperature: -271.3°C (1.9 K)Cooling: ~60T liquid HeMax. Beam energy: 7TeVCurrent beam energy: 3.5TeV
TOTEMLHCf
CERNs Cloud computing infrastructure - a status report - 5
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Introduction to CERN
Data Signal/Noise ratio 10-9
Data volume:High rate * large number of channels * 4 experiments
15 PetaBytes of new data each year
Compute powerEvent complexity * Nb. events * thousands users
100 k CPUs (cores)Worldwide analysis & fundingComputing funding locally in major regions & countriesEfficient analysis everywhere
GRID technology
CERNs Cloud computing infrastructure - a status report - 6
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES LCG computing GRID
Number of sites:
T0: 1 (CERN), 20% T1: 11 round the worldT2: ~160
http://lcg.web.cern.ch/lcg/public/
Requried computing capacity:~100 000 processors
CERNs Cloud computing infrastructure - a status report - 7
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES The CERN Computer Center
Disk and tape: 1500 disk servers5PB disk space16PB tape storage
Computing facilities:>20.000 CPU cores (batch only)Up to ~10000 concurrent jobsJob throughput ~200 000/day
http://it-dep.web.cern.ch/it-dep/communications/it_facts__figures.htm
CERNs Cloud computing infrastructure - a status report - 8
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Why virtualization and cloud computing ?
Service consolidation:
Improve resource usage by squeezing mostly unused machines onto single big hypervisors Decouple hardware life cycle from applications running on the box Ease management by supporting life migration
Virtualization of batch resources:
Decouple jobs and physical resources Ease management of the batch farm resources Enable the computer center for new computing models
This presentation is about virtualization of batch resources only
CERNs Cloud computing infrastructure - a status report - 9
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Batch virtualization
Type 1: Run my jobs in your VM
Type 2: Run my jobs in my VM
CERN batch farm lxbatch:~3000 physical hosts~20000 CPU cores>70 queues
CERNs Cloud computing infrastructure - a status report - 10
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES
Type 3:Give me my infrastructurei.e a VM or a batch of VMs
Towards cloud computing
CERNs Cloud computing infrastructure - a status report - 11
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Philosophy
Internal cloud
development other/cloud applicationsT0
I
(far) future ?
hypervisor cluster
Batch
Physical SLC4 WN
Physical SLC5 WN
SLC4 WN SLC5 WN
Batch
Near future: (
(SLC = Scientific Linux CERN)
CERNs Cloud computing infrastructure - a status report - 12
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Visions beyond the current plan
Reusing/sharing images between different sites (phase 2) HEPIX working group founded in autumn 2009 to define rules and boundary conditions (https://www.hepix.org/)
Experiment specific images (phase 3)
Use of images which are customized for specific experiments
Use of resources in a cloud like operation mode (phase 4)
Images directly join experiment controlled scheduling systems (phase 5)
Controlled by experiment Spread across sites
CERNs Cloud computing infrastructure - a status report - 13
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Virtual batch: basic ideas
Platform Infrastructure Sharing Facility (ISF)
For high level VM management
Virtual batch worker nodes:
Clones of real worker nodes, same setup
Mix with physical resources
Dynamically join the batch farm as normal worker nodes
Limited lifetime: stop accepting jobs after 24h
Destroy when empty
Only one user job per VM at a time
Note: The limited lifetime allows for a fully automated system which dynamically adapts to the current needs, and automatically deploys intrusive updates.
CERNs Cloud computing infrastructure - a status report - 14
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Virtual batch: basic ideas, technical
Platform Infrastructure Sharing Facility (ISF)
For high level VM management
Images:
staged on hypervisors
master images, instances use LVM snapshots
Start with few different flavors only
Image creation:
Derived from a centrally managed “golden node”
Regularly updated and distributed to get updates “in”
CERNs Cloud computing infrastructure - a status report - 15
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Virtual batch: basic ideas
Platform Infrastructure Sharing Facility (ISF)
For high level VM management
Images distribution:
Only shared file system available is AFS
Prefer peer to peer methods (more on that later)
SCP wave
Rtorrent
CERNs Cloud computing infrastructure - a status report - 16
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Virtual batch: basic ideas
Platform Infrastructure Sharing Facility (ISF)
For high level VM management
VM placement and management system
Use existing solutions
Testing both a free and a commercial solution
OpenNebula (ONE)
Platform's Infrastructure Sharing Facility (ISF)
CERNs Cloud computing infrastructure - a status report - 17
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Batch virtualization: architecture
Batch systemmanagement
VM worker nodesWith limited lifetime VM management system
Hypervisors / HW resources
VM kiosk
Centrally managedGolden nodes
Job submissionCE/interactive
Grey: physical resourceColored: different VMs
CERNs Cloud computing infrastructure - a status report - 18
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Status of building blocks (test system)
Submission and batch managemen
Hypervisor cluster
VM kiosk and image distribution
VM management system
Initial deployment OK OK OK OK
Central management OK OK Mostly
implemented
ISF OK, ONE
missing
Monitoring and alarming OK Switched off
for testsmissing missing
CERNs Cloud computing infrastructure - a status report - 19
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Image distribution: SCP wave versus rtorrent
Preliminary !
Slow nodes,(under investigation)
(BT = bit torrent)
CERNs Cloud computing infrastructure - a status report - 20
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES VM placement and management
OpenNebula (ONE): Basic model:
Single ONE master
Communication with hypervisors via ssh only
(Currently) no special tools on the hypervisors
Some scalability issues at the beginning (50 VM at the beginning)
Addressing issues as they turn up
Close collaboration with developers, ideas for improvements
Managed to start more than 7,500 VMs
CERNs Cloud computing infrastructure - a status report - 21
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Scalability tests: some first numbers
“One shot” test with OpenNebula: Inject virtual machine requests And let them die Record the number of alive machines seen by LSF every 30s
Units: 1h5min
CERNs Cloud computing infrastructure - a status report - 22
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES VM placement and management
Platforms Infrastructure Sharing Facility (ISF)
One active ISF master node, plus fail-over candidates
Hypervisors run an agent which talks to XEN
Needed to be packaged for CERN
Resource management layer similar to LSF
Scalability expected to be good but needs verification
Tested with 2 racks (96 machines) so far, ramping up
Filled with ~2.000 VMs so far (which is the maximum)
See: http://www.platform.com
CERNs Cloud computing infrastructure - a status report - 23
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Screen shots: ISF
Status: 1 of 2 racks available
VM status:
Note: one out of 2 racks enabled for demonstration purpose
CERNs Cloud computing infrastructure - a status report - 24
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Screen shots: ISF
CERNs Cloud computing infrastructure - a status report - 25
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Summary
Virtualization efforts at CERN are proceeding. Still some work to be done.
Main challenges
Scalability considerations
provisioning systems
of the batch system
No decision on provisioning system to be used yet
Reliability and speed of image distribution
General readiness for production (hardening)
Seamless integration into the existing infrastructure
CERNs Cloud computing infrastructure - a status report - 26
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Outlook
What's next ?
Continue testing of ONE and ISF in parallel
Solve remaining (known) issues
Release first VMs for testing by our users soon
CERNs Cloud computing infrastructure - a status report - 27
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Questions ?
?
CERNs Cloud computing infrastructure - a status report - 28
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Philosophy
Hypervisor cluster(physical resources)
OpenNebula
Platform Infrastructure Sharing Facility (ISF)
For high level VM management
pVMO
VM provisioning system(s)
CERNs Cloud computing infrastructure - a status report - 29
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Details: Some additional explanations ...
“Golden node”:A centrally managed (i.e. Quattor controlled) standard worker node
which Is a virtual machine Does not accept jobs Receives regular updates
Purpose: creation of VM images
“Virtual machine worker node”: A virtual machine derived from a golden node Not updated during their life time Dynamically adds itself to the batch farm Accepts jobs for only 24h Runs only one user job at a time Destroys itself when empty
CERNs Cloud computing infrastructure - a status report - 30
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES VM kiosk and Image distribution
Boundary conditions at CERN
Only available shared file system is AFS Network infrastructure with a single 1GE connection
No dedicated fast network for transfers that could be used (eg 10GE, IB or similar)
Tested options:
Scp wave: Developed at Clemson university Based on simple scp
rtorrent: Infrastructure developed at CERN Each node starts serving blocks it already hosts
CERNs Cloud computing infrastructure - a status report - 31
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Details: VM kiosk and Image distribution
Local image distribution model at CERN
Approved images are moved to a central image repository (the “kiosk”) Hypervisors check regularly for new images on the kiosk node The new image is transferred to a temporary area on the hypervisors When the transfer is finished, a new LV is created The new image is unpacked into the new LV The current production image is renamed (via lvrename) The new image is renamed to become the production image
Note: may need a sophisticated locking strategy in this process
Virtual machines are instantiated as LVM snapshots of a base image.This is process is very fast
Replacing a production image:
CERNs Cloud computing infrastructure - a status report - 32
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Image distribution with torrrent: transfer speed
7GB file compressed452 target nodes
90% finished after 25minPrelim
inary
Slow nodes, under investigation
CERNs Cloud computing infrastructure - a status report - 33
CERN IT DepartmentCH-1211 Genève 23
Switzerlandwww.cern.ch/it
PES Image distribution: total distribution speed
7GB file compressed452 target nodes
All done after 1.5h
→ Unpacking still needs some tuning !
Preliminary