Oxford UpdateHEPix

Pete GronbechGridPP Project Manager

October 2014

Oxford Particle Physics - Overview

• Oxford University has one of the largest Physics Departments in the UK– ~450 staff and 1100 students.

• Particle Physics sub department is the largest in the UK

• We now support Windows, Linux (Ubuntu) and Mac on the desktop across the department.– Ubuntu - Cobbler used to install base system, Cfengine used

for package control and configuration.• Two computational clusters for the PP physicists.

– Grid Cluster part of the SouthGrid Tier-2– Local Cluster (AKA Tier-3)

• A common Cobbler and Puppet system is used to install and maintain all the SL6 systems.

Lincoln - October 20142

UK GridPP Computing Structure

• One Tier-1 center - RAL• 18 University Particle Physics Departments• Each is part of a regional Grid Tier-2 center• Most have some local computing clusters (AKA Tier-3)

• Oxford is part of SouthGrid. • SouthGrid is comprised of all the non London based

sites in the South of the UK.• Birmingham, Bristol, Cambridge, JET, Oxford, RAL PPD,

Sussex.

Lincoln - October 20143

UKI Tier-1 & Tier-2 contributions

The UK the second largest contributor to the WLCG. (~11% cf. 28% for USA)

Accounting for the last year.

Tier-1 accounts for~31%

Tier-2s share as below

4Lincoln - October 2014

UK Grid Job Mix

Lincoln - October 20145

Lincoln - October 20146

UK Tier-2 reported CPU – Historical View to present

Comparing last update in 2012.

Lincoln - October 20147

SouthGrid SitesAccounting as reported by

APEL

Oxford Grid Cluster

• Upgrades over the last year to increase the capacity of the storage and some modest CPU upgrades.

• 15 Dell 720XDs servers (12*4TB raw capacity). Note we used SATA not SAS this time. Unlikely to do this in the future. (SAS becoming the default and support for SATA costs extra). SE is running DPM.

• Three ‘twin-squared’ Viglen Supermicro HX525T2i worker nodes have been installed. Intel E5-2650v2 8 core (16 Hyper-threaded cores each) provides 384 job slots with 2GB RAM.

• Two thirds of the Grid Cluster now running HT Condor behind ARC CE. Remaining third running legacy torque/maui driven by CREAM CE.

Lincoln - October 20148

Current capacity 16,768HS06 1300TB

Intel E5-2650 v2 SL6 HEPSPEC06

Average result 361.429% improvement over 2650 v1on SL5

Lincoln - October 20149

Power Usage – Twin squared chassis

Max 1165W

Idle 310W

Lincoln - October 201410

Oxford’s Grid Cluster

11Lincoln - October 2014

Begbroke Computer Room

12Lincoln - October 2014

Local Computer room showing PP cluster & cold

aisle containment

13

• Very similar h/w to the Grid Cluster.• Same Cobbler and Puppet management

setup.• Lustre used for larger groups• Capacity:- 7192 HS06, 716TB

Lincoln - October 2014

Networking

~900 MB/s = 7.2 Gbps.

• The University had a 10Gbit link to JANET with a 10 Gbit failover link.

• A third link was added and the Grid traffic routed exclusively down that in August 2013.

• Plots from March 2014 (Atlas transfer from BNL)

Lincoln - October 201414

• CMS Tier-3– Supported by RALPPD’s PhEDEx server – Useful for CMS, and for us, keeping the site busy in quiet times– However can block Atlas jobs and during accounting period not so desirable

• ALICE Support– There is a need to supplement the support given to ALICE by Birmingham.– Made sense to keep this in SouthGrid so Oxford have deployed an ALICE VO box

• UK Regional Monitoring– Oxford runs the nagios based WLCG monitoring for the UK– These include the Nagios server itself, and support nodes for it, SE, MyProxy and WMS/LB– Multi VO Nagios Monitoring added two years ago.

• IPv6 Testing– We take leading part in the IPv6 testing, many services enabled and tested by the community.– perfSONAR IPv6 enabled. RIPE Atlas probe also on IPv6.

• Cloud Development– Openstack test setup (Has run Atlas jobs)– VAC setup (LHCb, Atlas & GridPP Dirac server jobs)

Other Oxford Work

Lincoln - October 201415

Lincoln - October 201416

Conclusions

• Recent hardware purchases have provided both storage capacity and CPU performance improvements.

• Good Network connectivity• Solid computer rooms.• Medium sized Grid site but have involvement in many

development projects.

HEPix Spring 2015 is coming to Oxford

17

Other Oxford Attractions!

18

Including Oxford Physics

Lincoln - October 201419