[email protected] HPC in the UK: An Update Alan Gray, EPCC, The University of Edinburgh HPC User Forum, EPFL, October 2009

[email protected]

HPC in the UK: An Update

Alan Gray, EPCC, The University of Edinburgh

HPC User Forum, EPFL, October 2009

Contents

• EPCC

• UK National Services: HECToR and HPCx

• Case Studies– Environmental Modelling– Computational Materials Chemistry– Fractal Generated Turbulent Flows– Interactive Biomolecular Modelling– FireGrid: Next Generation Emergency Response Systems

Contents

• EPCC



EPCC

• The University of Edinburgh founded EPCC in 1990 to act as

the focus for its interests in simulation

• Today, EPCC is a leading centre for computational science in Europe– ~70 permanent staff– Working in academia and industry

• Managing both UK national HPC

facilities– HECToR: Cray XT5h– HPCx: IBM Power5 eServer

TechnologyTransfer

Training

European Coordination

HPCResearchFacilities

VisitorProgramme

4

Contents

• EPCC



HECToR

• HECToR: High End Computing Terascale Resource– 6 year service, funded by UK government. Commenced 2007– Used for wide variety of apps across academia and industry– Located at University of Edinburgh, managed & operated by EPCC

(with help from Daresbury Laboratory staff)

• XT4: 5664 quad-core Opterons– peak performance 208 TFlops

• X2: 112 Cray Vector Processors

Cray XT5h

6

HECToR Upgrade path

• Currently in Phase 2a

• Q1 2010: Cray ‘Baker’ (Phase 2b, stage 1)– 20 cabinets; 3612 AMD ‘Magny Cours’ 12-core chips, 44,544 cores

total – estimated peak performance of 338 TFlops.– ~30 cabinets of XT4 will be retained.

• Q4 2010: upgrade Baker to Gemini network (Phase 2b, stage

2)

• Q3 2011: Phase 3. ?????– No Hardware provision contract at this time

7

HPCx

• Located at Daresbury, managed and operated by EPCC and Daresbury

• 160 IBM e-Server p575 nodes– 16 Power5 1.5 GHz cores per node: 2560 cores total– IBM HPS interconnect (aka Federation)– 12.9 TFLOP/s Linpack

8

• UK policy is to have overlapping HPC services

• HPCx: previously main service, now in operation as secondary service

Complementarity

• Managing both HECToR and HPCx simultaneously provided

unique opportunity to maximise benefits for UK research.

Run as “Complementary services”:– HECToR is our leading HPC facility – HPCx is our "National Supercomputer", trading overall utilisation in

favour of a more flexible service– Very long jobs– Interactive use (computational steering, visualisation, debugging,

etc)– Advanced reservations– …

• Gain experience to advantage of future HPC services.

9

Contents

• EPCC



Environmental modelling

Lois Steenman-Clark, University of Reading

• HIGEM: seven UK academic groups plus UK Met Office– Aim: achieve a major advance in developing an Earth System model

of unprecedented resolution– capable of performing multi-century simulations.

• Increasing horizontal resolution of Earth System models

allows capture of climate processes and weather systems in

much greater detail. – but scientifically challenging

– new model created, tested, analysed, assessed, tuned and optimised

– The control experiment, 115 model years of HIGEM, run on HPCx

11

Environmental modelling

• sea surface temperature anomalies

associated with El-Nino events from – a) an observational climatology– b) the HIGEM control run– c) standard climate resolution UM

experiments

• HIGEM model now regularly used in

current research projects – will be used for some very high resolution

experiments as part of the input to the next IPCC (International Panel on Climate Change) report due in 2013.

12

Computational Materials Chemistry

Richard Catlow and Scott Woodley, University College London

• Materials Chemistry Consortium, comprises over 25 research

groups

• extensive applications portfolio, – energy and environmental materials, – catalysis and surface science, – quantum devices, – nano-science – biomaterials

• Highest users of HPCx by project over lifetime of service.– Currently heavily utilising HECToR

13

Computational Materials ChemistryEnergy and Environmental Materials

• Modelling radiation damage in materials using DL_POLY MD code– assessment and design of materials for use in nuclear reactors.

• Implemented effects of electronic stopping and electron–ion interactions

within radiation damage simulations of metals,

• investigated the evolution of the damage on annealing for SiO2, GeO2,

TiO2, Al2O3, and MgO,

Simulation of damage created by 50 keV recoil

atom in quartz.

14


Biomaterials

• explored fundamental factors relating to the structure of

bone, in particular the interface between apatite and

collagen.

MD simulation of the nucleation of hydroxyapatite in an aqueous environment at a collagen template, showing the clustering of the calcium and phosphate ions around the collagen functional groups.

15


Nano-Chemistry and Nucleation

• rapidly expanding area: exploiting computational tools to develop models

for the structures, properties and reactivities of nano-particulate matter.

• explored possible structures and properties of such nanoparticles, as

well as how particularly stable particles can be employed as building

blocks

Stable octahedral clusters are connected to create

microporous crystals

16

Fractal-Generated Turbulent Flows

Professor Christos Vassilicos, Imperial College London

• New industrial fluid flow solutions urgently needed to meet

unprecedented requirements– Increase energy savings, reduce environmental impacts– Industries which want to create or minimise turbulence

– Aerospace and automotive industries– Reduce noise, fuel consumption, pollutant emissions.

– Chemical and process industries– use turbulence for mixing

• New flow concept originating from UK: turbulent flows generated by

fractal grids– create intense turbulence with very little effort or power input

– only need very small changes to the grid to have enormous effect. – Size of simulations is so large that they are impossible without HPC

17


• Fractal Square

Grid– Fluids pass

through grid, turbulence created

18


• The first ever successful simulations of turbulence generated

by fractal grids performed on HECToR in 2008 and 2009.– Used Incompact3d code

19

Streamwise velocity in one of the planes normal to a turbulence-generating fractal square grid. (From Laizet & Vassilicos 2009.)

Interactive Biomolecular Modeling

Carmen Domene, University of Oxford

• Aim: further understanding of “ion channel“ proteins within

nervous system– These regulate ion flow through the cellular membrane, exerting

control on electrical signals in cells– dysfunction can cause diseases in muscles, kidney, heart or bones.– Improved understanding may lead to better drugs and treatment

• Use Computational Steering to manipulate simulation by

hand, to create specific starting configurations of interest.– IMD: VMD (run on user PC) connects to NAMD (on back end of

HPCx). Can manipulate molecules by hand.– Allows simulation of rare, but possible, mechanisms

20

Interactive Biomolecular Modeling

• Work has demonstrated that alternative pathways for ion conduction to

the one already proposed in the literature are possible

• Considering these results, it would be also interesting to revisit many of

the models proposed in the literature which did not successfully agree

with experimental data

21

FireGrid: Next Generation Emergency Response systems

• Retrospective analysis of every emergency poses question:

Was the response adequate?• Almost invariably the answer is:

Better information would have led to more effective response.

22

• Brutally illustrated when emergency

crews continued operations

oblivious to impending collapse of

WTC1 and WTC2.


Dense Sensor network for early detection and monitoring

Emergency Response

Super Real Time Simulationof fire growth and structure response

Incident Commander

Building Command and Control

Largedatabases

Alerts

23


• Partners:

– The University of Edinburgh: R&D for all areas of the project

– EPCC, the Institute for Infrastructure and Environment, the Institute for Digital

Communication, the National e-Science Centre, and the Artificial Intelligence

Applications Institute

– BRE (Building Research Establishment) project leader and also provided the state-of-

the-art experimental facilities that housed the fire

– ABAQUS UK Limited and ANSYS-CFX (structural mechanics and CFD software)

– Xtralis expertise on active fire protection systems, as well as sensor equipment in

support of experiments;

– the London Fire Brigade: principal user and guided the development of the command

and control interface.

• Initial project has completed

– prototype Integrated Emergency Response System

– Successful live demos (with real fires) utilised HPCx and local Edinburgh University

cluster HPC resources.

24

References

• EPCC: www.epcc.ed.ac.uk

• HECToR: www.hector.ac.uk

• HPCx: www.hpcx.ac.uk

• Environmental Modelling“Moving the capability boundaries”, Lois Steenman-Clark, HPCx Capability

Computing Issue 13, http://www.hpcx.ac.uk/about/newsletter/

• Computational Materials Chemistry“Computational materials Chemistry on HPCx”, Richard Catlow and Scott

Woodley, HPCx Capability Computing Issue 13, http://www.hpcx.ac.uk/about/newsletter/

• Fractal Generated Turbulent FlowsTo appear under “Casestudies” section at www.hector.ac.uk

• Interactive Biomolecular Modelling“Interactive Biomolecular Modeling with VMD and NAMD at HPCx”, Carmen

Domene, HPCx Capability Computing Issue 13, http://www.hpcx.ac.uk/about/newsletter/

• FireGrid: Next Generation Emergency Response Systemshttp://www.epcc.ed.ac.uk/research-collaborations/casestudies/firegrid ;http://www.firegrid.org/

Documents

[email protected] HPC in the UK: An Update Alan Gray, EPCC, The University of Edinburgh HPC User Forum, EPFL, October 2009