EPCC News

Issue 70 autumn 2011

2 ePCC becomes nVIDIa CuDa Research Center

3 We are under new management

4 Crayrsquos exascale Research Initiative in europe european exascale software Initiative

5 asynchronous algorithms PlanetHPC research and innovation strategy

6 nu-Fuse nuclear fusion simulations at exascale

7 CResta and the exascale Challenge

8 HPC-europa research visitor programme

10 BonFIRe building service test beds on FIRe

11 sPRInt data analysis on HPC systems

12 DeIsa virtual european supercomputer

14 Data-intensive engineering at ePCC

15 OGsa-DaI Visual Workbench

16 software sustainability Institute

17 understanding the brain

18 simulating rare events

19 eDIm1 Data Intensive Research machine

20 the msc in HPC ten years on

21 empowering research in the humanities

22 aPOs-eu application Performance Optimisation and scalability

23 Overcoming digital exclusion

24 HeCtoR the third and final phase

In this issue

The newsletter of EPCC the supercomputing centre at the University of Edinburgh

New horizonsNew management

at EPCC and a new era

in HPC

When EPCC was founded in 1990 computational simulation was in its infancy and parallel computing was a novel niche technology Today simulation is accepted as the third methodology of science complementing theory and experiment It is used widely in academic research and by industry and commerce for improving products and reducing time to market With the advent of multicore processors parallel computing has entered the mainstream with all of the challenges that this entails

EPCC has been through a number of distinct phases in its history The early days of of the centre were enabled through the Governmentrsquos Parallel Applications Programme which was designed to stimulate the use of parallel computing by industry and commerce By the mid-to-late 1990rsquos our focus had shifted to the European Commissionrsquos Technology Transfer Node network and subsequent projects that sought to cascade the benefits of HPC down to smaller companies using more readily available computers Our third phase focused on the UK e-Science Programme and the explosion of Grid computing research that occurred around the start of the new millennium and through its first decade

Throughout all of this time EPCC has followed a strategy of working with multiple partners from the public and private sectors to deliver innovative projects whilst also delivering National HPC services on behalf of the UK Research Councils

Today EPCC is entering a new phase not just with a new management team but also in terms of where current computing technologies are taking us In 1994 the Cray T3D

service had 512 processors In 2010 the end of the IBM HPCx service had 2500 processors Current systems such as HECToR have tens if not hundreds of thousands of processor cores This is having profound consequences on how we program and use modern high performance computers The multicore era is also affecting desktop computing users As the number of cores in processors has increased on laptop and desktop computers the speed of each core has slowed This means everyone has parallel computing challenges EPCCrsquos skills have never been in greater demand both from the scientific and the industrial domains

In the last year our Exascale Technology Centre with Cray has worked on some of the key technologies required for the next generation of massively parallel computers These computers are expected to contain millions of cores and this challenges all we know about numerical simulation and modelling The challenges are significant the potential benefits are enormous and the opportunities for the future are greater than ever

As EPCCrsquos directors we bring complementary skills in business management project management research management and commercialisation and are ably supported by a small team of Directors with specific responsibilities for key EPCC activities

We thank our former Director Professor Arthur Trew who having led EPCC for sixteen years has taken up a new position as Head of the School of Physics and Astronomy at the University of Edinburgh We wish him well in his new job

It is always interesting to find out what my colleagues at EPCC have been up to and EPCC News offers an opportunity to do just that

We are currently involved in a number of Exascale activities that will help determine how future HPC applications will reach 1018 flops and will also develop the types of systems and tooling required to make this possible For example Nu-FuSE (page 6) which will model the impact of the fusion process on the materials used to build fusion reactors will require Exascale performance

Cloud computing activities are discussed in the BonFIRE article (page 10) Several data projects are also currently active with an OGSA-DAI visual workbench (page 15) being developed to facilitate the construction of OGSA-DAI workflows The ADMIRE project which developed a data mining platform has recently come to an end and some of its work is being taken forward by EUDAT (both page 14) Finally EPCC in collaboration with the School of Informatics here in Edinburgh has designed and built a new machine called EDIM1 to investigate how this type of hardware may be used to deal with data-intensive research problems (page 19)

There are some anniversaries our MSc in HPC (page 20) has now been in running for ten years while the Software Sustainability Institute (page 16) headquartered at EPCC has reached its first year Articles for both of these activities discuss how productive they have been

You will find many other topics covered in this issue and I hope you will dip in to find out what is happening at EPCC As a final note we have come under new leadership Mark Parsons and Alison Kennedy have jointly taken on the directorship role - you can read about how they see the centre evolving in their article on the opposite page Arthur Trew who has ably directed EPCC for the last 16 years has recently become Head of the School of Physics amp Astronomy (of which EPCC is a part) here at Edinburgh We thank him for his leadership and wish him well in his new role

The one abiding image that I will take away from my editorial role is that of a million chickens pulling a plough as a metaphor of Exascale computing (see Cray Exascale research story on page 4) I hope you enjoy reading this issue of EPCC News and find it informative

editorial Mario Antonioletti

EPCCrsquos pioneering work in the area of GPU computing has gained it the status of NVIDIA CUDA Research Center [1]

Graphics Processing Units (GPUs) were originally designed to display computer graphics but they have developed into extremely powerful chips capable of handling demanding general-purpose calculations However the GPU architecture is somewhat different to that of the CPU and cannot be programmed with traditional computing languages

In 2007 NVIDIA made GPUs widely accessible with the release of the CUDA hardware and software environment and since then EPCC has been working hard to help researchers and commercial partners overcome the challenges of using this novel architecture to exploit its performance capabilities For example EPCCrsquos expertise in large-scale parallel computing has been combined with the powerful CUDA environment to create an application capable of simulating extremely complex fluid dynamic models on the largest of NVIDIA GPU-accelerated supercomputers allowing cutting-edge research into condensed matter physics [2] Furthermore EPCC is actively involved in the development of new programming techniques which promise to increase future productivity in particular the expansion of the OpenMP directive-based model to support accelerators such as GPUs [3]

Dr Alan Gray who leads EPCCrsquos GPU activities said ldquoWe are very excited by this new partnership with NVIDIA We believe that GPU accelerated architectures are becoming increasingly important computational resources and we aim to work closely with NVIDIA to benefit as wide a range of research areas as possiblerdquo

[1] NVIDIA CUDA Research Center Program httpresearchnvidiacomcontentcuda-research-center-crc-program

[2] A Gray A Hart A Richardson and K Stratford Lattice Boltzmann for Large-Scale GPU Systems to appear in the proceedings of ParCo 2011 (2011) www2epccedacuk~alangpublicationsGRAYParCo2011pdf

[3] J Beyer A Gray A Hart LM Levesque D Oemhke H Richardson lsquoProgramming hybrid architectures with OpenMP accelerator directivesrsquo talk presented at Many Core and Reconfigurable Supercomputing 2011 Conference (2011) slides available at wwwmrsc2011euProgramme

Further examples of EPCCrsquos GPU activities are described in the Spring 2011 issue of EPCC News wwwepccedacuklibrarynewslettersepcc-news

ePCC awarded nVIDIa CuDa Research Center status

ePCC is under new managementAlison Kennedy and Mark Parsons

alison Kennedy and mark Parsons assumed responsibility for the management of ePCC on 1 January 2011 as executive Directors they take the helm at a very exciting time for ePCC

Clockwise from right Arthur Trew Mark Parsons

and Alison Kennedy

Many large scale application codes rely on algorithms for solving sparse systems of linear equations But as core counts in HPC machines seem set to increase over the next few years will this type of algorithm scale up

One of the problems we face is that the algorithms are very synchronous processors frequently have to wait for data from other processes to arrive before they can proceed It turns out that some algorithms will still work albeit with slower convergence if we relax some of the synchronisation and compute using the most recent data which happens to have arrived These asynchronous (or chaotic) algorithms were studied theoretically back in the 1980s but with HPC machines typically having a few hundred processors they didnrsquot have much practical application at the time However as we head towards exascale systems with more than a million cores revisiting these ideas may prove beneficial

EPCC is a partner in a project which aims to do just this together with collaborators from the universities of Manchester Leeds Hull and Strathclyde The project will re-analyse some existing algorithms and try to develop new ones looking at theoretical properties practical implementations on modern

architectures and the potential of these algorithms to tolerate certain types of faults

The project also has two demonstrator applications ndash one in finite element modelling of stress in bones and another in the optimisation of electrical power grids ndash to which these alternative algorithms will be applied Together we hope to find new ways of solving this type of problem on future generations of very parallel but possibly unreliable hardware

This project is funded by the Engineering and Physical Sciences Research Council

asynchronous algorithmsMark Bull

One of the applications used by ASYNC calculates the maximum principal strain in a human mandible undergoing incisor biting using about half a million elements For more detail see F Groumlning et al 2011 ldquoWhy do humans have chins Testing the mechanical significance of modern human symphyseal morphology with finite element analysisrdquo American Journal of Physical Anthropology 144 593-606 Thanks to Michael Fagan for providing this image

The PlanetHPC team has been hard at work over the summer months producing a report for the European Commission to advise it on its strategy for HPC research and development

PlanetHPC is creating a roadmap for HPC research looking forward to 2020 and with the exascale era in mind This is likely to be a period of disruption in HPC with many challenges including energy consumption (an issue dominating much discussion across ICT as a whole) and how to program the new generations of HPC systems that will arise over the next decade

PlanetHPC held a workshop in May 2011 which was attended by leading industrialists and HPC experts The delegates took part in brain-storming sessions to discuss requirements for future HPC applications new programming models and techniques and how to engage industry in using HPC

A follow-up workshop was held in July 2011 at which EPCC presented a draft strategy paper based on the findings of the first workshop Delegates also heard presentations about the

HPC requirements of life sciences presented by Dr Peter Maccallum of Cancer Research UK and Phil Butcher of the Wellcome Trust Sanger Institute together with experiences of commercialising research presented by Dr Guy Lonsdale of scapos AG

PlanetHPC has now completed its strategy paper Entitled A Strategy for Research and Innovation through High Performance Computing it will shortly be released for public comment and will be available on the PlanetHPC website Central to the strategy is the recommendation to set up regionally based HPC pilot networks to stimulate the HPC market and prepare the foundation for future longer term RampD The aim is to encourage more industries (in particular SMEs) to try out HPC solutions and at the same time look towards the requirements of HPC in 2020

PlanetHPC website wwwplanethpceu

PlanetHPC is funded under Framework Programme 7 It receives funding from the European Commission under contract no 248749

PlanetHPC a strategy for HPC research and innovation Mark Sawyer

The European Exascale Software Initiative (EESI) is building a European vision and roadmap to address the challenges of developing and utilising future large-scale HPC systems As part of this EPCC and our colleagues at CSC in Finland have been working on a review of the current state of HPC providing a baseline and guide for this road mapping activity

The report provides an overview of the funding structures HPC centres and large scale HPC initiatives across Europe the US and Asia A brief summary is provided here to whet your appetite the full report is available on the EESI website

Asia has been investing heavily in supercomputing technology and now hosts the two fastest supercomputers and four of the five fastest in the world The two dominant countries in Asia are Japan and China In 2011 Japan reclaimed the top spot in the TOP500 (the list of the worldrsquos 500 fastest supercomputers) with the ldquoK Computerrdquo at the RIKEN Advanced Institute for Computational Science in Kobe This has a performance of 877

PFLOPS China whose Tianhe-1A supercomputer was ranked top in November 2010 remains the fastest developing country in Asia and the world China has invested heavily in supercomputing technology and now holds second position in the TOP500 list The Chinese government has actively promoted independent innovation to avoid reliance on foreign technology

However following significant investment over the last few years the US is still dominant in the supercomputing sector with five systems performing at the petaflop level and with two 20 PFLOPS systems anticipated by 2012

Europe has two petascale systems in France and Germany The Bull system (9th in the world) installed at the French CEA is the most powerful in Europe with a peak performance of 1254 PFLOPS Significant funding has been made available in Europe through the Framework 7 programme both in Petascale (eg PRACE) and Exascale terms (see CRESTA article on p7)

EESI website wwweesi-projecteu

european exascale software Initiative Lorna Smith

Crayrsquos exascale Research Initiative in europe Alistair Hart Cray

For the past two years Crayrsquos Exascale Research Initiative Europe has been working closely with EPCCrsquos Exascale Technology Centre (ETC) to understand and overcome the barriers to programming the next generation of supercomputers With the launch of the EU CRESTA project (see page 7) this work will continue to drive the development of new programming models and tools to push scientific application performance to the next level

Every ten years since 1988 supercomputers have got a thousand times faster Today computers like the HECToR Cray XE6 run real scientific programs at a speed measured in petaflops Continuing this trend by 2018 we expect another thousand-fold leap forward in performance when sustained application performance will be measured in exaflops Put another way this is one hundred million sums for every person on the planet every second

But the Road to the Exascale is a hard one Seymour Cray the pioneer of supercomputing famously once asked if you would rather plough a field with two strong oxen or five-hundred-and-twelve chickens Since then the question has answered itself power restrictions have driven CPU manufacturers away from ldquooxenrdquo (powerful single-core devices) towards multi- and many-core ldquochickensrdquo An exascale supercomputer will take this a step further connecting tens of thousands of many-core nodes Application programmers face the challenge of harnessing the power of tens of millions of threads

Crayrsquos Exascale Research Initiative Europe and EPCCrsquos ETC are finding ways to overcome these challenges New programming models like Fortran Coarrays simplify and accelerate the communication of data between processors The Exascale collaboration has pioneered work in this presenting tutorials at the Supercomputing Conferences in 2010 and 2011 and developing the first benchmark performance suite The CPUs in future Exascale systems may have more in common with todayrsquos GPUs (Graphics Processing Units beloved of high-end gamers) than the Opterons used in HECToR The collaborationrsquos work on new programming models for these is contributing to a new extended OpenMP international standard and has resulted in invited presentations at international conferences Network and IO performance are already important for applications and will be crucial at the Exascale Nevertheless they are poorly understood and hard to quantify By developing tools to monitor the performance of these subsystems Cray and EPCC can now present the information needed to tune the hardware and applications for maximum performance

The challenge and importance of Exascale scientific computing has now been recognised in the EU by the funding of research networks focusing on system and application software Cray is the Industrial Partner in the CRESTA project with EPCC as lead and many other Cray customers are partners By the time CRESTA reaches its final milestone in late 2014 Crayrsquos Exascale Research Initiative Europe will already be halfway down the Road to the Exascale

wwwplanethpceu

Nuclear fusion promises a low pollution route to generate a large fraction of the worldrsquos energy needs in a sustainable manner This is especially important as global energy consumption has grown by more than 20 times over the last century with the majority of current energy generation reliant on fossil fuels But the scientific and engineering challenges in designing a fusion reactor are formidable and commercial power plants are not expected before 2050

To be economically sustainable fusion energy reactors will need a lifetime of many years Therefore the materials that the reactor is built from must be able to withstand the likely impacts and damage that fusion plasmas will inflict on them Reproducing the actual conditions of extended high neutron flux (such as materials would encounter in a fusion reactor) is currently impossible in the laboratory so materials tests will require extrapolations from experiments conducted under alternate irradiation conditions Since radiation damage is a non-equilibrium process there is no simple analytic way to carry out such extrapolations Thus a detailed understanding of the fundamental processes involved is required and this can only be obtained from simulation and a full understanding of the processes involved in irradiation damage at atomic levels At its most ambitious this will enable the design of radiation-resistant materials but even at more pragmatic levels it will enable us to build models based on low-dose short timescale experiments for reliable exploration of longer term effects

At present stainless steels are by far the most likely materials to be used in the construction of fusion reactors However there is some evidence that ferritic (bcc) steel has better self-healing properties albeit with a reduction in the structural behaviour at high temperatures The main causes of radiation damage in steels are corrosion swelling creep precipitation and grain boundary segregation All of these are relatively slow processes and so performing accurate calculations requires simulation of years of material time - an average 250-atom quantum calculation of impurity interaction in steel might take a day of supercomputer time Typical commercial steel might have ten elements each of which interacts with the other elements present Vacancies interstitials helium and other defects also

need to be considered It is evident that very large amounts of computational resource will be needed to model the physics required to understand the radiation damage sustained by metals over the large time scales necessary for the damage to become apparent

Nu-FuSE is an international project (funded through the G8 Research Councils Initiative on Multilateral Research Funding) looking to significantly improve computational modelling capabilities to the level required by the new generation of fusion reactors

The projectrsquos focus is on three specific scientific areas fusion plasma the materials from which fusion reactors are built and the physics of the plasma edge This will require computing at the exascale level across a range of simulation codes collaborating to work towards fully-integrated fusion tokamak modelling The project is led by Professor Graeme Ackland at The University of Edinburgh and includes research teams in Caderache France (also the location of ITER the next generation of fusion reactors) Edinburgh Princeton USA Garching and Juumllich Germany Tsukuba Japan and Keldysh Russia

The largest supercomputers today can perform over a petaflop of calculations per second on real scientific applications But exascale systems are planned for 2016-2018 which means a million million million calculations per second a thousand times faster than todayrsquos systems Major challenges in scaling resiliency result validation and programmability must be addressed before exascale systems can be exploited effectively for fusion modelling

The Nu-FuSE project is focusing on meeting these challenges by improving the performance and scaling of community modelling codes to enable simulations far larger than are currently undertaken The materials simulation work where EPCC is working closely with Prof Graeme Ackland and his group of researchers is modelling the type of active particle impacts that a fusion reactor wall will encounter during the fusion process

NuFuSE website wwwnu-fusecom

nu-Fuse nuclear fusion simulations at exascaleAdrian Jackson

NuFuSE-

The Exascale Challenge is to build a supercomputer that can deliver an exaflop ndash that is a million million million calculations per second Meeting the challenge will require research and development on a global scale and so a number of exascale research programmes have been created The European Union for example announced the start of its exascale research programme in October 2011 with three complementary Framework 7 projects with a combined funding of euro25 million CRESTA (Collaborative Research into Exascale Systemware Tools and Applications) is one of these projects

Having demonstrated a small number of scientific applications running at the petascale (1015 flops) the HPC community particularly the hardware community is looking to the next challenge to reach the milestone of an exaflop (1018 flops)Formidable challenges are involved including scale (such systems could have over a million cores) reliability programmability power consumption and usability

The timescale for demonstrating the worldrsquos first exascale system is estimated to be 2018 From a hardware point of view we can speculate that such systems will consist of

bull large numbers of low-power many-core microprocessors (possibly millions of cores)

bull numerical accelerators with direct access to the same memory as the microprocessors (almost certainly based on evolved GPGPU designs)

bull high-bandwidth low-latency novel topology networks (almost certainly custom-designed)

bull faster larger lower-powered memory modules (perhaps with evolved memory access interfaces)

Only a small number of companies will be able to build such systems But it is crucial to note that software and applications are also part of the exascale computing challenge this is where the EPCC-led CRESTA project comes in

CRESTA will focus on the use of six applications with exascale potential and use them as co-design vehicles to develop an integrated suite of technologies required to support the execution of applications at the exascale The application set in CRESTA has been chosen as a representative sample from across the supercomputing domain including biomolecular systems

fusion energy the virtual physiological human numerical weather prediction and engineering Over the duration of the project the aim is to deliver key exploitable technologies that will allow the co-design applications to successfully execute on multi-petaflop systems in preparation for the first exascale systems towards the end of this decade

At the heart of the proposal is the co-design process The co-design vehicles are closely integrated to the systemware development The applications will exploit the output of the systeware tasks as well as providing guidance and feedback to direct future systemware work A key feature of CRESTA is its use of dual pathways to exascale solutions Many problems in HPC hardware and software have been solved over the years using an incremental approach Most of todayrsquos systems have also been developed incrementally growing larger and more powerful with each product release However we know that some issues at the exascale particularly on the software side will require a completely new disruptive approach CRESTA will therefore employ incremental and disruptive approaches to technical innovation sometimes following both paths for a particular problem to compare and contrast the challenges associated with each approach

CRESTA brings together four of Europersquos leading HPC centres (EPCC together with HLRS CSC and PDC) a world-leading supplier of HPC systems (Cray) seven application and problem owners from science and industry (DLR KTH ABO JYU UCL ECMWF and CRSA) Europersquos leading HPC tool company (Allinea) and Europersquos leading performance analysis organisation (TUD)

CRESTA project website wwwcresta-projecteu

a mountain to climb CResta and the exascale ChallengeLorna Smith

I started as a Research Scholar in the Computer Science Department at Carlos III University in Madrid in 2004 In July 2010 I finished my PhD thesis Dynamic optimization techniques to enhance scalability and

performance of MPI based application Since then I have been working in the same university as a Teaching Assistant

My research has focused on improving the scalability and performance of applications executed on multicore clusters applying several run-time techniques such as adaptive compression and data locality aware methods

Between February and May 2011 I had the opportunity to spend a research visit at the Data Intensive Research Group of the University of Edinburgh funded by the HPC-Europa2 program This group is led by Professor Malcolm Atkinson who was also my host during my visit

During my visit I improved one of the techniques that I developed during my doctoral work Moreover I learned about new methods to exploit the rapidly growing wealth of data Being exposed to the extensive experience of the Data Intensive Research Group in this area has been very beneficial to my understanding of the research problems associated with the challenge of dealing with vast amounts of data

One of the most important outcomes of my research visit is that I have joined this group as a Postdoctoral Research Associate Without the contacts made through the support of the HPC Europa-2 program and the collaboration with my future research group this opportunity would certainly not have presented itself to me so the visiting fellowship program has been extremely useful for my future career development

As I mentioned before my research proposal for this visit was to improve one technique developed in my doctoral work This means improving scalability and performance of MPI-based applications executed in multi-core architectures reducing the overhead of the IO subsystem by improving the performance of Two-Phase IO technique

I developed two new different aggregator patterns for this IO technique called aggregation-by-communication-number (ACN) and aggregation-by-volume-number (ACV) Further I studied the performance of these aggregator patterns using an MPI application called BISP3D that uses Two-Phase IO technique to write the results to a file

I have executed this application with different input data with different numbers of processes (ranging from 8 to 128) and by using the original Two-Phase IO and the modified Two-Phase IO with the ACN and AVN aggregator patterns

The evaluation (Figure 1) shows that in most of the cases the modified Two-Phase IO brings a speedup between 12 and 13 with one pattern that we have proposed This means that with the appropriate aggregator pattern the overall execution time is reduced In the worst cases our technique can lead to an additional overhead of up to 5 in computation time For these cases the default aggregator pattern seems to be good enough

When myself and my junior colleagues (Mr Andrey Frolov and Ms Anastasia Romanova) received our HPC-Europa grant in the beginning of 2011 I was working as a Group Leader (Group of Computational Physical Chemistry and Biophysics) at Max Planck Institute for Mathematics in the Sciences in Leipzig Germany My research interests include chemical physics and molecular biophysics with the main focus on modelling solvent and ion effects on biomolecules and nanoparticles

From February 2011 till May 2011 we worked together with our host Dr Alex Rozhin (School of Engineering and Applied Science) at Aston University in Birmingham on ion effects on carbon nanotube dispersions in organic solvents During our visit we fully exploited the power of the HECToR supercomputer by doing atomistic simulations of large solvent-ion-nanotube systems (hundred thousands of atoms) In parallel our host (a well-known expert in experimental nanoscience) and his colleagues ran experiments with the same systems This synergetic combination of simulations and experimental techniques helped us to reveal the main molecular-scale mechanisms of ion effects on nanotube dispersion stability We have shown that it is possible to regulate the nanotube dispersion stability and bundle formation by changing the type and concentration of ions

Because our findings provide an inexpensive and efficient route to bundle engineering of carbon nanotubes they are expected to have a high impact on several areas of nanoscience During our visit we generated a bulk of interesting results on molecular-scale interfacial phenomena in nanotube dispersions in organic electrolytes Some of the results we have already published in two papers (Fedorov et al PCCP (2011) 13 12399 Frolov et al Faraday Discussions accepted doi101039c1fd00080b) and several more manuscripts are under review and in preparation for publication The importance of the results

have also been appreciated by the poster evaluation panel (ldquoColloid chemistryrdquo division) on the 242nd ACS National meeting in Denver where Andrey Frolov was awarded a prize (sponsored by BASF) for our joint poster presentation ldquoEffects of salts on carbon nanotubes dispersions understanding molecular mechanismsrdquo during the session ldquoFundamental research in colloid and surface sciencerdquo

This HPC-Europa visit also helped me to strengthen my existing collaborations in the UK and establish some new contacts there particularly in Scotland (my very first visit to Scotland in 2005 was also supported by the HPC-Europa program) Therefore I cannot say that it is just a coincidence that shortly after my 2011 HPC-Europa visit I received a professorship offer from the University of Strathclyde in Glasgow where I am moving in October 2011 as a Professor in Physics and Life Sciences in the Department of Physics

I would like to emphasize that an interdisciplinary project such as this would be almost impossible to perform without the HPC-Europa support that gave us (i) access to the high-end supercomputer facilities (ii) a great opportunity to communicate on a daily basis with our experimental colleagues in Aston That was very important for the project and brought much lsquosynergyrsquo to our collaborative work We very much appreciate the highly qualified level of support from the HPC-Europa staff in Edinburgh I am also very grateful to them for organising my seminar there in May 2011 where for the first time I presented our new results in public We are particularly thankful to Catherine Inglis (administration) and Chris Johnson (high-performance computing) for their day-to-day support

More information about HPC-Europa including the application closing dates and details about how to apply can be found at wwwhpc-europaeu

HPC-europa short research visits long-term benefitsCatherine Inglis

Regular readers of EPCC News will know that HPC-Europa is an EC-funded programme which allows European computational scientists to undertake short collaborative research visits in a host group with similar interests while gaining access to some of the most powerful HPC facilities in Europe Most visits produce successful results with many

leading to long-lasting collaborations often resulting in scientific publications joint research proposals and job offers

We are always pleased to hear how past visitorsrsquo careers have developed following their visits Here two of our recent visitors share their experience in their own words with us We will publish further examples on the EPCC website

Left side Our simulations reveal that ndash due to the ion depletion from the carbon nanotube surface ndash the nanotubes should precipitate in organic dispersions upon addition of salts and form bundles

Right side Experiments confirm the simulation results NaI salt addition leads to precipitation of nanotubes and formation of bundles that are well recognizable by eyeLeft to right Anastasia Romanova Maxim Fedorov Andrey Frolov

Fig 1 Overall Speedup by using ACN and AVN aggregation pattern

Dr Rosa Filgueira Edinburgh Data Intensive Research Group School of Informatics University of Edinburgh

Maxim V Fedorov Andrey I Frolov Anastasia O Romanova Max Planck Institute for Mathematics in the SciencesProject Basic mechanisms of salt effects on carbon nanotube dispersions in organic solvents insights from large-scale Molecular Dynamics simulations

Following two years of development the SPRINT (Simple Parallel R INTerface) framework is nearing the release of its first full version SPRINT v100 due in October 2011 SPRINT an R package specifically designed for use in high performance computing (HPC) provides R users with the parallel tool kit needed to perform computationally challenging statistical analyses easily on HPC SPRINT lifts the limitations on data size and analysis time commonly found in many disciplines like bioinformatics where large volumes of highly parallel high throughput post-genomic data are generated SPRINTrsquos unique strengths are its HPC platform independence its focus on complex hard to parallelise data dependent functions and its user friendliness

SPRINT has two main components a library of parallelised R functions and an intelligent harness handling all aspects of using HPC The SPRINT harness is implemented using the MPI standard a parallel processing technology supported by a wide range of platforms This ensures SPRINT can run and provide performance benefits on multi-core desktop PCs shared memory platforms clusters clouds and supercomputers Use of SPRINT requires no HPC knowledge and allows the researcher to concentrate on biological questions rather than the technicalities of IT infrastructure Existing R scripts only need minimal changes to exploit the SPRINT library This enables the biologist to use tried and tested scripts on larger datasets and on HPC without further programming The functions implemented in SPRINT were selected in order to address analysis bottlenecks as identified by a survey of the R user community Previous releases of SPRINT have included parallel implementations of Pearson correlation (pcor) permutation testing (pmaxT) and a clustering function for partitioning around medoid (ppam) These SPRINT functions have shown extremely good performance and scalability

SPRINT version 10 includes three new parallel functions pApply is an implementation of the R apply function This allows a user to perform the same operation over all the

elements of data objects such as matrices or lists The other two functions are implementations of the machine learning methods random forest and rank product respectively prandomForest and pRP The random forest algorithm is an ensemble tree classifier that constructs a forest of classification trees from bootstrap samples of a dataset The random forest algorithm can be used to classify both categorical and continuous variables Rank products are a method of identifying differentially regulated genes in replicated microarray experiments Planned future additions to SPRINT include the R bootstrap and SVM functions as well as an investigation of algorithms for the processing of next generation sequencing

Recent paper publications and presentations at international conferences have increased SPRINTrsquos visibility The figures on the Comprehensive R Archive Network (CRAN) show an increased and sustained interest in the SPRINT package with downloads currently running at over 150 per week (cranr-projectorgreport_cranhtml) Clearly the field of application and usefulness of SPRINT will grow even further with the addition of new functionality The release of SPRINT v100 is therefore expected to add to the popularity of the framework In addition the SPRINT team are currently organising their first training courses in Edinburgh and Oxford Further details can be found on the SPRINT website

SPRINT is an open-source project and is open to contributions All sources are available from R-Forge r-forger-projectorg

Packaged releases are available from CRAN cranr-projectorg

SPRINT project wwwr-sprintorg

Contact the SPRINT team sprintedacuk

sPRInt R for HPC Muriel Mewissen Division of Pathway Medicine

The BonFIRE project is building a Future Internet Research and Experimentation (FIRE) facility for the large-scale testing of distributed applications services and systems To this end BonFIRE offers a multi-site Cloud test bed with heterogeneous compute storage and networking resources Users can run experiments on the BonFIRE facility to determine how their application will work with different types of Cloud infrastructure thus allowing them to understand and improve their applicationrsquos performance in these types of environments The BonFIRE facility has been operational since June 2011 Currently access is only available through Open Calls although the facility will open up in Quarter 3 of 2012 to researchers outside the BonFIRE consortium

BonFIRE aims to support three experimentation scenarios each offering the user a different level of network control

bull Basic experimentation involves a geographically dispersed multi-site Cloud facility whose heterogeneous resources are interconnected through the public Internet Dedicated resources are offered at four BonFIRE sites EPCC INRIA-Rennes HLRS in Stuttgart and HP Labs in Bristol

bull The second scenario allows users to experiment with their own virtualized controlled emulated networks It involves the ldquoVirtual Wallrdquo Cloud at IBBT in Ghent which is based on the Emulab software

bull The third type of scenario for which support is planned for May 2012 will allow experimentation with controlled physical networks

The pragmatic evolution of BonFIRE is important and three embedded experiments have driven its development since its inception The first is investigating the requirements arising from the composition of services at the different layers of the Future Internet ecosystem This includes Cloud and services composition The second experiment is working to classify virtualized resources in terms of their performance when running high-level benchmarks with a view to generating predictive models of how these resources will fit the needs of user applications The third experiment is investigating the elasticity requirements of Cloud-based web applications Twenty-seven use cases were captured from these experiments and from the experience and expertise of the BonFIRE partners from these 101 functional and non-functional requirements were elicited These have underpinned the BonFIRE architecture in the first twelve months of the project

Four more experiments joined BonFIRE in September as a result of the first Open Call The first new experiment will use the Virtual Wall in Ghent to validate the hypothesis that

proprietary desktop virtualization and dynamic virtual network path technologies can enhance user experience especially regarding multimedia applications The second new experiment will investigate the impact of using a distributed infrastructure on the high-throughput problem of calculating the dose of radiotherapy treatment The third new experiment will exploit the various hypervisors and Cloud managers on BonFIRE to test at a large scale a proprietary continuous security-monitoring framework Finally the fourth experiment will deploy and test two service optimization models across the distributed BonFIRE infrastructure

At present BonFIRE provides the following features to help users understand how their applications work in Cloud-type distributed environments

bull Advanced monitoring tools give access to performance metrics for the virtualized resources and to specific experiment monitoring results

bull Both real and emulated networking resources are available on interconnected test beds Controlled networks are expected to be available from June 2012

bull In-depth user documentation and support for experimenters facilitate uptake of the facility and increase the quality of the research

Access to BonFIRE resources is through the standard Open Cloud Computing Interface (OCCI) interface [1] BonFIRE additionally supports experimentation that can be done using the RESTfully scripting language [2] The BonFIRE Portal provides a higher level mechanism for experiment specification and also visualises resource deployment

In total 250 cores with 460GB of RAM and 12TB of storage are available permanently This will increase to 350 cores with 700GB of RAM and 30TB of storage by December 2012 On-request access to about 2300 additional multi-core nodes is also possible Single sign-on to all BonFIRE services root access to compute resources and support for cross-site elasticity come as standard

The BonFIRE facility is planned to evolve in four major phases each adding functionality based on the needs of its users The first release introduced the features and capabilities that are currently available in the system The first upgrade to the platform software is due in October 2011 Of particular interest is the planned support for experiment descriptors based on the Open Virtualization Format standard as well as increased

BonFIRe ndash building service test beds on FIRe Kostas Kavoussanakis

The BonFIRE infrastructure Picture

courtesy HLRS University of

Stuttgart

support for the experiment lifecycle A second upgrade to the platform software in May 2012 will be shaped by the experiences and results of the users gained though the first Open Call for experiments Additionally it will incorporate the ability to federate BonFIRE systems with external facilities like FEDERICA [3] while focusing on the deployment of a Cloud-to-network interface which will enable the control of different types of network infrastructures To this end a sixth facility at PSNC will be added to BonFIRE With EPCC leading the activity that builds BonFIRE these are very exciting times The final phase due in May 2013 will use results from the second Open Call for experiments together with any non-funded experiments to construct the final version of the facility

If you are interested in joining BonFIRE the second Open Call for funded experiments will open on 1 February 2012 and close at 1700 CET on 7 March 2012 During the third quarter of 2012 the facility is expected to open for experimenters outside the BonFIRE consortium

Announcements will be made on BonFIRErsquos website bonfire-projecteu

[1] Open Cloud Computing Interface (OCCI) wwwocci-wgorg

[2] Restfully developed by Cyril ROHR at INRIA httpsgithubcomcrohrrestfully

[3] FEDERICA wwwfp7-federicaeu

Photo Mark Sadowski

Continues opposite

The European collaborative project DEISA (Distributed European Infrastructure for Supercomputing Applications) came to an end in April this year after successfully deploying and operating a virtual supercomputer spanning the European continent over a 7-year period

To be deemed successful such an infrastructure has to operate as a transparent and non-disruptive layer on top of existing national services to hide the complexity of Grid technologies from the scientific end users and to guarantee persistence and portability of scientific applications since these are essential to the activities of research organizations Not only this but the platform has to be managed across a number of heterogeneous and autonomous HPC systems run by operators speaking different languages and observing different polices at their sites

These requirements were met by integrating a number of existing national high-end systems using Grid computing DEISA did not attempt to solve all the problems in Grid computing but rather applied the most efficient pragmatic solutions in order to enable the DEISA environment Linking the core infrastructure to other infrastructures or virtual organizations was a fundamental priority for DEISA Once achieved this integration of national resources contributed significantly to the enhancement of HPC capability and capacity in Europe and thus to European scientific achievements The HPC centres involved were also tightly coupled using a dedicated 10 Gbs network bandwidth provided by the European research network GEANT and the national research networks to reduce intra machine communication overheads By using the DEISA middleware the DEISA services hid the heterogeneous nature of the

infrastructure allowing developers and users to manage their applications compute jobs IO etc on the distributed infrastructure in almost the same way as they would have done at a single national HPC site

There were two main routes for accessing the DEISA infrastructure the DEISA Extreme Computing Initiative (DECI) and the Virtual Community access DECI-targeted applications were complex demanding and innovative simulations with a multi-national dimension Under DECI users received not only cycles on one or more HPC systems but also access to Europe-wide coordinated expert teams for operation technology developments and application enabling and support Virtual Community access was similar to DECI but each project ran for more than a year DEISA supported Virtual Communities in four important grand challenge areas life sciences fusionenergy research space sciencecosmology and climate research

DEISA thus operated as a virtual European supercomputing centre and was exploited by many scientists across a range of subjects such as cosmology earth science engineering material science particle physics plasma physics and life sciences These groups used DEISA to run simulations which would simply have not been possible otherwise In total DEISA was used by more than 200 research institutes and universities from over 25 European countries with collaborators from four other continents

By the end of DEISA a large number of HPC systems had been integrated into its infrastructure These included CRAY X2XT4XT5SE6 IBM Power5Power6 IBM Blue GeneP IBM PowerPC SGI ALTIX 4700 and NEC SX89 vector systems

The final DEISA environment gave an aggregated peak performance of over 282PFs with a fixed fraction of resources dedicated to DEISA usage This contributed a significant enhancement to the capabilities of European scientific research that needed high performance computing (HPC) capabilities DEISA also successfully demonstrated how a multi-terascale European computing platform could be deployed and operated through the use of Grid technologies that interfaced the DEISA research infrastructure with other European and US IT infrastructures over existing national supercomputers as a persistent production quality virtual European supercomputing centre that provided global reliable non-disruptive and transparent services to its community

Although DEISA has come to an end its work is carried forward by the PRACE (Partnership for Advanced Computing in Europe) project which will provide European researchers with access to so-called Tier-0 and Tier-1 systems funded by national agencies (with EC support for leadership-class computers) Indeed DEISA worked closely with PRACE and it has been agreed in principle that the vast majority of the DEISA infrastructure will be employed by PRACE The two DEISA user access programs ndash DECI (under PRACErsquos Project Access) and Virtual Communities (under PRACErsquos Programme Access) ndash will also continue where DECI now stands for Distributed European Computing Initiative

DEISA was judged a great success by the European Commission and scientists alike and we expect PRACE to be even more successful

DEISA was financed by the EUrsquos FP6 and FP7 programmes

Modelling nuclear fusion EUFORIA project Adrian JacksonEUFORIA is working on the design and construction of ITER the worldrsquos biggest fusion reactor Modelling the physical processes involved in nuclear fusion requires exceptional computational resources and EUFORIA drew on Grid and HPC technology to build a computer modelling platform that meets the needs of European fusion scientists

DEISA enabled the EUFORIA platform to access vital HPC resources through a comprehensive software infrastructure In particular the remote access and job submission infrastructure that DEISA provided combined with access to multiple computational resources from the same software enabled EUFORIA and European fusion scientists to easily integrate HPC resources with fusion simulations

wwweuforia-projecteu

Successful redocking of heparin with antithrombin Taken from DECI-6 project BLOODINH High throughput in-silico screening in HPC architectures for new inhibitors for treatment of blood diseases

DEISA ultimately consisted of the 11 project partners and 4 associated partners shown above It successfully integrated a number of existing national high-end systems using Grid technologies to form the DEISA environment DEISA then enabled scientific and technological discovery which would not have been possible otherwiseDeIsa homogenous access to

heterogeneous supercomputingGavin J Pringle

Simulation of fusion plasma undertaken using DEISA resources through EUFORIArsquos modelling platform

Illustration of density fluctuations in Ohmic Textor simulations (narrow simulation region is zoomed in radial direction for visualization purposes) Taken from DECI-6 project fullfgk Numerical simulation of tokamak fusion

The landscape of computationally-assisted research has changed over the last few years Where computational simulation once defined a complementary third paradigm to scientific theory and experiment the use of computers to distil new results from ever-increasing volumes of digital data has given rise to a fourth paradigm data-intensive research

Characterised by the work of Microsoftrsquos Jim Gray and collaborators and captured in the seminal book The Fourth Paradigm Data-Intensive Scientific Discovery [1] data-intensive research looks different in approach and applicability to any of the other scientific models In their 2010 report Riding the Wave the European High-Level Experts Group on data (HLEG) used the phrase ldquodata scientistrdquo to describe a new kind of researcher one who uses the ever-growing wealth of digital data as their research infrastructure using computers and software tools to examine huge corpora of data searching for links and patterns hidden among the bytes

Because of the sheer volume of digital data ldquodata sciencerdquo and data-intensive methods revolve not so much around maximising calculations-per-second but around moving bytes through systems intelligently and efficiently ndash or perhaps trying not to move them unless absolutely necessary Streaming data incremental processing long-running database queries ldquomove the computation to the datardquo ndash building efficient systems in this environment is the preserve of the data-intensive engineer

EPCC has been working in data-intensive engineering since the earliest days of the UKrsquos e-Science Core Programme and most recently through the ADMIRE project ADMIRE saw EPCC and the Universityrsquos School of Informatics lead a European team of researchers and engineers in exploring new methods to tame these large-scale data-intensive problems It finished this summer with an ldquoexcellentrdquo final review by its funders the European Commission

ADMIRE investigated how data sources could be brought together with the rich diversity of tools and applications used by data scientists in a way that could be controlled and engineered sensibly The result was an hourglass a system which introduces a common canonical description of data-intensive workflow processes which both tools and systems can understand ADMIRE introduces DISPEL the data-intensive systems process engineering language to provide the necessary canonical description for data-intensive processes DISPEL documents describing complex data combination and analysis tasks can be sent to a ldquoGatewayrdquo service compiled and executed on a distributed enactment platform built on top of the OGSA-DAI data workflow engine Users can interact with the Gateway with a wide variety of tools from simple web-based portals to a fully-fledged development environment derived from the popular Eclipse workbench

ADMIRErsquos ideas and approaches were successfully demonstrated by addressing data-intensive problems in fields including customer relationship management astronomy flood prediction and gene annotation A full account of the development of ADMIRErsquos architecture and language and its use to address real-life data-intensive challenges will be published early next year in a major book

ADMIRErsquos technology will be further developed by upcoming projects For example the JISC-funded An Enhanced Visual Workbench for OGSA-DAI is improving the usability of ADMIRErsquos Eclipse-based workbench (see opposite) On a larger scale the pan-European seismology project VERCE is one of a number of so-called ldquoresearch data infrastructurerdquo projects funded recently by the European Commission With so much digital data now being produced by researchers across the world what tools and systems are needed to allow it to be exploited to create new science or deliver genuine sociological impact VERCE addresses these questions in the context of

never mind the flops wherersquos my baudData-intensive engineering at ePCCRob Baxter

The OGSA-DAI Visual Workbench project which started in August aims to create a powerful highly usable and intuitive development environment for OGSA-DAI The basis for this environment will be the Workbench developed for the ADMIRE project as seen in the figure If you are unfamiliar with OGSA-DAI it is a mature platform for integrating distributed data by building workflows from composable functions (a full explanation can be found at the project website [1]) A common request from OGSA-DAI users has been for some form of visual editor to aid the creation of simple workflows similar to that provided by Taverna [2]

In contrast to OGSA-DAI the ADMIRE project [3] had broader aims It was a research project that developed a distributed streaming platform for knowledge discovery The project created a new language DISPEL designed to succinctly describe data-intensive workflows To support users of the platform the project also created the ADMIRE Workbench essentially a large set of co-operating plug-ins for the Eclipse IDE [4]

The ADMIRE Workbench contains both a graphical workflow editor and a text editor for the DISPEL language as well as various plug-ins for monitoring running processes and visualising their results In the ADMIRE project DISPEL workflows were sent to ldquoGatewaysrdquo which converted DISPEL into OGSA-DAI workflows for execution on OGSA-DAI installations (in the ADMIRE parlance OGSA-DAI is an ldquoEnactment Platformrdquo) Because of this it is reasonably straight-forward to integrate the Gateway with the Workbench to seamlessly provide OGSA-DAI developers with the ADMIRE tools an OGSA-DAI Workbench

The project work will concentrate on addressing various usability and robustness issues as well as hiding ADMIRE architectural details Part of the focus is on improving the maintainability of the ADMIRE code which will lower the barrier for future projects and developers to continue work on the Workbench The project will deliver a virtual machine image (or ldquoWorkbench-in-a-boxrdquo) that will allow users to get started straight away without the need to waste time installing various dependencies The virtual image will include everything needed to run a simple but complete example including documentation sample data and code

Development started with an evaluation of the usability of the software and the quality of the underlying code These evaluations were used to identify the areas in greatest need of improvement You can follow the OGSA-DAI Workbench project on the OGSA-DAI blog [5] which touches on general software development and usability engineering issues as well as providing progress updates

The OGSA-DAI Visual Workbench is a four-and-a-half-month project funded by JISC

[1] wwwogsadaiorguk

[2] wwwtavernaorguk

[3] wwwadmire-projecteu

[4] wwweclipseorg

[5] sourceforgenetappswordpressogsa-dai (Look at posts with the ldquoworkbenchrdquo tag)

OGsa-DaI Visual WorkbenchAdrian Mouat

ADMIRE workbench in action

seismology the EUDAT project takes a broader look and asks what should a common data infrastructure for European research look like

EUDAT seeks to build on results from a number of key ldquoverticalrdquo discipline-specific data projects and create prototypes of the first genuine ldquohorizontalrdquo research data infrastructure Its mission is to create a concrete example of the vision laid out in the HLEG report a common infrastructure platform that will allow researchers to work with data from a wide variety of existing and future archives in a standard way using common services and their own preferred tools While a lot of the

technology to create such a platform already exists integrating the policies systems and protocols to provide a seamless user experience is EUDATrsquos main challenge EPCC has a leading role in EUDAT our particular remit being to create a strategy for the long-term sustainability of research data infrastructure The ideas approaches and experiences of ADMIRE will stand us in good stead for EUDAT as we continue to engineer solutions to the problems being washed up by the rising tides of digital data

[1] researchmicrosoftcomen-uscollaborationfourthparadigm

EUDAT website wwweudateu

14 15Continues opposite

If getting good information is difficult getting good information from across disciplines is near impossible Huge organisations ndash the Googles and Microsofts of this world ndash get round the problem by employing lots of experts from different backgrounds We donrsquot have those kinds of resources at the Institute so we came up with a different solution

The Institute helps researchers (from all disciplines) build better software To do this we need to know whorsquos using software and who needs our help The easiest way to learn about these developments is to attend conferences ndash after all thatrsquos the way the researchers do it But what are the right conferences to attend and when you get there who are the right people to talk to Unless yoursquore already an expert in the field these questions are difficult to answer To solve our information problem the Institute needed experts We donrsquot have the resources to employ them so we sent them packing instead

We came up with the idea of a swap the Institute needs information and researchers want to travel We agreed to pay researchers up to pound3000 a year in travelling expenses in exchange for the benefit of their expertise Itrsquos a fair deal we get information that has been sifted and verified by experts in the field and the researchers can go to conferences that they would not otherwise have been able to attend

In July this year the hunt for our Agents began We based the application process around the two skills that an Agent must possess they have to learn new information quickly and they have to effectively communicate that information to non-specialists (ie us) In the first round of judging we asked the applicants to describe their research and the way they use

software in a way that a non-expert audience could understand The second round was more difficult we asked the applicants to write a blog about someone elsersquos research There were other factors to consider We needed a good coverage of the research community so we wanted researchers from a wide range of organisations and as wide a range of backgrounds as possible We also gave preference to early career researchers because we have found that they tend to be active in emerging technologies and are more likely to write the reports we need

The application process was a great success We were inundated with applications in the first round ndash over 130 of them for the 10 places we had on offer This was whittled down to 30 applications in the first round of judging and then with the help of the NGSrsquos Claire Devereux as our independent adjudicator we chose our 10 Agents

Now that wersquove recruited our Agents getting good information is a lot easier We have access to experts in fields as diverse as climate change and sports science This is an incredible resource We have massively increased the breadth of research that we can deal with and have exposure to and we have a team to call on when we face problems that are outside of our field of expertise

Details about our Agents are available on the Software Sustainability Institutersquos website If you would like to meet the Agents the best opportunity will take place in the UK next March at the Collaborations Workshop 2012

wwwsoftwareacukagents

not so secret agentsSimon Hettrick Software Sustainability Institute

To diagnose and treat severe brain conditions we need an understanding of the structure and operation of the brain However there is still much that we donrsquot understand about the brainrsquos function not only when we are ill but also when we are healthy

The Brain Research Imaging Centre (BRIC) based at the Division of Clinical Neurosciences at Edinburghrsquos Western General Hospital develops and uses a wide variety of medical imaging techniques to better understand neurological disorders such as multiple sclerosis stroke and prion diseases including BSECJD BRIC is a significant contributor to research into brain disease and is a member of SINAPSE the Scottish Imaging Network ndash A Platform for Scientific Excellence an initiative to promote collaboration in brain imaging across Scotland

BRIC has developed many applications for medical image processing and analysis In our role as members of the Software Sustainability Institute (SSI) EPCC worked with BRIC to improve the sustainability of two of its applications

DICOM Confidential takes sensitive medical data ndash DICOM images ndash and allows users to remove any information that could be used to identify individual patients We assessed how easy it is to use extend and modify DICOM Confidential and reviewed its code We also looked at effective ways to support DICOM Confidential and ensure its long-term availability The result of this is that DICOM Confidential is now hosted by the open source project repository SourceForge and there is on-going development to address the matters our review raised

MCMxxxVI (Multispectral Colour Modulation and Variance Identification) is a suite of innovative image manipulation and analysis tools They colourise and fuse together medical images of a brain generated by MRI (magnetic resonance imaging) scanners Tissues of interest can then be identified and extracted from these images MCMxxxVI tools are currently used by students and are also of interest to clinicians One tool the Automatic Lesion Extractor runs under MATLAB We ported this tool to be a standalone Windows executable written in C++ and using the open source VTK image visualisation toolkit and wxWidgets GUI toolkit Distributing ALE as a stand-alone application makes it far easier for users for example students researchers and clinicians to download and use We also created a new SourceForge project BRIC1936 to host ALE

Our collaboration benefited BRIC in contributing to their goal of promoting the wider adoption of these applications within the medical imaging community The SSI also benefited as the collaboration allowed project processes to be exercised and refined and our knowledge of using open source packages and setting up projects on SourceForge to be enhanced

The Software Sustainability Institute wwwsoftwareacuk

Brain Research Imaging Centre wwwbricedacuk

SINAPSE wwwsinapseacuk

DICOM Confidential project sourceforgenetprojectsprivacyguard

BRIC1936 project sourceforgenetprojectsbric1936

understanding the brain in sickness and in health Mike Jackson

MCMxxxVIrsquos automatic lesion extractor showing an MRI scan of a brain (left) tissue of interest (right) and a brain mask that extracts the tissue from the brain scan (centre)

Institute

SoftwareSustainability

The Software Sustainability Institute (SSI) which is headquartered at EPCC was set up to identify and develop the software considered to be important to researchers working with them to make it easier to maintain It recently celebrated its first birthday In the last 12 months we have worked with software from domains including fusion energy climate policy chemistry psychology social simulation and distributed computing

Therersquos a strong emphasis at the SSI on software engineering and collaborative development two principles which have guided EPCC over the last 20 years An example of this approach is illustrated in Mike Jacksonrsquos case study of a project wersquove recently completed with the Brain Research Imaging Centre [2] ndash see page opposite for more information

Looking to the future the SSI is always seeking to expand its domain-specific knowledge through networks such as the ProductArea Liaisons (PALs) and SeIUCCR Community Champions [3] Simon Hettrick the SSI Dissemination and Outreach Officer explains below why wersquove just recruited a set of Agents [4] and how theyrsquoll gathering information that might just mean the difference between life and deathhellip for software

[1] wwwsoftwareacuk[2] wwwbricedacuk[3]wwwsoftwareacukwhat-do-we-doseiuccr-project[4] wwwsoftwareacukagents

the software sustainability Institute a year of progress Neil Chue Hong SSI Director

The above headline was recently spotted by your correspondent on the newspaper billboards in the streets of Edinburgh At first sight the event would seem rather unlikely particularly for the long-suffering followers of the Scottish national team But how likely or unlikely is it really Such questions are very important to those considering events which may be extremely rare but potentially have a very wide-ranging impact (For interested readers at the time of writing a bookmaker was offering 300-1 for Scotland to win the next FIFA World Cup in 2014)

A more serious example [1] of a rare event is the credit crunch of 2008 a global event of a size not seen probably since the 1930s A part of the systematic failure to predict this event is arguably attributable to the failure of the banksrsquo computer models to account properly for these events which while having very low probability cannot be ignored From the scientific perspective such a failure is understandable Simulating rare events is extremely hard This is for the simple but fundamental reason that in a lsquobrute-forcersquo simulation the vast majority of the computational effort is spent simulating the uninteresting waiting time between the events that do matter So few if any occurrences of the rare event itself are observed in any one simulation So most (if not all) of your computational time and effort is wasted

While high-performance computing has led to impressive advances in our ability to simulate larger systems for longer it is clear that these advances need to be combined with more intelligent methods than brute-force simulation to make it possible to simulate rare events effectively In the area of microscopic simulation a number of such methods have been developed Broadly these are based on statistical sampling methods which try to help the simulation toward the final goal However these methods tend to suffer from limitations

including the inability to deal with systems subject to some external driving

A new project based in Edinburgh funded by the UK Engineering and Physical Sciences Research Council will attempt to improve the situation To do this we will combine a powerful new technique for capturing rare events with the use of large supercomputers The goal of the work is to provide software which will make it easier for scientists to apply this technique (known as ldquoForward Flux Samplingrdquo [2]) The method works by generating many different simulations and discarding those which do not make progress against the final goal In this way sampling is concentrated on the simulations which do make progress allowing the whole system to be ratcheted forward As forward flux sampling does not care about the details of the underlying simulation algorithm (provided it has a random component) the software should be relevant to many different types of application As an example the method can be used in conjunction with molecular dynamics to study conformational (shape) changes in proteins rare mistakes in the folding of proteins are thought to play an important role in neurodegenerative diseases

It is hoped our new software will become available to help study such problems in the near future However it will almost certainly not predict the next financial crisis as for Scotland winning the next World Cup that remains in the lap of the footballing gods Oh yes and that headline referred to the Homeless World Cup

[1] Diehard Scotland supporters may beg to differ

[2] RJ Allen et al Journal of Physics Condensed Matter 21 463102 (2009)

scotland wins World CupKevin Stratford

ss Wor

fficia

eDIm1 a new machine for Data Intensive ResearchAdam Carter

The use of computers in science is now ubiquitous and the amount of simulation data is increasing at a staggering pace The advent of more automated sensing and screening techniques means ever more digital output is being produced by devices such as telescopes microscopes and atmospheric sensors as well as the growing data collected from experiments There is now so much data that itrsquos not possible to process and analyse it without computers The sheer quantity however is allowing new methodologies to emerge by joining data together by automating analysis and by identifying patterns in the data we are discovering new ways to understand the world around us

We believe that Data Intensive Research will become increasingly important in coming years and so building on the success of EPCCrsquos various data-related projects and its HPC expertise we have installed a new machine known as EDIM1 designed specifically for Data Intensive Research

EDIM1 was built from commodity components to a custom specification The 120-node system has over 6TB of fast storage directly connected inside every node and by using low-cost low-power Atom processors it has been possible to create a machine which is more balanced in favour of IO-intensive applications As well as commissioning this system we have been working with research groups in fields such as astronomy and geosciences to evaluate how it can be applied to existing and future research problems

While the processing power of computers continues to double every couple of years not all components have kept up In particular hard disk technologies continue to improve both for spinning disks and the more recent solid state disks but the speed at which data can be read from and written to disk has not increased nearly so quickly as the rate at which it can be processed by CPUs (and now GPUs) Computers are becoming increasingly unbalanced and in the case of data-intensive problems the processor can spend a larger proportion of its time sitting idle waiting for input and output (IO) to disk

EDIM1 attempts to address this issue by concentrating on IOops rather than FLOPs In particular we aimed to make the machine ldquoAmdahl-balancedrdquo Gene Amdahl famous for noting that the performance of a parallel program is ultimately limited by the serial part of the code also made several other observations about the balance between the rate at which instructions are performed by a computer and its memory and IO characteristics Referred to by some as ldquoAmdahlrsquos other lawrdquo he

described three quantities (all ratios) which should be close to one in order to make the best use of a computer

bull Amdahl Number bits of sequential IO per second per instruction per second

bull Amdahl Memory Ratio megabytes of main memory per million instructions per second

bull Amdahl IOops ratio IOops per 50000 instructions

Building such a machine from commodity parts had its challenges One of these was trying to interpret how the manufacturersrsquo part specifications actually translate into machine performance Itrsquos quite common for different manufacturers to quote their own set of incomparable specifications designed to show their hardware in its best light IOops can be reads and writes sequential and random and of different sizes The IO bandwidth can also be quoted in various ways some of which take into account a diskrsquos onboard cache for example It has also been a challenge to ensure that all the parts are compatible not just that they work together but that they actually perform well together To give one example we had to track down (with the help of our suppliers) why our disks were negotiating transfer rates down to SATA1 when they were actually capable of communicating at twice that speed Fixing this involved not only flashing chips in the hardware but actually replacing nearly all of the 480 SATA cables in the machine

EDIM1 is now up and running and one of our MSc students has been putting it through its paces Measurements [1] have shown that its empirical Amdahl number is around 12 close to the theoretical value of 133 deduced from the manufacturersrsquo specifications and in excess of unity (the point where a machine becomes balanced in favour of IO) These numbers are far higher than would be expected from a PC with a high-end processor and a single hard disk These results suggest that for problems that are too big to fit in a computerrsquos main memory EDIM1 could out-perform many machines and clusters that are considerably more costly and more power-hungry

EDIM1 was jointly funded by EPCC and the School of Informatics at the University of Edinburgh

[1] Kulkarni O MSc Dissertation Report Available soon from wwwepccedacukmsc

EDIM1 built and hosted by EPCC

The MSc in High Performance Computing is celebrating its tenth anniversary this year In 2001 we welcomed our first intake of six students Since then the programme has grown year on year and we will see more than thirty students graduate in November The MSc also continues to have a broad-based and international appeal Over the years we have had students from across Europe Asia Central and North America the Middle East and Africa This yearrsquos class is possibly one of the most diverse with 15 nationalities represented

We recently decided to take a step back and review the curriculum and structure of the whole MSc This review has now been completed As a result space has been created for new content reflecting new trends and developments in HPC while existing material has been consolidated and rationalised Although many of the changes are evolutionary most course names have been changed and in some cases there has been a change of focus and the inclusion of a significant quantity of new material

At the same time we have moved away from our previous model of delivering each course in intensive half-day blocks taught over a semester Although each course still runs for a

semester it is now structured into separate weekly lecture and practical sessions This gives students more time to consolidate and reflect on new material after the lectures and to get to grips with programming and other practical work before tutorials The new structure has also enabled us to integrate our timetable more closely with related MSc courses for example in Computer Science Informatics and Mathematics As a result students can now choose from a wide range of optional courses to complement the core subjects taught in the MSc

One of the highlights of the MSc for most students ndash and staff ndash is the dissertation project These cover a wide range of topics reflecting the extensive range of staff skills and interests at EPCC Below and opposite we highlight a selection of this yearrsquos dissertations Dissertation reports of these should be available from November on the MSc website

Past dissertation reports ndash and videos of recent MSc graduates talking about living and studying in Edinburgh ndash can be downloaded from the MSc website wwwepccedacukmsc

Dynamic Loop Nesting in Shared Memory Programming Orestis AgathokleousRegions of nested loops are a common feature of HPC codes In shared memory programming models such as OpenMP these structures are the most common source of parallelism

This work investigated the possibility of applying code transformations before compilation to enable a dynamic choice to be made at runtime on how parallelisation should be applied on the nested loops A source-to-source compiler was developed to perform the transformations automatically using a directive based approach similar to OpenMP

Krylov Subspace Solvers in OP2 on GPUs Lesleis Nagy

An extensible framework for the execution of Krylov Subspace solvers is presented which implements the Conjugate Gradient method using OP2 and CUSPARSE amp CUBLAS libraries

Timings were taken for OP2 CUSPARSE amp CUBLAS and serial versions of the implemented Conjugate Gradient method OP2 was found to be sensitive to the structure of the input matrix for the linear system being solved A metric called the lsquostructure effect factorrsquo was derived to quantify the performance of OP2 with respect to the structure of the input matrix

the msc in HPC ten years on Judy Hardy EPCC

Benchmarking an Amdahl-balanced Cluster for Data Intensive Computing Omkar KulkarniThis project evaluated the performance of an Amdahl-balanced cluster (see EDIM1 on page 19) designed for data-intensive research It used numerous benchmarks to characterise individual components of a node and the cluster as a whole Special attention was paid to the performance of the secondary storage A custom benchmark was written to measure the balance of the cluster ndash the extent of difference in performance between the processor and IO The tests also included distributed computing benchmarks based on the popular MapReduce programming model using the Hadoop framework The results of these tests were very encouraging and showed the practicality of this configuration

MPI Visualisation Thomas RichardThis project aimed to generate a real-time visualisation tool for MPI codes It used the MPI Profiling interface to wrap MPI functions collect the data and display it in a real-time display as the program is executed The tool aimed to be easy to use and accessible to MPI learners It could handle most of the MPI point-to-point communications functions The tool also proposes a memory registration system the code registers arrays and the tool then displays which parts are accessed during communications Some derived MPI data types are also supported such as Vectors and Contiguous in order to be useful for the first steps of programming with MPI

empowering research in the humanitiesMark Hedges Centre for e-Research Kingrsquos College London

Taught Courses

bull Advanced Parallel Programming

bull HPC Architectures

bull HPC Ecosystem

bull Message Passing Programming

bull Parallel Design Patterns

bull Performance Programming

bull Parallel Numerical Algorithms

bull Project Preparation

bull Parallel Programming Languages

bull Programming Skills

bull Software Development

bull Threaded Programming

Recent years have seen several collaborations in humanities e-research between EPCC and the Centre for e-Research (CeRch) at Kingrsquos as personified by CeRch researchers Mark Hedges and Tobias Blanke Collaboration began back in 2009 with the Linking and Querying Ancient Texts (LaQuAT) project funded as part of the JISC ENGAGE project at EPCC LaQuAT looked at the thorny problem of integrating the very diverse and fuzzy datasets created by researchers into classical antiquity such as documents written on stone or papyrus LaQuAT raised more questions than it answered but these questions led in turn to a more recent collaboration ndash the Supporting Productive Queries for Research (SPQR) project funded by the JISC Managing Research Data programme

While LaQuAT used OMII-UKrsquos OGSA-DAI software to attempt a relational database model of integrating various database and XML resources SPQR instead used a linked data approach based on semantic web technologies The relational database model had proved too rigid to deal with the ldquofuzzyrdquo incomplete and inconsistent data that we possess about antiquity and it was hoped that this model which used RDF as the fundamental form for representing information and ontologies to provide the semantic ldquogluerdquo between datasets and their divergent representations would be more successful

The EPCC team of David Scott and Bartosz Dobrzelecki led by Mike Jackson developed scripts to convert the source XML and databases into over fifty-six thousand RDF triples incorporating links to geographically linked datasets provided by Pleiades and GeoNames An off-the-shelf linked data browser was used as the basis for evaluation by researchers who concluded that this approach offers an intuitive and usable means of exploring and understanding the data although of course more work remains

to be done The SPQR triples were also incorporated by the PELAGOIS project into a wider set of geographically annotated linked data which can be explored at pelagiosdmeaitacatgraph-explore

Our other success was the JISC-funded TEXTvre (pronounced ldquotexturerdquo) project which was an international collaboration between Kingrsquos EPCC and the University of Sheffield in the UK and the State and University Library at Goumlttingen in Germany TEXTvre builds upon the successful e-humanities project TextGrid which developed a collaborative environment for creating critical editions of literary texts (think big XML files and annotations) that used the German grid infrastructure EPCCrsquos contribution was to replace this underlying infrastructure with a repository based on Fedora which means that TEXTvre can be set up more easily as an institutional environment initially for the digital humanists at Kingrsquos but also potentially elsewhere This work involved many complexities notably including the mapping of the semantic structure within TextGrid onto Fedorarsquos data model and was carried out by EPCCrsquos Malcolm Illingworth under the supervision of Kostas Kavoussanakis TEXTvre recently finished with the successful installation of the software at CeRch But we have been awarded continuation money from the Software Sustainability Institute (SSI) at Edinburgh which will make the TEXTvre software more robust modular and easier to set up and which will thus increase uptake

Inscription on the Archive Wall in the Theatre in AphrodisiasTranscriptions of such inscriptions into electronic forms are helping classicists undertake research into the ancient world Photograph (c)

1977 Mossman Rouecheacute

Continues opposite

EPCC has been working with the Edinburgh University Business School for almost 20 years applying high-performance computing to business problems and building a better understanding of consumer behaviour

With the advent of Grid computing and globally distributed infrastructures for secure communication and computation the possibility of collaborations around shared data views of global markets became a technical possibility leading to the creation of the ESRC-funded INWA Grid connecting EPCC to Perth in Western Australia and the Chinese Academy of Sciences in Beijing [1] [2]

Current funding from the EPSRC has enabled a link with an industry partner in China and creation of a secure collaborative environment allowing exploration of consumer behaviour in response to the introduction of new channels ndash the methods by which consumers find select and purchase products and services These channels range from shops established technologies such as phone mail and fax and newer technologies such as the Internet

From a corporate perspective there is a real value in migrating consumers from expensive channels such as shops to more efficient channels such as the Internet but each stage in that migration has the potential to exclude significant parts of society ndash for example in 2010 92 million adults in the UK had never used the Internet In the same year Martha Lane Fox the UK Digital Champion published a report entitled Directgov 2010 and Beyond Revolution Not Evolution The tension that a step change to a lsquoDigital by Defaultrsquo assumption implies for parts of society that have not yet chosen or had the opportunity to engage are severe ndash companies cannot easily design for consumers they do not know about and if the populations are seen as small then there is no economic incentive to do so These parts of society stand the risk of becoming permanently invisible to the designers of future channels and hence permanently excluded

In a global village the problem can get worse as companies have the possibility of using Internet channels to find more customers that are of the same type as their existing customers ndash growing larger more homogeneous markets rather than addressing the differences in society from which exclusion stems

Martha Lane Foxrsquos focus on lsquorevolutionrsquo aptly describes just how rapid the changes in industry structure are expected to be This is shown in Figure 1 in a study of over 150 banks across Europe Despite the presence of direct channel technologies such as telephone banking electronic banking and ATMs for many years their share of total transactions has grown very slowly and is still relatively small This contrasts with the planned restructuring of the banking industry where for example the 16 of savings accounts sold across direct channels in 2010 is expected to rise to 58 in only 5 years while all the channels that involve contact with a person including branches and call centres are expected to decline

If this lsquosupply-sidersquo revolution is not able to include new types of customer then those that it excludes may well become permanently invisible The BRIDGE (Building Relations with the Invisible in the Digital Global Economy) project a collaboration between the University of Edinburgh together with Yvonne Barnard from the University of Leeds and Mike Bradley from Middlesex University is exploring this Leeds and Middlesex are investigating the impact of interface design on adoption while Edinburgh is performing adoption studies at a scale that allows even small segments within the UK to be related to potential demand in much larger export markets Establishing this potential would make it economically attractive for UK business to invest in inclusion and hence contribute to a smoother migration to being digitally engaged for those caught up in the lsquorevolutionrsquo

More details can be found at wwwbridge-relateorg

[1] wwwepccedacuknewsannouncementsinwa-grid

[2] wwwtein3netuploadpdfINWA_final_webpdf

4 9 100

10072 13 5 10

11 4 4

Currentaccounts

Savingsaccounts

Mortgages

Product Purchasing

Current sales breakdown by channel Expected dominant Channel2010 of sales 2015 of respondents

Branches AgentsBrokers Call Centre Direct Channels (Internet ATM Mobile)

SOURCE Efma online survey across 150+ European Banks Face-to-Face A euro15-20BN Multichannel Opportunity McKinsey amp Co April 2011

Figure 1 Channel use in 150+ European Banks Adapted from McKinsey amp Co report ldquoFace to face A euro15-20Bn Multichannel Opportunityrdquo April 2011

Digital exclusion building consumer relations with the invisibleAshley D Lloyd University of Edinburgh Business School

In scientific computing faster and more powerful computers do not immediately lead to better results Incompatibility between the requirements of existing software and the capabilities of new supercomputers is a growing problem that will be addressed by a pioneering new Russian-European collaboration The European branch of this collaboration is called APOS-EU

The future of high-performance computing means two things massive parallelism and heterogeneity Processor manufacturers are squeezing ever more computational cores onto their CPUs and HPC vendors are looking to augment these many-core chips with GPU-like accelerators to deliver the next push forward for computing capacity Although such developments give scientists the potential to run ever bigger faster or more detailed simulations there are significant challenges to be overcome before todayrsquos most important application codes can exploit these hardware advances efficiently

APOS-EU in collaboration with a peer project funded by the Ministry of Education and Science of the Russian Federation will target popular codes from the strategically important application areas of seismic modelling oil- and gas-reservoir simulation computational fluid dynamics fusion energy and nanotechnology Through a programme of research activities and workshops the joint EU-Russian team will enable representative

codes from the five application areas to efficiently exploit the capabilities of tomorrowrsquos supercomputers These activities will drive the creation of new tools for code sustainability enabling next-generation science on next-generation supercomputers

The suite of codes that we select will be exemplars for the techniques that we advocate ndash for example illustrating how to properly exploit the memory hierarchy of a NUMA platform or how to structure computations for maximum acceleration by GPGPUs Furthermore case studies will be developed using representative distributed-memory applications to showcase the innovation in our tool chain Looking beyond the project our work will make an important contribution to the wider field of scientific computing streamlining and simplifying the upgrade path for many other application codes

APOS-EU is a two-year project which runs from February 2011 until January 2013 The consortium has five partners EPCC (project coordinator representing the University of Edinburgh) CAPS entreprise Uniwersytet Warszawski TOTAL SA and Houmlchstleistungsrechenzentrum Stuttgart

For more information see the project website apos-projecteu

Or follow the group on Twitter aposeu

Initial water saturation of an oil-bearing reservoir where blue indicates 100 water Image courtesy of Total Exploration Production

aPOs-eu application Performance Optimisation and scalabilityGeorge Beckett

EPCC is a European centre of expertise in developing high performance novel computing solutions managing advanced systems and providing HPC training Our clients and partners include local and global industry government institutions and academia EPCCrsquos

combination of advanced computing resources and expertise is unique and unmatched by any European university

wwwepccedacukinfoepccedacuk

Somewhere over the rainbow Liz Sim EPCC User Support

ldquo skies are blue and the dreams that you dare to dream really do come truerdquo

The HECToR project started back in 2007 and in late 2011 it will enter its third and final phase which is due to run until 2013 Back at the start of the HECToR project we could only have dreamed about what the third phase of HECToR would comprise but now those dreams are about to become reality

An upgrade to the HECToR system was announced in May In late 2011 the existing 20-cabinet Cray XE6 will be expanded and upgraded to a 30-cabinet system utilizing the next generation AMD Interlagos processor and Crayrsquos Gemini interconnect The machine will have a projected theoretical peak of 820TF and as such represents a greater than two-fold increase in capability over the current Phase 2b system

To mark the unveiling of the third phase the UK Research Councils launched a competition to produce a design for the front of HECToR The competition was open to children aged 11ndash16 years old The response to the competition was very positive and over 70 entries were received from school children across the UK The winning designs were chosen by a panel of judges from the Research Councils and the University of Edinburgh

The prizes were awarded to the designs that best demonstrated

bull an understanding of the breadth of science undertaken on HECToR

bull that science can be inspiring and benefit society as a whole

bull originality and creativity

Congratulations to the overall winner Lily Johnson of Heathersett Old School in Norwich whose winning design appears at the top of this page

Lilyrsquos design has been converted by a professional designer to fit the cabinet layout As the winner Lily will receive a contribution towards travel costs to attend a launch ceremony in Edinburgh in 2012 when the finalised artwork will be unveiled She will also receive a wall plaque for her school to mark her contribution to the understanding of science in the UK

Congratulations also to Harriet Wheatley and Alice Salt (who took second place with a joint entry) and Katie Knox (third place) Last but not least a huge thanks to all of those who entered the competition We were delighted by the number and quality of entries we received

PRACE and HECToR As one of the ldquoTier 1rdquo sites within PRACE HECToR will be playing an important role Back in June our call for applications for DECI (Distributed European Computing Initiative) resources attracted over 50 proposals within Europe with a good response from within the UK

Each of the 17 participating Tier 1 sites is providing at least 5 of its annual resources to DECI Successful projects will start running in November utilising a whole range of architectures distributed across Europe including IBM BlueGeneP and Power 6 a number of clusters (some with GPGPUs) and of course Cray XTXE machines such as HECToR

The next call for applications opens on 2nd November and information on applying can be found on the PRACE website wwwprace-rieu

The HECToR system is funded by the UK Research Councils For more information on the HECToR service please see wwwhectoracuk

_GoBack