Supporting computational science in Nordic area

CSC CSC –– the the FinnishFinnish IT center IT center for sciencefor science

ORAP Paris 8.11.2005

Kimmo KoskiManaging Director, CSC Finland

Contents

Nordic support structuresCase Norway, Sweden and DenmarkCase CSC – the Finnish IT center for scienceGrowing need for international collaborationOpinions on hot topics in Nordic area and in Europe

• Distribution vs. centralization• Is there a superior computer architecture: varying needs of top science• Respect for the national infrastructures• How to build competence to meet the grand challenges

Nordic characteristics

Tradition in Nordic collaboration: Nordic Minister council, Nordic funding, Nordunet, …Very different national systems in supporting scientific computing

• Finland: fully centralized (one national center)• Sweden and Norway: distributed (4-6 national small centers)• Denmark: no center at all, funding distributed to research groups

Some challenges in fitting together the national systems

Still growing interest in tighter collaboration in HPC and science infrastructures

Nordic development plans

Nordic collaboration structure is forming: funding bodies, governments, HPC centersNordic Data Grid Facility proposal for 2006->Common interest in:

• Resource exchange (cycles, application usage, databases, persons,…)• Unified coordination of international projects: CERN LHC, ESO, ITER, …)• Better interactions through Nordic forums, such as HPC conferences• Research network development, dark fiber and lambda networks• Future data challenges: opportunity for workload optimization?• Tighter links between Nordic computing centers

The Nordic collaboration is targeted to contribute in European level• In some cases one or more Nordic countries together, in some cases individually

Nordic Collaboration - NDGFNDGF- Nordic Data Grid Facility

http://www.ndgf.orgLong-term initiative run by the Joint Committee of the Nordic Natural Science Research Councils (NOS-N) in 2002 to establish a Nordic Data Grid Facility and to involve Nordic countries in European and global co-operation in data sharing in a variety of fields

Participants: Denmark, Finland, Norway, Sweden

National funding: prototype phase (2003-2005)Funding for the year 2006 proposedManagement infrastructure - Nordic Steering Committee

National test-beds for R&D in Grid-technology are beingset up in all the Nordic Countries

• Using the NORDUGrid middleware (funded by Nordunet2)• Storage elements: app. 10 TB of public available storage

• Using the NORDUGrid middleware (funded by Nordunet2)• Storage elements: app. 10 TB of public available storage

The e VITA Initiative (Norway)

eScience; Infrastructure, Theory and Applications (coordinated action)Motivation: ”…computational science is one of the most important technical fields of the 21st century because it is essential to advances throughout society.” (PITAC, 2005)Scope 2006-2015Budget~10 MEURO/year

NOTUR – The Norwegian High Performance Consortium

Operated as a Metacenter, coordinated by UNINETT Sigma (Trondheim)4 universities and Met office are partnersHardware investments are co-funded by the Research Council and the universitiesResources are allocated by the Research CouncilCurrent funding from Research Council 4 MEURO/yearNational coordination ensures diversity of hardware and sharing of resources

SNIC – Swedish National Infrastructure for Computing

Six national centers • PDC Stockholm• NSC Lindköping• HPC2N Umeå• UNICC Gothenburg• LUNARC Lund• UPPMAX Uppsala

SNIC Metacenter projects• National helpdesk for SNIC • SweGrid• HPC portals • National storage solutions • National application support in Computational Chemistry • National application support in Bioinformatics• EGEE Regional Operations Center

Grid research projectshttp://www.snic.vr.se/

DCSC – Danish Center for Scientific Computing

No national centersFunding distributed directly to research groups, coordination by DCSCRegional operation centers:

• Technical University of Denmark • University of Copenhagen • University of Southern Denmark • Aarhus University

The Danish Center for Scientific Computing has been set up as a national centre under the Ministry of Science, Technology and Innovation with government funding allocated for data processing capacity within the area of scientific computing for research assignmentshttp://www.dcsc.dk/

Case CSC – the Finnish IT Center for Science

Special for the Finnish model

In Finland there is only one national IT center for science (CSC) in which most activities are concentrated

• Supercomputing and data repositories• Research network (FUNET)• Scientific support• Scientific library systems• Scientific applications, databases

Strong organization for scientific support for researchers• Scientists to support researchers in using the infrastructure• Tools and methods• Multidisciplinary approach

CSC Fact Sheet

First supercomputerCray X-MP/EA 416

in 1989

Since March 2005, facilities

in Keilaniemi, Espoo

Funet startedin1984

CSC Turnover in 200414.0 M€,

137 employees in April 2005

Operated on a non-profit principle

All shares to the Ministryof Education of Finland

in 1997

Reorganized as alimited company,

CSC-Scientific ComputingLtd. in 1993

Founded in 1970 as a technical support unit for

Univac 1108

CSC Fields of Services

UNIVERSITIES

POLYTECHNICS

RESEARCH INSITUTES

COMPANIES

UNIVERSITIES

POLYTECHNICS

RESEARCH INSITUTES

COMPANIES

FUNET SERVICESFUNET SERVICES

COMPUTATIONAL SERVICESCOMPUTATIONAL SERVICES

APPLICATIONS SERVICES APPLICATIONS SERVICES

INFORMATION SYSTEMS MANAGEMENT

INFORMATION SYSTEMS MANAGEMENT

EXPERTISE IN SCIENTIFIC COMPUTING

EXPERTISE IN SCIENTIFIC COMPUTING

CSC Statistics 2004:Processor Time per Discipline

23 %

12 %

6 % 4 % 2 %

48 %

Nanoscience

Physics

Chemistry

Biosciences

Weatherforecasts (FMI)

Computationalfluid dynamics

331

157

12788

71

67

56

45

43

42

160

Biosciences

Physics

Chemistry

Nanoscience

Engineering

Linguistics

Computationalfluid dynamicsComputationaldrug design Weatherforecasts (FMI) Mathematics

Other

CSC Statistics 2004:Ten Largest Disciplines by Users

2004, IBM eSuper Cluster 1600 (16 IBM p690 nodes witha theoretical peak performance of 2.2 Tflop/s) – switch upgrade withFederation technology 2004, coordination and support of the Finnish Physics and Chemistry Material Science Grid (PC clusters in 8 laboratories, ~ 500 CPUs totally)2005, cluster capacity at CSC (~ 700 CPU’s cluster)2006 – 2008, new supercomputer purchase:• Grant of 10 M€ from Ministry of Education • RFP will be out at turn of the year 2006• First phase is aimed to be

installed at 3Q/2006• Possible second (and third)

phase at 2007/2008

2004, IBM eSuper Cluster 1600 (16 IBM p690 nodes witha theoretical peak performance of 2.2 Tflop/s) – switch upgrade withFederation technology 2004, coordination and support of the Finnish Physics and Chemistry Material Science Grid (PC clusters in 8 laboratories, ~ 500 CPUs totally)2005, cluster capacity at CSC (~ 700 CPU’s cluster)2006 – 2008, new supercomputer purchase:• Grant of 10 M€ from Ministry of Education • RFP will be out at turn of the year 2006• First phase is aimed to be

installed at 3Q/2006• Possible second (and third)

phase at 2007/2008

CSC Strategy of the Future HPC CapacityUpdates

Funet - Finnish University and research NETwork

High quality academic Internet services for Finland• Advanced services like IPv6 and

IP multicast• Core line speed – 2,5 Gbps• Funet member access at

speeds up to 1 Gbps• International connectivity

through NORDUnet85 member organizations:

• All Finnish universities (20) and polytechnics (30)

• Public research institutions and administrative organizations close to higher education (35)

• Over 350 000 end users

High quality academic Internet services for Finland• Advanced services like IPv6 and

IP multicast• Core line speed – 2,5 Gbps• Funet member access at

speeds up to 1 Gbps• International connectivity

through NORDUnet85 member organizations:

• All Finnish universities (20) and polytechnics (30)

• Public research institutions and administrative organizations close to higher education (35)

• Over 350 000 end users

M- Grid – Finnish Grid in production use

Participants: CSC, 7 Finnish universities, Helsinki Instituteof Physics (HIP) – 9 sites totallyObjectives: To build a homogeneous PC-clusterenvironment with the theoretical computing capacity ofappr. 1.5 Tflops / 410 nodesEnvironment

Hardware: Dual AMD Opteron 1.8-2.2 Ghz nodes with2-8 GB memory, 1-2 TB shared storage, separate 2xGE(communications and NFS), remote administrationOS: NPACI Rocks Cluster Distribution / 64 bit, based onRedHat Enterprise Linux 3Grid middleware: NorduGrid ARC Grid MW compiledWith Globus 3.2.1 libraries, Sun Grid Engine as LRMS

http://www.csc.fi/proj/mgrid/

Material Sciences National Grid Infrastructure (M- grid) is a national joint project funded by the Finnish Academy for the National Research Infrastructure Program in the Grid area

CSC:Administration tasksMaintain Operating System, LRMS, Grid middleware, certain librariesSeparate small test cluster for testing new software releasesTools for system monitoring, integrity checking, etc.

CSC:Administration tasksMaintain Operating System, LRMS, Grid middleware, certain librariesSeparate small test cluster for testing new software releasesTools for system monitoring, integrity checking, etc.

DEISA objectives

To enable Europe’s terascale science by the integration of Europe’s most powerful supercomputing systems

Enabling scientific discovery across a broad spectrum of science and technology is the only criterion for success

DEISA is an European Supercomputing Service built on top of existing national services

DEISA deploys and operates a persistent, production quality, distributed supercomputing environment with continental scope

EU Collaboration - FP6 EMBRACE ProjectEMBRACE – a European Model for Bioinformatics Research and

Community Education

Funded by EC with 8.3 M€Launched on February 1, 2005

Designed as a Network of Excellence toaccumulate experience and professional skills ofhighly qualified biologists of 17 institutes from 11European countries.

Coordinated by EMBL, the European MolecularBiology Laboratory and its part EBI, the EuropeanBioinformatics Institute, UK.

http://www.embracegrid.infoThe project’s objective is “to draw together a wide group of experts throughout Europe who are involved in the use of information technology in the biomolecular sciences”

• Create APIs to DBs• Adjust SW to use APIs• Grid-enabled Web–services

- EMBRACEgrid• Test problem driven

development processCSC: Technology Evaluation and Watch WP, PairsDB, technology assessments, test problems, workshops

• Create APIs to DBs• Adjust SW to use APIs• Grid-enabled Web–services

- EMBRACEgrid• Test problem driven

development processCSC: Technology Evaluation and Watch WP, PairsDB, technology assessments, test problems, workshops

HAKA Infrastructure and Middleware inthe Finnish Higher Education

http://www.csc.fi/suomi/funet/middleware/english/index.phtml

The purpose of the HAKA infrastructure is to support the teaching and research activities in the Finnish universitiesand polytechnics by developing and maintaining the common infrastructure for user identification.

HAKA isis a part of the FFinnish innish EElectronic lectronic IDIDentificationentification in in HHigher igher EEducation ducation (FEIDHE), a national funded joint project of the academic commun(FEIDHE), a national funded joint project of the academic community in Finland.ity in Finland.

With his own user ID, the user can log in to services connected to the HAKA infrastructure regardless of the organization that maintains the service - in conjunction with the log in, the user’s current ID data is transferred to the service provider.

The HAKA infrastructure is based on the SAML standard and Shibboleth Open Source code solutions. Shibboleth technology wasdeveloped in universities in the United States.

The HAKA infrastructure is based on the SAML standard and Shibboleth Open Source code solutions. Shibboleth technology wasdeveloped in universities in the United States.

The system has been built up as a collaboration project between CSC and academic IT centers since March 2002. 12 different pilots have been carried in 2003-2005.

CSC Software Development : SOMA Project

http://www.csc.fi/proj/drug2000

The aim of Soma project (part of the drug2000-program)is to organize and update CSC's modeling environment to better meet the European standard in computer-aided drug design and increase its usability by non-experts in computer modeling.

Funded by CSC and the National TechnologyAgency (Tekes)Tailored structure-based small-molecule designenvironment

Consists of several molecular modeling tools (e.g. MDL,CSDS, Corina, Gold, Volsurf, Bodil) and allows designingof hypothetic new ligand molecules and predicts theirproperties

SOMA can be accessed from the CSC Scientist’s interfaceSOMA can be accessed from the CSC Scientist’s interface

CSC Software Development : MIKA Project

http://www.csc.fi/physics/mika/

MIKA is the real-space, multigrid-based program package for electronic structure calculations with applications in research of quantum dots, nanostructures and positron physics.

3 year collaboration project with COMP group of the Laboratory of Physics atHelsinki University of Technology and CSC to prepare a free academic SWthrough the Open Source code

Snapshots from a simulation of nanowire breaking by the ultimate jellium model. A catenoid surface (a), cluster-derived structures (b) and (d), and uniform cylindrical shape (c) can be seen. Green rectangles mark the lead-constriction boundary

Snapshots from a simulation of nanowire breaking by the ultimate jellium model. A catenoid surface (a), cluster-derived structures (b) and (d), and uniform cylindrical shape (c) can be seen. Green rectangles mark the lead-constriction boundary

Multigrid Instead of K-spAce

MIKA consists of several dierent modules for solving the Kohn-Sham equationsof the density- functional theory in dierent geometries

a b c d

CSC Software Development : ELMER

Elmer is a finite element computational tool for multi-physics problems developed in collaboration with Finnish universities, research laboratories and industry

Elmer - Finite Element Software for Multiphysical Problems

Elmer includes physical models of fluid dynamics, structural mechanics,electromagnetics, heat transfer etc. These are described by partial differentialequations which Elmer solves by the Finite Element Method (FEM)

http://www.csc.fi/elmer

The program has been developed at CSC over the past 10 years.ELMER 5.0 is currently released as an open source SW under the GNU GeneralPublic License.Elmer comprises of several different parts: thegeometry, boundary conditions and physicalmodels are defined in ElmerFront; the resultingproblem definition is solved by ElmerSolver.Finally, the results are visualized by ElmerPost.Additionally, a utility ElmerGrid may be used forsimple mesh manipulations

Growing need for international collaboration in providing infrastructure

Motivation to develop national infrastructure AND form international networks/grids is growing in Nordic area

• Growing need and impact of computational science (PITAC 2006)

• Research groups are working increasingly internationally, so the support organisations also need to provide internationally competitive resources

• Global players are looking for strong collaboratorsSolving Grand challenge problems might require investments beyond country budgets

• Survey for Nordic Grand challenges of computingBoosting Nordic area attractiveness for top researchers

• Competitive infrastructure• Stimulating research environment

Hot topics in Nordic and Europe

Distribution vs. centralization• The optimal solution to might be different depending on the

location• History often decisive: for example it is very difficult to close

centers and human capital is still the major asset• Efficient co-usage of centralization and (human) networks,

solution is a combination adapted to national needsIs there a superior computer architecture: varying needs of top science

• No single architecture solves all problems• Not all the problems can (usually) be addressed sufficiently

within the available budget• Requirement for priorities: what are the most important grand

challenges (in the opinion of the funding authority)

Hot topics in Nordic and in Europe (cont.)

Respect for the national infrastructures• Existing investment for hardware, software and peopleware in

all of the European countries can not be bypassed• New infrastructures need to be built on top of the existing ones

• Top specialists usually have a choice• Number of top specialists is limited – starting from scratch

requires human capital• Grid projects linking national infrastructures can be very useful,

if get to address concrete issues

How to build competence to meet the grand challenges of computing with sustainable infra

Major problems might require parallel computing with 100000+ processors and we do not even know what new problems this will cause How to build the competence to solve top-end problems

• Two lower layers very important (majority of person resources, applications development etc.)

Balanced system with smooth interoperability between layers is important

• Lower layers feed upper layers with scalable applications

EuropeanHPC center(s)

National/regional centers,Grid-collaboration (DEISA etc.)

Local centers