View
1
Download
0
Category
Preview:
Citation preview
PROF. TAN TIN WEE, DIRECTOR, NSCC SINGAPORE
JON LAU, DEPUTY DIRECTOR, NSCC SINGAPORE
DR. GABRIEL NOAJE, SENIOR COMPUTATIONAL SCIENTIST, A*STAR CRC
NATIONAL SUPERCOMPUTING
CENTRE SINGAPORE
National Supercomputing Centre Singapore
• State-of-the-art national facility with computing, data and
resources to enable users to solve science and technological
problems, and stimulate industry to use computing for
problem solving, testing designs and advancing technologies.
• Facility will be linked by high bandwidth networks to connect
these resources and provide high speed access to users anywhere
and everyone.
National Supercomputing Centre (NSCC) Singapore
August 27, 2015 2 GTC Asia South 2015
National Supercomputing Centre Singapore
Supporting National R&D Initiatives 1
Objectives
Attracting Industrial Research Collaborations 2
Enhancing Singapore’s Research Capabilities 3
August 27, 2015 3 GTC Asia South 2015
National Supercomputing Centre Singapore
Petascale 2015 National Supercomputing Centre (NSCC)
2006 A*STAR Computational Resource Centre (ACRC)
Mid 2000s HPCC of Nanyang Technological University (NTU)
1998 Institute for High Performance Computing (IHPC)
Early 1990s National Supercomputing Resource Centre
(NSRC)
Late 1980s DISCS and Computer Centre, National University
of Singapore (NUS)
Historical Trajectory
August 27, 2015 4 GTC Asia South 2015
Terascale
Gigascale
National Supercomputing Centre Singapore
NSCC Datacentre - Fusionopolis
5
Level 17
August 27, 2015 GTC Asia South 2015
National Supercomputing Centre Singapore
NSCC Datacentre
August 27, 2015 6 GTC Asia South 2015
National Supercomputing Centre Singapore
NSCC components
NSCC Supercomputer • New 1+ PetaFLOP Supercomputer
• Recurrent investment every 3-5 years
• Co-investment (OpEx) from primary stakeholders
• Remote login nodes in stakeholders campuses
Science, Technology & Research
Network (STAR-N)
• High bandwidth InfiniBand fabric to connect the distributed login nodes
• High speed network access to users (both public & private) anywhere
• Support transfer of large data-sets (both locally & internationally)
• Build local & international network connectivity (Internet 2 USA, TEIN*CC to
GEANT, to Japanese Networks)
• ASEAN, USA, Europe, Australia, Japan, Middle East
August 27, 2015 7 GTC Asia South 2015
National Supercomputing Centre Singapore
NSCC Supercomputer
~1 PFlop
System
• 1,288 nodes (dual socket,
12 cores/CPU E5-2690v3)
• 128 GB DDR4 / node
• 10 Large memory
nodes (1x6TB, 4x2TB, 5x
1TB)
~10PB Storage
• HSM, Tiered, Tier 3
SSD disks
• I/O 500 Gbps flash
burst buffer , 10x
Infinite Memory
Engine (IME)
EDR
Interconnect
• Mellanox EDR Fat Tree
• 500 Gbps pipeline
between Biopolis &
Fusionopolis
• InfiniBand connection
to all end-points (login
nodes) at three campuses
August 27, 2015 8 GTC Asia South 2015
National Supercomputing Centre Singapore
GPU nodes
Accelerators nodes
• 128 nodes with NVIDIA GPUs (identical to the compute nodes)
• NVIDIA K40 (2880 cores)
• 368,640 total GPU cores
Visualization nodes
• 2 nodes Fujitsu Celsius R940 graphic workstations
• Each with 3 x NVIDIA Quadro K4200
•NVIDIA Quadro Sync support
August 27, 2015 9 GTC Asia South 2015
National Supercomputing Centre Singapore
Singapore InfiniBand Connectivity
Connecting all National Supercomputing Centre stakeholders:
A*STAR, NUS, NTU, SUTD and others with InfiniBand links.
NUS
A*STAR
Fusionopolis
SingAREN
Global Switch
100Gbps InfiniBand enabled
500Gbps InfiniBand enabled
A*STAR
Biopolis
WOODLANDS
10/40/100Gbps InfiniBand/IP
SELETAR
CHANGI
NOVENA
OUTRAM
ONE-NORTH
JURONG
NTU
August 27, 2015 10 GTC Asia South 2015
National Supercomputing Centre Singapore
Example of Accessibility NSCC-GIS Integration - Big Data with HPC
NGSP Sequencers at B2
(Illumina + PacBio)
NSCC
Gateway
STEP 2: Automated
pipeline analysis once
sequencing completes.
Processed data resides in
NSCC
500Gbps
Primary
Link
Data Manager
STEP 3: Data manager index
and annotates processed data.
Replicate metadata to GIS.
Allowing data to be search and
retrieved from GIS
Data Manager Compute Tiered Storage
POLARIS, Genotyping &
other Platforms in L4~L8
Tiered Storage
STEP 1: Sequencers
stream directly to
NSCC Storage
(NO footprint in GIS)
Compute
1 Gbps per
sequencer
10 Gbps
1 Gbps per
machine
100 Gbps
10 Gbps
NSCC
GIS
National Supercomputing Centre Singapore
Global Connectivity as Anchor Participant of InfiniCortex Project
August 27, 2015 GTC Asia South 2015 12
NSCC
National Supercomputing Centre Singapore
Pillars of NSCC
1 PF
Supercomputer
HPC Domain
Expertise
STAR-N
Network
August 27, 2015 13 GTC Asia South 2015
National Supercomputing Centre Singapore
HPC Expertise
• Engineers available to help users initiate, optimize, migrate
their applications on the new supercomputer
• Support team available for hardware and software issues
through a ticketing system
• Partnerships with experts like NVIDIA, Mellanox,
Obsidian, DDN, etc.
• Collaborative projects with international academic and
research supercomputer centres
August 27, 2015 14 GTC Asia South 2015
National Supercomputing Centre Singapore
GPU Outreach Efforts
• NSCC’s system will have 128 GPU accelerated nodes as a start
• Capacity will be increased once the demand is high enough
• GPU Computing is still quite a new domain in Singapore
• Exploring opportunities to develop competency of Parallel
Computing and GPU acceleration locally and in the region
• Provide HPC training sessions as well as advanced courses (GPU
programming, code optimization)
• Exploring collaboration projects to use the 128 GPUs
August 27, 2015 15 GTC Asia South 2015
National Supercomputing Centre Singapore
GARUDA from Systems Biology Institute
Garuda – The way biology connects
• open, community-driven, common platform that provides a
framework to interface, discover, & navigate through different
applications, databases and services in bio-medical research
• provides language independent API to connect software as
gadgets
• Gadget = software integrated as a node in a more complex
workflow
• explore gadgets through the gateway and operate them
through the dashboard
• supported by a global alliance of leaders in computational
biology and informatics
August 27, 2015 16 GTC Asia South 2015
National Supercomputing Centre Singapore
GARUDA on GPUs
• Extend the gadget portfolio to Deep Learning applications
• Install the entire framework on the NSCC supercomputer to
allow for a low entry point into HPC for researchers
August 27, 2015 17 GTC Asia South 2015
• Integrate GPU-enabled
bioinformatics applications as
gadgets into the Garuda
framework
• Provide Garuda users with
access to workflows that
include GPU-enabled
applications
National Supercomputing Centre Singapore
VASP
• VASP = Vienna Ab-initio Simulation Package
• a complex package for performing ab initio quantum-
mechanical molecular dynamics (MD) simulations using
pseudopotentials or the projector-augmented wave method
and a plane wave basis set
• VASP+GPU version 5.4.1 developed by Univ of Vienna with
newer collaborators (ENS-Lyon, IFPEN, U of Chicago, CMU,
URCA, Materials Design, Inc., NVIDIA and Aachen-RTWH)
• A*CRC / NSCC part of the closed beta test program
• Work by Michael Sullivan – A*STAR IHPC
August 27, 2015 GTC Asia South 2015 18
National Supercomputing Centre Singapore
GROMACS
• GROMACS = GROningen MAchine for Chemical Simulations
• a versatile package to perform molecular dynamics, i.e.
simulate the Newtonian equations of motion for systems with
hundreds to millions of particles
• version 5.1 released on August 14, 2015 with many GPU
improvements
• Improvements to mdrun performance
– Improved performance auto-tuning with GPUs
– Added kernel and compiler support for K80
– Added optional NVIDIA Management Library (NVML) Integration
– Allowed increasing CUDA thread block size
– Added support for flushing WDDM queue on Windows
– Optimized atomic accumulation in CUDA short-ranged kernels
– Improved the intra-GPU load balancing
August 27, 2015 GTC Asia South 2015 19
National Supercomputing Centre Singapore
Unit A of human estrogen receptor A
• 78.000 atoms
• Explicit solvent
• OPLS-aa wer-a-a-bpa-ion-
pull.groforce field
• Gromacs 5.0.6 with CUDA 7.0
Performance
• CPU-only run: 56 ns/day
• With 1x K40m: 85 ns/day
• With 2x K40m: 130 ns/day
• With 1x K80: 120 ns/day
• With 2x K80: 146 ns/day
Challenges
• Finding the best ration between
number of OpenMPI/OpenMP
tasks
• Work by Jernej Zidar – A*STAR
IHPC
GROMACS
August 27, 2015 GTC Asia South 2015 20
National Supercomputing Centre Singapore
Blender
• Collaboration with digital media SME to fit open source
Blender into animation pipeline
• Leveraging on Blender optimized on NVIDIA GPUs
• Benchmarking of Blender with and without GPUs
• POC one of the largest Blender rendering farms
August 27, 2015 GTC Asia South 2015 21
http://www.blender.org/wp-content/uploads/2012/11/Modelling1.png
National Supercomputing Centre Singapore
Demo and prototyping novel services
InfiniCloud-GPU
• Rapid scheduling of GPU accelerated cloud instances on
InfiniCloud – InfiniBand interconnected cluster
• GPU passthrough support at the KVM
Altair PBS Pro & Bright Computing BCM • Scheduling of accelerated cloud instances on High
Memory Machines (12 TB RAM)
• Scheduling of accelerated cloud instances on GPU nodes
August 27, 2015 GTC Asia South 2015 22
National Supercomputing Centre Singapore
GPU Meetups Around the World
August 27, 2015 23 GTC Asia South 2015
National Supercomputing Centre Singapore
Singapore GPU Users Group
August 27, 2015 GTC Asia South 2015 24
Singapore GPU Users Group in numbers:
• 2 organizers (Dr. Gabriel Noaje - A*CRC & Oxana
Plis – NVIDIA)
• 10 meetups since October 17th, 2013
• 20+ speakers
• 213 registered members, ~30 participants/meeting
• Giveaways: GPU, Shield tablet, NVIDIA Shield
National Supercomputing Centre Singapore
HPC democratisation
Co-organize/sponsor local events:
• GPU Technology Conference Asia South
• Deep Learning Workshop
Trainings:
• Comprehensive training program in both HPC in general and GPU computing in particular
• Collaborate with partners: University of Illinois Urbana-Champagne (GPU Center of Excellence), IDA, NVIDIA
• Different levels: beginners, advanced, topic oriented
• Different platforms: classic face to face courses, self-paced online courses, interactive sessions
• Challenges for students, young entrepreneurs to promote and incentivize HPC usage across Singapore and SE Asia
• Many initiatives to come …
August 27, 2015 25 GTC Asia South 2015
National Supercomputing Centre Singapore
Conclusion
• NSCC = National Supercomputing Centre
• 1 PF Supercomputer (not very big, but unique in many
ways)
• Objective: To democratize HPC in all research and
industry domains across Singapore
• Create an HPC ecosystem in the region through:
–Aggressive promotion and partnership development strategy
–High speed interconnect with the world and local sites
islandwide
– Lead GPU centered projects and initiatives
August 27, 2015 26 GTC Asia South 2015
National Supercomputing Centre Singapore
We want to work with you!
August 27, 2015 GTC Asia South 2015 27
THANK YOU
Recommended