Innovation Intelligence®
RADIOSS Structural Analysis Solver
Performance Optimizations
Eric Lequiniou
June 16, 2013
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Contents
• About Altair
• About RADIOSS
• HPC Advisory Council Benchmark Testing and Results
• Introducing HyperWorks Unlimited
• Q&A / For More Information
“Our vision is to radically change
the way organizations design
products and make decisions.”
– James R. Scapa, Chairman & CEO, Altair
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Overview
Founded ...
In 1985 as a product design consulting company
Today ...
A global software, services & technology leader
with over 40 offices in 19 countries and
3,200+ customers worldwide
‘85 ‘12
$100M
Est. $240M
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Innovation Intelligence®
27+
Years of Innovation
40+
Offices in 19 Countries
1800+ Employees Worldwide
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Customers
Automotive Aerospace Heavy Equipment Government Life/Earth Sciences Consumer Goods Energy
3,200+ customers worldwide
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Altair Knows HPC
Altair is the only company that:
makes HPC tools…
develops HPC applications…
…and uses these to solve real HPC
problems
500 Altair engineers worldwide
use HPC every day for
real-world modeling
& simulation
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
HPC Advisory Council RADIOSS Performance Study
Research performed under the HPC Advisory Council activities
• Participating vendors: Intel, Dell, Mellanox
• Compute resource: HPC Advisory Council Cluster Center
Objectives
• Give an overview of RADIOSS Performance
• Compare different MPI libraries
• Compare different interconnects
• Understand RADIOSS communication patterns
• Provide Best Practices for running RADIOSS and increase productivity
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
About RADIOSS
Compute-intensive simulation software for Manufacturing
• For 20+ years an established standard for automotive crash and impact
• Differentiated by its high scalability, quality and robustness
• Supports multiphysics simulation and advanced materials
• Used across all industries to improve safety and manufacturability
Companies use RADIOSS to simulate real-world scenarios (crash
tests, climate effects, etc.) to test the performance of a product
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Hybrid MPP Parallelization
Highly parallel code
• Multi-level parallelization
• Domain decomposition MPI parallelization
• Multithreading OpenMP
Enhanced performance
• Best scalability in the marketplace
• High efficiency on large HPC clusters
• Unique, proven method for rich scalability over thousands of cores for FEA
• Flexibility -- easy tuning of MPI & OpenMP
• Robustness -- parallel arithmetic allows perfect repeatability in parallel
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
HPC Advisory Council Test Configuration
Dell™ PowerEdge™ R720xd 32-node (512-core) “Jupiter” cluster
• Dual-Socket Eight-Core Intel E5-2680 @ 2.70 GHz CPUs
• Memory: 64GB memory, DDR3 1600 MHz
• OS: RHEL 6.2, OFED 1.5.3 InfiniBand SW stack
• Hard Drives: 24x 250GB 7.2 RPM SATA 2.5” on RAID 0
Mellanox Infiniband
• Mellanox ConnectX-3 FDR InfiniBand VPI adapters
• Mellanox SwitchX SX6036 InfiniBand VPI switch
Intel Cluster Ready certified cluster
MPI Libraries
• Intel MPI 4.1.0
• Platform MPI 8.2
Application: Altair RADIOSS 12.0 Single Precision
Benchmark dataset: Neon Refined 1 million elements 8ms
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Performance – Interconnect (MPP)
FDR InfiniBand provides better scalability performance than Ethernet
• 743% better performance than 1GbE at 8 nodes
• 214% better performance than 10GbE at 16 nodes
• 1GbE does not scale beyond 4 nodes with pure MPI
Higher is better
Platform MPI
16 Processes/Node
743%
214%
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Performance – Hybrid MPP Version
Enabling Hybrid MPP unlocks RADIOSS
scalability
• At larger scale, productivity improves as more
threads involves
• As more threads involved, amount of
communications by processes are reduced
• At 32 nodes (or 512 cores), the best
configuration is 2 PPN with 8 threads each
The following environment setting and
tuned flags are used:
I_MPI_PIN_DOMAIN=auto
I_MPI_ADJUST_BCAST=1
I_MPI_ADJUST_REDUCE=2
KMP_AFFINITY=compact
KMP_STACKSIZE=400m
+ user environment: “ulimit -s unlimited”
Higher is better
Intel MPI
FDR InfiniBand
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Performance – Interconnect (HMPP)
FDR InfiniBand provides better scalability performance than Ethernet • 290% better performance than 1GbE at 32 nodes
• 93% better performance than 10GbE at 32 nodes
Higher is better Intel MPI
8 OpenMP Threads/MPI process
93% 290%
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Performance – Processors (HMPP)
Intel E5-2680 (Sandy Bridge) cluster outperforms prior generations • Performs up to 50% better than X5670 cluster at 16 nodes
• Jupiter: 2-socket Intel E5-2680 @ 2.7GHz, 1600MHz DIMMs, FDR IB, 24 HDDs
• Janus: 2-socket Intel X5670 @ 2.93GHz, 1333MHz DIMMs, QDR IB, 1 HDD
Higher is better 8 OpenMP Threads/MPI process
Intel MPI
47%
50%
38%
42% 38%
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Performance – OFA vs DAPL (HMPP)
“OFA provider” under Intel MPI delivers better scalability performance • Up to 15% better application performance than DAPL provider at 32 nodes
Higher is better 8 OpenMP Threads/MPI process
15%
Intel MPI
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Profiling – % Time Spent on MPI
RADIOSS utilizes non-blocking
communications in most data transfers
• The most time MPI calls is MPI_Waitany() and
MPI_Wait()
MPI_Waitany(43%), MPI_Wait(22%), MPI_Bcast(18%),
MPI_Recv(12%)
• Time spent on MPI_Wait and Waitany are for
MPI_Isend/Irecv
Wait time are accounted for time spent on pending
non-blocking transfers
16 Processes/Node
Pure MPP
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
RADIOSS Profiling – MPI Message Sizes
RADIOSS uses small and medium MPI message sizes • Most message sizes are between 0B to 64B, and 257B to 1KB
Pure MPP
16 Processes/Node
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Summary of the HPC Advisory Council Study
RADIOSS is designed to perform at large scale HPC environment
• Shows excellent scalability over 512 cores (32 nodes) and beyond with Hybrid MPP
• Hybrid MPP version enhanced RADIOSS scalability (1 MPI per socket, 8 threads each)
Intel Xeon E5-2600 series and FDR InfiniBand enable RADIOSS to scale
• The E5-2680 cluster outperforms X5670 cluster by up to 50%
Network and MPI comparisons
• Using fast Infiniband interconnect like Mellanox is the key to get good scalability
• Tuning environment parameters is important to maximize performance
• With Intel MPI it is easy to setup RADIOSS Hybrid MPP for optimal performance
A Big Thanks to the HPC Advisory Council for their competencies and great work to study
RADIOSS Scalability!
For more information, please visit
http://www.hpcadvisorycouncil.com/pdf/RADIOSS_Analysis_and_Profiling.ppt.pdf
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
HPC
Hardware
HPC
Workload
Management
Introducing HyperWorks Unlimited
…for Infinite Exploration!
Lease Roll-in Power-Up Connect Simulate Innovate
Unlimited
HyperWorks
Software
Copyright © 2013 Altair Engineering, Inc. Proprietary and Confidential. All rights reserved.
Thanks for your time!
• Eric Lequiniou
• Director, High Performance Computing
• Ravi Kunju
• Managing Director of Strategy and Marketing