High Performance Computational Biomechanics and PRACE The … · 2019. 9. 24. · Diagnose and treatment planning Understanding biological systems Physiological models They provide

High Performance Computational Biomechanics and PRACE

The Perfect Marriage

M. Vázquez, G. Houzeaux and R. de la Cruz

Barcelona Supercomputing Center Centro Nacional de Supercomputación

Spain

BoF “PRACE - The European HPC Infrastructure”SC11 Conference

Seattle - USA

November 2011

Friday, November 25, 11

Bio-mechanics researchers:

Develop the Physiological modelsDeal with medical image processingDesign data acquisition tools

Computational scientists:

Developing computational tools to run simulations

Provide the required simulation capacity

Motivation - Our role

Medical doctors:

Healing is the final objectiveDiagnose and treatment planning

Understanding biological systemsPhysiological models

They provide the main motivation and insight to the problem


Physical understanding

Mathematical modeling

Numerical schemes

Write a code

Make it run efficiently

Upgrade it to new models

Upgrade it to cope with new architectures

Computational scientists:

Developing computational tools to run simulations

Provide the required simulation capacity

High Performance Computational Mechanics

Motivation - Our role


Our tools - ALYA

BSC In-house code

Multi-physics modular code for High Performance Computational Mechanics & DesignBorn in 2004

Numerical solution of PDE’sVariational methods are preferred (FEM, FVM)Hybrid meshes, non-conforming meshesExplicit and Implicit formulationsCoupling between multi-physics (loose or strong)Advanced meshing issues

Parallelization by MPI and OpenMPAutomatic mesh partition using MetisPortability is a mustPorting to new architectures: CELL, GPUs, ...


The Project: Biomechanics

Highly complex geometries

Very fine grain discretizations:

High resolution scanners

Improved geometry definition

Improved anatomical information

Multiscale phenomena

Transient problems

The Goal:

Be capable of

Easily producing good quality refined meshes

Running large scale problems

Postprocessing the results


Project name:

Parallel uniform mesh subdivision in Alya

Preparatory type C

Code development with support from PRACE-1IP experts

Started May 2011

Resources assigned in :

Jugene (Germany)

IBM 72 rack Blue Gene/P294912 Cores, 144 TB memory~1 PFlop Peak, 825.5 TFlops Rmax

Curie (France)

Fat nodes: 360 S6010 bullx nodes, 11520 Cores, 46 TB memory, 105 Tflops PeakHybrid nodes: 16 bullx B chassis, 288 Intel® + 288 Nvidia processors, 192 Tflops PeakThin nodes: 5040 B510 bullx nodes, 80640 cores, 322 TB memory (March 2012)

The Project: Biomechanics


Strategy: Parallel mesh subdivision

Global Mesh partition with METIS


Strategy: Parallel mesh subdivision

Global Mesh partition with METIS

3M P1/P1

24M P1/P1

Local Mesh partition with METIS


Alya runs on Jugene

Physics: Incompressible flow

Numerical strategy: Algebraic fractional momentum BiCGStab + Deflated CG

Geometry: Aneurism, provided by J. Cebral (GMU - USA)

525K nodal points3M tetrahedra

Mesh partitioned on Marenostrum due to memory requirements

Files transferred to Jugene

Run with Mesh Division on Jugene

2.9M P1/P1

23.5M P1/P1

188M P1/P1

1.5B P1/P1

MD (1)

MD (2)

MD (3) 4000

6000

8000

10000

12000

14000

16000

2048 8192 16384

730k 183k 92k

Spee

d up

Number of CPU’s

Average # elements per CPU

IdealJugene BG


Why High Performance Computational Biomechanics?


The Projects

Five projects on Biomechanics:

Cardiac Electromechanical model

Respiratory system

Brain circulatory system

Chemo-mechanics

Mitral valve simulations


Debora Gil Centro de Visión por Computador

Universitat Autonoma de Barcelona (Spain)

Francesc CarrerasUnitat Imatge Cardiaca

Htal. de Sant Pau (Spain)

Manel BallesterUniv. de Lleida (Spain)

Johan Hofmann and Jeannette SpuhlerKTH (Sweden)

Thomas Franz and Jeroen KortsmithUniversity of Cape Town (South Africa)

Daniel AugerUniversity of Cape Town (South Africa)

Antoine Jerusalem and Denny TjahhantoIMDEA Materials (Spain)

Example I

Cardiac Electromechanical model

The Projects


Cardiac Electromechanical Model

Electrical Propagation

Mechanical Deformation Blood Flow

VolumeBo

unda

ries

Coupling

Macro-micro models and bridging scales“Live” tissue: metabolism, perfusion...

Geometrical problemsCoupling: volume and boundaries

Large variabilityUncertainties

Personalized models

Particular features


Oxford ventricular rabbit model

Prof. D. Gavaghan (Computing Laboratory, University of Oxford)

Prof. P. Kohl (Department of Physiology, Anatomy & Genetics, University of Oxford)

Dr. J. Schneider (Department of Cardiovascular Medicine, University of Oxford) BBSRC-funded Oxford 3D Heart Project

Provided by M. Bernabeu and M. Bishop

Downsampled mesh: 431 000 tetrahedra

200 CPUs - Coupled problem

1.5hs for 1.5secs on Marenostrum

Coupled parallel electromechanical model of the heartP. Lafortune, R. Arís, M. Vázquez, G. Houzeaux Submitted to Int. J. Num. Meth. Bioeng. 2011




Three coupled problems:Electrophysiology, Solid Mechanics and Blood flow (with ALE, under development)

One single mesh for EP and SM, solid ALE for the fluidOne parallel code to simulate the full problem

Non-structured mesh coming from medical image processing

Data assimilation through Kalman Filters or Transient Adjoint






Improved fiber model:

DT-MRI fiber field produced

from Johns Hopkins datasets

by D. Gil, A. Borràs and A. Andaluz

CVC - Univ. Autònoma de Barcelona

14 M tetrahedra



0

100

200

300

400

500

0 100 200 300 400 500

430000 4300 2150 1433 860

Spee

d U

p

CPU

Elements/CPU

simulationsideal

Very small problem for

running efficiently in 500

cores!!


Scalability of the coupled model


Results: Implicit scheme scalability

Ideal

Spee

d up

100

200

300

400

500

10 50 100 200 500Number of CPUs

15.5K

2.8K

1.4K 0.7K 0.28KAverage number of elements/CPU

Spee

d up

100

200

300

400

500

Number of CPUs

Average number of elements/CPU

0

Ideal

010 50 100 200 500

124.4K

22.8K

11.4K 5.6K 2.24K

Krylov = 5, toler = 10–5




Spee

d up

100

200

300

400

500

10 50 100 200 500Number of CPUs

1M

179K

90K 44.8K 18KAverage number of elements/CPU

Ideal

0

Legend:

Mesh = 1.12M elementsMesh = 140K elements

Mesh = 9.0M elements

Ideal

Spee

d up

100

200

300

400

500

10 50 100 200 500Number of CPUs

15.5K

2.8K

1.4K 0.7K 0.28KAverage number of elements/CPU

Spee

d up

100

200

300

400

500

Number of CPUs

Average number of elements/CPU

0

Ideal

010 50 100 200 500

124.4K

22.8K

11.4K 5.6K 2.24K





Spee

d up

100

200

300

400

500

10 50 100 200 500Number of CPUs

1M

179K

90K 44.8K 18KAverage number of elements/CPU

Ideal

0

Legend:

Mesh = 1.12M elementsMesh = 140K elements

Mesh = 9.0M elements

With D. Tjahjanto and A. Jersusalem, IMDEA Materials, Spain


D. Taylor, D.J. Doorly, R.C. Schroter, J. Peiró, V. Franke,

R. Ameyda, N. Tolley

Imperical College London (UK)

W. Gedroyc (MRI), A. Wright (CT)

Radiology, St. Mary’s Hospital Paddington London (UK)

D. Firmin (MRI)

Royal Brompton Hospital London (UK)

The Projects

Example II

Respiratory system


Transient1M->27M elementsPrism boundary layerInlet: periodic pressure or velocityOutlet: zero pressure

zero pressure

velocity or pressure

∝ cos(2π/T)

Respiratory System


Respiratory System


Respiratory System


0

0.5

1

1.5

2

2.5

3

3.5

0 0.5 1 1.5 2 2.5

Point08

1.1M4.4M

8.6M <= 1.1M14.2M

Respiratory System

Transition captured by uniform mesh refinement

Computational Modelling of Airflow in theRespiratory Tract D. Doorly, D. Taylor, G. Houzeaux, C. Rennie, N. Tolley and M. Vázquez

Presented at ParCFD 2011 Conference


Design variables:

15 - 30 parameters + Windkessel’s

Fiber field, obtained with high definition scanners

Initial impulse (Purkinje system terminals)

High definition geometries


Conclusion


Design variables:






Not one... but

hundreds or tens of runs for one single heart

Smart data assimilation algorithms

Conclusion


Design variables:






Unprecedented availability of computing power...

... and rising!

Exotic architectures, new programming models

Not one... but

hundreds or tens of runs for one single heart

Smart data assimilation algorithms

Conclusion


Sensitivity Analysis:

Fiber distribution from functional analysis

Initial conditions: Purkinje system

Compressibility

Model parameters

Geometry variability


Conclusion





Compressibility

Model parameters



Conclusion

Model improvement:

Electrical + Mechanical + Fluid

Valves

Multiscale: we can arrive at cell level


Model improvement:


Valves





Compressibility

Model parameters



Conclusion





Compressibility

Model parameters



Conclusion

Biomedical applications:

Hydrogel injection on infarcted hearts

Effects of drugs on ionic channels

Patient specific models

Model improvement:


Valves



High Performance Computational Biomechanics and PRACE

The Perfect Marriage

M. Vázquez, G. Houzeaux and R. de la Cruz

Barcelona Supercomputing Center Centro Nacional de Supercomputación

Spain

BoF “PRACE - The European HPC Infrastructure”SC11 Conference

Seattle - USA

November 2011


Documents

High Performance Computational Biomechanics and PRACE The … · 2019. 9. 24. · Diagnose and treatment planning Understanding biological systems Physiological models They provide