Running OpenFOAM in parallel on the grid: Running OpenFOAM in parallel on the grid: CFD in the study of cardiovascular diseaseCFD in the study of cardiovascular disease

11Bogdan Ene-IordacheBogdan Ene-Iordache, , 11Massimo Rizzi, Massimo Rizzi, 22Daniele Cesini,Daniele Cesini,22Emidio Giorgio, Emidio Giorgio, 22Luciano Gaido and Luciano Gaido and 22Giuseppe La Rocca Giuseppe La Rocca

11 Mario Negri Institute for Pharmacological Research, Italy;Mario Negri Institute for Pharmacological Research, Italy;22 INFN and IGI, ItalyINFN and IGI, Italy

1

The The Mario Negri Institute Mario Negri Institute for Pharmacological Research is a not-for-for Pharmacological Research is a not-for-profit biomedical research organization founded in 1961profit biomedical research organization founded in 1961

• The Institute’s research programs span from the molecular level to The Institute’s research programs span from the molecular level to the whole human being, and the findings help build up the basis for the whole human being, and the findings help build up the basis for developing new drugs or making existing ones more effectivedeveloping new drugs or making existing ones more effective

• Main research headings are the battle against cancer, nervous and Main research headings are the battle against cancer, nervous and mental illnesses, cardiovascular and kidney diseases, rare diseases mental illnesses, cardiovascular and kidney diseases, rare diseases and the toxic effects of environmental contaminants, mother and and the toxic effects of environmental contaminants, mother and child’s healthchild’s health

2

Mario Negri Institute Mario Negri Institute is organized in Departments, Laboratories and Research Units is organized in Departments, Laboratories and Research Units

Department of Biomedical EngineeringDepartment of Biomedical Engineering

Conducts research and development in biomedicine, both at experimental and clinical level Conducts research and development in biomedicine, both at experimental and clinical level

The ongoing studies are related to four main areas: The ongoing studies are related to four main areas:

• study of the mechanisms involved in the progression of chronic nephropathystudy of the mechanisms involved in the progression of chronic nephropathy

• studies on the role of haemodynamics in the development of vascular diseasesstudies on the role of haemodynamics in the development of vascular diseases

• development of laboratory techniques for tissue engineeringdevelopment of laboratory techniques for tissue engineering

• development of information systems to manage clinical data and images generated in the development of information systems to manage clinical data and images generated in the context of controlled clinical trials and in routine clinical practicecontext of controlled clinical trials and in routine clinical practice

3

CLINICAL BACKGROUNDCLINICAL BACKGROUNDIntroducing atherosclerosisIntroducing atherosclerosis

Coronary Heart DiseaseCoronary Heart Disease

Carotid Artery DiseaseCarotid Artery Disease

• Atherosclerosis is the thickening of the arterial wall due Atherosclerosis is the thickening of the arterial wall due to formation of an atherosclerotic plaqueto formation of an atherosclerotic plaque

Source: National Heart Lung and Blood Institutehttp://www.nhlbi.nih.gov/health/health-topics/topics/atherosclerosis/

Peripheral Arterial DiseasePeripheral Arterial Disease

• Complex pathological process Complex pathological process in the walls of blood vessels in the walls of blood vessels that develops over many years that develops over many years

• Fatty material and cholesterol Fatty material and cholesterol are deposited inside the lumen are deposited inside the lumen of blood vessels. These of blood vessels. These deposits cause deposits cause stenosisstenosis of the of the blood vesselblood vessel

• The plaque can trigger The plaque can trigger formation of blood clotformation of blood clot

• Atherosclerosis leads to Atherosclerosis leads to heart attackheart attack, , strokestroke, both with high mortality, both with high mortality

4

CLINICAL BACKGROUNDCLINICAL BACKGROUNDGlobal burden of Cardiovascular DiseaseGlobal burden of Cardiovascular Disease

• An estimated An estimated 17 million17 million people die of people die of cardiovascular diseases, particularly heart cardiovascular diseases, particularly heart attacks and strokes, every year.attacks and strokes, every year.

• Coronary heart disease kills more than Coronary heart disease kills more than 7 7 millionmillion people each year, and strokes kills people each year, and strokes kills nearly nearly 6 million6 million. Most of these deaths are . Most of these deaths are in developing countries.in developing countries.

WHO - The Atlas of Heart Disease and StrokeWHO - The Atlas of Heart Disease and Stroke (http://www.who.int/cardiovascular_diseases/resources/atlas/en/)(http://www.who.int/cardiovascular_diseases/resources/atlas/en/)

What about the future ?What about the future ?

5Year 2002Year 2002

CLINICAL BACKGROUNDCLINICAL BACKGROUNDHaemodynamics and atherosclerosisHaemodynamics and atherosclerosis

• The entire arterial tree is exposed to atherogenic The entire arterial tree is exposed to atherogenic effects of systemic risk factorseffects of systemic risk factors

• The haemodynamic conditions play a decisive The haemodynamic conditions play a decisive role in vascular remodelling and in the role in vascular remodelling and in the development of atherosclerosisdevelopment of atherosclerosis

• Understanding of the patho-biologic processes Understanding of the patho-biologic processes responsible for atherosclerosis might allow responsible for atherosclerosis might allow identification of new therapies identification of new therapies

• However, the atherosclerotic lesions form at However, the atherosclerotic lesions form at specific arterial sitesspecific arterial sites

6

Pressure (p) is the force per unit area normal to the vessel wall, and results in circumferential stretching of the vessel wall

Shear stress () is the tangential force per unit area of the vessel wall and is exerted in the direction of blood flow

(r) = drdu

[ dynes/cm2]r

u(r) u

C Hahn* and MA Schwartz, NatRev, 2009

= Shear stress

= blood viscosity

Shear rate

BACKGROUNDBACKGROUNDMechanical forces acting on the vessel wallMechanical forces acting on the vessel wall

7

Regions in which Regions in which complex flow patterns complex flow patterns

developdevelop((oscillatory flowoscillatory flow))

Regions in which flow is always Regions in which flow is always in the same direction and in the same direction and patterns are patterns are laminarlaminar

Endothelial cells (Endothelial cells (ECEC) act as ) act as mechanoreceptorsmechanoreceptors detecting and responding to detecting and responding to shear stressshear stress

After mechanoreceptor activation, a complex network of intracellular pathways is After mechanoreceptor activation, a complex network of intracellular pathways is triggered (triggered (mechanotransductionmechanotransduction))

C Hahn* and MA Schwartz, NatRev, 2009

BACKGROUNDBACKGROUNDEffect of different blood flow patternsEffect of different blood flow patterns

8

DISTURBED FLOWDISTURBED FLOW LAMINAR FLOWLAMINAR FLOW

BACKGROUNDBACKGROUNDAthero-prone and athero-protective waveformsAthero-prone and athero-protective waveforms

9Dai G et al., PNAS, 2004

Athero-prone shear stress waveform induces:Athero-prone shear stress waveform induces:

Dysregulation of EC cytoskeletal andDysregulation of EC cytoskeletal andjunctional proteinsjunctional proteins

Proinflammatory phenotypeProinflammatory phenotypeCytokine inducible cell surface expression of Cytokine inducible cell surface expression of adhesion molecules associated with adhesion molecules associated with atherosclerosisatherosclerosis

BACKGROUNDBACKGROUNDCFD in the study of arterial diseasesCFD in the study of arterial diseases

Malek et al, Malek et al, JAMAJAMA, 1999 (review), 1999 (review)10

T

w dtTTAWSS

0

1

T

w

T

w

dt

dtOSI

0

0121

1])21[( TAWSSOSIRRT

TAWSSTAWSS = Time Averaged Wall Shear Stress = Time Averaged Wall Shear Stress

Lee, Lee, J Biomech Eng, 2009J Biomech Eng, 2009

OSIOSI = Oscillatory Shear Index = Oscillatory Shear Index

quantifies the deviation of WSS from its “natural” quantifies the deviation of WSS from its “natural” directiondirection during the cardiac cycle. Varies from 0 e during the cardiac cycle. Varies from 0 e 0.5.0.5.

He and Ku, J Biomech Eng, 1996He and Ku, J Biomech Eng, 1996

RRTRRT = Relative Residence Time = Relative Residence Time

is a combination of both TAWSS and OSIis a combination of both TAWSS and OSI

H. A. Himburg, Am J Physiol Heart Circ Physiol, 2004H. A. Himburg, Am J Physiol Heart Circ Physiol, 2004

BACKGROUNDBACKGROUND““Disturbed-flowDisturbed-flow”” indicators indicators(haemodynamic wall parameters)(haemodynamic wall parameters)

11

OpenFOAM: overviewOpenFOAM: overview• OpenFOAM (Open Field Operation and Manipulation) OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox is a free, open CFD Toolbox is a free, open source CFD software package produced by a commercial company,source CFD software package produced by a commercial company, OpenCFD Ltd OpenCFD Ltd(latest news: was acquired by ESI group on Sept 12) (latest news: was acquired by ESI group on Sept 12) and distributed by theand distributed by the OpenFOAM OpenFOAM Foundation (www.openfoam.org)Foundation (www.openfoam.org)

www.openfoam.comwww.openfoam.com 12

• Large user base across most areas Large user base across most areas of engineering and science, from both of engineering and science, from both commercial and academic commercial and academic organizationsorganizations

• OpenFOAMOpenFOAM has an extensive range has an extensive range of features to solve from complex of features to solve from complex fluid fluid flowsflows involving involving chemical reactionschemical reactions, , turbulenceturbulence and and heat transferheat transfer, to , to solid dynamicssolid dynamics and and electro-electro-magneticsmagnetics

Installing OpenFOAM on grid infrastructureInstalling OpenFOAM on grid infrastructureThanks to the NGI_IT User Support team, OpenFOAM (v. 2.0.1) with Thanks to the NGI_IT User Support team, OpenFOAM (v. 2.0.1) with openmpiopenmpi support (ver. support (ver. 1.5.3) and 1.5.3) and ParaViewParaView (ver. 3.10.1), has been successfully installed on the NGI_IT computing (ver. 3.10.1), has been successfully installed on the NGI_IT computing infrastructureinfrastructure

• Due to the peculiar directory structure and the need to appropriate script sourcing on Due to the peculiar directory structure and the need to appropriate script sourcing on all the grid nodes before to start the analyses, the installation and the configuration all the grid nodes before to start the analyses, the installation and the configuration of OpenFOAM on the grid infrastructure was not easyof OpenFOAM on the grid infrastructure was not easy

• Additional SW packages were requested:Additional SW packages were requested:OpenFOAM-2.0.1 builds on many Linux distributions OpenFOAM-2.0.1 builds on many Linux distributions butbut the ParaView-3.10.1 the ParaView-3.10.1

supplied in ThirdParty required the following add-ons:supplied in ThirdParty required the following add-ons:• cmake-2.8.6 or higher;cmake-2.8.6 or higher;• Qt-4.6.4;Qt-4.6.4;• Compiler GNU ver. 4.0.0.Compiler GNU ver. 4.0.0.

• Defined a new MPI wrapper for OpenFOAM:Defined a new MPI wrapper for OpenFOAM:It has been created in It has been created in /opt/i2g/etc/mpi-start/, the following MPI wrapper the following MPI wrapper

/opt/i2g/etc/mpi-start/openmpi_openfoam.mpi to run OpenFOAM in parallelto run OpenFOAM in parallel

• Post-configuration on the LSF master nodePost-configuration on the LSF master nodeIn order to properly set-up the OpenFOAM environment on all WNs In order to properly set-up the OpenFOAM environment on all WNs

dedicated to the parallel execution, the following setting dedicated to the parallel execution, the following setting . /opt/exp_soft/gridit/OpenFOAM/OpenFOAM-2.0.1/etc/bashrc has been has been included in the included in the /etc/bashrc of the LSF master nodeof the LSF master node 13

OpenFOAM in grid: more detailsOpenFOAM in grid: more details• The open-source software has been installed for the The open-source software has been installed for the GRIDITGRIDIT VO VO

For technical information about installation and configuration of For technical information about installation and configuration of OpenFOAM in grid, please refer to the following wiki:OpenFOAM in grid, please refer to the following wiki:

https://wiki.italiangrid.it/twiki/bin/view/UserSupport/OpenFoamhttps://wiki.italiangrid.it/twiki/bin/view/UserSupport/OpenFoam

14

OpenFOAM in grid: validationOpenFOAM in grid: validation• To test the installation of OpenFOAM, we have used OpenFOAM tutorial casesTo test the installation of OpenFOAM, we have used OpenFOAM tutorial cases

• To validate the installation of OpenFOAM in a grid cluster, we have used parallel jobs with To validate the installation of OpenFOAM in a grid cluster, we have used parallel jobs with very big cylindrical meshes (very big cylindrical meshes (1,824,200 cells1,824,200 cells))

3-D big mesh - 1,824,200 cells3-D big mesh - 1,824,200 cells

icoFoamicoFoam solver solver

transient case - 100 timestepstransient case - 100 timesteps

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

8 CPU

4 CPU

1 CPU

speed-up x 3.0 speed-up x 3.0

speed-up x 3.9 speed-up x 3.9

(serial case x 1.0) (serial case x 1.0)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

8 CPU

4 CPU

1 CPU

speed-up x 1.3 speed-up x 1.3

speed-up x 1.1 speed-up x 1.1

(serial case x 1.0) (serial case x 1.0) pitzDailypitzDaily

2-D mesh 2-D mesh -- 12,200 cells 12,200 cells

simpleFoam simpleFoam solversolver

steady state casesteady state case

15

Application’s workflowApplication’s workflow

From a technical point of view, a From a technical point of view, a parallel CFDparallel CFD runrun is performed in three different steps: is performed in three different steps:

• Initially, the master node prepares the case by using OpenFOAMInitially, the master node prepares the case by using OpenFOAM’’s domain s domain decomposition decomposition decomposePardecomposePar utility: utility:

processorNNprocessorNN directories for each CPU are created with decomposed mesh, directories for each CPU are created with decomposed mesh, fields, solution controls, model choice and discretization parameters; fields, solution controls, model choice and discretization parameters; distribution of input data to each slave node is subsequently guaranteed distribution of input data to each slave node is subsequently guaranteed by the grid infrastructure (by the grid infrastructure (mpi_startmpi_start))

• The CFD case is then solved with The CFD case is then solved with icoFoamicoFoam solver on a per-processor basis, solver on a per-processor basis, where each CPU node uses its local disk space to store results data in time where each CPU node uses its local disk space to store results data in time directoriesdirectories

• At the end of the run, results are collected from the slave nodes to the master At the end of the run, results are collected from the slave nodes to the master node, and the resulting archive is then saved on a grid SE using user-defined node, and the resulting archive is then saved on a grid SE using user-defined post hookspost hooks

The numerical and graphical post-processing is performed locally after using the The numerical and graphical post-processing is performed locally after using the reconstructParreconstructPar utility of OpenFOAM to recreate the case to a single CPU utility of OpenFOAM to recreate the case to a single CPU

16

CFD simulation on Carotid BifurcationCFD simulation on Carotid BifurcationMETHODSMETHODS

DensityDensity

ViscosityViscosityρρ = 1.045 g/cm= 1.045 g/cm33

= 0.033 Poise (Newtonian) = 0.033 Poise (Newtonian)

BLOOD PROPERTIESBLOOD PROPERTIES

UNSTEADY CFD RUNSUNSTEADY CFD RUNS

simulation of simulation of 3 cardiac cycles3 cardiac cycles

icoFoamicoFoam solver solver -parallel-parallel implicit, backward timeintegrationimplicit, backward timeintegration velocity waveform in CCA:velocity waveform in CCA:

CCA

ECAICA

0.51.01.52.02.53.0

0 0.2 0.4 0.6 0.8 1

t/T

Q/Q

mea

n

17

3-D mesh 3-D mesh ~ ~ 200,000 cells200,000 cells

Ford MD, et al., Physiol Meas, 2005

0.51.01.52.02.53.0

0 0.2 0.4 0.6 0.8 10.51.01.52.02.53.0

0 0.2 0.4 0.6 0.8 1

Simulation on Carotid BifurcationSimulation on Carotid BifurcationRESULTS: WSSRESULTS: WSS

Peak systolic WSSPeak systolic WSS Diastole WSSDiastole WSS18

[dyne/cm2]

Simulation on Carotid BifurcationSimulation on Carotid BifurcationRESULTS: TAWSSRESULTS: TAWSS

19

[dyne/cm2]

Simulation on Carotid BifurcationSimulation on Carotid BifurcationRESULTS: OSIRESULTS: OSI

20

Simulation on Carotid BifurcationSimulation on Carotid BifurcationRESULTS: RRTRESULTS: RRT

21

Low and oscillating wall shear stress is a powerful stimulus for atherogenesis progression of early atherosclerotic plaques differentiation to high-risk plaque

Patient specific CFD models can be used for describing the haemodynamics in those arteries prone to atherogenesis and for predicting potential risks of developing and sustaining atherosclerosis

NGI_IT infrastructure has been successfully set up and used to efficiently run parallel CFD simulations in large arteries

Future improvements of our numerical simulations will include setting of fluid-structure interaction (FSI) for blood vessels, a new challenge for the cardiovascular biomechanics research

CONCLUSIONSCONCLUSIONS

22

ACKNOWLEDGEMENTSACKNOWLEDGEMENTS

23

Thank you !Thank you !