Zellescher Weg 12

Willers-Bau A106

Tel. +49 351 - 463 - 31945

Matthias Lieber (matthias.lieber@tu-dresden.de)

Tobias Hilbrich (tobias.hilbrich@zih.tu-dresden.de)

Center for

Information Services and High Performance Computing

(ZIH)

The Heat Example

HRSK Practical on Debugging, 03.04.2009

2

Content

Heat Equation

The Example Code

OpenMP Version

MPI Version

Login to HRSK

Building the Heat Example

Running and Plotting

3

Heat Equation

Heat equation solver as debugging example

Heat equation describes heat distribution in a region over time

Equation (2D):

Example implements solution on a 2D grid

Time step for a grid cell (x,y):

Visualization as 3D chart:

⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

+∂∂

=∂∂

2

2

2

2

yu

xuk

tu

⎟⎟⎠

⎞⎜⎜⎝

⎛Δ

−++−+

Δ−++−

=Δ

Δ22

],[2]1,[]1,[],[2],1[],1[],[y

yxuyxuyxux

yxuyxuyxukt

yxu

Time

Steps

4

The Example Code

Available in C and Fortran90

3 Versions: Serial, OpenMP, and MPI

Key object is the structure for the 2D grid

Functions:

– heatAllocate

& heatDeallocate

–

Creates/Frees the grid

– heatInitialize

& heatInitFunc

–

Sets the initial heat distribution

– heatPrint

& heatOutput

–

Prints the grid to stdout

or a data file

– heatTimestep

–

Calculates one timestep

for the full grid

– heatBoundary

–

Exchanges boundary data

– heatTotalEnergy

–

Calculates overall energy amount

– Main function –

Contains main loop

5

The Example Code - Grid

Grid Structure contains 2 grids and the grid size

Grid:

Ghost cells used as neighbors, needed for border cells of actual

grid

Time steps are calculated for all actual grid cells, results stored in second grid

xsize

ysize

Ghost

Cell

(Boundary

Cell)

Grid

Cells

(1,1)

(3,2)

Current Grid: Temporary Grid:⎟⎠⎞

⎜⎝⎛ +

Δ−++−

Δ+ ...],[2],1[],1[],[ 2xyxuyxuyxutkyxu

Grids switched after calculation for all grid cells

5 point stencil

6

The Example Code - Ghost Cells

5 point stencil needs left, right, up, down neighbor for each grid point

Ghost cells serve as these neighbors

Their values can either be constant or calculated from the grid

The code uses periodic ghost cells

After each time step a copy operation is necessary to update the

ghost cells

Implemented in “heatBoundary”

Update:

Grid:(1) Left border (2) Right border (3) Top border (4) Bottom border

7

The Example Code - Main Program

Starts initialization

Iterates time steps and boundary exchange (heatTimestep, heatBoundary)

Each time step calculation returns the max temperature difference

Iterations stop when max difference lower than a set value

Prints the initial and final grid to standard output

Prints 4 snapshots of the heat grid to a file “data-<#Iterations>.dat”

A provided gnuplot

script visualizes this data

8

OpenMP Version

OpenMP version is simple

Both grids are kept in shared memory

Time step calculation uses multiple threads

No data races, due to saving the results in the temporary grid

Calculation of the maximum temperature difference needs adaption

9

MPI Version - Overview

Each process has its own copy of both grids

Initialization and set-up done by all processes

Each process calculates on a part of the overall grid:

MPI functions used to create this 2d cartesian

grid

Communications of a data row(Fortran)/column(C) uses a derived datatype

Consequences:

– New border exchange

– Gathering of data necessary for output

P0 P1

P3P2

10

MPI Version – Border Exchange

MPI processes need to exchange borders with their neighbors

The cartesian

grid provides up, down, left, right neighbors

Periodic boundaries created with periodic cartesian

grid

– The 2D grid is in fact a donut !

Boundary exchange for process “Pi”:

Pk

PiPj Pm

Pn

Neighborhood of Pi Exchange for Pi

To PkFrom Pk

To Pm

From Pm

To Pn From Pn

To Pj

From Pj

11

MPI Version – Gather

Data distributed between processes after first time step

Printing either requires IO by all processes or gathering data on one process

Implementation uses data gathering on process 0

P0 Copy

P1Copy

P3Copy

P2

P1

P2 P3

P0: P1:

P3:P2:

12

Login to HRSK

Login into your PC

Open a SSH connection to mars.hrsk.tu-dresden.de

Start -> Programme

-> SSH Secure Shell -> Secure Shell Client

Enable X11 forwarding:

Switch it on!

13

Connect to mars:

Login to HRSK 2

mars.hrsk.tu-dresden.de

14

Building the Heat Example

If you need an editor use: vi, mcedit, nano, nedit, gedit, or kwrite

Use the Intel compilers

– module load intel

C users: cd

~/debuggingC/serial

Fortran users: cd

~/debuggingF/serial

The folder contains all example codes for serial debugging

“exampleC.c”/”exampleF.F90”

contain the working version of the heat code

“exampleC-XX.c”/”exampleF-XX.F90”

are versions with various errors

Build the working code with:

– C: icc

exampleC.c

–o example.exe

– Fortran: ifort

exampleF.F90 –o example.exe

15

Running and Plotting

Start an interactive batch session (optional for the serial part

of the course):

– bsub -U schulung#0 -n 4 -Is -W 2:00 bash

Execute: ./example.exe

The code prints the initial and final grid as well as the initial and final energy

It also creates data files named “data-XXXXXXXX.dat”

Visualize the data files with:

– bash build-gnuplot.sh

– gnuplot

gnuplot.ini

– This should give you an animation from 4 different timesteps

– The scripts also generate a postscript, open it with gv

heat-conduction.ps