The Centre for Australian Weather and Climate ResearchA partnership between CSIRO and the Bureau of Meteorology
On the efficient tracking of grid point storm issues
Ilia Bermous
13 October 2011
2
Grid point storm issuesGrid point storm issues
The right modelling of physical processes and finite difference parameter settings to avoid grid point storm issues
Task of efficient handling/tracking grid point storm issues
Informative diagnostic to be able to track grid point storm issues
Efficient usage of computer resources when numerical analysis becomes unstable and produces rubbish
3
Ineffiency of the UM TRAP_W subroutine Ineffiency of the UM TRAP_W subroutine
Practical aspects
Implementation aspects
4
Ineffiency of the UM TRAP_W subroutine:practical aspects
Ineffiency of the UM TRAP_W subroutine:practical aspects
UM model uses the following capping condition
trap_option flag (0|1|2):
0 – reset, no diagnostic; 1 – reset + diagnostic
2 – diagnostic only
if trap_option=2 => additional meaningless (based on a free choice of Cw) output is produced, so nothing new seen so far
if trap_option=0|1 => the model uses artificial resetting (“cooking” in numerical analysis have not seen before), this kind of actions may only be useful for debugging purposes of the implemented algorithm
NOTE: in tracking down the grid point storm issue we have not been interested on how condition (1) was fullfield, we are interested on a max value of the vertical wind component w.
t
zCw kwijk
|| (1)
5
Ineffiency of the UM TRAP_W subroutine:practical aspects (cont #2)
Ineffiency of the UM TRAP_W subroutine:practical aspects (cont #2)
Cw=1000 – default setting; Cw=2&4 – in Gary’s last meeting report
Let’s consider 2 cases for the “Jan 13” original crash job
Cw=1 (job completes making 900 time steps => 75hour)w_max level proc position run w_max level timestep
0.377E+01 44 113 42.3% East 60.1% North 0.553E+01 46 872
Cw=10 (job completes)w_max level proc position run w_max level timestep
0.479E+01 41 160 40.4% East 81.9% North 0.186E+02 40 174
Cw time step at which W
started to reset
Total points for upward resetting and number of time steps used for
resetting
Total points for downward resetting and number of times
steps used for resetting
1 1 6,020,289 900 1,958,920 900
10 79 17,491 415 none
6
Ineffiency of the UM TRAP_W subroutine:implementation aspects
Ineffiency of the UM TRAP_W subroutine:implementation aspects
Let’s take a piece of a UM code referred by Gary:
IF ((k>Cw_test_lev).OR.(ic_xy(i,j)>0)) THEN ic_xy(i,j) = 1
ic = ic + 1
w_adv(i,j,k) = w_test(i,j)
k>Cw_test_lev condition should be moved outside and merged with the loop
Do k=model_levels-1,1,-1 & k>Cw_test_lev condition =>
Do k=model_levels-1,max(1, Cw_test_lev+1),-1
Condition ic_xy(i,j)>0 is redundant
There is a number of other redundant statements in the subroutine
7
Concepts of a new designConcepts of a new design 2 new terms have been introduced
w_soft_limit – value to trigger a diagnostic to be produced if |w| > w_soft_limit
w_hard_limit – value to stop execution if |w| > w_hard_limit condition is achieved on a PE
New development has been included in the sources by using a pre-processor logic with
#if defined (BOM_TRAP)
statements
Only 4 Fortran files have been modified ./atmosphere/dynamics_diagnostics/trap_w.F90./control/top_level/atm_step.F90./control/top_level/readlsta.F90./include/common/cruntimc.h
8
ResultsResults
New implementation was tested using UM7.5 R12 sources
Chris R12 xbawq job (13 Jan 2011 case) has been used
Here is some output with w_soft_limit=20 and w_hard_limit=100 setting
( “soft_limit” output string was used for an easy grapping)
w_max level proc position N points timestepsoft_limit 1.5E+02 56 149 48.7% East 78.4% North 4 218
On each PE a single output line for each level is produced with a k_total number of points for which the soft limit condition was fulfilled
Execution in this run was stopped after reaching w_hard_limit at time step 218 and 1407 soft_limit messages were produced from all PEs
. . ./dataw1 > grep "soft_limit " *fort6* | wc -l1407
9
Some numerical/algorithm aspects Some numerical/algorithm aspects
R12: Δt=300sec, 70 levels, 400m< Δzk<8000m, k>29 where grid point storm starts
capping
Cw=1 for k=70, Cw=10 for k=30
Semi-implicit finite-difference schema stabilityΔt=600sec => |w| > 100 m/sec at T=3*Δtas |w| may grow during integration this shows that the chosen Δt is not small enough to satisfy the stability requirement of the finite-difference operators used in the modellingran case with Δt=60sec using “tip” from Stuart’s presentation (it takes ~9.5hours for 75h integration on 192 cores), note that for this value of Δt a different range of Cw values should be used for trap_w condition (1)
w_max level proc position run w_max level timestep0.886E+01 42 162 41.1% East 87.4% North 0.218E+02 37 700
wjiwjik
wijk CwCwt
zCw 3
270,,30,, 26||,
3
4||,||
10
ConclusionsConclusions
A new developed version of TRAP_W subroutine produces useful information on tracking relatively large values of w vertical wind component
A moderate size of the output data is produced
Numerical analysis is stopped when “unreal” values of vertical wind component w are produced
One of the possible ways to avoid a grid point storm problem is to use a relatively small time step Δt according to the stability requirement of the finite-difference operators used in the modelling Results comparison between Cw=10, Δt=300sec “cooked” case
and Δt =60sec shows some large instability near the domain boundary
11
Conclusions (cont #2)Conclusions (cont #2)
12
Conclusions (cont #3)Conclusions (cont #3)