Build and Run GSI on S4 · • Untar both tarball files: > tar –xvf comGSIv3.6_EnKFv1.2.tar > tar –xvf CRTM_2.2.3.tar • Go into comGSIv3.6_EnKFv1.2/dtc/ and run configure script

Build and Run GSI on S4

Ting-Chi Wu ([email protected]) CIRA Data Assimilation Group

Feb 15, 2018

2018 NOAA Data Assimilation Training at CIRA

•  Go to: https://dtcenter.org/com-GSI/users/downloads/index.php •  Type your e-mail address to begin download •  Go for the community GSI Version 3.6 and EnKF Version 1.2 •  Download both 1) comGSIv3.6_EnKFv1.2_tarball and 2)

CRTM_2.2.3_Big_Endian_coefficients_tarball •  Make sure to check out Release notes and Known issues •  You may also want to download GSI v3.6 user guide for more

information: https://dtcenter.org/com-GSI/users/docs/index.php

•  Put both tarball files onto your S4 account (scp or sftp) > scp name_of_tarball [email protected]:/data/users/username/

Build GSI on S4 (1/3)

•  Set up environment in your .bashrc (or .cshrc, .kshrc, etc), which usually sits in your home directory at /home/username/ –  WRF_DIR –  NETCDF –  LAPACK_PATH

Example .bashrc file using Intel build: module load license_intel intel/14.0-2 impi/5.0.3.048 module load udunits2 hdf hdf5 netcdf4/4.1.3 nco parallel-netcdf ncview export NETCDF=/opt/netcdf4/4.1.3-intel-14.0-2 export PNETCDF=/opt/pnetcdf/1.6.1-intel-14.0-2 module load shellb3 module load grads export WRF_DIR=/home/twu/hwrfrun/sorc/WRFV3/ export LAPACK_PATH=/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64


•  Reload your .bashrc by typing “source .bahsrc” or simply log out and log back in

•  Untar both tarball files: >  tar –xvf comGSIv3.6_EnKFv1.2.tar >  tar –xvf CRTM_2.2.3.tar

•  Go into comGSIv3.6_EnKFv1.2/dtc/ and run configure script to select a compiler option, then run the compile script and redirect output to a log file: > cd comGSIv3.6_EnKFv1.2/dtc/ > ./configure > Select option 7. Linux x86_64, Intel compiler (ifort & icc)

(dmpar, optimize) > ./compile |& tee compile.log

•  If the build is successful, you should see an executable named gsi.exe sitting in both src/ and dtc/run/.


•  Before running GSI, you should know: –  gsi.exe is available (located at src/ or dtc/run) –  Setup gsiparm.anl (the namelist file) –  Decide on a work directory ($WORK_ROOT) –  Location and name of background files (e.g., wrfout) –  Location and name of observation (e.g., *bufr files) –  Location of the fix/ directory (e.g., *info files) –  Location of the CRTM coefficient directory (should be located

at CRTM_v2.2.3/) for satellite radiance assimilation

•  run_gsi_*.ksh has it all! ü A sample run script to start with (under dtc/run/) ü Will prepare gsiparm.anl for you using comgsi_namelist*.sh

Running GSI on S4 (1/4)

•  Download data for practice cases here: https://dtcenter.org/com-GSI/users/tutorial/online_tutorial/releaseV3.6/data/download_data.php –  We will use Practice case 3: ARW Hybrid 3DEnVar –  I have downloaded and put them on S4 at /data/users/twu/

GSIv3.6_PracticeCase3/ –  You are recommended to try other practice case(s) after the class

•  > cd comGSIv3.6_EnKFv1.2/dtc/run/ •  > cp run_gsi_regiona.ksh run_gsi_regiona_wrfarw.ksh •  Then, we will modify run_gsi_regiona_wrfarw.ksh to run the

practice case and go over the gsiparm.nml file together


•  Modifications to the run_gsi_regiona_wrfarw.ksh: –  ANAL_TIME=2017051312 –  WORK_ROOT=/scratch/username/gsi_wrfarw_$ANAL_TIME –  OBS_ROOT=/data/users/twu/GSIv3.6_PracticeCase3/obs –  BK_ROOT=/data/users/twu/GSIv3.6_PracticeCase3 –  BK_FILE=${BK_ROOT}/wrfinput_d01.dms –  CRTM_ROOT=PATH_TO_YOUR_CRTM_2.2.3_DIR –  GSI_ROOT=/data/users/username/comGSIv3.6_EnKFv1.2 –  FIX_ROOT=${GSI_ROOT}/fix –  GSI_EXE=${GSI_ROOT}/dtc/run/gsi.exe –  GSI_NAMELIST=${GSI_ROOT}/dtc/run/comgsi_namelist.sh –  If_hybrid=Yes –  ENS_ROOT=/data/users/twu/GSIv3.6_PracticeCase3 –  ENSEMBLE_FILE_mem=${ENS_ROOT}/gfsens/sfg_2017051306_fhr06s


•  Include the below MPI script in run_gsi_regional_wrfarw.ksh

•  Add S4 SLURM option in the $ARCH section & set ARCH=‘S4’

•  Submit job with sbatch > sbatch run_gsi_regional_wrfarw.ksh •  Monitor your job status > squeue –u username

#!/bin/bash #SBATCH --job-name=YOUR_JOB_NAME #SBATCH --partition=s4 #SBATCH --account=star #SBATCH --ntasks=90 #SBATCH --mem-per-cpu=6000 #SBATCH --time=02:00:00 #SBATCH --output=PATH_TO_OUTPUT_DIR/YOUR_JOB_NAME.%j

’S4’) ##### JCSDA S4 SLURM RUN_COMMAND="srun --cpu_bind=core --distribution=block:block" ;;

Ø Check out S4 User Guides for more info (need username/password to log in): https://groups.ssec.wisc.edu/acl_users/credentials_cookie_auth/require_login?

came_from=https%3A//groups.ssec.wisc.edu/groups/S4/s4-user-guides


•  The standard output file: stdout •  Structure of stdout: – Read in all data and prepare analysis:

•  Read in configuration (gsiparm.anl namelist) •  Read in background (wrf_inou*) •  Read in observations (bufr files) •  Partition domain and data for parallel analysis •  Read in constant fields (fix files) •  Read in ensemble files (if hybrid)

–  GSI analysis –  Update guess and save analysis result

Begin J table inner/outer loop 0 2 J term Jbackground 1.3012353608618746E+03surface pressure 1.9513868407565806E+03temperature 2.4889488235135345E+03wind 4.1005694521652695E+03moisture 7.8109360307634040E+02----------------------------------------------------- J Global 1.0623234080373601E+04 End Jo table inner/outer loop 0 2...observer_final:successfullyfinalizedglbsoi:complete[000]gsisub()::complete. ENDING DATE-TIME FEB 06,2018 22:53:23.661 37 TUE 2458156PROGRAMGSI_ANLHASENDED.

Ø But… this doesn’t guarantee a successful assimilation Ø you’ll need to look into more details

Standard Output and Diagnostics (1/4)

•  fort.2** files: innovation (O-B and O-A; fit to observations) statistics from each outer loop –  fort.201: surface pressure data (ps) –  fort.202: wind data (uv) –  fort.203: temperature data (t) –  fort.204: moisture data (q) –  fort.205: precipitable water data (pw) –  fort.206: ozone observations (oz) –  fort.207: satellite radiance data (rad) –  fort.208: precipitation data (pcp) –  fort.209: radar radial wind (rw) –  fort.210: doppler lidar wind (dw) –  fort.211: radar wind superobs data (srw) –  fort.212: GPS data (gps_ref) –  fort.213: conventional SST data (sst) –  fort.214: tropical cyclone central pressure data (psfc) –  … –  fort.220: output from the inner loop minimization (pcgsoi) –  fort.221: fit to pbl height (pblh) –  …


Ø  characterized as conv (see convinfo table)

currentfitofsurfacepressuredata,rangesinmb--------------------------------------------------pressurelevels(hPa)=0.02000.0o-gitobsusetypstypcountbiasrmscpenqcpeno-g01psasm120000083-0.15310.65350.51130.5113o-g01psasm1800000453-0.61570.89820.55800.5580o-g01psasm1800001688-0.22000.82961.25381.2538o-g01psasm1810000382-0.08450.77690.57940.5794o-g01psasm18700008848-0.02750.56020.20180.2018o-g01asmall10454-0.06870.60930.30270.3027o-g01psrej120000019.09789.09780.00000.0000o-g01psrej18000011-11.878711.87870.00000.0000o-g01psrej181000014-2.279834.73610.00000.0000o-g01psrej18300002-15.186515.18950.00000.0000o-g01rejall18-3.615131.24960.00000.0000o-g01psmon18000002-0.39980.82061.99261.9926o-g01psmon18000012-0.52510.52510.41580.4158o-g01psmon1810000259-0.08200.82320.87270.8727o-g01psmon1830000760-0.59061.33880.00000.0000o-g01psmon18700001510.01510.43560.60970.6097o-g01monall1174-0.40011.15580.27510.2751numberofpsfcobsthatfailedgrosstest=18nonlinqctest=0typepsfcjiter1nread17904nkeep11646num10454typepsfcpen=0.316472951696346036E+04qcpen=0.316472951696346036E+04r=0.302729qcr=0.302729currentfitofsurfacepressuredata,rangesinmb--------------------------------------------------….o-g02….o-g03

Exampleoffort.201

costtermsJb,Jo,Jc,Jl=100.000000000000000000E+001.436036611019270822E+040.000000000000000000E+000.000000000000000000E+00cost,grad,step,b,step?=101.436036611019270822E+045.758921198958813825E+002.856226974284829012E+010.000000000000000000E+00good...costtermsJb,Jo,Jc,Jl=2101.345410379470981525E+039.256931413187994622E+030.000000000000000000E+000.000000000000000000E+00cost,grad,step,b,step?=2101.060234179265897546E+044.464976649873005771E-026.666238574241830861E+014.648847532482277001E-01good

Exampleoffort.220


•  Built-in diagnostic tools (comGSIv3.6_EnKFv1.2/util/Analysis_Utilities/): -  Located at read_diag/: read in diag _<obstype>_ges and diag _<obstype>_anl and then write out

ascii files results_<obstype>_ges and diag _<obstype>_anl -  plot_cost_grad/: read fort.220 and plot cost function and gradient versus inner/outer loop -  plots_ncl/: plot analysis increment (analysis minus background/first guess)

•  In addition to the fort.2** files, you’ll also have many pe00nn.<obstype>_0m in the work/ directory: -  Where nn = MPI nodes ID and m = outer loop ID -  For example, due to domain partition, conv observations were distributed into 4 nodes

(0000, 0001, 0002, 0003):

•  pe0000.conv_01, pe0000.conv_02, pe0000.conv_03 •  pe0001.conv_01, pe0001.conv_02, pe0001.conv_03 •  pe0002.conv_01, pe0002.conv_02, pe0002.conv_03 •  pe0003.conv_01, pe0003.conv_02, pe0003.conv_03

-  They will be cat into binary files diag _<obstype>_ges and one diag _<obstype>_anl (check run_gsi_regional_wrfarw.ksh)

-  They include innovation info (O-B and O-A) for all <obstype> observation data points over the domain

Standard Output and Diagnostics (4/4) Example using plot_cost_grad/

GSI_cost_gradient.ncl

Example using plots_ncl/Analysis_increment.ncl

ps@21914:1800.1735.95234.351023.6011023.600.28ps@46770:1800.0037.09232.551025.0011025.000.16ps@MASKSTID:1801.0035.70234.401023.6011023.600.27ps@MASKSTID:1800.0035.70234.201024.0011024.000.46t@MASKSTID:1801.0035.70234.401023.601286.35-0.58t@MASKSTID:1800.0035.70234.201024.001286.55-0.44q@MASKSTID:1801.0035.70234.401023.6015.90-0.00q@MASKSTID:1800.0035.70234.201024.0015.74-0.00uv@MASKSTID:2801.0035.70234.401021.0012.00-1.81-11.60-1.51uv@MASKSTID:2800.0035.70234.201021.2313.50-0.25-9.700.30ps@MASKSTID:1800.0040.50231.701023.0011023.000.03ps@46059:180-0.6737.90230.301026.5011026.50-0.15ps@21530:1800.0038.88234.061023.3011023.300.79…

Example results_conv_ges file

Ø  As long as you know how to manipulate NetCDF and ASCII data, your can use other software to process these output data

Ø  My personal preference is Python (I’m happy to share my scripts)

ü Observation process and BUFR ü Single observation test/experiment ü Configure GSI to run hybrid 4DEnVar (NCEP/GFS

operational configuration) •  Advanced topics: –  Observation sensitivity study –  Satellite radiance assimilation –  Introducing new observation(s)

•  Past GSI Tutorial documents: https://dtcenter.org/com-GSI/users/docs/tutorial_presentations_2017.php

•  Attend Annual Joint GSI/EnKF Community Tutorial

Suggest GSI Reading Topics

Reference: •  The materials of this presentation come from: -  GSI Community Version 3.5 Advanced User’s Guide. Aug 2016 -  GSI Community Version 3.6 User’s Guide. September 2017 -  Derber J. C.: Overview of GSI. 2017 GSI Summer Tutorial -  Auligne T. and Descombes G.: Background Error Covariance and

GEN_BE. 2014 GSI Community Tutorial -  Whitaker J.: GSI Hybrid EnVar Data Assimilation. 2014 GSI Tutorial -  Liu Z.: Hybrid Variational/Ensemble Data Assimilation. 2011 WRFDA

tutorial -  Kleist D.: Background and Observation Errors: Estimation and Tuning.

2011 GSI Tutorial

Useful links: •  http://4dvarenkf.cima.fcen.uba.ar/course/en/index.php?m=6 •  https://dtcenter.org/com-GSI/users/docs/index.php

http://www.inverseproblems.info/

Documents

Build and Run GSI on S4 · • Untar both tarball files: > tar –xvf comGSIv3.6_EnKFv1.2.tar > tar –xvf CRTM_2.2.3.tar • Go into comGSIv3.6_EnKFv1.2/dtc/ and run configure script