Upload
jenny-silsby
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Verification and calibration of probabilistic precipitation forecasts derived from neighborhood and object based
methods for a convection-allowing ensemble
Aaron Johnson and Xuguang Wang
School of Meteorology and Center for Analysis and Prediction of Storms
University of Oklahoma, Norman, OK
Acknowledgement: F. Kong, M. Xue, K. Thomas, K. Brewster, Y. Wang, J. Gao
Warn-on-Forecast and High Impact Weather Workshop9 February 2012
2
Outline• Motivation and convection-allowing ensemble overview• Non-traditional methods of generating probabilistic forecasts• Calibration methods• Results
– Neighborhood based• Full ensemble without calibration• Full ensemble with calibration• Sub-ensembles without calibration• Sub-ensembles with calibration
– Object based• Full ensemble without calibration• Full ensemble with calibration• Sub-ensembles without calibration• Sub-ensembles with calibration
3
Forecast example
• Hourly accumulated precipitation
• Near-CONUS domain• Subjective impressions of
storm structures
4
Motivation• Numerous calibration studies for meso- and global-scale ensembles (e.g.,
Wang and Bishop 2005, Wilks and Hamill 2007, Sloughter et al. 2007)– How do different probabilistic forecast calibrations compare at convection-
allowing resolution?
• Neighborhood methods relax grid point sensitivity of high resolution forecasts (e.g., Ebert 2009) while object based methods retain storm scale features but are typically applied to deterministic forecasts (e.g., Davis et al. 2006, Gallus 2010).– How skillful are such non-traditional probabilistic forecasts before and after
calibration?– How to generate probabilistic forecasts at convection-allowing resolution?
• 2009 CAPS ensemble forecasts for HWT Spring Experiment clustered according to WRF model dynamics (Johnson et al. 2011)– Is multi-model necessary? Is the conclusion changed before and after
calibration?
5
20 members initialized 00 UTC, integrated 30 hours over near-CONUS domain on 26 days from 29 April through 5 June 2009, on 4 km grid without CP.
MemberIC LBC R MP PBL SW Rad. LSM
ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC
NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH
10 members are from WRF-ARW, 8 members from WRF-NMM, and 2 members from ARPS.
MemberIC LBC R MP PBL SW Rad. LSM
ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC
NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH
Initial background field from 00 UTC NCEP NAM analysis.Coarser (~35 km) resolution IC/LBC perturbations obtained from NCEP SREF forecasts
MemberIC LBC R MP PBL SW Rad. LSM
ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC
NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH
Assimilation of radar reflectivity and velocity using ARPS 3DVAR and cloud analysis for 17 members
MemberIC LBC R MP PBL SW Rad. LSM
ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC
NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH
• Perturbations to Microphysics, Planetary Boundary Layer, Shortwave Radiation and Land Surface Model physics schemes.
MemberIC LBC R MP PBL SW Rad. LSM
ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC
NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH
6
Object based probabilistic forecasts• Event being forecast: object of interest
• Probability obtained from: percentage of ensemble members in which the forecast object occurs
Methods of Generating Probabilistic Forecasts
• Neighborhood based probabilistic forecasts– Event being forecast: accumulated
precipitation exceeding a threshold– Probability obtained from:
percentage of grid points within a search radius (48 km) from all members that exceed the threshold
Figure 8 from Schwartz et al. (2010)
7
Definition of Objects
8
Calibration Methods• Reliability diagram method: forecast probability is replaced
with observed frequency during training
• Schaffer et al. (2011) method: Extension of the reliability diagram method by including more parameters.
• Logistic Regression: Neighborhood based
x1 = mean of NP0.25 x2 = standard deviation of NP0.25
Object basedx1 = uncalibrated forecast probabilityx2 = ln(area)
• Bias Adjustment of each member: Adjust values so CDF of forecasts matches observations (Hamill and Whitaker 2006)
9
Neighborhood Results: Uncalibrated Full Ensemble
• Diurnal cycle for most thresholds• Skill also depends on threshold/accumulation period
10
Neighborhood Results: Calibrated Full Ensemble
• Skill improvement limited to the periods of skill minima• During skill minima, similar improvements from all calibrations
11
Neighborhood Results: Uncalibrated Sub-Ensembles
• ARW significantly more skillful than NMM for almost all lead times and thresholds• Multi-Model is not significantly more skillful than ARW
12
Neighborhood Results: Calibrated Sub-Ensembles
• Differences among different sub-enembles are reduced.• Multi-Model only shows advantages at 27-30 hour lead times.
13
Object Based Results: Full Ensemble• Uncalibrated:• Skill minimum during first 6
hours when members tend to be too similar (i.e., underdispersive)
• Lower skill than neighborhood based
• Lower skill for hourly than 6 hourly
• Calibrated:• Bias adjustment is the least
effective and LR is the most effective.
14
Object Based Results: Sub-Ensembles
Uncalibrated:• ARW significantly more
skillful than NMM.• Multi-model did not show
advantage compared to ARW
Calibrated:• Again, more skillful after
calibration and more skillful for longer accumulation period.
• Like the neighborhood probabilistic forecasts, differences in skill among different sub-ensembles are reduced by calibration.
15
Conclusions• Probabilistic precipitation forecasts from a convection allowing ensemble for the 2009
NOAA HWT Spring Experiment were verified and calibrated.• Probabilistic forecasts were derived from both the neighborhood method and a new
object based method.• Various calibrations including reliability based, logistic regression, and individual
member bias correction methods were implemented.
• For both the neighborhood and the object based probabilistic forecasts, calibration significantly improved the skill of the forecasts compared to the non-calibrated forecast during skill minima.
• For the neighborhood probabilistic forecasts, skill of different calibrations were similar• For the object based probabilistic forecast, the LR method is most effective.
• Sub-ensembles from ARW and NMM are also verified and calibrated for the purpose of guiding optimal ensemble design.
– ARW was more skillful than NMM for both neighborhood and object based probabilistic forecasts
– The difference in skill was reduced by calibration– Multimodel ensemble of ARW and NMM members only shows advantages compared to
single model ensemble after the 24 hour lead time for the neighborhood based forecasts.
• Probability of occurrence is forecast for control forecast objects, A and B.
• Other panels are forecasts from the other members.
• Forecast probability of A is 1/8=12.5%
• Forecast probability of B is 7/8=87.5%
16
Example of object based method
CONTROL FORECAST
A B
17
Sensitivity of Neighborhood based calibrations to training length
18
Sensitivity of Object based calibrations to training length