A STATISTICAL EVALUATION OF TAMDAR DATA IN SHORT-RANGE MESOSCALE NUMERICAL MODELS

A STATISTICAL EVALUATION OF TAMDAR DATA INSHORT-RANGE MESOSCALE NUMERICAL MODELS

Neil A. Jacobs1 and Yubao Liu2

1AirDat, LLC, Morrisville, NC 275602National Center for Atmospheric Research, Boulder, CO 80307

A STATISTICAL EVALUATION OF TAMDAR DATA INSHORT-RANGE MESOSCALE NUMERICAL MODELS

Neil A. Jacobs1 and Yubao Liu2

1AirDat, LLC, Morrisville, NC 275602National Center for Atmospheric Research, Boulder, CO 80307

Vertical Resolution Case Studies

Hypothesis: Increasing the number of model -levels in the lower to mid-troposphere will better utilize the greater observation density provided by TAMDARs, and result in a more accurate forecast.

Vertical Resolution Case Studies

Hypothesis: Increasing the number of model -levels in the lower to mid-troposphere will better utilize the greater observation density provided by TAMDARs, and result in a more accurate forecast.

Model Description:

RT-FDDA First-guess field

Little_R (MM5 Objective analysis) INTERPF MM5

Domains 1/2 (36-km/12-km)

Grell CP, MRF PBL, Mixed-phase (Reisner-1) microphysics

4 Simulations (96 h…was 14 h):

“TAMDAR” (36 -levels)

“Cntl” No TAMDAR (36 -levels…same as “TAMDAR”)

“TAMDAR+” (48 -levels: 6 1.5 km / all 12 5.5 km)

“Cntl+” No TAMDAR (48 -levels…same as “TAMDAR+”)

Initialized 1100 UTC 22 April 2005

Great Lakes late-season snow event (April 22-25, 2005)Great Lakes late-season snow event (April 22-25, 2005)

TAMDAR

TAMDAR+

Stage IV (4 km)

Cntl+

Cntl

1-h precipitation forecast (mm) and sea-level pressure (mb), as well as the 1-h Stage-IV analysis (mm), valid 1200 UTC 22 April 2005.

Cntl / Cntl+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The Cntl is blue, and the Cntl+ is red.

Cntl / Cntl+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The Cntl is blue, and the Cntl+ is red.

TAMDAR / TAMDAR+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The TAMDAR is blue, and the TAMDAR+ is red.

TAMDAR / TAMDAR+Scatter plot of 1-h QPF totals versus the 1-h Stage-IV analysis comparing matching grid point magnitudes (above 5 mm threshold) summed over each of the 14 forecast hours. The TAMDAR is blue, and the TAMDAR+ is red.

5

10

15

20

25

30

35

5 10 15 20 25 30 35

Matching grid point magnitude20050422 TAMDAR (Blue) and TAMDAR+ (Red)

Mo

de

l fo

reca

ste

d p

reci

p (

mm

)

Stage IV precip analysis (mm)

y = 9.21 + 0.127x R = 0.0869 (535)y = 3.44 + 0.621x R = 0.6732 (1102)

5

10

15

20

25

30

35

5 10 15 20 25 30 35

Matching grid point magnitude20050422 Cntl (Blue) and Cntl+ (Red)

Mo

de

l fo

reca

ste

d p

reci

p (

mm

)

Stage IV precip analysis (mm)

y = 8.79 + 0.196x R = 0.1765 (464)y = 7.05 + 0.325x R = 0.2346 (657)

2 f (x,y) 2 f (x,y)

x 2

2 f (x,y)

y 2

2[g(x,y) f (x,y)] [2g(x,y)] f (x,y) x 2 y 2 2 2

4e (x 2 y 2 )2 2

We can isolate cells based on magnitude, and retain only the precipitation associated with that cell

Laplacian edge detection:

Can also detect noise, so a smoothing Gaussian filter was tested…Laplacian of Gaussian:

•Both methods yield near-identical results

•All forecasts are regridded to 12-km

•An assumption is made that the closest cell (radial search) was the predicted cell.

•A weighted score was applied to the magnitude of the cell based on the linear distance (to the maximum) from “truth” (Stage-IV).

•For example, Stage-IV compared against itself would receive full weight. A score of 0 would mean either no cell was detected, or the distance was > 2(dm_cell+ds4_cell).

Precipitation cell isolation: who cares?A crude method to quantify QPF performance

Total

EXAMPLE: Raw Stage-IV 3-h accumulated precipitation data (Total), and the postprocessed (no minimum) isolated cells. The domain-2 data are mapped on the x-y grid.

5-mm

10-mm 15-mm

20-mm

Comparison of 12-h (4x3-h) QPF between TAMDAR+, TAMDAR, Cntl+, Cntl, and other various models regridded to the smallest grid (12-km), as well as the

Stage-IV analysis (“truth”) for 20-mm cells with 2-mm minimum bound.

Comparison of 12-h (4x3-h) QPF between TAMDAR+, TAMDAR, Cntl+, Cntl, and other various models regridded to the smallest grid (12-km), as well as the

Stage-IV analysis (“truth”) for 20-mm cells with 2-mm minimum bound.

The results presented here are consistent with preliminary findings from similar studies conducted at NCAR.

-100

0

100

200

300

400

500

600

700

0 10 20 30 40 50

20-mm cells (2-mm min) 12-h fcst sum

TAMDAR+TAMDARCntl+CntlRUCNAM218GFSStage IV

Precip (mm) per grid point

Wei

ghte

d gr

id p

oint

val

ue s

core

Results suggest that the addition of TAMDAR data in conjunction with increased vertical resolution improves the forecast skill for certain output parameters.

GFS, NAM, and RUC were included as “reference” models, and the improvement of the Cntl over these models is attributed to the 4DVAR ingestion technique of the RT-FDDA system.

However, the TAMDAR+ run shows significant improvements of 18-22% over the TAMDAR, Cntl, and the Cntl+ for this case.

This suggests that proper utilization of TAMDAR data plays a crucial role in forecast skill.

Results suggest that the addition of TAMDAR data in conjunction with increased vertical resolution improves the forecast skill for certain output parameters.

GFS, NAM, and RUC were included as “reference” models, and the improvement of the Cntl over these models is attributed to the 4DVAR ingestion technique of the RT-FDDA system.

However, the TAMDAR+ run shows significant improvements of 18-22% over the TAMDAR, Cntl, and the Cntl+ for this case.

This suggests that proper utilization of TAMDAR data plays a crucial role in forecast skill.

Model Description:

RT-FDDA First-guess field

Little_R (MM5 Objective analysis) INTERPF MM5

Domains 1/2 (36-km/12-km)

Grell CP, MRF PBL, Mixed-phase (Reisner-1) microphysics

4 Simulations (96 h…was 14 h):

“TAMDAR” (36 -levels)

“Cntl” No TAMDAR (36 -levels…same as “TAMDAR”)

“TAMDAR+” (48 -levels: 6 1.5 km / all 12 5.5 km)

“Cntl+” No TAMDAR (48 -levels…same as “TAMDAR+”)

Initialized 2300 UTC 29 August 2005

Hurricane Katrina (August 29, 2005)Hurricane Katrina (August 29, 2005)

TAMDAR+ Cntl+

Stage-IV

Sea-level pressure (mb) and 1-h precip. (in)Valid 0600 UTC 30 AUG 2005 (7-h Fcst)

Stage-IV

Cntl+TAMDAR+


Stage-IV

Cntl+TAMDAR+


Stage-IV

Cntl+TAMDAR+


Stage-IV

Cntl+TAMDAR+


Precipitation bands

TAMDAR+ is 4 mb deeper

850-hPa Relative Humidity (%)Valid 1800 UTC 30 AUG 2005 (19-h Fcst)

TAMDAR+ Cntl+

850-hPa Relative Humidity (%)Valid 0600 UTC 30 AUG 2005 (7-h Fcst)

Cntl+TAMDAR+

850-hPa RH Analysis DifferenceTAMDAR+ minus Cntl+2300 UTC 29 AUG 2005

850-hPa T Analysis DifferenceTAMDAR+ minus Cntl+2300 UTC 29 AUG 2005

From outer 36-km grid

Regions of RH responsible for future band formation

RAOB verification of RH band (case 2) is tough because of the space-time void.

TAMDAR was meant to fill this void, but verification against itself is a last choice.

Grell CP scheme trigger function is dependent on saturation (or near saturation) of moisture fields.

Minor differences in magnitude that exist near the CP scheme’s trigger threshold can tip the scales in a huge way hours later, which can be good or bad. Thus, proper assimilation of accurate data is key!

RAOB verification of RH band (case 2) is tough because of the space-time void.

TAMDAR was meant to fill this void, but verification against itself is a last choice.

Grell CP scheme trigger function is dependent on saturation (or near saturation) of moisture fields.

Minor differences in magnitude that exist near the CP scheme’s trigger threshold can tip the scales in a huge way hours later, which can be good or bad. Thus, proper assimilation of accurate data is key!

Why was the cyclone in TAMDAR+ 4 mb deeper when increased RH was the only difference seen in the analysis?

An increase in lower-tropospheric PV seen in the TAMDAR+ run appears to be linked to latent heat release from the precipitation bands around the cyclone (e.g., Bretherton 1966).

Preliminary findings suggest that the majority of geopotential height difference can be attributed to this additional PV.

Why was the cyclone in TAMDAR+ 4 mb deeper when increased RH was the only difference seen in the analysis?

An increase in lower-tropospheric PV seen in the TAMDAR+ run appears to be linked to latent heat release from the precipitation bands around the cyclone (e.g., Bretherton 1966).

Preliminary findings suggest that the majority of geopotential height difference can be attributed to this additional PV.

Precipitation Forecast Comparison

46 cases (22 March 2005- 4 June 2005)

AIRDAT (RT-FDDA-MM5 - TAMDAR)AIRNOT (RT-FDDA-MM5 - no TAMDAR)RUC, NAM, GFSStage-IV "truth"

Originally 49 cases, but 3 cases in May omitted based on initialization errors.

Precipitation Forecast Comparison

46 cases (22 March 2005- 4 June 2005)

AIRDAT (RT-FDDA-MM5 - TAMDAR)AIRNOT (RT-FDDA-MM5 - no TAMDAR)RUC, NAM, GFSStage-IV "truth"

Originally 49 cases, but 3 cases in May omitted based on initialization errors.

-1 104

0

1 104

2 104

3 104

4 104

5 104

6 104

0 10 20 30 40 50

5-mm cells (2-mm min) 12-h Fcst sum

AIRDAT TotalAIRNOT TotalGFS TotalNAM218 TotalRUC TotalStage4 Total

Wei

ghte

d gr

id p

oint

val

ue s

core


-1 104

0

1 104

2 104

3 104

4 104

5 104

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


-5000

0

5000

1 104

1.5 104

2 104

2.5 104

3 104

3.5 104

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


-5000

0

5000

1 104

1.5 104

2 104

2.5 104

3 104

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


-5000

0

5000

1 104

1.5 104

2 104

2.5 104

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


-5000

0

5000

1 104

1.5 104

2 104

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


-2000

0

2000

4000

6000

8000

1 104

1.2 104

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


-2000

0

2000

4000

6000

8000

0 10 20 30 40 50



Wei

ghte

d gr

id p

oint

val

ue s

core


Object-oriented verification technique

Developed by Barbara Brown et al. (NCAR), and presented at previous GLFE.

All 49 cases…3 May outliers not removed

GFS not shown

0

50

100

150

200

0 2 4 6 8 10 12 14 16

AIRDATAIRNOTRUCNAMStage4

Nu

mb

ers

Intensity (mm) 12-h Fcst

NCAR verification 0.2 to 15-mm threshold

Improvement of QPF accuracy for short-range severe precipitation

8-11% - Control RT-FDDA-MM5 w/out TAMDAR (apples-to-apples)

34-55% - RUC w/out TAMDAR (apples-to-oranges)

71-84% - NAM (apples-to-squash)

>90% - GFS (apples-to-spaghetti)

A conservative estimate of potential improvement because…

• 4-month study utilized only 36 -levels• Stage-II/IV bias adjustment typically on low side (Smith and Krajewski 1991)

• Weighting/optimization of ingestion/parameterizations are still being refined

AirDat and NCAR findings are consistent despite different techniques

Improvement of QPF accuracy for short-range severe precipitation

8-11% - Control RT-FDDA-MM5 w/out TAMDAR (apples-to-apples)

34-55% - RUC w/out TAMDAR (apples-to-oranges)

71-84% - NAM (apples-to-squash)

>90% - GFS (apples-to-spaghetti)

A conservative estimate of potential improvement because…

• 4-month study utilized only 36 -levels• Stage-II/IV bias adjustment typically on low side (Smith and Krajewski 1991)

• Weighting/optimization of ingestion/parameterizations are still being refined

AirDat and NCAR findings are consistent despite different techniques

Cold-Start-MM5 Sensitivity Studies

Cumulus PBL NOTAMDAR TAMDAR

(TempError) (TempError)

Kain-Fritsch Blackadar 2.2C 2.0C

Grell Blackadar 2.3C 2.0C

Betts-Miller Blackadar 2.5C 2.4C

Kain-Fritsch2 Blackadar 2.2C 2.1C

Kain-Fritsch MRF 2.4C 2.3C

Grell MRF 2.5C 2.4C

Betts-Miller MRF 2.8C 2.6C

Kain-Fritsch2 MRF 2.5C 2.4C

Kain-Fritsch ETA-MY 2.8C 2.5C

Grell ETA-MY 2.6C 2.4C

Betts-Miller ETA-MY 2.9C 2.8C

Kain-Fritsch2 ETA-MY 2.9C 2.8C

(PrecipError) (PrecipError)

Kain-Fritsch Blackadar 0.45Ó 0.36Ó

Grell Blackadar 0.48Ó 0.42Ó

Betts-Miller Blackadar 0.64Ó 0.59Ó

Kain-Fritsch2 Blackadar 0.46Ó 0.38Ó

Kain-Fritsch MRF 0.52Ó 0.49Ó

Grell MRF 0.54Ó 0.50Ó

Betts-Miller MRF 0.69Ó 0.60Ó

Kain-Fritsch2 MRF 0.53Ó 0.51Ó

Kain-Fritsch ETA-MY 0.52Ó 0.47Ó

Grell ETA-MY 0.54Ó 0.49Ó

Betts-Miller ETA-MY 0.67Ó 0.59"

Kain-Fritsch2 ETA-MY 0.54Ó 0.48Ó

168 simulations on a CONUS 36-km grid

7 winter events (12 combinations)

Table: 144-h average of 3-h error

Error = | Forecast - ASOS* |

Blackadar good in winter despite 5-layer LSM

Snow cover / lack of veg. LSM’s influence

KF -> Grell may = feedback in warm-start

12/12 combinations: error with TAMDAR

*Automated Surface Observing System (NWS/FAA/DOD)

Cold-Start-MM5 Sensitivity Tests

0

0.5

1

1.5

2

2.5

3

12 24 36 48 60 72 84 96 108 120

2m-Temperature Error (C) CntlExp

Avg

. | T

emp.

Err

or |

(C)

Forecast Hour

2-m temp. error averaged for all 7 cases using KF/BlackadarCntl (No TAMDAR) Exp (TAMDAR)

This is a trend seen in all 7 cases! …not just an artifact of one “outlier”.

?• Better QPF > more accurate snow cover, albedo, and/or surface radiation > long- range surface temp. impact ?

• Better forecasted feedback from downstream “blocking” ?

• Weird Hovmoller teleconnection ?

• Lucky-7 ?

Objective was to obtain CPU speed benchmark for new 3GHz dual-core

Upcoming studies…

• Sampling Rate Impact Study 3 Parallel Simulations on Cold-Start MM5 36-km CONUS 12-km GLFE 48 -levels CNTL = No TAMDAR data EXP1 = TAMDAR data at “old” original sample rate EXP2 = TAMDAR data at “new” increased sample rate

• Variable Sampling Rate ? Based on forecasted dynamics

• Weighting Studies… Independent testing of ascent/descent …and independent testing of RH, T, winds Testing of radius and magnitude w.r.t. seasonal and diurnal variations

• QPF Verification: Round 2 Additional QPF verification studies, as well as other surface and upper-level verification will be needed after OSSE / weighting.

Upcoming studies…

• Sampling Rate Impact Study 3 Parallel Simulations on Cold-Start MM5 36-km CONUS 12-km GLFE 48 -levels CNTL = No TAMDAR data EXP1 = TAMDAR data at “old” original sample rate EXP2 = TAMDAR data at “new” increased sample rate

• Variable Sampling Rate ? Based on forecasted dynamics

• Weighting Studies… Independent testing of ascent/descent …and independent testing of RH, T, winds Testing of radius and magnitude w.r.t. seasonal and diurnal variations

• QPF Verification: Round 2 Additional QPF verification studies, as well as other surface and upper-level verification will be needed after OSSE / weighting.

Acknowledgments

Barbara Brown, Randy Bullock, and Wei Yu (NCAR’s object-based QPF verification)

Stan Benjamin and William Moninger (NOAA/ERL/FSL/GSD)

NCAR (OSSE computer support, etc.)

NASA Aeronautics Research Office’s Aviation Safety ProgramFAA Aviation Weather Research ProgramAirDat, LLC

Acknowledgments

Barbara Brown, Randy Bullock, and Wei Yu (NCAR’s object-based QPF verification)

Stan Benjamin and William Moninger (NOAA/ERL/FSL/GSD)

NCAR (OSSE computer support, etc.)

NASA Aeronautics Research Office’s Aviation Safety ProgramFAA Aviation Weather Research ProgramAirDat, LLC

Documents

A STATISTICAL EVALUATION OF TAMDAR DATA IN SHORT-RANGE MESOSCALE NUMERICAL MODELS