IMPACT OF TAMDAR ON THE RUC MODEL: A LOOK INTO SOME OF THE STATISTICS WITH CASE STUDIES

AMS Annual Meeting 2007 – San Antonio 11th IOAS-AOLS18 January 2007

IMPACT OF TAMDAR ON THE RUC MODEL:

A LOOK INTO SOME OF THE STATISTICS WITH CASE STUDIES

Ed Szoke,* Stan Benjamin, Randy Collander*, Brian Jamison*, Bill Moninger, Tom Schlatter, and Tracy

Smith*

NOAA/ESRL Global Systems DivisionBoulder, CO USA

*Joint collaboration with the Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, CO


Overview The main issue:

Objective evaluation (statistics) of relative humidity (RH) has occasionally shown poorer performance for RUC runs with TAMDAR

Statistics - calculated by comparing RUC forecasts with and without TAMDAR to RAOBs at the standard pressure levels (850, 700, 500 mb)

Is this really worse performance with TAMDAR OR

are there other reasons for the poorer scores?

Procedure: Find days that stand out with poorer scores Examine individual RAOBs with forecast soundings to

see where the errors occur Concentrate on the Great Lakes subset (13 RAOBs)


Verification areas:for this study we used the inner (blue) box containing 13 RAOBs


RUC withTAMDAR (“dev2”, blue line)

RUC run without TAMDAR (“dev”, red line) for the GreatLakes area.

Bottom plot showsthe difference, positiveif dev2 is a better forecast than dev.

Starred days highlightpoorer scores for dev2.

3-h RMS error statistics for June-October 2006 at 700 mb for Great Lakes area – 13 RAOBs

Dev - control

Dev2 – add TAMDAR


Statistics (RMS error)for RH for 6-h RUC forecasts valid at 0000 UTC at 700 mbfor the Great Lakes area.

Starred days highlightpoorer scores for dev2.

6-h RMS statistics for June-October 2006 at 700 mb

* * * *

Dev - control

Dev2 – add TAMDAR


Case 1: 23 June 06 00z – 3h forecasts

RMS score for dev2 is 7% worse than dev for 3-h forecasts valid at 0000 UTC 23 June.RAOB comparisonshowed 2 sites account for most of this error.

Peoria, Illinois (ILX)comparison is shown here. For all plots RAOB is green, dev (RUC w/o TAMDAR) in bluedev2 (RUC w/TAMDAR) in black.

The shape (character)of the dev2 RH appears to be a better match tothe RAOB, but is offby ~50 mb so scores poorly at 700 mb.

RH at 700 mb: RAOB = 74 % dev = 94 % dev2 = 34 % (60% error!)

raob

dev

dev2


Case 1: 23 June 06Pittsburgh (PIT) was theother RAOB where theRH is significantly worsefor dev2 than for dev. In this case, dev2 is 39% drier than the RAOB, while dev is only 12% excessively moist.

While one could arguethat the shape of the dev2 RH profile bettermatches the changes inthe vertical shown bythe RAOB, the excessivedrying for dev2 is probably simply not as good a forecast in this case.

raob

dev

dev2



Case 2: 14 July 06 – 3 h 700 mb forecasts RMS scores for dev2 again were ~7% worse than for dev, for 3-h forecasts valid at 0000 UTC 14 July.RAOB comparisonshowed 4 sites account for most of this error.

Buffalo, NY (BUF)comparison is shown here.

In this case the dev2 follows the RAOB RH profile nicely until there is a more moist shift exactly at 700 mb, yielding what apperas tobe an unrepresentative error at 700 mb for dev2while dev happens to geta perfect match.

RH differences: dev2 ~20% dev almost no error

raob

dev

dev2


Case 2: 14 July 06Aberdeen, South Dakota(ABR) comparison is shown here.

The dev2 RH forecastmore closely matches the RAOB up to ~770 mb, then both forecasts dry out, while the RAOB does not. While both forecasts dry at about thesame rate in the vertical,it happens that the dev forecast crosses the RAOB at 700 mb. Butthis is because it iserroneously more moistbelow 750 mb! So thebetter score at 700 mb is not representative (withdev2 being 23% drier than the RAOB).

raob

dev

dev2


Case 2a: 14 July 06 – 6 h 700 mb forecasts RMS scores for dev2 forecasts were ~5% worse than for dev, for 6-h forecasts valid at 0000 UTC 14 July.RAOB comparisonshowed about half thesites accounting for smaller errors.

Buffalo, NY (BUF)6-h comparison is shownas it illustrates the errorthat happens to occurwith a sharp but vertically shallow more moist layer in the RAOB just at 700 mb. Nothingin other observations toknow if this is real. Without this layer thedev2 forecast followsthe RAOB moisture more closely than dev.

raob

dev

dev2


Case 3: 12 October 06 – 6h forecasts valid 0000 UTC Green Bay, Wisconsin(GRB) comparison is shown here.

The RMS error at 700 mbfor dev2 on 12 Oct was 7% worse than for dev.Almost all of this error comes from the GRB comparison.

RH & Differences at 700 mb: RAOB: 88% RH dev: 83% (-5% diff) dev2: 22% (-66% diff)

The difference at 700 mb is the largest found during this period. It occurs as the dev2 forecast dries out a deep portion of the troposphere in nw flow.

Is the forecast asbad as it looks?

raob

dev

dev2



Case 3: 12 October 06 0000 UTC 700 mb plot The forecast from dev2may not be as bad as itappeared. There is significant drying to thewest and northwest of GRB behind the deep 700 mb upper level low.

(Dewpoint is number below the temperatureon the station plots.)

So the main issue may be that the forecast from dev2 is just off a smallamount in timing.


Case 3: 12 October 06 0000 UTC – RAOB and dev2 comparison illustrating drying

Another way to show this drying is illustratedhere with an overlay ofthe GRB RAOB and 2upstream RAOBs (MPXand INL), along with thedev2 6-h forecast.

MPX, more to the west of GRB, is drier above700 mb. INL, more tothe nw, shows the drierlayer reaching all the way down past 700 mb.Note that the dev2 forecast compares ratherwell to the INL RAOB,verifying nearly exactlyat 700 mb.


Case 4: 20 October 06 0000 UTC 3-h forecasts at 700 mbRMS score for dev2 is 4.5% worse than for dev for 3-h forecasts valid at 0000 UTC 20 October.RAOB comparisonshowed 2 sites account for most of this error (INL & MPX).

International Falls (INL)comparison is shown.

The deep layer of dryingin the RAOB is better captured by the dev2 RH forecast, while dev appears to mainly missthis dry layer, but happens to verify betterat 700 mb.

RH & Differences at 700 mb: RAOB: 12% RH dev: 42% (+30% diff) dev2: 70% (+58% diff)

raob

dev

dev2


Case 4: 20 October 06 0000 UTC 500 mb plot with IR

Similar to the last case,drying is occurring behind a trough axispassing INL, so could argue that the character of the dev2 forecast is more representative of what is really happening thanthe dev forecast, thoughscoring worse at 700 mb.


Case 5: 28 June 06 0000 UTC 3-h forecasts at 700 mb Better verification for dev2 (RUC w/TAMDAR)

This time the RMS score for dev2 is 10% better than for dev for 3-h forecasts valid at 0000 UTC 28 June.RAOB comparisonsfound that a lot of variability, but some bigerrors for dev.

Wilmington, Ohio (ILN)comparison is shown.

Both forecasts have thedrying beginning lowerthan observed, but because it does not start for dev2 until just above 700 mb it scores much better thandev.

RH & Differences at 700 mb: RAOB: 78% RH dev: 13% (-55% diff) dev2: 61% (-17% diff)

raob

dev

dev2


Case 6: 18 Oct 06 0000 UTC 3-h forecasts at 700 mb Better verification for dev2 (RUC w/TAMDAR)

In this case..dev2 is 4% better than dev (3-h forecasts )

Minneapolis, Minnesota(MSP)

This case illustrates the effect of a • very sharp drylayer in the RAOB (whichmay or may not be real).• shape of both RUCforecasts is similar, but • dev2 moisture profileis shifted ~30 mb lower and happens to closely match the RAOB at 700 mb, yielding an RHvalue 31% better than dev at 700 mb.

raob

dev

dev2


What this all means... RH often varies strongly in the vertical as shown in RAOB profiles Calculating error statistics only at the mandatory levels makes them more vulnerable to unrepresentativeness

It can only take 1 or 2 bad RAOB comparisons (out of 13 in the Great Lakes area) to yield a large RMS error

With only mandatory levels being used, slight shifts of the RH in the vertical can be severely penalized

The RAOBs often have some very sharp RH variations in the vertical that may or may not be real and can result in huge errors if they fall at a mandatory level Additionally, it is unrealistic to expect the RUC model to

resolve some of these fluctuations (if they are indeed real)

Considering the above reasoning, we decided to change the verification to a layer method

Calculations made at 10-mb intervals


Comparison of new scoring method with old for October 2006

Days of interest are highlighted. For 12 Oct and 20 Oct dev still scores better, but the error is much reduced (~2% RMS for layer on 12 Oct vs. 5% at 700 mb only; for 20 Oct also ~2% RMS for layer on vs. ~5% at 700 mb only). For 18 Oct, when dev2 scored better at 700 mb, the difference is also reduced by more than half. These results appear to be consistent with the sounding comparisons shown earlier.

New method: 900 to 650 mb averaging Old method: 700 mb single level


Summary We began the study as a forensic pathology study

to try to better understand why the RMS RH scores were substantially worse for the RUC runs with TAMDAR on some days

Used the Great Lakes area with 13 RAOB/forecast comparisons

Focused on 3-h and 6-h forecasts valid at 0000 UTC since TAMDAR data in abundance for these initialization times

Discovered issues with the mandatory-only scoring method Change to a layer average method have produced more representative results Found

no characteristic problems with TAMDAR data or with RUC no misdesign with RUC assimilation or model additional TAMDAR in upstream airports would decrease aliasing

Documents

IMPACT OF TAMDAR ON THE RUC MODEL: A LOOK INTO SOME OF THE STATISTICS WITH CASE STUDIES