Using Ensembles in a Deterministic Forecast Era

National Weather Service

Using Ensembles in a Using Ensembles in a Deterministic Forecast EraDeterministic Forecast Era

Bernard N. MeisnerScientific Services DivisionNWS Southern RegionFort Worth, Texas


The views expressed herein are those of the author and do not necessarily reflect the position of the National Weather Service.



What Information is Available in What Information is Available in Ensemble Guidance?Ensemble Guidance?

• Obviously, the Obviously, the ensemble mean, spread and ensemble mean, spread and extremesextremes..

• But what about the results of a But what about the results of a cluster cluster analysis?analysis?

• Might the ensemble MOS guidance be Might the ensemble MOS guidance be calibratedcalibrated to provide useful information? to provide useful information?

• What about applyingWhat about applying pattern recognition? pattern recognition?


Model Flip-FlopsModel Flip-Flops



Model Flips and FlopsModel Flips and FlopsHow to define them?How to define them?

Current model runlies outside envelopeof previous ensemble.

Run-to-run changein MOS max/mintemperatures >10oF.

Run-to-run changein MOS max/mintemperatures exceedsmonthly Mean Absolute Error.



Identifying Model Flip-flopsIdentifying Model Flip-flops

• Model flip flops Model flip flops usually result from forecast usually result from forecast uncertainty,uncertainty, rather than radical changes in the rather than radical changes in the initial data. initial data.

• Operational forecast Operational forecast will often resemble one of will often resemble one of the ensemble membersthe ensemble members from a previous model from a previous model runrun valid at the same time, and almost always valid at the same time, and almost always fall within the range of forecasts given by the fall within the range of forecasts given by the prior ensemble run. prior ensemble run.

• The The ensemble mean and spreadensemble mean and spread is a better is a better measure of fundamental run-to-run model measure of fundamental run-to-run model forecast change than consecutive operational forecast change than consecutive operational forecasts. forecasts.


VerificationVerification

108 Hr108 HrForecastForecast




Sfc VorticitySfc Wind Speed

Confidence Graph, Confidence Graph, Spaghetti ChartsSpaghetti ChartsHigh & Low Confidence High & Low Confidence

ExamplesExamples

FSU/NWS TLH Web SiteFSU/NWS TLH Web Sitehttp://moe.met.fsu.edu/confidence/http://moe.met.fsu.edu/confidence/

Spread of Current Ensemble RunAverage Ensemble Spread

for time of year

Climatological SpreadNCEP/NCAR Reanalysis



• Analysis has indicated that forecast errors are Analysis has indicated that forecast errors are greater when the spread of the ensemble is greater when the spread of the ensemble is relatively large.relatively large.

• But we don’t yet know how to minimize that But we don’t yet know how to minimize that error, given any additional information that may error, given any additional information that may be contained within the ensemble guidance.be contained within the ensemble guidance.


Model Flip FlopsModel Flip Flops

Data Used:Data Used:MEXMOS Guidance for DFW and PIT Airports (Have also MEXMOS Guidance for DFW and PIT Airports (Have also looked at a few other sites in NWS Southern Region.)looked at a few other sites in NWS Southern Region.)

Flip (and Flop) Criterion:Flip (and Flop) Criterion:

Run-to-run change in Max/Min temperature guidance Run-to-run change in Max/Min temperature guidance of 10of 10ooF or greater.F or greater.


DFW Max/Min TemperatureDFW Max/Min Temperature1010ooF ThresholdF Threshold

MonthMonth FlipsFlips

Number Number CorrectCorrect

Flip – Flip – FlopsFlops

Number Number CorrectCorrect

Oct 2005Oct 2005 33 33 -- --

NovNov 77 33 -- --

DecDec 99 44 77 44

Jan 2006Jan 2006 88 55 -- --

FebFeb 55 55 11 --

MarMar 1010 55 33 22

AprApr 22 00 -- --

TOTALTOTAL 4242 25 (60%)25 (60%) 1111 6 (55%)6 (55%)


Model Flips…and FlopsModel Flips…and Flops

Summary:Summary:

• Flips (and flops) occur during the cool Flips (and flops) occur during the cool season.season.

• Typically occur for Typically occur for just one verification timejust one verification time..

• Model flips are rare; flip-flops are very rare.Model flips are rare; flip-flops are very rare.• 53 Flips; 11 Flops (out of 400+ model runs/7000+ forecasts)53 Flips; 11 Flops (out of 400+ model runs/7000+ forecasts)

• Flips are most common for Days 4-6. Flips are most common for Days 4-6.

• Flip-flops are most common for Days 5-6.Flip-flops are most common for Days 5-6.


Forecast OpportunitiesForecast Opportunities




Data Used:Data Used:MEXMOS Guidance for DFW and PIT Airports (Have also MEXMOS Guidance for DFW and PIT Airports (Have also looked at a few other sites in NWS Southern Region.)looked at a few other sites in NWS Southern Region.)

Forecast Opportunity Criterion:Forecast Opportunity Criterion:Max/Min temperature guidance error 10Max/Min temperature guidance error 10ooF or greater.F or greater.

Caveat:Caveat:Smaller errors at certain thresholds can be significant!Smaller errors at certain thresholds can be significant!

2828ooF vs 33F vs 33ooF; 99F; 99ooF vs 104F vs 104ooFF


Forecast OpportunitiesForecast OpportunitiesThree Days in AugustThree Days in August



August 2006August 2006

1011018080

1011017878

1011017979

1021027979

1031037979

1031037979

94947575

1001007777

1031038080

1041048383

1051058080

1031038282

1021027979

1031037979

1051057979

1041048080

1051057979

1051058181

1021028282

1021028282

1031038383

1031038080

1021027777

1041048181

1031038383

1021028282

96967878

83837676

89897373

92926767

97977171

ClimoClimo94.894.874.074.0



1011018080

1011017878

1011017979

1021027979

1031037979

1031037979

94947575

1001007777

1031038080

1041048383

1051058080

1031038282

1021027979

1031037979

1051057979

1041048080

1051057979

1051058181

1021028282

1021028282

1031038383

1031038080

1021027777

1041048181

1031038383

1021028282

96967878

83837676

89897373

92926767

97977171


August 24, 2006August 24, 2006

Period One MOS Guidance:Period One MOS Guidance:Operational:Operational: 94 OVC 94 OVC

Cntrl:Cntrl: 99 PC 99 PC

P1–P4P1–P4 100 PC 100 PC 98 PC98 PC 99 PC 99 PC 98 OVC98 OVC

P4-P8P4-P8 99 PC 99 PC 99 PC99 PC 100 PC100 PC 99 PC99 PC

P9-P12P9-P12 97 PC 97 PC 98 PC98 PC 99 PC 99 PC 99 PC99 PC

P13-P14P13-P14 99 PC 99 PC 99 PC99 PC

While the OVC sky cover was triggered in the While the OVC sky cover was triggered in the operational run, it was only triggered in one operational run, it was only triggered in one ensemble member.ensemble member.



1011018080

1011017878

1011017979

1021027979

1031037979

1031037979

94947575

1001007777

1031038080

1041048383

1051058080

1031038282

1021027979

1031037979

1051057979

1041048080

1051057979

1051058181

1021028282

1021028282

1031038383

1031038080

1021027777

1041048181

1031038383

1021028282

96967878

83837676

89897373

92926767

97977171




Cntrl:Cntrl: 94 OVC 94 OVC

P1–P4P1–P4 93 OVC 93 OVC 94 OVC 94 OVC 94 OVC 94 OVC 93 OVC93 OVC

P4-P8P4-P8 96 OVC 96 OVC 94 OVC94 OVC 98 OVC 98 OVC 93 OVC93 OVC

P9-P12P9-P12 95 OVC 95 OVC 93 OVC93 OVC 96 OVC 96 OVC 93 OVC93 OVC

P13-P14P13-P14 96 OVC 96 OVC 95 OVC95 OVC

All forecast runs triggered the OVC sky cover.All forecast runs triggered the OVC sky cover.



1011018080

1011017878

1011017979

1021027979

1031037979

1031037979

94947575

1001007777

1031038080

1041048383

1051058080

1031038282

1021027979

1031037979

1051057979

1041048080

1051057979

1051058181

1021028282

1021028282

1031038383

1031038080

1021027777

1041048181

1031038383

1021028282

96967878

83837676

89897373

92926767

97977171




Cntrl:Cntrl: 95 OVC 95 OVC

P1–P4P1–P4 95 OVC 95 OVC 95 OVC 95 OVC 95 OVC 95 OVC 95 OVC95 OVC

P4-P8P4-P8 96 PC 96 PC 95 OVC95 OVC 95 OVC 95 OVC 95 PC95 PC

P9-P12P9-P12 96 PC 96 PC 95 OVC95 OVC 97 PC 97 PC 94 OVC94 OVC

P13-P14P13-P14 95 OVC 95 OVC 97 PC97 PC

All but five of the forecast runs triggered the OVC sky All but five of the forecast runs triggered the OVC sky cover. Runs with PC sky cover still substantially cover. Runs with PC sky cover still substantially underestimated the maximum temperature of 103underestimated the maximum temperature of 103ooF.F.



Forecast Opportunities:Forecast Opportunities:

• Typically occur for Typically occur for calendar dayscalendar days rather than rather than model runsmodel runs..

• Are most common for Days 3-7. Are most common for Days 3-7.

• Frequently occur for days on which the Frequently occur for days on which the observed max/min temperature departs observed max/min temperature departs substantially from the climatological norm.substantially from the climatological norm.• MEXMOS guidance typically underestimates the observed MEXMOS guidance typically underestimates the observed

departure from normal.departure from normal.

• Rarely occur for days when the temperature Rarely occur for days when the temperature change from the previous day is large.change from the previous day is large.• GFS seems to handle these events well.GFS seems to handle these events well.


Calibrated Ensemble MOS Calibrated Ensemble MOS PoP GuidancePoP Guidance



Calibrated PoP GuidanceCalibrated PoP Guidance

WFO Jackson, Miss. Procedure:WFO Jackson, Miss. Procedure:

• Initialize each forecast package with 0000 Initialize each forecast package with 0000 UTC MEX MOS guidance.UTC MEX MOS guidance.

• Only change grids when/where necessary.Only change grids when/where necessary.(Forecast Opportunities)(Forecast Opportunities)• Increase forecast PoP when ensemble mean PoP Increase forecast PoP when ensemble mean PoP

exceeds predetermined thresholds.exceeds predetermined thresholds.

• Decrease forecast PoP when emsemble mean PoP Decrease forecast PoP when emsemble mean PoP is below predetermined thresholds.is below predetermined thresholds.

• Technique applicable for cool season only.Technique applicable for cool season only.






Improvement over MOSImprovement over MOSKJAN PoP 2005 vs 2004KJAN PoP 2005 vs 2004

2005

2004


Calibrated PoP GuidanceCalibrated PoP Guidance

Results:Results:

• Big improvement over previous year.Big improvement over previous year.

• Removed previous dry bias for all forecast Removed previous dry bias for all forecast periods.periods.

• Average forecast PoP was greater than MOS Average forecast PoP was greater than MOS guidance on rainy days for all forecast guidance on rainy days for all forecast periods.periods.

• Average forecast PoP was less than MOS Average forecast PoP was less than MOS guidance on dry days for all forecast periods.guidance on dry days for all forecast periods.



• Even in a probabilistic forecast era, some users Even in a probabilistic forecast era, some users will continue to require the most likely scenario. will continue to require the most likely scenario. Will that always be the ensemble mean? Will that always be the ensemble mean?

• As the size of the ensembles increase it will not As the size of the ensembles increase it will not be possible to manually examine the output be possible to manually examine the output from each member. from each member. (No more spaghetti charts!)(No more spaghetti charts!)

• How might we glean all the useful information How might we glean all the useful information from an ensemble – much more than just the from an ensemble – much more than just the means and extremes?means and extremes?


Questions?Questions?

mailto: [email protected]


Documents

Using Ensembles in a Deterministic Forecast Era