National Weather Service
Using Ensembles in a Using Ensembles in a Deterministic Forecast EraDeterministic Forecast Era
Bernard N. MeisnerScientific Services DivisionNWS Southern RegionFort Worth, Texas
National Weather Service
The views expressed herein are those of the author and do not necessarily reflect the position of the National Weather Service.
National Weather Service
Using Ensembles in a Using Ensembles in a Deterministic Forecast EraDeterministic Forecast Era
What Information is Available in What Information is Available in Ensemble Guidance?Ensemble Guidance?
• Obviously, the Obviously, the ensemble mean, spread and ensemble mean, spread and extremesextremes..
• But what about the results of a But what about the results of a cluster cluster analysis?analysis?
• Might the ensemble MOS guidance be Might the ensemble MOS guidance be calibratedcalibrated to provide useful information? to provide useful information?
• What about applyingWhat about applying pattern recognition? pattern recognition?
National Weather Service
Model Flip-FlopsModel Flip-Flops
National Weather Service
National Weather Service
Model Flips and FlopsModel Flips and FlopsHow to define them?How to define them?
Current model runlies outside envelopeof previous ensemble.
Run-to-run changein MOS max/mintemperatures >10oF.
Run-to-run changein MOS max/mintemperatures exceedsmonthly Mean Absolute Error.
National Weather Service
Using Ensembles in a Using Ensembles in a Deterministic Forecast EraDeterministic Forecast Era
Identifying Model Flip-flopsIdentifying Model Flip-flops
• Model flip flops Model flip flops usually result from forecast usually result from forecast uncertainty,uncertainty, rather than radical changes in the rather than radical changes in the initial data. initial data.
• Operational forecast Operational forecast will often resemble one of will often resemble one of the ensemble membersthe ensemble members from a previous model from a previous model runrun valid at the same time, and almost always valid at the same time, and almost always fall within the range of forecasts given by the fall within the range of forecasts given by the prior ensemble run. prior ensemble run.
• The The ensemble mean and spreadensemble mean and spread is a better is a better measure of fundamental run-to-run model measure of fundamental run-to-run model forecast change than consecutive operational forecast change than consecutive operational forecasts. forecasts.
National Weather Service
VerificationVerification
108 Hr108 HrForecastForecast
102 Hr102 HrForecastForecast
96 Hr96 HrForecastForecast
National Weather Service
Sfc VorticitySfc Wind Speed
Confidence Graph, Confidence Graph, Spaghetti ChartsSpaghetti ChartsHigh & Low Confidence High & Low Confidence
ExamplesExamples
FSU/NWS TLH Web SiteFSU/NWS TLH Web Sitehttp://moe.met.fsu.edu/confidence/http://moe.met.fsu.edu/confidence/
Spread of Current Ensemble RunAverage Ensemble Spread
for time of year
Climatological SpreadNCEP/NCAR Reanalysis
National Weather Service
Using Ensembles in a Using Ensembles in a Deterministic Forecast EraDeterministic Forecast Era
• Analysis has indicated that forecast errors are Analysis has indicated that forecast errors are greater when the spread of the ensemble is greater when the spread of the ensemble is relatively large.relatively large.
• But we don’t yet know how to minimize that But we don’t yet know how to minimize that error, given any additional information that may error, given any additional information that may be contained within the ensemble guidance.be contained within the ensemble guidance.
National Weather Service
Model Flip FlopsModel Flip Flops
Data Used:Data Used:MEXMOS Guidance for DFW and PIT Airports (Have also MEXMOS Guidance for DFW and PIT Airports (Have also looked at a few other sites in NWS Southern Region.)looked at a few other sites in NWS Southern Region.)
Flip (and Flop) Criterion:Flip (and Flop) Criterion:
Run-to-run change in Max/Min temperature guidance Run-to-run change in Max/Min temperature guidance of 10of 10ooF or greater.F or greater.
National Weather Service
DFW Max/Min TemperatureDFW Max/Min Temperature1010ooF ThresholdF Threshold
MonthMonth FlipsFlips
Number Number CorrectCorrect
Flip – Flip – FlopsFlops
Number Number CorrectCorrect
Oct 2005Oct 2005 33 33 -- --
NovNov 77 33 -- --
DecDec 99 44 77 44
Jan 2006Jan 2006 88 55 -- --
FebFeb 55 55 11 --
MarMar 1010 55 33 22
AprApr 22 00 -- --
TOTALTOTAL 4242 25 (60%)25 (60%) 1111 6 (55%)6 (55%)
National Weather Service
Model Flips…and FlopsModel Flips…and Flops
Summary:Summary:
• Flips (and flops) occur during the cool Flips (and flops) occur during the cool season.season.
• Typically occur for Typically occur for just one verification timejust one verification time..
• Model flips are rare; flip-flops are very rare.Model flips are rare; flip-flops are very rare.• 53 Flips; 11 Flops (out of 400+ model runs/7000+ forecasts)53 Flips; 11 Flops (out of 400+ model runs/7000+ forecasts)
• Flips are most common for Days 4-6. Flips are most common for Days 4-6.
• Flip-flops are most common for Days 5-6.Flip-flops are most common for Days 5-6.
National Weather Service
Forecast OpportunitiesForecast Opportunities
National Weather Service
National Weather Service
Forecast OpportunitiesForecast Opportunities
Data Used:Data Used:MEXMOS Guidance for DFW and PIT Airports (Have also MEXMOS Guidance for DFW and PIT Airports (Have also looked at a few other sites in NWS Southern Region.)looked at a few other sites in NWS Southern Region.)
Forecast Opportunity Criterion:Forecast Opportunity Criterion:Max/Min temperature guidance error 10Max/Min temperature guidance error 10ooF or greater.F or greater.
Caveat:Caveat:Smaller errors at certain thresholds can be significant!Smaller errors at certain thresholds can be significant!
2828ooF vs 33F vs 33ooF; 99F; 99ooF vs 104F vs 104ooFF
National Weather Service
Forecast OpportunitiesForecast OpportunitiesThree Days in AugustThree Days in August
National Weather Service
National Weather Service
August 2006August 2006
1011018080
1011017878
1011017979
1021027979
1031037979
1031037979
94947575
1001007777
1031038080
1041048383
1051058080
1031038282
1021027979
1031037979
1051057979
1041048080
1051057979
1051058181
1021028282
1021028282
1031038383
1031038080
1021027777
1041048181
1031038383
1021028282
96967878
83837676
89897373
92926767
97977171
ClimoClimo94.894.874.074.0
National Weather Service
August 2006August 2006
1011018080
1011017878
1011017979
1021027979
1031037979
1031037979
94947575
1001007777
1031038080
1041048383
1051058080
1031038282
1021027979
1031037979
1051057979
1041048080
1051057979
1051058181
1021028282
1021028282
1031038383
1031038080
1021027777
1041048181
1031038383
1021028282
96967878
83837676
89897373
92926767
97977171
National Weather Service
August 24, 2006August 24, 2006
Period One MOS Guidance:Period One MOS Guidance:Operational:Operational: 94 OVC 94 OVC
Cntrl:Cntrl: 99 PC 99 PC
P1–P4P1–P4 100 PC 100 PC 98 PC98 PC 99 PC 99 PC 98 OVC98 OVC
P4-P8P4-P8 99 PC 99 PC 99 PC99 PC 100 PC100 PC 99 PC99 PC
P9-P12P9-P12 97 PC 97 PC 98 PC98 PC 99 PC 99 PC 99 PC99 PC
P13-P14P13-P14 99 PC 99 PC 99 PC99 PC
While the OVC sky cover was triggered in the While the OVC sky cover was triggered in the operational run, it was only triggered in one operational run, it was only triggered in one ensemble member.ensemble member.
National Weather Service
August 2006August 2006
1011018080
1011017878
1011017979
1021027979
1031037979
1031037979
94947575
1001007777
1031038080
1041048383
1051058080
1031038282
1021027979
1031037979
1051057979
1041048080
1051057979
1051058181
1021028282
1021028282
1031038383
1031038080
1021027777
1041048181
1031038383
1021028282
96967878
83837676
89897373
92926767
97977171
National Weather Service
August 12, 2006August 12, 2006
Period One MOS Guidance:Period One MOS Guidance:Operational:Operational: 94 OVC 94 OVC
Cntrl:Cntrl: 94 OVC 94 OVC
P1–P4P1–P4 93 OVC 93 OVC 94 OVC 94 OVC 94 OVC 94 OVC 93 OVC93 OVC
P4-P8P4-P8 96 OVC 96 OVC 94 OVC94 OVC 98 OVC 98 OVC 93 OVC93 OVC
P9-P12P9-P12 95 OVC 95 OVC 93 OVC93 OVC 96 OVC 96 OVC 93 OVC93 OVC
P13-P14P13-P14 96 OVC 96 OVC 95 OVC95 OVC
All forecast runs triggered the OVC sky cover.All forecast runs triggered the OVC sky cover.
National Weather Service
August 2006August 2006
1011018080
1011017878
1011017979
1021027979
1031037979
1031037979
94947575
1001007777
1031038080
1041048383
1051058080
1031038282
1021027979
1031037979
1051057979
1041048080
1051057979
1051058181
1021028282
1021028282
1031038383
1031038080
1021027777
1041048181
1031038383
1021028282
96967878
83837676
89897373
92926767
97977171
National Weather Service
August 22, 2006August 22, 2006
Period One MOS Guidance:Period One MOS Guidance:Operational:Operational: 94 OVC 94 OVC
Cntrl:Cntrl: 95 OVC 95 OVC
P1–P4P1–P4 95 OVC 95 OVC 95 OVC 95 OVC 95 OVC 95 OVC 95 OVC95 OVC
P4-P8P4-P8 96 PC 96 PC 95 OVC95 OVC 95 OVC 95 OVC 95 PC95 PC
P9-P12P9-P12 96 PC 96 PC 95 OVC95 OVC 97 PC 97 PC 94 OVC94 OVC
P13-P14P13-P14 95 OVC 95 OVC 97 PC97 PC
All but five of the forecast runs triggered the OVC sky All but five of the forecast runs triggered the OVC sky cover. Runs with PC sky cover still substantially cover. Runs with PC sky cover still substantially underestimated the maximum temperature of 103underestimated the maximum temperature of 103ooF.F.
National Weather Service
Forecast OpportunitiesForecast Opportunities
Forecast Opportunities:Forecast Opportunities:
• Typically occur for Typically occur for calendar dayscalendar days rather than rather than model runsmodel runs..
• Are most common for Days 3-7. Are most common for Days 3-7.
• Frequently occur for days on which the Frequently occur for days on which the observed max/min temperature departs observed max/min temperature departs substantially from the climatological norm.substantially from the climatological norm.• MEXMOS guidance typically underestimates the observed MEXMOS guidance typically underestimates the observed
departure from normal.departure from normal.
• Rarely occur for days when the temperature Rarely occur for days when the temperature change from the previous day is large.change from the previous day is large.• GFS seems to handle these events well.GFS seems to handle these events well.
National Weather Service
Calibrated Ensemble MOS Calibrated Ensemble MOS PoP GuidancePoP Guidance
National Weather Service
National Weather Service
Calibrated PoP GuidanceCalibrated PoP Guidance
WFO Jackson, Miss. Procedure:WFO Jackson, Miss. Procedure:
• Initialize each forecast package with 0000 Initialize each forecast package with 0000 UTC MEX MOS guidance.UTC MEX MOS guidance.
• Only change grids when/where necessary.Only change grids when/where necessary.(Forecast Opportunities)(Forecast Opportunities)• Increase forecast PoP when ensemble mean PoP Increase forecast PoP when ensemble mean PoP
exceeds predetermined thresholds.exceeds predetermined thresholds.
• Decrease forecast PoP when emsemble mean PoP Decrease forecast PoP when emsemble mean PoP is below predetermined thresholds.is below predetermined thresholds.
• Technique applicable for cool season only.Technique applicable for cool season only.
National Weather Service
National Weather Service
National Weather Service
National Weather Service
National Weather Service
Improvement over MOSImprovement over MOSKJAN PoP 2005 vs 2004KJAN PoP 2005 vs 2004
2005
2004
National Weather Service
Calibrated PoP GuidanceCalibrated PoP Guidance
Results:Results:
• Big improvement over previous year.Big improvement over previous year.
• Removed previous dry bias for all forecast Removed previous dry bias for all forecast periods.periods.
• Average forecast PoP was greater than MOS Average forecast PoP was greater than MOS guidance on rainy days for all forecast guidance on rainy days for all forecast periods.periods.
• Average forecast PoP was less than MOS Average forecast PoP was less than MOS guidance on dry days for all forecast periods.guidance on dry days for all forecast periods.
National Weather Service
Using Ensembles in a Using Ensembles in a Deterministic Forecast EraDeterministic Forecast Era
• Even in a probabilistic forecast era, some users Even in a probabilistic forecast era, some users will continue to require the most likely scenario. will continue to require the most likely scenario. Will that always be the ensemble mean? Will that always be the ensemble mean?
• As the size of the ensembles increase it will not As the size of the ensembles increase it will not be possible to manually examine the output be possible to manually examine the output from each member. from each member. (No more spaghetti charts!)(No more spaghetti charts!)
• How might we glean all the useful information How might we glean all the useful information from an ensemble – much more than just the from an ensemble – much more than just the means and extremes?means and extremes?