View
64
Download
0
Category
Tags:
Preview:
DESCRIPTION
Verification & Weather Impacts: Past and Future. Mark Fox Regional Training Officer. National Weather Service. Currently Verify: TOR / SVR / FFW ( POD – FAR – CSI ) Max / Min T ( in various forms ) PoP ( brier, reliability, and other forms ) Marine Wind Speed / Wave Heights - PowerPoint PPT Presentation
Citation preview
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 1
Verification &Weather Impacts:Past and Future
Verification &Weather Impacts:Past and Future
Mark Fox
Regional Training Officer
Mark Fox
Regional Training Officer
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 2
National Weather ServiceNational Weather Service
Currently Verify:– TOR / SVR / FFW ( POD – FAR – CSI )– Max / Min T ( in various forms ) – PoP ( brier, reliability, and other forms )– Marine Wind Speed / Wave Heights– TAF (various forms)
• 50% Offices say their “Best Service” is Direct and Personal Communications
Currently Verify:– TOR / SVR / FFW ( POD – FAR – CSI )– Max / Min T ( in various forms ) – PoP ( brier, reliability, and other forms )– Marine Wind Speed / Wave Heights– TAF (various forms)
• 50% Offices say their “Best Service” is Direct and Personal Communications
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 3
A Goal: A Goal:• Begin the culture change of “beating the model” to ‘Get
it right when it matters.’• Focus should be on:
– Impact Weather– Accurate Grids
• How ?– Change Habits– Change perceptions of the models
• “Garbage that the model spits out”
– Change definition of success ?– Change what we measure ?– Reward what we do (even if it fails!) ?
• Good AFD with a “bad” MAE• Or: Praise a “bad” forecast for the right reasons
• Begin the culture change of “beating the model” to ‘Get it right when it matters.’
• Focus should be on: – Impact Weather– Accurate Grids
• How ?– Change Habits– Change perceptions of the models
• “Garbage that the model spits out”
– Change definition of success ?– Change what we measure ?– Reward what we do (even if it fails!) ?
• Good AFD with a “bad” MAE• Or: Praise a “bad” forecast for the right reasons
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 4
Weather ImpactsWeather Impacts
• The Customer Defines the Impact– Not always what we think the impact is
– We know the meteorology (and the limitations of the meteorology)
• SSDs Working Definition of Impact Weather:– A weather event that causes a substantial departure
from the normal routine. The event must be forecastable with sufficient lead time to allow decision makers to take appropriate action that results in a saving of lives and/or a reduction in property damage or other adverse economic impacts.
• The Customer Defines the Impact– Not always what we think the impact is
– We know the meteorology (and the limitations of the meteorology)
• SSDs Working Definition of Impact Weather:– A weather event that causes a substantial departure
from the normal routine. The event must be forecastable with sufficient lead time to allow decision makers to take appropriate action that results in a saving of lives and/or a reduction in property damage or other adverse economic impacts.
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 5
Forecasting for Weather ImpactsForecasting for Weather Impacts• SR Grid policy is designed to allow
forecasters to concentrate on the weather that will create impacts
• WISE Concept came out in 2005– SSD / CWWD survey in early 2006:
• 90% said our office are using the grid policy, but our neighbors are not.
• Idea behind WISE, or looking for “windows of opportunity” is to let the model do what it does best and allow the forecaster to do what he/she does best.
• SR Grid policy is designed to allow forecasters to concentrate on the weather that will create impacts
• WISE Concept came out in 2005– SSD / CWWD survey in early 2006:
• 90% said our office are using the grid policy, but our neighbors are not.
• Idea behind WISE, or looking for “windows of opportunity” is to let the model do what it does best and allow the forecaster to do what he/she does best.
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 6
WISE? What about ConOps ?WISE? What about ConOps ?
• ConOps is discussing infrastructure (grid size, cluster, equipment, etc.), but not operations
• We have the opportunity to influence operations for the ConOps
• Already being done in many offices
• ConOps is discussing infrastructure (grid size, cluster, equipment, etc.), but not operations
• We have the opportunity to influence operations for the ConOps
• Already being done in many offices
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 7
Southern Region Grid PolicySouthern Region Grid Policy• Does the Model Make Meteorological
Sense ?– Yes : Use it– No : Don’t
• Everyone uses the same thing as the starting point to collaborate the forecast
• Concentrate on the Impact Events or where the model is off base– Routine errors minimized
• Does the Model Make Meteorological Sense ?– Yes : Use it– No : Don’t
• Everyone uses the same thing as the starting point to collaborate the forecast
• Concentrate on the Impact Events or where the model is off base– Routine errors minimized
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 8
Southern Region Grids - BeforeSouthern Region Grids - Before
Max / Min T Forecasts by RegionSeptember 2004 - May 2005
-10-8-6-4-202468
10
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR CR ER WR
Max / Min T Forecasts by RegionSeptember 2004 - May 2005
-10-8-6-4-202468
10
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR CR ER WR
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 9
Southern Region Grids - AfterSouthern Region Grids - After
Max / Min T Forecasts by RegionSeptember 2005 - May 2006
-10-8-6-4-202468
10
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR CR ER WR
Max / Min T Forecasts by RegionSeptember 2005 - May 2006
-10-8-6-4-202468
10
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR CR ER WR
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 10
Why the Jump ?Why the Jump ?• Regime ?
– Western drought – Above normal winter– Model MAE
• 4.2 in 2005 / 2006• 4.1 in 2004 / 2005
• Man / Machine Mix ?– Forecast more than 4 degrees from MOS– 13 % of the time in 2005 – 2006– 17 % of the time in 2004 - 2005
• Regime ?– Western drought – Above normal winter– Model MAE
• 4.2 in 2005 / 2006• 4.1 in 2004 / 2005
• Man / Machine Mix ?– Forecast more than 4 degrees from MOS– 13 % of the time in 2005 – 2006– 17 % of the time in 2004 - 2005
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 11
A Possible ReasonA Possible Reason
Temperature ForecastsWhen Differing from Guidance > 4 Degrees
-20
-10
0
10
20
30
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
2004 - 2005 2005 - 2006
Temperature ForecastsWhen Differing from Guidance > 4 Degrees
-20
-10
0
10
20
30
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
2004 - 2005 2005 - 2006
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 12
Southern Region Grids - BeforeSouthern Region Grids - Before
PoP Forecasts by RegionSeptember 2004 - May 2005
-10
-8
-6
-4
-2
0
2
4
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR ER CR WR
PoP Forecasts by RegionSeptember 2004 - May 2005
-10
-8
-6
-4
-2
0
2
4
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR ER CR WR
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 13
Southern Region Grids - AfterSouthern Region Grids - After
PoP Forecasts by RegionSeptember 2004 - May 2005
-10
-8
-6
-4
-2
0
2
4
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR ER CR WR
PoP Forecasts by RegionSeptember 2004 - May 2005
-10
-8
-6
-4
-2
0
2
4
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
SR ER CR WR
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 14
Why the Jump ?Why the Jump ?• Regime ?
– Western drought – Above normal winter– Model
• 0.11 Brier Score in 2005 / 2006• 0.08 Brier Score in 2004 / 2005
• Man / Machine Mix ?– Forecast more than 20 % from MOS– 6 % of the time in 2005 – 2006– 10 % of the time in 2004 - 2005
• Regime ?– Western drought – Above normal winter– Model
• 0.11 Brier Score in 2005 / 2006• 0.08 Brier Score in 2004 / 2005
• Man / Machine Mix ?– Forecast more than 20 % from MOS– 6 % of the time in 2005 – 2006– 10 % of the time in 2004 - 2005
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 15
A Possible Reason ?A Possible Reason ?
PoP ForecastsWhen Differing from Guidance > 20 %
-20
-15
-10
-5
0
5
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
2004 - 2005 2005 - 2006
PoP ForecastsWhen Differing from Guidance > 20 %
-20
-15
-10
-5
0
5
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14
Forecast Period
% I
mpr
ovem
ent
over
MO
S
2004 - 2005 2005 - 2006
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 16
The Future ?The Future ?• But…wouldn’t grid PoP verification
make more sense ??
• Jack.settelmaier@noaa.gov
• But…wouldn’t grid PoP verification make more sense ??
• Jack.settelmaier@noaa.gov
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 17
Severe Weather WarningsSevere Weather Warnings• Public Forecast Verification Nice for
Forecasters– Not Tied to GPRA Goals
• Hazardous Weather Warnings are in the Bread and Butter of NWS– Tied to GPRA Goals
• Public Forecast Verification Nice for Forecasters– Not Tied to GPRA Goals
• Hazardous Weather Warnings are in the Bread and Butter of NWS– Tied to GPRA Goals
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 18
Tornado WarningsTornado Warnings
All Tornado Events
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year
POD LT > 0
All Tornado Events
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year
POD LT > 0
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 19
Tornado WarningsTornado Warnings
Tornado EventsWith One or More Fatalities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year
POD LT > 0
Tornado EventsWith One or More Fatalities
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Year
POD LT > 0
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 20
Making the ShiftMaking the Shift• Get away from the routine
– Need a database instead of ‘Stats on Demand’• Brent.macaloney@noaa.gov
• Verify the stuff that matters– Grids instead of points (lots of issues here)
• BOI Verify
– Large temperature swings– Critical Temperatures– PoP timing– Hazards and threats
• Get away from the routine – Need a database instead of ‘Stats on Demand’
• Brent.macaloney@noaa.gov
• Verify the stuff that matters– Grids instead of points (lots of issues here)
• BOI Verify
– Large temperature swings– Critical Temperatures– PoP timing– Hazards and threats
April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 21
Making the ShiftMaking the Shift• Verify the stuff that matters
– Mississippi Valley Tornado Outbreak– DFW Snow / Sleet– West Texas December Cold Snap– Florida Fires– Fire Weather Season in General– Anything else I can’t think of…
• How many of us were told:– Great job beating MOS !!!
• Verify the stuff that matters– Mississippi Valley Tornado Outbreak– DFW Snow / Sleet– West Texas December Cold Snap– Florida Fires– Fire Weather Season in General– Anything else I can’t think of…
• How many of us were told:– Great job beating MOS !!!
Recommended