The Value of Air Quality The Value of Air Quality Forecasting in the Mid-Atlantic Forecasting in the Mid-Atlantic

RegionRegionGregory Garner – Penn State MeteorologyGregory Garner – Penn State Meteorology

Advisers: Anne Thompson, William RyanAdvisers: Anne Thompson, William Ryan

2011 National Air Quality Conference: San Diego, CAMarch 10, 2011

22

The problem…The problem…Adapted from Yorks (2009)

Adapted from Tang et al. (2008)

Charlotte, NC (Eder, 2010)

33

……so is it valuable?so is it valuable?• How to assess the value of a forecast system?How to assess the value of a forecast system?

– The “Umbrella Problem”The “Umbrella Problem”

LossLossCostCost

00CostCost

ProtectProtect Do Not Do Not ProtectProtect

GoodGoodBadBad

CostCost LossLoss

44

Cost? Loss? Ok…now what?Cost? Loss? Ok…now what?• ““How much would I expect to lose on any given day?”How much would I expect to lose on any given day?”

– ProtectProtect

– Do not protectDo not protect

– ……and if you had to chooseand if you had to choose

( Prob {Good} x Cost ) + ( Prob {Bad} x Cost )= ( Prob {Good} + Prob {Bad} ) x Cost

= Cost

( Prob {Good} x 0 ) + ( Prob {Bad} x Loss )

= Prob {Bad} x Loss

Minimum ( Cost, Prob {Bad} x Loss )

55

Where does the forecast come in?Where does the forecast come in?• You can make a better decision with a forecast!You can make a better decision with a forecast!

– No forecastNo forecast

– ForecastForecast

Prob {Bad} Climatological probability that a Bad event occurs

Prob {Bad} Probability that a Bad event occurs with the given forecast

66

……but what about “Value”?but what about “Value”?• You expect to lose with You expect to lose with climatologyclimatology……

• You expect to lose with the You expect to lose with the forecastforecast……

• So the difference between the two is the value!So the difference between the two is the value!

77

A little more on “Value”A little more on “Value”• Expected “Value” Expected “Value”

of the forecast of the forecast system depends system depends on the probability on the probability of a forecasted of a forecasted “Bad” event“Bad” event

• Sum this up over Sum this up over all levels above the all levels above the threshold:threshold:

$VALUE!

88

Interpreting ValueInterpreting Value• Value (V) is a function of Cost and LossValue (V) is a function of Cost and Loss

– A ratio of Cost to Loss (C/L) is used to calculate VA ratio of Cost to Loss (C/L) is used to calculate V

• Value as the “percent savings from climatology”Value as the “percent savings from climatology”• C/L as actions or “protective measures”C/L as actions or “protective measures”

CostCost LossLoss

??

99

DataData• 41 sites, 9 forecast 41 sites, 9 forecast

regions (overlap)regions (overlap)– DDOE, MDE, VADEQDDOE, MDE, VADEQ

• 3 forecast systems3 forecast systems– NAQFC, Human, NAQFC, Human,

PersistencePersistence• Convert to AQI to set Convert to AQI to set

thresholds for value thresholds for value calculationcalculation

1010

Forecast VerificationForecast Verification

1111

Forecast ValueForecast Value

1212

Spatial / Temporal Variation in ValueSpatial / Temporal Variation in Value

1313

Summary / Future ApplicationsSummary / Future Applications• Assess value of forecasts with cost-loss ratio modelAssess value of forecasts with cost-loss ratio model

• Biased model still has valueBiased model still has value

– Although human forecasters prevail!Although human forecasters prevail!

• Variability of value in time and spaceVariability of value in time and space

• General methods General methods Many applications Many applications

• Evaluation tool for NAQFC – MOSEvaluation tool for NAQFC – MOS

• Monitor-location optimizationMonitor-location optimization

1414

AcknowledgementsAcknowledgements• Data provided byData provided by

– Maryland Department of the EnvironmentMaryland Department of the Environment– District Department of the EnvironmentDistrict Department of the Environment– Metropolitan Washington Council of GovernmentsMetropolitan Washington Council of Governments– Virginia Department of Environmental QualityVirginia Department of Environmental Quality– NOAA – National Operational Model Archive and Distribution SystemNOAA – National Operational Model Archive and Distribution System

• Bill Ryan (PSU), Duc Nguyen (MDE), and Laura Landry (MDE)Bill Ryan (PSU), Duc Nguyen (MDE), and Laura Landry (MDE)

– Valuable insight into AQ forecasting operationsValuable insight into AQ forecasting operations

• This study was funded by NSF DRU Program Award I.D. 0729413 with This study was funded by NSF DRU Program Award I.D. 0729413 with added support from NASA’s Tropospheric Chemistry Program and Aura added support from NASA’s Tropospheric Chemistry Program and Aura Validation.Validation.

1515

ReferencesReferencesR Development Core Team (2010). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org/.

Original S code by Richard A. Becker and Allan R. Wilks. R version by Ray Brownrigg. Enhancements by Thomas P Minka (2010). maps: Draw Geographical Maps. R package version 2.1-5. URL http://CRAN.R-project.org/package=maps.

Thomas Williams, Colin Kelley et al. (2004): gnuplot. URL http://gnuplot.info

Berger, J. O., 2006: Basic concepts. Statistical Decision Theory and Bayesian Analysis, Springer Science+Business Media LLC, 2d ed., 1–34.

Eder, B., et al., 2010: Using national air quality forecast guidance to develop local air quality index forecasts. Bulletin of the American Meteorological Society, 91 (3), 313–326, doi:10.1175/2009BAMS2734.1.

Katz, R. W. and A. H. Murphy, 1997: Forecast value: prototype decision-making models. Economic Value of Weather and Climate Forecasts, Cambridge University Press, 183–217.

Tang, Y., et al., 2008: The impact of chemical lateral boundary conditions on cmaq predictions of tropospheric ozone over the continental united states. Environ. Fluid Mech., doi:10.1007/ s10652-008-9092-5.

Thompson, J. C. and G. W. Brier, 1955: The economic utility of weather forecasts. Mon. Wea. Rev., 83, 249–254.

Thornes, J. E., 2001: How to judge the quality and value of weather forecast products. Journal of Applied Meteorology, 8, 307–314.

Yorks, J. E., et al. 2009: The variability of free tropospheric ozone over Beltsville, Maryland (39N, 77W) in the summers 2004–2007. Atmospheric Environment, 43, 1827 - 38

1616

Extra(s)Extra(s)• Extra slide(s) to follow…Extra slide(s) to follow…

1717

Value as a metric…Value as a metric…