Robert W. Pinder, Alice B. Gilliland, Robert C. Gilliam, K. Wyat AppelAtmospheric Modeling Division, NOAA Air Resources Laboratory,

in partnership with USEPA National Exposure Research Laboratory

2007 CMAS ConferenceOctober 2, 2007

Evaluating uncertainty predictions using an ensemble of CMAQ model

configurations

Objective• Uncertainty: what is the

likelihood that the observed value is within a given range?

• Applications: Exposure studies Diagnostic evaluation tool

• One measure of uncertainty is the error when the model is compared with observations.

• Can we use an ensemble approach to make a better estimate of this range?

Histogram of CMAQ O3 Model Error 8-hour max O3, 228 AQS Sites from the SE US

Sources of UncertaintyStructural Uncertainty:

VOC species lumpingPhysical processesApproach: vary representation

Parameter Uncertainty:EmissionsMeteorologyChemical rate constantsApproach: Monte Carlo methods

Challenge: Monte Carlo methods are not feasible given CMAQ’s computational requirements.

Method

• Variety of CMAQ / MM5 model configurations

• Direct sensitivity calculations• Use observations to remove

spurious ensemble members

Generate Ensemble Members for Structural Uncertainty using Multiple Model Configurations

Planetary Boundary Layer / Land Surface Model Pleim-Xiu Land Surface Model; ACM: Asymmetric

Convective Model (Pleim and Chang, 1992) Miller-Yamada-Janjic (Janjic, 1994), NOAH Land

Surface Model Medium Range Forecast (Hong and Pan, 1996), NOAH

Land Surface Model

Chemical Mechanism Carbon Bond IV SAPRC-99

Six structural uncertainty cases

Generate Ensemble Members for Parametric Uncertainty using Direct Sensitivity Calculation

VOCNO

OVOCNO

VOCO

2VOC

NOO

2NO

VOCOVOC

NOONOO

x

32

x

23

22

2x

322

x

3

x

3x3

Use the Direct Decoupled Method (DDM) to calculate sensitivity to:• NOx Emissions• VOC Emissions• Second-order sensitivity• O3 Boundary conditions

Compared to brute-force calculation, errors are 5-10% (Cohan et al., 2005)

At each grid cell, calculate ozone response to emissions and boundary concentrations

Direct Calculation of Ozone SensitivityJuly 16, 2002, 2 PM

>30 -50510152025

)NO(EmissionO

x

3

)OCEmission(VO3

)O(BoundaryO

3

3

O3 (ppb)

Use Observations to Constrain Ensemble

Used to evaluate boundary conditions

Used to evaluate ensemble quality

AQS O3 Monitoring Sites

Repeat 200 times

Structural uncertainty simulations (6)

Use DDM to calculate O3 sensitivity to NOx, VOC, and boundary conditions.

Randomly sample from range of uncertain NOx emissions, VOC emissions, boundary concentrations, and structural uncertainty cases

Generate an ensemble member by calculating the O3 field acrossSE US domain

Use observations to remove spurious ensemble members

Example: Atlanta, GeorgiaJuly 1-28, 2002

Structural Uncertainty

Structural + Parametric Uncertainty

Spread is large – can we use the observations to narrow this range?

Prune ensemble members not consistent with observations

Remove ensemble members that do not constrain the range

Pruned ensemble has narrow range while still including observations

200 member ensemble10 member ensemble

Compare with +/-30%

Range is 40% lower± 30% of base case CMAQ10 member ensemble

Analysis at all sitesDataset:• 38 locations, 28 days• 1064 observations

Evaluation:• Randomly reserve 50% of dataset• Derive ensemble, prune using half

of observations• Evaluate using the reserve dataset

Ensemble range includes 85% of observations

Range is 40% smaller than ±30% of base case

AQS O3 Monitoring Sites

Trade-off between coverage of observations and spread in range

Ensemble Size

Observations within Range

Average Ensemble Range

10 70% ± 12%

20 72% ± 12%

50 75% ± 15%

100 83% ± 18%

200 85% ± 18%

± 30% CMAQ

91% ± 30%

Conclusions• This ensemble generation and pruning technique

provides a more robust uncertainty range: Observed value is within the range 85% of the time 40% reduction in spread compared to +/- 30% rule

• Simultaneously narrowing these bounds and improving the performance depends on reducing structural errors in CMAQ

• Locations and times that fall outside of the ensemble range should be targeted for uncovering structural errors in the model

DISCLAIMER: The research presented here was performed under the Memorandum of Understanding between the U.S. Environmental Protection Agency (EPA) and the U.S. Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) and under agreement number DW13921548. This work constitutes a contribution to the NOAA Air Quality Program. Although it has been reviewed by EPA and NOAA and approved for publication, it does not necessarily reflect their policies or views.

Acknowledgements:

Sergey Napelenok, Jenise Swall, Kristen Foley