AN EVALUATION OF THE ETA-CMAQ AIR QUALITY FORECAST … · CMAQ Brian Eder* Ken Schere* Robert...

AN EVALUATION OF THE AN EVALUATION OF THE ETAETA--CMAQ AIR QUALITY FORECAST MODELCMAQ AIR QUALITY FORECAST MODELAS PART OF NOAAAS PART OF NOAA’’S NATIONAL PROGRAMS NATIONAL PROGRAM

CMAQ CMAQ Brian Brian EderEder**Ken Ken SchereSchere**Robert Gilliam*Robert Gilliam*Jonathan Jonathan PleimPleim**

AIRNOWAIRNOW Atmospheric Sciences Modeling DivisionAtmospheric Sciences Modeling Division

NOAA NOAA –– Air Resources LaboratoryAir Resources Laboratory* On assignment to NERL, U.S. EPA* On assignment to NERL, U.S. EPA

DaiwenDaiwen Kang Kang UCAR Visiting ScientistUCAR Visiting ScientistU.S. EPA, R.T.P., NCU.S. EPA, R.T.P., NC

August 26,2003August 26,2003 Although this work was reviewed by EPA and approved for publication, it may not necessarily reflect official Agency policy.

Forecast ConfigurationForecast Configuration

- Eta Meteorology- CBIV Mechanism- SMOKE Emissions (Offline)- 12 km grid resolution - 22 Vertical Layers- 48 Hr. Forecast (12Z Init.)

Simulation PeriodSimulation Period

- 7 July – 30 September, 2003 - 12 – 19 August (Rerun with land-use correction)

Domain

Models-3 CMAQ

This evaluation used:This evaluation used:

Hourly O3 concentrations (ppb) from EPA’s AIRNOW network

521 stations

7 July - 30 September

A suite of statistical metrics for both:

discrete forecasts and categorical forecasts

for the:

hourly, maximum 1-hr, maximum 8-hr O3 simulations

Two Forecast / Evaluation TypesTwo Forecast / Evaluation Types

-- Discrete Forecasts Discrete Forecasts

[Observed] [Observed] versus versus [Forecast][Forecast]

-- Category Forecasts Category Forecasts (Two Category) (Two Category)

Observed Exceedances, NonObserved Exceedances, Non--ExceedancesExceedancesversusversus

Forecast Exceedances, NonForecast Exceedances, Non--ExceedancesExceedances

Discrete Forecast / Evaluation Discrete Forecast / Evaluation

StatisticsStatistics- Summary- Regression

- Biases

- Errors

AIRNOW

MB Model ObsN

N= −∑1

1( )

NMBModel Obs

Obs

N

N=−

⋅∑

∑

( )

( )

1

1

100%

RMSE Model ObsN

N= −

⎛

⎝⎜⎜

⎞

⎠⎟⎟∑1 2

1

0 5

( ).

NMEModel Obs

Obs

N

N=−

⋅∑

∑1

1

100%( )

[Observed] versus [Forecast]

Category Forecast / Evaluation Category Forecast / Evaluation

-- Two Category Forecasts Two Category Forecasts

Observed Exceedances, NonObserved Exceedances, Non--ExceedancesExceedances

versusversus

Forecast Exceedances, NonForecast Exceedances, Non--ExceedancesExceedances

a b

c d

Fore

cast

Exc

eeda

nce

No

Yes

No YesObserved Exceedance

a ba b

c dc d

Category ForecastCategory ForecastAccuracyPercent of forecasts that correctly predict event or non-event.

Bias Indicates if forecasts are under-predicted (false negatives) or over-predicted (false positives)

False Alarm Rate

Percent of times a forecast of high ozone did not occur

Ab c

a b c d%=

++ + +

⎛⎝⎜

⎞⎠⎟ ⋅100

Ba bb d

=++

⎛⎝⎜

⎞⎠⎟

a ba b

c dc d

FARa

a b%=

+⎛⎝⎜

⎞⎠⎟ ⋅100

Critical Success Index

How well the high ozone events were predicted.

Probability Of Detection

Ability to predict high ozone events

CSIb

a b d=

+ +⎛⎝⎜

⎞⎠⎟ ⋅100%

Category ForecastCategory Forecasta ba b

c dc d

PODb

b d%=

+⎛⎝⎜

⎞⎠⎟ ⋅

100

a

c

a ba b

c d c d a= 154b= 1c= 36,023d= 4n= 36,182

CMAQ = 34.5 + 0.63(AIRNOW)

Max 1Max 1--hr Ohr O33

7 July – 30 September

SummaryStatistics

Discrete Evaluation

Categorical Evaluation

CMAQ = 34.5 + 0.63 (AIRNOW)

Ozone 125 ppb

A 99.6%

B

FARCSI

POD

r

N 26.0

BIASES

MB 99.4%0.6%

16.7%

NMB

ERRORS

RMSE

NME

[ppb] CMAQ AIRNOW

Mean 68.1 53.1 0.62

SD 17.3 16.7 36,814

15.028.2%

21.1

32.2%

CV 25.3 31.5Max 182.9 132

95th 99.2 8175th 78.7 65

50th 66.0 53

25th 55.6 41

5th 44.2 27

Min 0 1

Max 1Max 1-- hr Ohr O33

Spatial Evaluation Spatial Evaluation

Max 1Max 1-- hr Ohr O33CorrelationCorrelation

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

0 to 0.25 0.25 to 0.5 0.5 to 0.75 0.75 to 1

0.00 – 0.250.25 – 0.500.50 – 0.750.75 – 1.00

Mean = 0.62

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-10 to 10 10 to 20 20 to 30 30 to 40 40 to 50

Max 1Max 1-- hr Ohr O33Mean BiasMean Bias

Spatial Evaluation Spatial Evaluation

-10 – 1010 - 2020 - 3030 – 4040 – 50

Mean = 15.0

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

0 to 10 10 to 20 20 to 30 30 to 40 40 to 50

Spatial Evaluation Spatial Evaluation

Max 1Max 1-- hr Ohr O33Root Mean Square ErrorRoot Mean Square Error

0 – 1010 - 2020 - 3030 – 4040 – 50

Mean = 21.1

CMAQ = 35.1 + 0.62(AIRNOW)

a ba b

c d c d a= 3537b= 152c= 33,242d= 67n= 36,998

Max 8Max 8--hr Ohr O33

SummaryStatistics

Discrete Evaluation

Categorical Evaluation

CMAQ = 35.1 + 0.62 (AIRNOW)

Ozone 85 ppb

A 89.6%

B

FARCSI

POD

r

n 10.3

BIASES

MB 96.0%3.7%

41.0%

NMB

ERRORS

RMSE

NME

[ppb] CMAQ AIRNOW

Mean 64.0 46.7 0.59

SD 15.8 15.0 36,998

17.437.3%

22.2

39.9%

CV 24.6% 32.2%

Max 162.2 108.4

95th 92.1 71.675th 73.9 57.2

50th 62.2 46.1

25th 52.6 35.7

5th 42.1 23.3

Min 0 1

Max 8Max 8-- hr Ohr O33

Spatial Evaluation Spatial Evaluation

Max 8Max 8-- hr Ohr O33CorrelationCorrelation

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-0.1 to 0.25 0.25 to 0.5 0.5 to 0.75 0.75 to 1

0.00 – 0.250.25 – 0.500.50 – 0.750.75 – 1.00

Mean = 0.59

Spatial Evaluation Spatial Evaluation

Max 8Max 8-- hr Ohr O33Mean BiasMean Bias

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

-10 to 10 10 to 20 20 to 30 30 to 40 40 to 50

-10 – 1010 - 2020 - 3030 – 4040 – 50

Mean = 17.4

-92 -90 -88 -86 -84 -82 -80 -78 -76 -74 -72 -70 -68

34

36

38

40

42

44

46

48

0 to 10 10 to 20 20 to 30 30 to 40 40 to 50

Spatial Evaluation Spatial Evaluation

MaxMax--8 hr O8 hr O33Root Mean Square ErrorRoot Mean Square Error

0 – 1010 - 2020 - 3030 – 4040 – 50

Mean = 22.2

LandLand--Use ErrorUse ErrorApproximately two months into the forecast period, AMDB discoverApproximately two months into the forecast period, AMDB discovered ed the the landthe the land--use fields associated with Eta were being postuse fields associated with Eta were being post--processed processed incorrectly by NCEP. As a resultincorrectly by NCEP. As a result::

-- Most of the domain was classified as water.Most of the domain was classified as water.-- Dry deposition was greatly under simulatedDry deposition was greatly under simulated-- Concentrations were over predictedConcentrations were over predicted

This error was corrected on Sept. 9This error was corrected on Sept. 9thth..

-- An eight day period (12An eight day period (12--19 August) was re19 August) was re--simulated.simulated.-- Positive biases were cut in half, errors reduced also.Positive biases were cut in half, errors reduced also.

Erroneous Corrected

Temporal EvaluationTemporal Evaluation

–– Max 1 hr OMax 1 hr O33

Land-use Correction

Run rMB

(ppb)NMB(%)

RMSE(ppb)

NME(%)

A(%)

B FAR(%)

- 100.0

100.0-

CSI(%)

POD(%)

Initial 0.64 16.2 27.5 23.0 31.7 99.0 0.0 -

Corrected 0.66 7.6 13.0 16.6 21.7 99.6 0.0 -

Max 1-hr O3

Max 8-hr O3

Comparison BetweenInitial and Corrected Simulations

August 12 –19 2003

Run rMB

(ppb)NMB(%)

RMSE(ppb)

NME(%)

A(%)

B FAR(%)

- 100.0

92.03.5

CSI(%)

POD(%)

Initial 0.62 19.2 37.2 24.6 39.9 76.2 0.0 -

Corrected 0.64 10.4 20.1 17.1 26.3 90.7 6.6 28.0

SummarySummaryThe Eta-CMAQ modeling system performed reasonably well, in this, its first attempt at forecasting ozone concentrations:

Correlation: 0.59 - 0.62Bias: 15.1 ppb (28.2%) - 17.4 ppb (37.3%)

Error: 21.1 ppb (32.2%) - 22.2 ppb (39.9%)Accuracy: 89.6 - 99.6%

An error was discovered in Eta’s post processed land-use designation that resulted in the:

– under-estimation of dry deposition and – hence over-simulation of concentrations

Once corrected, the positive biases and errors were greatly reduced when the model was re-run for an eight day period:

Correlation: 0.64 - 0.66Bias: 7.6 ppb (13.0%) - 10.4 ppb (20.1%)

Error: 16.6 ppb (21.7%) - 17.1 ppb (26.3%)Accuracy: 90.7 - 99.6%