Calculating Statistics: Concentration Related

Performance Goals

James W. BoylanGeorgia Department of Natural Resources

PM Model Performance WorkshopChapel Hill, NC

February 11, 2004

Outline

• Performance Statistic– Standard Bias and Error Calculations

• Model Performance Goals for PM– Speciated Bias and Error Goals– Relative Proportions Goals

Performance Metrics Equation

Mean Bias (g/m3)

Mean Error (g/m3)

Mean Normalized Bias (%) (-100% to +) Mean Normalized Error (%) (0% to +)

Normalized Mean Bias (%) (-100% to +) Normalized Mean Error (%) (0% to +)

Mean Fractional Bias (%) (-200% to +200%) Mean Fractional Error (%) (0% to +200%)

N

iom CC

NMB

1

1

N

i mo

om

CC

CC

NMFE

1

2

1

N

i mo

om

CCCC

NMFB

1

2

1

N

io

N

iom

C

CCNME

1

1

N

io

N

iom

C

CCNMB

1

1

N

i o

om

C

CC

NMNE

1

1

N

i o

om

C

CC

NMNB

1

1

N

iom CC

NME

1

1

Example

GT showed a positive bias of 11 points

NB = 14.3%FB = 13.3%

North Carolina 77Georgia Tech 88

Performance Metrics

• Mean Normalized Bias and Error – Usually associated with observation-based minimum

threshold• Some components of PM can be very small making it

difficult to set a reasonable minimum threshold value without excluding a majority of the data points

– Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small

• A few data points can dominate the metric

– Overestimations are weighted more than equivalent underestimations

Performance Metrics

• Normalized Mean Bias and Error – Biased towards overestimations

• Mean Fractional Bias and Error– Bounds maximum bias and error – Gives additional weight to underestimations and

less weight to overestimations

Example Calculations

• Mean Normalized Bias and Error– Most biased and least useful of the three metrics

• Normalized Mean Bias and Error • Mean Fractional Bias and Error

– Least biased and most useful of the three metrics

Model g/m3

)

Obs. g/m3

)

MBg/m3)

NMB (%)

MNB (%)

MFB (%)

ME g/m3

)

NME (%)

MNE (%)

MFE (%)

0.05 1.0 -0.95 -95 -180.95 +0.95 +95 +180.95

1.0 0.05 +0.95 +1900 +180.95 +0.95 +1900 +180.95

1.0 0.01 +0.99 +9900 +196.04 +0.99 +9900 +196.04

0.683 0.353 +0.33 +93.4 +3901.7 +65.3 0.96 272.9 3965.0 186.0

SAMI Model Performance Summary

Species

# Obs

Mean

g/m3

MB g/m

3

NMB (%)

MNB (%)

MFB (%)

ME g/m

3

NME (%)

MNE (%)

MFE (%)

SO4 134 6.71 -1.16 -17.3 1.1 -22.7 2.48 37.0 55.1 50.2

NO3 134 0.63 -0.30 -47.6 6.8 -73.6 0.52 81.8 112.8 107.2

NH4 134 2.70 -1.25 -46.4 -27.4 -57.4 1.43 53.1 61.6 70.0

NH4 Bi

134 1.44 0.01 0.4 34.2 -2.6 0.62 42.9 70.4 44.4

ORG 132 3.41 -0.27 -7.8 15.8 -6.0 1.37 40.4 53.8 43.9

EC 132 0.56 -0.05 -8.6 15.1 -12.7 0.27 48.3 61.9 50.4

Soils 135 0.55 0.25 46.2 171.6 21.9 0.57 102.9 207.4 72.5

PM2.5 130 17.05 -4.79 -28.1 -9.1 -28.8 6.8 39.8 48.9 47.6

PM10 130 23.44 -5.21 -22.2 -6.2 -21.0 9.18 39.1 44.2 43.5

PMC 126 6.98 -0.48 -6.9 43.9 7.8 3.86 55.2 78.7 54.1

bext 132 133.1 -27.91 -21.0 -10.2 -23.7 43.70 32.8 40.0 40.4

Proposed Performance Goals• Based on Mean Fractional Error (MFE) and Mean

Fractional Bias (MFB) calculations• Performance goals should vary as a function of

species concentrations– More abundant species should have a MFE +50%

and MFB ±30%– Less abundant species should have less stringent

performance goals

• Goals should be continuous functions with the features of:– Asymptotically approaching +50% MFE and ±30%

MFB when the concentrations (mean of the observed and modeled concentrations) are greater than 2.5 g/m3

– Approaching +200% MFE and ±200% MFB when the concentrations (mean of the observed and modeled concentrations) are extremely small

Proposed Mean Fractional Error and Bias Goals

301703/5.0

)(5.0

mg

CC mo

eMFB

501503/75.0

)(5.0

mg

CC mo

eMFE

Example Calculations

Species X Model g/m3)

Obs. g/m3)

FB (%) FE (%)

Day 1 – Site A 2.0 1.0 +66.7 +66.7

Day 1 – Site B 1.0 2.0 -66.7 +66.7

Day 2 – Site A 1.0 0.4 +85.7 +85.7

Day 2 – Site B 0.5 1.5 -100.0% +100.0%

Average 1.125 1.225 -3.6% 79.8%

Average CO + CM = 0.5*(1.125 + 1.225) = 1.175

MFE performance goal for “Species X” = 81.3%MFB performance goal for “Species X” = ±46.2%

Mean Fractional Error Goal

Speciated Fine PM Performance

0

50

100

150

200

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Mean Fractional Bias Goal

Speciated Fine PM Performance

-200

-100

0

100

200

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

SAMI – 6 Episodes

Speciated Fine PM Performance

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Ammonium (Sulfate)

Nitrate

Soils

Elem. Carbon

Organics PMC PM2.5 PM10

Ammonium (Bisulfate) Sulfate

SAMI – 6 Episodes

Speciated Fine PM Performance

-200

-100

0

100

200

0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Elem. Carbon

PMC

Organics

Nitrate

Soils

Ammonium (Sulfate)

PM2.5 PM10Ammonium (Bisulfate)

Sulfate

VISTAS – July 1999 Episode

Speciated Fine PM Performance

0

50

100

150

200

0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Ammonium (Sulfate)

Nitrate

Soils

Elem. Carbon

Organics

PMCPM2.5 PM10

Sulfate

VISTAS – July 1999 Episode

Speciated Fine PM Performance

-200

-100

0

100

200

0.0 5.0 10.0 15.0 20.0 25.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Elem. Carbon

PMC

OrganicsNitrate

Soils

Ammonium (Sulfate)

PM2.5 PM10

Sulfate

VISTAS – January 2002 Episode

Speciated Fine PM Performance

0

50

100

150

200

0.0 2.0 4.0 6.0 8.0 10.0 12.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Err

or

Ammonium (Sulfate)

Nitrate

Soils

Elem. Carbon

Organics

PMC PM2.5 PM10

Sulfate

VISTAS – January 2002 Episode

Speciated Fine PM Performance

-200

-100

0

100

200

0.0 2.0 4.0 6.0 8.0 10.0 12.0Average Concentration (g/m3)

Me

an

Fra

cti

on

al

Bia

s

Elem. Carbon

PMC

Organics

Nitrate

Soils

Ammonium (Sulfate)

PM2.5 PM10

Sulfate

Relative Proportions (RP) PERF Goals

• EPA draft guidance (2001)– “For major components (i.e., those observed to

comprise at least 30% of measured PM2.5), we propose that the relative proportion predicted for each component averaged over modeled days with monitored data agrees within about 20% of the averaged observed proportion. For minor observed components of PM, we suggest a goal that the observed and modeled absolute proportion of each minor component agree within 5%.”

N

i

oN

i m

m

Total

component

Total

component

Co

C

NC

C

NBias

11

11 0.2 RP%(5%)

Example Calculation

• Calculating component proportions based on concentrations averaged over multiple days can hide poor model performance

Observed RP (%) Modeled RF (%)

Day 1 50% 95%

Day 2 50% 95%

Day 3 50% 5%

Day 4 50% 5%

Average 50% 50%

Observed Simulated

26.7%

47.1%12.3%

2.8%

4.3%6.8%

25.6%

50.5%

10.8%

4.7%

4.2% 4.1% SO4

ORG

NH4

NO3

EC

Soils

Relative Proportions for SAMI

Relative Proportions for SAMI

Relative Proportions Evaluation

-15

-10

-5

0

5

10

15

0 10 20 30 40 50 60

Relative Proportion of Fine PM (%)

Bia

s (

%)

Sulfate

Ammonium (Bisulfate)

Organics

Nitrate

Elem. Carbon

Soils

Proposed Relative Proportions Performance Goals

• Propose to use an equation that accounts for the day-to-day variability of species relative proportions:

RP 30%, Error 10%RP 15%, Error 5%RP 15% - 30%, Error [RP]/3

N

i o

o

m

m

Total

component

Total

component

C

C

C

C

NError

1

1

Proposed Relative Proportions Performance Goals

Relative Proportions Evaluation

0

5

10

15

0 10 20 30 40 50 60

Relative Proportion of Fine PM (%)

Err

or

(%)

Sulfate

Ammonium (Bisulfate)

OrganicsNitrate

Elem. Carbon

Soils

Concluding Remarks

• Recommended performance values are model goals, not model criteria– Failure to meet proposed performance goals

should not prohibit the modeling from being used for regulatory purposes• Help identify areas that can be improved upon

in future modeling

• If performing episodic modeling, performance evaluation should be done on an episode-by-episode basis

• If performing annual modeling, performance evaluation should be done on a month-by-month basis

Concluding Remarks (cont.)

• As models mature, performance goals can be made more restrictive by simply: – Adjusting the coefficients in the MFE and MFB

goal equations – Lowering the relative proportion error goals

• Q: Is there a need for performance goals for gaseous precursors or wet deposition species?– “One-atmosphere” modeling system– If not, still should be evaluated to help identify

potential problems with PM model performance

Questions?