Data Quality Report Card: Just How Good Are We?

January 15, 2015 Aman Bains

1

Presentation Outline

Background on Quality Measurements and Objectives

Data Quality Report Results for 2013 ARB’s PQAO Challenges Looking Ahead

2

3

Measurement Quality Objectives

Measurements are made in order to assess the quality of the data produced One-point QC checks Flow rate verifications Performance audits “Objectives” are acceptance criteria for quality attributes, usually dictated in CFR, for rendering data for record to be of “good quality.”

Precision, Accuracy and Bias

Precision Good Accuracy Good

Precision Good Accuracy Bad

Precision Bad Accuracy Good

4 Bias distortion in one direction

Tools for Assessing Precision

5

Pollutant Precision

1-pt QC checks

Collocated Measurements

Gaseous CO, NO2, O3, SO2

Continuous PM2.5 PM10

Manual PM2.5

PM10 (high-vol) PM10 (low-vol)

Tools for Assessing Accuracy/Bias

6

Pollutant Accuracy/Bias

1-pt QC checks

Flow Rate Verifications (required in

AQS)

Flow Rate Verifications

(not required in AQS)

Performance Audit

Gaseous CO, NO2, O3, SO2

Continuous PM2.5 PM10

Manual PM2.5

PM10 (high-vol) PM10 (low-vol)

Measurement Quality Objectives (Criteria)

7

Pollutant Precision % (CV) Bias % Accuracy via

Audits %

Gaseous 1-pt QC Checks Through- the-Probe

CO 10 ± 10 ± 15 NO2 15 ± 15 ± 15 O3 7 ± 7 ± 10

SO2 10 ± 10 ± 15

Particulates Collocated Samples Flow Rate Flow Checks

PM10 10 ± 4 ± 10 PM2.5 10 ± 4 ± 4

Data Quality Report Provides data producers and users an assessment

of data quality Identify where improvements are needed Ambient data capture rates Precision and accuracy of criteria pollutants Compared against measurement quality

objectives (federal criteria) Gases (CO, O3, NO2, SO2) Particulates (PM10, PM2.5)

8

AMP 430 “Ambient” Data Capture Data producers should review data quarterly to ensure

that data are uploaded in a timely manner.

Goal is to have at least 75% of the possible ambient data uploaded to AQS.

9

AMP 256 Data Quality Indicator Report

10

2013 Ambient Data Reported

11

*

*Includes ARB and 21 other air monitoring districts reporting under ARB’s PQAO

*

2013 Ambient Data Reported

12

* *

*Includes ARB and 21 other air monitoring districts reporting under ARB’s PQAO

2013 Data Quality Results – Gases

Percentage of sites uploading 100% 1-pt QC Precision Criteria

Met?

Accuracy/ Bias

Criteria Met?

Pollutant ARB Bay Area

San Diego

South Coast

CO 90 100 67 86 NO2 87 100 88 69 O3 93 100 89 90

SO2 80 100 100 63 13

2013 Ambient Data Reported

14

*Includes ARB and 21 other air monitoring districts reporting under ARB’s PQAO

* *

2013 Data Quality Results – Particulates

Pollutant ARB* Bay Area

South Coast

San Diego

Precision Criteria

Met?

Accuracy / Bias

Criteria Met?

PM10 100 100 100 100

PM2.5 99 100 100† 100 x

15 *Collocation requirement (15%) not met in ARB’s PQAO

†South Coast met criteria for one method for the year 2013

Three-Year Average at ARB’s PQAO Level

16

Pollutant 2011-2013 Result (%) Criteria (%)

Gaseous Precision Bias Precision Bias

CO 4.70 ±3.61 10 ± 10

NO2 5.35 ±3.89 15 ± 15

O3 3.75 ±2.80 7 ± 7

SO2 4.18 ±3.22 10 ± 10

Particulates Precision Bias Precision Bias

PM10 5.19 ±1.28 10 ± 4

PM2.5 20.56 ±0.50 10 ± 4

Percent of Sites Meeting the Precision Criteria for Gases – 2013

Pollutant US ARB Bay Area

South Coast

San Diego

CO 98 97 100 100 100 NO2 99 98 100 100 100 O3 98 98 100 100 100

SO2 98 100 100 100 100

17 This assessment is performed at the site level.

Percent of Sites Meeting the Precision Criteria for PM – 2013

Pollutant US Region 9* ARB Bay

Area South Coast

San Diego

PM10 69 50 100 100 100 100

PM2.5 55 14 18 0 67 0

18 Percentage is based on the number of sites with collocated samplers.

* Includes Arizona, Nevada, and Hawaii. Excludes California.

Why PM2.5 Precision Criteria More Difficult to Meet?

Lower concentrations lead to higher imprecision Lower cut-off limits for evaluation: PM2.5: 3 µg/m3

PM10: 15 µg/m3

Higher cut-off limits lead to better precision

Improve precision through operational practices Consistency in practice should help

19

Summary: Precision and Accuracy

20

Precision Accuracy

Gases

Particulates

o Less than required collocated sites

o Annual precision criteria not met for PM2.5, but improving

*

* PM flow rate verification data should be uploaded

Challenges Air Quality System (AQS) Accurate reflection of network Uploading data Correct coding (e.g., collocated samplers) Important to get accurate info in AQS

Other Investigate PM2.5 precision

21

Looking Ahead AQS training and corrections Adjust PM collocated monitors Improve PM2.5 precision Upload PM flow rate verification data Develop additional assessment tools (graphics) Provide timely data quality reviews Continue using 3-year average to track trends

and identify possible data quality issues

22

Questions? Aman Bains, (916) 323-0032, abains@arb.ca.gov Patrick Rainey, Manager (916) 327-4756, prainey@arb.ca.gov Reports available at ARB’s website: www.arb.ca.gov/aaqm/qa/dqreports/dqreports.htm

23