Data Quality Screening ServiceChristopher Lynnes, Richard Strub, Thomas Hearty, Bruce Vollmer

Goddard Earth Sciences Data and Information Sciences CenterRobert Wolfe, Suraiya Ahmad, Neal Most

MODIS Adaptive Processing SystemPeter Fox, Stephan Zednik, Tetherless World Constellation, RPI

Edward Olsen, Jet Propulsion Laboratory

Goal: Help users apply proper screening to data using quality flags

0

Cloud Mask Status Flag0=Undetermined1=Determined

Cloud Mask Cloudiness Flag0=Confident cloudy1=Probably cloudy2=Probably clear3=Confident clear

Day/Night Flag0=Night1=DaySunglint Flag0=Yes1=NoSnow/Ice Flag0=Yes1=No

Surface Type Flag0=Ocean, deep lake/river1=Coast, shallow lake or river2=Desert3=Land

Level 1 and 2 satellite data products typically keep all retrieved values.

Quality Control “flags” are often available for these data

Describe instrument performance and calibration

Reflect observing conditions (e.g., cloud fraction)

Are based on algorithm “happiness”

Statistically, the better the quality flag, the less likely it contains systematic biases.

Easy:Quality level for total precipitable

water

Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS)

Not So Easy:Highest pressure of

“Best” quality values in moisture profiles

The Data Quality Screening Service (DQSS) filters out bad pixels for the user

An ontology organizes the variations in quality schemes and drives both the selection interface and the Masker algorithm.

DQSS Ontology

Funded by NASA ACCESS (Accelerating Collaborative Connections for Earth System Science)

Output file has the same format and structure as the input file (except for the extra mask and original data fields)

Original Data Array:Total Column Precipitable H2O

Mask Based on User Criteria(Quality level < 2)

Good quality data pixels retained

Percent of Biased Data in MODIS Aerosols Over Land Increases as Confidence Flag Decreases

Bad

Marginal

Good

Very Good

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Compliant*Biased LowBiased High

*Compliant data are within + 0.05 + 0.2τAeronet

Statistics derived from Hyer, E., J. Reid, and J. Zhang, 2010, An over-land aerosol optical depth data set for data assimilation by filtering, correction, and aggregation of MODIS Collection 5 optical depth retrievals, Atmos. Meas. Tech. Discuss., 3, 4091–4167.

Objective: Provide a quaiity screening service

(a) Quality Control flags can be complicated to handle

MODIS Quality Bitfieldsin Cloud_Mask_SDS

Ocean Land

AIRS Quality Levels

0 Best Data Assimilation1 Good Climatic Studies2 Do Not Use

MODIS Aerosols Confidence Flags

3 Very Good2 Good1 Marginal0 Bad

3 Very Good2 Good1 Marginal0 Bad

Ocean Land Use these flags to have 2/3 of values within expected error bounds

±0.05 ± 0.15 t ±0.03 ± 0.10 t

Ocean Land

(b) Interpretations and recommendations vary across and within instruments

Two Different AIRS Quality Schemes

Deployment: Distributed architecture supports DQSS at a diverse set of data

providersMODIS Operational Environment

GES DISC Operational Environment

Sustainment: Ontology-driven software reduces the cost of adding datasets to

DQSS

deep description of data fields and variables

QualityView ties ScreeningAssertions together

QualityLevel is the simplest of quality schemes

Status: DQSS is available for AIRS Level 2 data at GES DISC

MODIS Level 2 Water Vapor at MODAPS

Use of Java also helps portability...*

Coming Soon (Nov-Dec 2011)...

Microwave Limb Sounder (relatively easy)

MODIS Level 2 Aerosols (not easy)

Software Release? (Need a requestor)

AcknowledgmentsPatrick West of RPI; Karen Horrocks, Cid Praderas, Ivan Tcherednitchenko, Greg Ederer, Gang Ye, Ali Rezaiyan-Nojani of MODAPS

AIRS Level 2 Quality Contol Selection Interface in GES DISC’s Mirador search tool

MODAPS Post-Processing Selection Interface

0

1

2

3

4

5

6

7

OPeNDAP access to DQSS via the OPeNDAP Gateway

Allows OPeNDAP to access REST services on back-end

Ozone Monitoring Instrument (account for row anomalies)?

Link Quality Control ontology with other ontologies?

Quality Assessment Ontology?

Data and Services Ontology (deep description of data fields)?

Collaborative Screening (Dr. Alice shares screening criteria with Dr. Bob)

Down the Road*...but not nearly as much as we expected