TEMPO Science Data Processing Center

TEMPO Science DataProcessing Center

John C. HouckEwan O’Sullivan

John E. DavisRaid M. Suleiman

Smithsonian AstrophysicalObservatory

Outline

Role of the SDPC

Development timeline

Driving requirements

Data processing flow

Status

2018 June 6-7 TEMPO Science Team Meeting, Boulder, CO 2/16

Role of the SDPC

Observation planning support• Schedule radiance scans, including special observations,

and irradiance, dark, and linearity measurements

Process data autonomously• Receive Level 0 TEMPO science data from IOC• Obtain GOES imagery, weather forecasts, snow & ice data• Generate Level 1,2,3 data products• Maintain mission data archive at SAO

Data distribution• Atmospheric Science Data Center, ASDC @NASA/LaRC• Remote Sensing Information Gateway, RSIG @EPA• Web mirror @SAO


Development Timeline


Late 2013 Requirements definitionEarly 2014 System designJuly 2014 PDR = Preliminary design reviewMay 2016 CDR = Critical design review

now⇒ Implement, test, revise2018 V1 complete2019 V2 complete; Requirements verification

1 yr pre-launch Operational facility setup and testingpost-launch Updates to support operations

NASA project development process, adapted for TEMPO:

Driving Requirements

Produce specified Level 1,2,3 data products

Distribute Level 1,2,3 data products within 3hr/24hr/1mo

Autonomous operation, 40 hr/wk staff

Capacity to support simultaneous reprocessing

Storage for planned mission lifetime

Recovery within 96 hours


Data Processing Flow:Level 0 to Level 2


a) Format L0 data from the IOC

b) Download ancillary data c) Process L0 to L1

d) Process L1 to L2 on iiith host

IOC

4.1.10:hk′ fmt_hk

4.6.10:hk

0:ccd′ fmt_ccd

4.6.10:irr

0:drk

0:rad

eph′ fmt_eph ephem

0:iru′ fmt_iru 0:iru

0:dit′ fmt_dit 0:dit

pkt′ fmt_pkt pkt

L0_ccd 1:irr

L0_ccd 0:drk+

L0_ccd

4.5.21:rad′′′

L1_inr_prep

4.5.21:rad′′

INR (smoothover full scan)

4.5.4

1:rad′

L1_inr_post

4.5.21:rad

scantailoring

GOES tiepoints

dl_goes4.1.2

GOES

dl_iers4.1.2

IERS Bulletin A

dl_snow4.1.2

Snow & Ice

dl_met4.1.2

Meteorology

1:rad

1:irr

L1_cloud4.5.5

cloud

L1_o3_profile4.5.5

O3 profile

L1_o3_total4.5.5

O3 total

L1_trace_gas4.5.5

H2CO

L1_trace_gas4.5.5

NO2 L2_split4.5.5

NO2

CSC archived file external source

CSCI temporary file internal copyKey

Daily, copyto each hostfor L1 to L2processing

ProcessL1 to L2on iiith host

Receive TEMPOLevel 0 data

from IOC

Download ancillarydata from web

Level 0 to Level 1processing







IOC

4.1.10:hk′ fmt_hk

4.6.10:hk

0:ccd′ fmt_ccd

4.6.10:irr

0:drk

0:rad




pkt′ fmt_pkt pkt

L0_ccd 1:irr

L0_ccd 0:drk+

L0_ccd

4.5.21:rad′′′

L1_inr_prep

4.5.21:rad′′


4.5.4

1:rad′

L1_inr_post

4.5.21:rad

scantailoring

GOES tiepoints

dl_goes4.1.2

GOES

dl_iers4.1.2

IERS Bulletin A

dl_snow4.1.2

Snow & Ice

dl_met4.1.2

Meteorology

1:rad

1:irr

L1_cloud4.5.5

cloud

L1_o3_profile4.5.5

O3 profile

L1_o3_total4.5.5

O3 total

L1_trace_gas4.5.5

H2CO

L1_trace_gas4.5.5

NO2 L2_split4.5.5

NO2






from IOC




Radiance Data Granularity

Standard one hour East/West scan:• 1280 mirror positions• 10 radiance granules per scan• Each granule: 6 minutes of data,

128 mirror positions

Special observations:• For shorter scans, granule size

may vary

Data products:• Level 1, 2 products are per-granule:TEMPO_rad_L1_V01_20200715T180000Z_S001G01.nc

TEMPO_no2_L2_V01_20200715T180000Z_S001G01.nc

• Level 3 products are per-scan:TEMPO_no2_L3_V01_20200715T180000Z_S001.nc







IOC

4.1.10:hk′ fmt_hk

4.6.10:hk

0:ccd′ fmt_ccd

4.6.10:irr

0:drk

0:rad




pkt′ fmt_pkt pkt

L0_ccd 1:irr

L0_ccd 0:drk+

L0_ccd

4.5.21:rad′′′

L1_inr_prep

4.5.21:rad′′


4.5.4

1:rad′

L1_inr_post

4.5.21:rad

scantailoring

GOES tiepoints

dl_goes4.1.2

GOES

dl_iers4.1.2

IERS Bulletin A

dl_snow4.1.2

Snow & Ice

dl_met4.1.2

Meteorology

1:rad

1:irr

L1_cloud4.5.5

cloud

L1_o3_profile4.5.5

O3 profile

L1_o3_total4.5.5

O3 total

L1_trace_gas4.5.5

H2CO

L1_trace_gas4.5.5

NO2 L2_split4.5.5

NO2






from IOC




Level 0 to Level 1:Radiometric calibration


1. Convert CCD frames into radiance, irradiance measurements• Radiometric calibration• Stray light correction• For details, see X. Liu presentation

2. Wavelength calibration• Spectral fitting calibration method has been implemented• Impractical to fit λi for every radiance spectrum

⇒{

calibrate irradiance λi using spectral fittingcalibrate radiance λi using lookup tables: ∆λ (Tbench, . . .)

3. Prepare Level 1 radiance granules for geolocation by including:• scan mirror position vs time• spacecraft gyroscope rates vs. time• spacecraft predicted ephemeris

Level 0 to Level 1:Geolocation (INR)


TEMPO geolocation is based on GOES imagery; uses softwareprovided by Carr Astronautics

• INR sets minimum spatial coverage required per granule

GOES-East & GOES-West imagery downloaded via PDA• Level 2 cloud and moisture imagery, bands 1 & 2• Level 2 cloud top height product• custom ROI to cover TEMPO field of regard• 5-minute update interval (GOES mode-dependent)

Two pass algorithm using a Kalman filter• Radiance granules are processed in time order• Smoothing pass at end of scan⇒ processing lag

INR-defined content in Level 1 radiance file:• longitude, latitude, altitude coordinates; quality flag• GOES-E/W ABI imagery resampled to TEMPO pixels, bands• GOES cloud top height; cloud mask derived from RGB channels

Level 0 to Level 1:Post-INR

With longitude, latitude coordinates:1. Compute:

• viewing, solar angles• mean terrain height (GMTED 2010)

2. Define ground pixel quality flags:• ground cover type, land/water mask (MODIS MCD12Q1)• snow & ice cover (NSIDC SSM/I-SSMIS)• glint, eclipse flags

3. Apply polarization correction• Radiation transfer model lookup table provides predicted

Stokes parameters Q(λ ), U(λ ), as a function of viewinggeometry, surface albedo, ozone column, surface pressure,cloud pressure, and cloud fraction

• For details, see X. Liu presentation







IOC

4.1.10:hk′ fmt_hk

4.6.10:hk

0:ccd′ fmt_ccd

4.6.10:irr

0:drk

0:rad




pkt′ fmt_pkt pkt

L0_ccd 1:irr

L0_ccd 0:drk+

L0_ccd

4.5.21:rad′′′

L1_inr_prep

4.5.21:rad′′


4.5.4

1:rad′

L1_inr_post

4.5.21:rad

scantailoring

GOES tiepoints

dl_goes4.1.2

GOES

dl_iers4.1.2

IERS Bulletin A

dl_snow4.1.2

Snow & Ice

dl_met4.1.2

Meteorology

1:rad

1:irr

L1_cloud4.5.5

cloud

L1_o3_profile4.5.5

O3 profile

L1_o3_total4.5.5

O3 total

L1_trace_gas4.5.5

H2CO

L1_trace_gas4.5.5

NO2 L2_split4.5.5

NO2






from IOC




Level 1 to Level 2


Level 2 products are generated by legacy retrieval codes, updatedfor TEMPO

• Clouds: rotational Raman scattering (J. Joiner et al.)• Trace gas algorithm updated for TEMPO (see G. Gonzalez Abad

presentation)• O3 profile algorithm updated to include 540-740 nm band (see X. Liu

presentation)• TOMS V8 total O3 algorithm

Parallel processing is necessary to meet requirements• Radiance granules are processed in parallel• O3 profile is further parallelized by subdividing each radiance granule• Slurm workload manager used for task scheduling

Per-scan post-processing:• NO2 stratospheric/tropospheric separation (see J. Geddes et al.

presentation)• Level 3 products

Data Distribution

Web mirror @SAO – ftp access to recent data• e.g. within the last 30 days

RSIG interface to SAO mission archive• supports quick visualization of archive products

Frequent transfer of TEMPO science products to ASDC• access to full archive


Status


Installed ∼1/3 scale compute cluster to support testing• Test cluster: 168 cores

3 compute nodes (56 cores/256 GB RAM) + 1 master node• Operations: 565 cores (CDR estimate)

Generated high fidelity synthetic test data• synthetic TEMPO radiances (C. Chan Miller)• matching synthetic GOES imagery; realistic TEMPO host spacecraft

dynamics, gyroscope, and scan mirror time series (Carr Astronautics)

Version 1 completion is scheduled for Aug 2018• complete work on O3 profile code• process synthetic test data on the test cluster

Version 2 is scheduled for completion by Oct 2019• Incorporate instrument characterization: radiometric calibration,

stray light correction

Documents

TEMPO Science Data Processing Center