AMSR-E SIPS Processing Status
Presented by Helen ConoverInformation Technology and Systems Center
at the University of Alabama in Huntsville
AMSR-E Joint Science Team MeetingSept 4-5, 2013
Oxnard, CA
[email protected] www.itsc.uah.edu ghrc.nsstc.nasa.gov 256.961.7807
2
Outline
• Introduction• AMSR-E Reprocessing Status
– L2A with Wentz V7 complete– Standard Browse Imagery Updated– Collecting Data Provenance– Inserting Data Object Identifiers
• Global Imagery Browse Services (GIBS) Overview
• AMSR2 Processing Plans
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
3
AMSR-E SIPS Personnel
• NASA MSFC: H. Michael Goodman, PI• UAHuntsville ITSC (Staff shared with GHRC DAAC):
Management: Dr. Sara Graves, Director, ITSCDevelopment: Helen Conover, Lead
Bruce BeaumontEvans CriswellMichael McEniry
Mary NairEngineering: Kathryn Regner
Sandra Harper Data Management: Lamar Hawkins Sherry HarrisonSystems Administration: Will Ellett, Lead Michele Garrett
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
AMSR-E SIPS Processing and Dependencies
4
Level-2A Brightness Temperatures(with Metadata and QA files)
L2B Ocean(1/2-orbit)
L2B Land(1/2-orbit)
L2B Rain(1/2-orbit)
L3 Ocean(Daily)
L3 Land(Daily)
L3 Snow(Daily)
L3 Sea Ice(Daily, 3 Res)
L3 Snow(Pentad)
L3 Ocean(Weekly)
L3 Ocean(Monthly)
L3 Rain(Monthly)
L3 Snow(Monthly)
Once per day
Once every
five daysOnce per
week
Once permonth
ProductGeneration
at SIPS-GHRC
-------------~ 29 times
per day
Level-2A Brightness Temperatures
L3 Sea IceDrift
~ 29 times per day at SIPS-RSS
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
5
Algorithm DAP Version Summary
Algorithm Short Name Current Version
L2A Tb V12
L2B Land V10
L2B Ocean V07
L2B Rain V11
L3 Land V07
L3 Ocean V05
L3 Rain V07
L3 Sea Ice V14
L3 Sea Ice Drift B03
L3 Snow V10
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
Reprocessing
• Earlier this year, the SIPS began the most comprehensive reprocessing effort to date.
• The ESDT Version 3 data set will feature the following enhancements:– L2A products generated with Wentz
Version 7 algorithm– Inclusion of ISO-compliant lineage
information and digital object identifiers (DOIs) in data files
– Updated browse imagery
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
6
Wentz Version 7 algorithm
• The new brightness temperature products generated at SIPS-RSS – use AMSR-E L1A V03 dataset as source input – use the new RSS Version 7 radiative
transfer model algorithm, intercalibrated with other microwave radiometers, such as SSM/I F13 and WindSat, thus bringing all satellite microwave radiometers to a common calibration standard
– show improved calibration over land, based on heavily vegetated tropical rainforest scenes
– improve Radio Frequency Interference (RFI) Flagging in response to new geostationary sources.
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
7
Provenance Capture
• All AMSR-E ESDT Version 3 products will include new metadata routines that contain or capture dataset DOIs and ISO Lineage.– The digital object identifier (DOI) uniquely
identifies a dataset. Metadata about the data is correlated with the DOI. The DOI for an object is permanent, whereas its location and other metadata may change.
– Storing ISO-compliant lineage information in data files means that processing history stays with the files.
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
8
Enhanced Browse Imagery
• All AMSR-E ESDT Version 3 products will benefit from a new browse image process which includes– new 24-bit PNG images with embedded metadata, such
as source science data file name, parameters, date/time stamp, projection, etc.
– KMZ files suitable for display on map or globe– the traditional “framed” image with title, color bar and
annotations • Full resolution browse imagery, data only, will be
provided as map layers to NASA’s Global Image Browse Service (GIBS)– Selected channels or variables from each product,
suitable for overlay on MODIS background images
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
9
Reprocessing Plan
• L2A reprocessing (addition of metadata and browse image generation for RSS-produced L2A data) completed at SIPS-GHRC in July.
• L2B and L3 datasets will be reprocessed individually throughout 2014-15 as algorithm updates are received from TL-SCF.
• Assuming a final L2A dataset update by SIPS-RSS in late 2015, SIPS-GHRC will perform a full mission reprocessing of all products in 2016 using final L2A as input with latest available science algorithm.
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
10
Global Imagery Browse ServicesOverview
AMSR Science Team MeetingMatt Cechini, GIBS Lead
Helen Conover, AMSR-E SIPS
AMSR2 Processing Planning at the SIPSPresented by Helen Conover
Information Technology and Systems Center at the University of Alabama in Huntsville
AMSR-E Joint Science Team MeetingSept 4-5, 2013
Oxnard, CA
[email protected] www.itsc.uah.edu ghrc.nsstc.nasa.gov 256.961.7807
Introduction
• The AMSR-E SIPS is currently negotiating with NASA HQ to do standard and near real-time product generation– SIPS in Huntsville, AL will generate climate
research quality products from JAXA Level-1R data, using algorithms adapted from AMSR-E processing by the US Science Team. These products will be archived at the NSIDC DAAC.
– SIPS will also generate LANCE (Land Atmosphere Near real-time Capability for EOS) products for users who are interested in monitoring natural and man-made hazards in near real time (no archive).
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
13
Proposed AMSR2 Data Flow
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
14
TLSCFScience SoftwareIntegration and Test,Metadata & QA software
SIPS-Huntsville
JAXA-EOCL1R Product Generation
L2 and L3 Product Generation(science, metadata, qa, ph, subsets, browse)
NSIDC DAAC L2 and L3
Data Archive and
Distribution
L2 & L3
L1R
L0
Product Delivery Record Server (PDRS)
Science AlgorithmsAlgorithm Teams
Ocean Sea Ice
Rain
Land
Snow
Users
Products and Dependencies
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
15
L2B Ocean(1/2-orbit)
L2B Land(1/2-orbit)
L2B Rain(1/2-orbit)
L3 Ocean(Daily)
L3 Land(Daily)
L3 Snow(Daily)
L3 Sea Ice(Daily,3 Res)
L3 Snow(Pentad)
L3 Ocean(Weekly)
L3 Ocean(Monthly)
L3 Rain(Monthly)
L3 Snow(Monthly)
Once per day
Once every
five daysOnce per
week
Once permonth
ProductGeneration
-------------~ 29 times
per day
Level-1R Brightness Temperatures
Ingest from JAXA~ 29 times per day
L3 Sea Ice Drift
AMSR2 Near-Real-Time Processing
• NASA’s LANCE system is eager for AMSR2 products. AMSR-E was second only to MODIS with LANCE users:– “Our flood detection system depends on AMSR-E data,
and is used by governments in many countries (e.g., http://www.gdacs.org/flooddetection/ ).”
Tom De Groeve, Joint Research Centre of the European Commission– “At the Cooperative Institute for Meteorological Satellite
Studies at the University of Wisconsin, we make continuous use of the AMSR-E retrievals of total precipitable water shared in real-time on the LANCE servers... Microwave-frequency retrievals of low-level water vapor are usually the earliest indicator available for tropical cyclone development in remote ocean environments.”
Chris Velden, Cooperative Institute for Meteorological Satellite Studies at the University of Wisconsin
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
16
Discussion
• SIPS plans to use JAXA L1R as input to the US Level-2B and Level-3 algorithms – Already signed GCOM-W1 AMSR2 User Agreement and
registered with JAXA GCOM-W1 data system.– What other operations agreements do we need to
prepare and with whom do we coordinate?– What arrangements do we need to make for routine,
automated data delivery and with whom do we coordinate?
– Can we receive L1R per ½ orbit in near real (similar to L1B delivery to GSFC PPS)?
• LANCE data latency requirement is within 3 hours of observation, but sooner is better…
– What else do we need to consider as we plan to receive Level-1R data directly from JAXA?
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
17
Thank you!
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
18
AMSR2 Primary POCs
• Helen Conover, Technical [email protected]
• Kathryn Regner, Systems [email protected]
• Sherry Harrison, Data Management and User [email protected]
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
19
Backup
TLSCF ScienceSoftware I&T SIPS-GHCC
PO.DAAC
JAXA-EOCL1A Product Generation
SIPS-RSSL2A Product Generation
L2B and L3 Product Generation(science, metadata, qa, ph, subsets, browse)
L2 and L3algorithms
L2A algorithm
NSIDC DAAC Data Archive and Distribution
Level-1
Level-2 and Level-3
Level-1
Level-1
NASAEDOS
GBAD (Pre Ephem) & PDS
GroundStations
L2B and L3 Algorithms
Level-0
Product Delivery Record Server (PDRS)
Algorithm Teams
Ocean Sea Ice
Rain
Land
Snow Data
NASA Flt Dynamics
GBAD(Def Ephem)
AMSR-E Data Flow
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
22
SIPS-GHCC HardwareOperational Configuration (Mar
‘13)
Multiple processing environments across three machines results in much greater flexibility for processing and
reprocessing.
Priv
ate
Netw
ork
(1 G
B/se
c)
Forward Processing
Reprocessing
Dev, Integration & Test
Sun Fire x4270 x642 x 2.93 GHz Sun Intel Xeon Processors
4 x 2GB DIMM , RHEL 5AMSR 1-3
AMSR-E Joint Science Team Meeting Oxnard, CA Sept. 4-5, 2013
Near line storage for all Level-2A, Level-2B and Level-3 AMSR-E products
Sun Storage 7310With J4400 Disk
Array10TB NAS
0123
4567
89
10USB
13
1112
1415
REARPS
TOPFAN
Storage 7110
micros ystems
SysSN: 0829Q
ASO
OC
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
OK
Sun Storage 712022TB NAS