Upload
others
View
9
Download
5
Embed Size (px)
Citation preview
European Space Agency (ESA) Landsat MSS/TM/ETM+/OLI Archive: 42 years of our history
Sébastien Saunier, Amy Northrop, Samantha Lavender, Luca Galli, Riccardo Ferrara, Stefano Mica, Roberto Biasutti, Philippe Goryl, Ferran Gascon, Marco Meloni, Bas Altena
MultiTemp 2017, Bruges, June 2017
Landsat MSS Dataset Improvements and Multi Temporal Analysis
This image cannot currently be displayed.
Contents
2
• From 1975, ESA has used its multi-mission ground systems to acquire, process and archive data from the Third Party Missions (TPM).
• Landsat data continuity missions are part of ESA’s TPM.
3
4
1972
1973
197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000200120022003200420052006200720082009201020112012
Landsat 1 - MSSLandsat 2 - MSSLandsat 3 - MSSLandsat 4 - MSSLandsat 5 - MSSLandsat 5 - TMLandsat 6 - ETMLandsat 7 - ETM+
Landat 8 - OLI/TIRS
ESA Operation in operational lifeOperational life
5
Landsat Sensor
RBV (LS-1-3)
MSS (LS-1-5)
TM (LS-4-5)
ETM+ (LS-7)
Spectral Bands (µm) 1) 0.48-0.57 VNIR
2) 0.58-0.68 VNIR 3) 0.70-0.83 VNIR P) 0.505-0.75 VNIR (available on LS-3)
4) 0.50-0.60 VNIR 5) 0.60-0.70 VNIR 6) 0.70-0.80 VNIR 7) 0.80-1.1 VNIR 8) 10.4-12.5 TIR (available on LS-3)
1) 0.45-0.52 VNIR 2) 0.52-0.60 VNIR 3) 0.63-0.69 VNIR 4) 0.76-0.90 VNIR 5) 1.55-1.75 VNIR 6) 10.4-12.5 TIR 7) 2.09-2.35 SWIR
1) 0.45-0.52 VNIR 2) 0.52-0.60 VNIR 3) 0.63-0.69 VNIR 4) 0.76-0.90 VNIR 5) 1.55-1.75 VNIR 6) 10.4-12.5 TIR 7) 2.09-2.35 SWIR P) 0.50-0.90 VNIR
Swath Width 185 km 185 km 185 km 185 km
Spatial resolution (Across Track / Along Track)
40 m (PAN) 80 m (VNIR)
59x79 m VNIR 240 m TIR
30 m VNIR/SWIR 120 m TIR
15 m PAN 30 m VNIR/SWIR 60 m TIR
Radiometric resolution Analogic 6 bit 8 bit 9 bit ( 8 bit transmitted)
Band to band registration 0.2 pixel (90 %) 0.2 pixel (90 %) Geodectic Accuracy (without Ground Control Point)
500 m (90 %) 400 m (90 %)
Data Rate 15 Mbits/s 85 Mbits/s 2*75 Mbits/s Average Power 50 W 332 W 590 W
Telescope Aperture 23 cm 40.6 cm 40.6 cm
Context
• The ESA collection of Landsat Historical data observed in Continental Europe, African Region and Arctic has become massive.
• Major efforts have been undertaken, to improve processing s/w and then produce the most up to date Level 1 products for the community.
• (As part of the International Ground Station (IGS), ESA is continuously maintaining efforts to acquire, process and distribute new real time Landsat 8 data
6
7
This archive was a black box
8
Status prior improvements / Solutions
9
Item Status Solution Reference
Format CEOS Format -Aligned to USGS: GeoTIFF and ancillary files within a zip archive
LS-DFCB
GEO Accuracy
Orbit and attitude data are mostly inaccurate preventing operational use of data. Image Geometry is distorted and “Panoramic Effect” observed
-Re evaluate / Re-estimate Orbit data -Improve GCP set, DEM, GCP Filtering -Improve Geometric Model (attitude model) -Use Multi Scene Refinement concept
ACS ATBD
Status prior improvements / Solutions 2/2
10
Item Status Solution Reference
RADIO Accuracy
Large variations in pre flight calibration Image Saturation
Ground Model Lifetime Calibration Update of calibration
Helder , Saunier
Image Quality Image Anomalies Detect and Flag with BQA ESA s LANDSAT Product Definition
Product generation
A lot of products has never been generated
Systematic generation and quality control
TPM website https://tpm-ds.eo.esa.int/collections/
Helder “Absolute Radiometric Calibration of Landsat Using a Pseudo Invariant Calibration Site” , IEEE Transactions on Geoscience and Remote Sensing Volume 51, Issue 3 (March 2013).
Saunier, “Landsat MSS, radiometric calibration processing improvement” in Proceeding of ESA Living planet Symposium 2016.
Difference with USGS products
• Geometry : Instead of systematically applying scene based ortho-rectification – it is in some cases more appropriate to calibrate the geo-referencing model by using data from the whole satellite pass. The distortions for the whole pass are modelled using a weighted, constrained, least squares fit with iterative outlier removal. Also, an accurate ortho rectification is applied for scenes partially occluded with cloud, and for scenes with large portion of water.
• Radiometric calibration: The analysis of Level 0 data, has revealed that MSS calibration was not optimal, in a way that for bright area target RED / NIR1 bands included a lot of saturation. For these regions, the calibration coefficients have been revised and most data are now almost free from saturations.
• BQA: Quality information attached to Level 1 MSS pixel has been developed for a better filtering of input pixels.
• Image Reparation: Pixels flag as corrupted for a certain class of anomaly (sticky bit) are repaired – Development / Validation is ‘on going’
11
Geometric Processing (1/2)
12
• Objective is simple – maximize number of Terrain Corrected products (L1T).
Geometric Processing (2/2)
Quicklooks for same pass v3.04 (top) and v3.06 (bottom)
– Note the scenes with large distortions for v3.04
13
Geometric Processing Accuracy
• Methodology has been validate for TM and Improved for MSS.
14
15 Calibration Stability of MSS data after alignment to TM Data (Helder, 2013)
Radiometric Processing 1/2
• Intensive work of the SDU / usgs to harmonized MSS calibration. Calibration methodology implemented at ESA.
16 Selected Landsat WRS scenes for calibration updates
Radiometric Calibration 2/2
• Objective is simple – maximize number of Saturation Free products.
17
18
Bit Value USGS Landsat Collection 1 Description
ESA MSS Description
0 1 Designated Fill Designated Fill
1 2 Dropped Pixel Dropped Pixel
2 4 Radiometric Saturation
Radiometric Saturation
3 8
4 16 Cloud Cloud
5 32 Cloud Confidence Cloud Confidence
6 64
7 128 Cloud Shadow Confidence
Cloud Shadow Confidence
8 256
9 512 Snow / Ice Confidence
Snow / Ice Confidence
10 1024
11 2048 Scan Line Anomaly (SLA)
striping / saturation /
hotspot / sticky bit
12 4096
13 8192
14 16584
15 32786 Land / Water mask
BQA – Pixel based information
19
– Scan Line Artefacts (SLA) – hot spot, sticky bit
– Dropped lines
– Saturation
– Cloud cover
– Land / Water mask (ESA CCI product)
Insight on the archive
• Analyze temporal coverage of MSS data for specific countries and considering winter and summer period.
• EU countries covered since the beginning and almost uniformly.
• Improvements of the coverage of Eastern Countries (Russian, Kazakstan), Northern Countries (Greenland) and Southern Countries (Western part of Africa ) and Islands.
22
23
24
“Proof of concept”
• Outline : Several multi temporal applications for which the use of MSS is valuable have been selected in order to demonstrate that data are fit for application purposes.
• Thematic Area : Phenology Monitoring, Snow / Ice Monitoring, and urban mapping
25
Phenology : Monitoring of Forest (Case 1) / Grass land (Case 2) • Approach : Collect a large MSS, TM, LS08 dataset
over forest (Case 1), grass land (Case 2). Apply preprocessing, compute NDVI, perform analysis.
• Case 1 Study site: – Bouconne, Buzet Forest
around Toulouse (France)
• Case 2 Study site: – The northern coast of East Anglia,
(Eastern North Sea coastline of the United Kingdom)
26
Phenology Case 1– Results 1/2
27
Phenology Case 1– Results 2/2
• Agreements between measurements from two different dataset, involving different measurements.
• Good shape of the average NDVI with more uncertainties at growing season
• Outliers removed because of – In some cases, very small NDI values on MSS1 / MSS2
– 1985/1987 Long and in late freezed period
– One LS08 NDVI (Summer 2016) very high compared to previous ones (0.76)
28
Phenology Case 2 – Results
• Strong Discrepancies between TM and MSS
• MSS NDVI appears lower than TM and its temporal variability is important.
• Variability of MSS calibration is an issue, strong changes are mostly due to weather conditions and year to year seasonal shift.
29
• Landsat 5 data , MSS (1990-1993)/ TM (1985-2010)
Snow/Ice : Equilibrium Line
• Approach : Collect a MSS, Sentinel 2 data acquired (if possible) in Autumn, Segment Equilibrium line, and assess if MSS data radiometry allows to perform this task.
30
Snow/Ice : Equilibrium Line
31
Snow/Ice : Equilibrium Line
• Results : – MSS2/3/4/5 data allows to search this line at large scale
– Problem is more the period of data, two observation per year and sometimes cloudy.
– Dataset, MSS 5, can be densified with TM data.
– Landsat and S2 Results are in agreement.
32
Urban Mapping
• Approach : Collect a large MSS, TM, LS08 dataset over Toulouse City, more than 100 products, create a movie and assess visually urban dynamic – Evaluate if details seen with MSS are sufficiently well defined to characterized changes.
• Results : – No geometric correction applied, except for one product
– The registration of images are with 1.5 pixels if one consider all the series.
– See next slides.
33
Conclusions
34
• The ESA MSS data reprocessing project has been presented. The work brings new knowledge regarding our data stored at ESA.
• We hope it will benefit to a larger user audience.
• The same approach has been successfully applied for TM / ETM+ data. However, MSS required very careful / specific treatment in term of product development, product processing and quality control.
• The reprocessing of MSS data will take place by the end of Summer.
• LS08 data Near Real time service proposed by ESA
• IDEAS+ QC activities performed on a monthly basis: – Absolute Geolocation Accuracy Assessment;
– Multi temporal Geolocation Accuracy Assessment ;
• Short term geolocation stability – data from the same path, observed on the same date are analysed
• Cross track variability – different regions in the image located at several locations in the field view are compared
– Band to band Geometric Registration Accuracy Assessment;
– Radiometric Calibration stability over PICS;
• Statistical comparison between results of the two different ROIs is systematically performed
• Influence of terrain content is better assessed.
– BQA band inspection, Image Artefact Analysis and Thermal band inspection.
• Monthly Performance Report is issued every month and available @ https://earth.esa.int/web/sppa/mission-performance/esa-3rd-party-missions/landsat-8/oli-tirs/cyclic-quality-reports
35
LS08 Processing and QC at ESA
Team Introductions
Processor development
Instrument Data quality Evaluation and Analysis Service eXpert Product Reprocessing Scalable Service
Tool verification
L1 QC
Expert support
Project management
Systematic QC tool development (AMALFI)
Project management
Data collection (L0 & AUX)
Bulk processing
Repatriation (consolidated L0 & processed L1)
Data consolidation
L0 QC
Processor integration
Data Access
• ESA’s TPM online dissemination service: https://tpm-ds.eo.esa.int/collections/
• ESA Landsat 8 Web Portal: https://landsat8portal.eo.esa.int/portal/
• ESA Landsat 8 RealTime Viewer http://landsat8-realtime.eo.esa.int/observer
37
Thank you for your attention
This image cannot currently be displayed.
• Estimate LS05 MSS Calibration parameters over Libya 4 PICS applying cross calibration methodology;
• Validation on a small dataset observed over Libya 4 and in Iceland;
• Validation on a large dataset observed in all available PICS test sites;
• Define most affected WRS scenes in the archive broken down into Desert / Ice-snow regions;
• For these scenes apply new calibration parameters depending for L1 product generation depending on the season (Summer / Winter)
39
MSS v3.05 IPF – Enhancements Calibration Coefficients
40
Oversampled LANDSAT 8
Surface Reflectance
Measurements
ECMWF Water vapor
Ozone
MSS Spectral Band Definition
MSS Top of
Atmosphere Measurement
SIMULATED Top of
Atmosphere Measurement
Radiative Transfer
Code (6s)
MSS Observation
date and geometrie
Data Collected for all ROIs
Statistical Comparison
Calibration Parameters
41 Author: SEBASTIEN
MSS v3.05 IPF – Enhancements Calibration Coefficients
Libya 4 dataset, RED band
Libya 4 dataset, NIR1 band
42
CPF patch with changes on Min / Max Radiance
Latest CPF patch with changes on Min / Max Radiance and
Radiometric Calibration Gain
MSS v3.05 IPF – Enhancements Calibration Coefficients
Timeline of Events
43
2014 2015 2016 2017
IPF, QC Tool & CPF
verification
v3.05 specification definition &
design
MSS stress test (v3.03)
v3.04 development &
verification
Processing
QC of full MSS archive
MSS L1 ACTIVITIES
TM/ETM+ L1 ACTIVITIES
MTI Delta
QUALITY CONTROL
TM BSK
CPF definition & preliminary verification
QC tool update
ETM+ KSE & MPS
TM MPS TM MTI
ETM+ MTI
Mid-orbit gap analysis
Regeneration of TM GCP AUX
IDEAS+ activity X-PReSS activity
Processing Processing
AUX SV generation
44
Iceland Results (snow – ice) Percentage of saturated pixels
for 3 L1 scenes: Green and Red bands Little reduction of saturation
Percentage of saturated pixels for 3 L1 scenes: NIR1 and NIR2 bands Significant improvement for NIR1
MSS v3.05 IPF – Enhancements Calibration Coefficients
MSS v3.05 IPF – Enhancements Calibration Coefficients • LS05 MSS CPF files have been updated (applicable for a
restricted number of WRS scenes): – LM5CPF19840406_19841108 – LM5CPF19841109_19940428
• Revised Band 2 and Band 3 scaling parameters improve product quality
• Minimum Radiance and Maximum Radiance have been changed for the RED and NIR1 bands
• The NIR1 calibration gain of the first CPF has been changed • Validation on a extended PICS dataset is now on going • Regarding methodology, improvements have been tested by
considering varying AOTs value depending on the location in the scene, it increases the accuracy of the results
45
BQA
• Additional logging – To ensure full traceability between L0 and L1 (log files not available to users)
• Full resolution jpeg QuickLook – In addition to existing medium res. png
• Quality Assurance Band (BQA) – 16 bit GeoTIFF file for all MSS L1 products – Sticky bit correction (TBC) – Updated L1 product quality classification
• Updated calibration coefficient – To reduce L1-introduced saturation
• Geometry – To provide more details on GCPs – To give greater confidence on accuracy
46
Bit Value USGS Landsat Collection 1 Description
ESA MSS Description
0 1 Designated Fill Designated Fill
1 2 Dropped Pixel Dropped Pixel
2 4 Radiometric Saturation
Radiometric Saturation
3 8
4 16 Cloud Cloud
5 32 Cloud Confidence Cloud Confidence
6 64
7 128 Cloud Shadow Confidence
Cloud Shadow Confidence
8 256
9 512 Snow / Ice Confidence
Snow / Ice Confidence
10 1024
11 2048 Scan Line Anomaly (SLA)
striping / saturation /
hotspot / sticky bit
12 4096
13 8192
14 16584
15 32786 Land / Water mask
MSS v3.05 IPF – Enhancements Geometry • The nominal geometrical correction procedure of the
ESA Landsat processor is to: – Automatically locate GCPs from a GCP database within a
scene by normalized absolute image correlation in orthorectified map geometry.
– Model the distortion with an affine transform using a weighted, constrained, least squares fit with iterative outlier removal.
– Apply the correction using DEM to produce L1T product
• We call this single scene refinement
47
MSS v3.05 IPF – Enhancements Geometry • The overall accuracy of the single scene fit is
estimated considering: – The number of found GCPs
– Their correlation SNR
– Their distribution over the scene
– The chi squared goodness of fit
– The number of outliers
48
MSS v3.05 IPF – Enhancements Geometry • If the estimated accuracy obtained by single scene fit
is not considered sufficient to produce an optimal quality L1T, several back up possibilities are available: – Label the L1T as suboptimal
– Use multiscene refinement to produce L1T
– Produce a L1G+ correcting geometry without DEM (not used for ETM)
– Produce a L1Gt using DEM and nominal geometry (used only for ETM)
– Produce a L1G with no geometric correction or DEM
49
MSS v3.05 IPF – Enhancements Geometry • In multiscene refinement:
– The affine transform distortions estimated by single scene fitting are used to estimate 2 distortions at 2 points along track per scene in a common SOM projection geometry.
– The distortions for the whole pass are modelled using a weighted, constrained, least squares fit with iterative outlier removal.
– A generic fourier series is now used to model the variation of distortions along track.
50
MSS v3.05 IPF – Enhancements Geometry • The main enhancements w.r.t. v3.04 (delta MSS processor)
are: – Stronger distortion constraints are used for single scene fitting
(this can avoid the production of very distorted scenes and help outlier removal in difficult situations)
– Use general Fourier series instead of b-splines for multiscene fitting (this works much better in particular when distortions have to be extrapolated for several scenes at start and end of pass)
– Add accuracy estimates which also consider distribution of used GCPs or multiscene refinement in MTL annotations, for L1G+ products also.
– Add other statistics to MTL annotations regarding GCP distribution – Use another orbit propagator (works better for longer
propagations)
51
MSS v3.05 IPF – Enhancements Geometry
Example of multiscene fit with estimated errors (L0 pass with several scenes covered by clouds)
– Note that for MSS the attitude information are missing
52
MSS v3.05 IPF – Enhancements Geometry • The multiscene refinement accuracy for each scene is
estimated considering: – The number of used single scene distortion estimates
– Their input single scene estimated accuracies
– Their distribution over the scene
– The chi squared goodness of fit
– The number of outliers
– The position of the scene within the pass
53
MSS v3.05 IPF – Enhancements Geometry • New MTL fields per product level
54
Level L1G/L1GT L1G+/L1T L1G+/L1T
Fit type None Multi Single MODEL_FIT_TYPE x x x GEOMETRIC_MAX_ERR x x GROUND_CONTROL_POINT_DISCARDED x x NUMBER_EMPTY_GROUND_CONTROL_POINT_WINDOWS x GROUND_CONTROL_POINT_WINDOW<n> x GROUND_CONTROL_POINT_RESIDUALS_SKEW_X x GROUND_CONTROL_POINT_RESIDUALS_SKEW_Y x GROUND_CONTROL_POINT_RESIDUALS_KURTOSIS_X x GROUND_CONTROL_POINT_RESIDUALS_KURTOSIS_Y x