Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Validation Report | Research Applications
PUBLIC Page 1 of 41
Coastal Waters Research Synergy Framework
Validation Report
Research Applications
Document Code: CORESYF-ACR-VVR-STR03-E-R
Date of delivery: SAR-2 meeting (T0+33) Deliverable identifier: D4.08 Version of document: 2.0 – last updated 5/11/2018 Dissemination level for document: PU
Table of Signatures
Name Function Signature
Prepared by
Romain Serra Anne Vallette
Nadim Dayoub Helen Snaith
Chiara Pratola Rory Scarrott
Walther Camaro Eimear Tuohy
Francisco Sancho Alberto Azevedo Florent Birrien
Ana Moura Nuno Grosso
ACRI-HE ACRI-HE
NOC NOC UCC UCC UCC UCC LNEC LNEC LNEC
IH Deimos
Reviewed by
Romain Serra
Anne Vallette
ACRI-HE, Application
Developer
Approved by Miguel Terra-Homem Executive Board Chair
Signatures and approvals appear on original
Validation Report | Research Applications
PUBLIC Page 2 of 41
Project start date: 01/01/2016
Project duration: 36 months
Revision Records
Version Date Changes Authors
1.0 14/11/2017 First issue of document
Romain Serra Nadim Dayoub Paolo Cipollini Rory Scarrott Eirini Politi Eimear Tuohy Francisco Sancho Alberto Azevedo Ana Moura Nuno Grosso
1.1 29/11/2017 Minor additions to optical bathymetry acceptance tests results, §3.2 (comments added in steps 15 and 16).
Romain Serra
2.0 5/11/2018 Second issue of document Romain Serra Anne Vallette
Validation Report | Research Applications
PUBLIC Page 3 of 41
Table of Contents
1 Introduction ............................................................................................................................. 5
1.1 Purpose and Scope .......................................................................................................... 5
1.2 Document Structure ........................................................................................................ 6
2 ACCEPTANCE TESTS RESULTS .................................................................................................. 6
3 Acceptance Test Procedures ................................................................................................... 7
3.1 Bathymetry determination from SAR imagery ................................................................ 8
3.2 Bathymetry, benthic classification and water quality determination from optical
imagery ...................................................................................................................................... 14
3.3 Vessel detection & Oil Spill Detection with SAR & Optical ............................................ 20
3.4 Time series processing for hyper temporal optical data analysis ................................. 27
3.5 Ocean and Coastal altimetry ......................................................................................... 32
4 References ............................................................................................................................. 40
List of Tables
Table 2-1: Test case to Test procedure traceability ........................................................................ 6
List of Figures
Aucune entrée de table d'illustration n'a été trouvée.
Validation Report | Research Applications
PUBLIC Page 4 of 41
Acronyms and Abbreviations
AIS Automatic Identification System
AOI Area Of Interest
BBP Backscattering coefficient of particulates
BOA Bottom of Atmosphere
CDOM Coloured Dissolved Organic Matters
Co-ReSyF Coastal Waters Research Synergy Framework
DEM Digital Elevation Model
ECMWF European Centre for Medium-Range Weather Forecasts
EMSA European Maritime Safety Agency
EO Earth Observation
FFT Fast Fourier Transform
GUI Graphical User Interface
IH Instituto Hidrografico, Portugal
LNEC Laboratório Nacional de Engenharia Civil, Portugal
NE North East
NOC National Oceanography Centre, UK
NW North West
QGIS Geographic Information Systems Software
RGB Red Green Blue composite (i.e. in true colours)
RMSE Root Mean Square Error
SAR Synthetic aperture radar
SAR-1, SAR-2 System Acceptance Reviews
SE South East
SGDR Sensor Geophysical Data Records
SNAP SentiNel Application Platform
SSH Sea Surface Height
SSHA Sea Surface Height Anomaly
SST Sea Surface Temperature
SW South West
SWH Significant Wave Height
TS Time Series
UCC University College Cork, Ireland
UT Untested
VTP Validation Test Procedures
VTS Validation Test Specifications
Validation Report | Research Applications
PUBLIC Page 5 of 41
1 Introduction The Co-ReSyF project will implement a dedicated data access and processing infrastructure,
with automated tools, methods and standards to support research applications using Earth
Observation (EO) data for monitoring of Coastal Waters, levering on the components deployed
SenSyF (www.sensyf.eu). The main objective is to facilitate the access to Earth Observation data
and pre-processing tools to the research community, towards the future provision of future
Coastal Waters services based on EO data.
Through Co-ReSyF‘s collaborative front end, even inexperienced researchers in EO will be able
to upload their applications to the system to compose and configure processing chains for easy
deployment on the cloud infrastructure. They will be able to accelerate the development of
high-performing applications taking full advantage of the scalability of resources available in the
cloud framework. The system’s facilities and tools, optimized for distributed processing, include
EO data access catalogues, discovery and retrieval tools, as well as a number of pre-processing
tools and toolboxes for manipulating EO data. Advanced users will also be able to go further and
take full control of the processing chains and algorithms by having access to the cloud back-end,
and to further optimize their applications for fast deployment for big data access and
processing.
The Co-ReSyF capabilities will be supported and initially demonstrated by a series of early
adopters who will develop new research applications on the coastal domain, guide the
definition of requirements and serve as system beta testers. A competitive call will be issued
within the project to further demonstrate and promote the usage of the Co-ReSyF release.
These pioneering researchers in will be given access not only to the platform itself, but also to
extensive training material on the system and on Coastal Waters research themes, as well as to
the project's events, including the Summer School and Final Workshop.
1.1 Purpose and Scope The main aim of this document is to check the correct implementation of the first version of the
Research Applications, demonstrating that the requirements have been satisfied.
Within the Co-ReSyF project, five research applications are being developed and deployed for
operation onto the Co-ReSyF platform:
i) Bathymetry using SAR data,
ii) Bathymetry using optical imagery,
iii) Oil Spill and Vessel detection using SAR and optical data,
iv) Time series processing for hyper temporal optical data analysis, and
v) Coastal Altimetry.
This document focuses solely on the V1 requirements of the Co-ReSyF Research Applications,
and presents the results of the acceptance tests, based on the plan defined in the frame of the
Co-ReSyF Validation Plan for Research Applications (D3.04).
Validation Report | Research Applications
PUBLIC Page 6 of 41
1.2 Document Structure The structure of the document is as follows:
Chapter 2: describes the Validation and Acceptance approach.
Chapter 3: describes the Results of the Acceptance Tests executed.
Chapter 4: details the Reference Documents.
2 ACCEPTANCE TESTS RESULTS The plan for the acceptance of the software version is fully described in (Co-ReSyF, 2016c). The
following section presents the test procedures executed during the acceptance tests in order to
execute the test cases for V1 as defined in (Co-ReSyF, 2016c). The following table traces the
test procedures for each of the test cases.
Table 2-1: Test case to Test procedure traceability
VTS No. VTP Id. APPLI1-VTS-001 to
APPLI1-VTS-007 APPLI1-VTP-001
APPLI2-VTS-001 to APPLI2-VTS-006
APPLI2-VTP-001
APPLI3-VTS-001 to APPLI3-VTS-011
APPLI3-VTP-001
APPLI4-VTS-001 to APPLI4-VTS-007
APPLI4-VTP-001
APPLI5-VTS-001 to APPLI5-VTS-008
APPLI5-VTP-001
Validation Report | Research Applications
PUBLIC Page 7 of 41
3 Acceptance Test Procedures Each acceptance test procedure covers one or more test cases and their respective requirements, and has a unique ID in the form of APPLI#-VTP-xxx (where xxx is a sequential number). The possible results for a test step are mainly PASS or FAIL. However, due to the various degrees of maturity of the research applications, some could not be integrated in the Co-ReSyF platform, at the time of release of this deliverable. In this case, the test was performed offline, the test step result was noted “UT” for “untested online in the Co-ReSyF platform” and offline results are presented in the procedure comments as much as possible. These will be further detailed in the final version of this document (V2). In the case that the step is verifying a requirement, the Pass/Fail criteria will need to be specified (this is mandatory) and will be used to assess the result of the step. On cases where no P/F criteria is specified the result of the step is assessed by the ability to execute the step.
Validation Report | Research Applications
PUBLIC Page 8 of 41
3.1 Bathymetry determination from SAR imagery
TEST PROCEDURE
VTP Id.: APPLI1-VTP-001 Id of Associated VTS:
APPLI1-VTS-01
APPLI1-VTS-02
APPLI1-VTS-03
APPLI1-VTS-04
APPLI1-VTS-05
APPLI1-VTS-06
APPLI1-VTS-07
Functionality to be Tested: This procedure tests the implementation and working functionality of the SAR-Bathymetry application
Required Test Environment: Desktop with a command line interface
Overview of the test procedure: The application was not still integrated in the platform, although specific module tests have been performed offline in a desktop environment using a command line interface. Thus, the following reports refer when applicable to the platform tests or to the offline environment tests.
Detailed description of the test procedure, including how to observe and verify the results:
Step Nb.
Description Result VTS Verified P/F Criteria Comments
1
Test the application as a whole. The aim is to check if everything is running without error and a bathymetry map is delivered.
FAIL APPLI1-VTS-01
The outputs of all processing steps shall be georeferenced maps, loadable in QGIS or SNAP software (GeoTIFF format)
The application has been integrated into the platform, but it fails running for a region with more than approximately 25 grid-points. Also, the GeoTIFF output format was not implemented..
Validation Report | Research Applications
PUBLIC Page 9 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
2
Check of the external wind
velocity at the centre point of
each SAR image. PASS APPLI1-VTS-02
Wind velocity should be comprised between 3 and 10 m/s for an image to be a good candidate for the application of the proposed methodology.
The module is running and gathers data from the ECMWF or the NCEP re-analysis databases. The module calculates the wind at the position closest to that at the centre of each image, for the KOMPSAT-5 images. The procedure remains to be tested for SENTINEL-1 images and other possible SAR image sources (e.g. TERRASAR-X). However, this module is not integrated in the running platform version.
3 Check external wave data for
each SAR image date PASS APPLI1-VTS-03
Existence of a well-defined ocean swell with one predominant direction and wavelengths adequate for the application of the method.
For the test region (the West coast of Portugal), a procedure was defined to extract data from an online available numerical model forecast (which can be used as hindcast) at http://ariel.lnec.pt/. Also, wave-related in situ past data have been collected from the Hidrográfico database, from the Leixões buoy, and made available for this project for the period 2002-2016. An algorithm was developed by Hidrografico in order to attribute to each wave condition a pass/fail criterion, based on the measured wave period and spectral width, which indicated the likelihood of the presence of a visible swell for a candidate image. However, this module is not integrated in the running platform version.
Validation Report | Research Applications
PUBLIC Page 10 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
4 Check of image selection
protocol FAIL APPLI1-VTS-04
1. The SAR image is visualized to verify the existence of wind shadow areas, in the bathymetry AOI. 2. The presence of swell is verified visually. 3. The presence of strong wave diffraction or wave refraction zones near the shore should be verified. 4. Other features are checked, such as the presence of different signatures like oil spills, vessels, internal waves, river plumes or other coastal processes that could affect the algorithm application.
This functionality was not implemented.
Validation Report | Research Applications
PUBLIC Page 11 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
5
Radiometric, speckle, sigma0 and crop and creation of the Bathymetry AOI with– visual check
PASS APPLI1-VTS-05
1. The polygon of the AOI
must be closed and
properly defined.
2. The grid size must be
within a default range,
depending on image
resolution.
3. The image within the AOI shall show clear wave crest patterns
For a given application, a North-South oriented rectangular polygon needs to be created as an area of interest (AOI). If this rectangular AOI includes land points, these are masked and excluded from the computation. It is expected that this algorithm evolves and allows for another closed-polygon shape that may adjust better to any given coastal configuration. The procedure also includes the SAR image calibration georeferencing and filtering. This step is incomplete, as remains the need to perform the flip or ellipsoid-correction. The grid size is presently defined by the user, within a default range given by the algorithm, which depends on the image resolution. This has been tested for SENTINEL-1 images. The version of the currently available processing modules does not mask the land pixels or perform the necessary ellipsoid correction and therefore the test of this module has failed. The other two functions (radiometric correction and speckle filtering) are already implemented and are working correctly.
Validation Report | Research Applications
PUBLIC Page 12 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
6
Determination of the FFT
boxes for each point of the
final grid PASS APPLI1-VTS-06
1. The image is cropped in
N FFT boxes which are
dependent of the
resolution of the grid.
2. Check if the wavelength
is comprised between a
pre-defined interval.
3. Check if the direction of the swell is comprised between an expected interval, from the visual inspection (APPLI1 -VTS-04).
The computation grid definition at the AOI is computed automatically, once the AOI and grid size are defined. For each grid point, a squared (box) sub-scene or image tilling with a user-defined size is cropped from the main image, centred at the grid point location. This is referred to as the “FFT boxes definition”. This box is further shifted (or linearly translated) a pre-defined spatial-lag length, along all the 4 main cardinal directions, and the four intermediate directions (NW, NE, SE, SW). This produces nine FFT boxes around each grid point. It follows the computation of the wave spectrum, through an image Fast Fourrier Transformation (FFT), yielding the peak wave length and wave direction at all the 9 FFT boxes, for each grid point. Several tests have been performed and several algorithms were developed to determine the peak wavelength and wave direction, using 1D and 2D filtering techniques and spectral estimates. The average of the valid results is then used as the grid-point wave length to compute the local bathymetry. The deep-water wave lengths are determined in order to retrieve the local wave peak period for each image AOI. Several methods to determine the wave length, based on 2D, and multiple 1D spectral estimates have been tested.
Validation Report | Research Applications
PUBLIC Page 13 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
7 Test of the depth inversion
algorithm PASS APPLI1-VTS-07
1. Creation of the
bathymetry map. The
estimated depths must be
coherent, particularly at
adjacent grid points.
2. Validation of the final
outputs with in situ data.
3. The outputs of all processing steps shall be georeferenced maps.
For a given image, areas (grid points) where the APPLI1 -VTS-06 protocol fails are flagged out from the bathymetry computation. These areas appear in maps as without a valid bathymetry, and are marked visually by a different colour. Validation of the final bathymetry outputs with field data has not been done yet, due to the lack of proper validation data.
8
Comparison of output with
EMODNET and local
soundings. PASS
Comparison of
output with
EMODNET and
local soundings.
The comparison of outputs with the EMODNET chart for the AOI is implemented. However, this functionality was not implemented in the running platform version.
Date of execution: 5/11/2018
Validation Report | Research Applications
PUBLIC Page 14 of 41
3.2 Bathymetry, benthic classification and water quality determination from optical imagery
TEST PROCEDURE
VTP Id.: APPLI2-VTP-001 Id of Associated VTS: APPLI2-VTS-01
APPLI2-VTS-02
APPLI2-VTS-03
APPLI2-VTS-04
APPLI2-VTS-05
APPLI2-VTS-06
Functionality to be Tested: The good performance of the Optical Bathymetry application.
Required Test Environment: Co-ReSyF online platform available, with a valid user. No special input needed, Co-ReSyF catalogue is sufficient.
Overview of the test procedure: Test if the whole processing chain delivers a bathymetry map as expected. If this test fails, there must be an error so looking into other tests is important in order to find the failure.
Detailed description of the test procedure, including how to observe and verify the results:
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 15 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
1
Test the application as a whole. The aim is to check if everything is running without error and a bathymetry map is delivered
PASS APPLI2-VTS-01
The outputs of all processing steps shall be georeferenced maps (GeoTIFF format)
The application is running as expected, without any error. A GeoTIFF showing the resulting bathymetry is delivered. As GeoTIFF is not fitted for presentation display, a JPEG is also produced, showing the assessed bathymetry.
This test indicates a good general behaviour of the application as it provides the expected result as an output.
Validation Report | Research Applications
PUBLIC Page 16 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
2 Test if images are corrected from Rayleigh scattering and
gaseous absorption. PASS APPLI2-VTS-02
1. Source images contain land and water pixels
2. Atmospheric correction provides bottom of atmosphere data in the Area Of Interest
3. The lowest quality pixels are flagged (clouds, shadows)
The atmospheric correction algorithm developed by ACRI searches for water and land pixel to find a dark target and correct the atmospheric effects. Scenes not providing this target (because of absence of water, presence of cloud…) are not treated.
The others are treated and corrected. As an output of the application, intermediate images are delivered in order to evaluate the processing over the scene. For most modules, a before/after study can be done so the user can understand how the correction works and why it is necessary.
For example, images of a Before/After atmospheric correction are delivered such as the ones below (Sentinel2, over Venice area, Italy) and can be used to visually assess the quality of the correction or the presence of artefacts or of sun glint over water (like visible on the eastern part of these images).
Validation Report | Research Applications
PUBLIC Page 17 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
3
Test if external bathymetry data is available and
associated with each satellite image considered
NA APPLI2-VTS-03
The Co-ReSyF application allows access to bathymetry data available on the area of interest, and when possible at a date close to the one of the considered satellite image used.
Pairing In-situ bathymetry data with the area of Interest was useful for the V1 as the bathymetric model required in-situ data for calibration. For V2, the bathymetric model is autonomous and is only based on water colour. So even if this module was already integrated (test was PASSED for V1), it has finally been decided to remove this module as it is finally useless.
4 Water Masking PASS APPLI2-VTS-04 1. A Water mask covers all the water areas.
The water masking is running well and produce a GeoTIFF with masking non water pixels. Only remain the water pixels that can be assessed for bathymetry retrieval. The following example shows quite well how water pixels are successfully distinguished thanks to the mask.
Validation Report | Research Applications
PUBLIC Page 18 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
5 Bathymetry and water quality
inversion PASS APPLI2 -VTS-07
1. The combined inversion maps (water depths, chlorophyll-a, bbp and CDOM) have continuous values at the limits of two clusters. If the analysis shows spatial heterogeneity over the map, the inversion process shall be adapted (e.g. change number of cluster).
2. The areas with water depths deeper than 15 meters are not considered (flagged).
The water depth (top left), Albedo (top right), CDOM (middle left), Chl-A (middle right) and BBP (bottom) are adjusted in the model to fit satellite radiometry. Areas not suitable for bathymetry retrieval are masked on the Water Depth map.
Validation Report | Research Applications
PUBLIC Page 19 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
6 Water level correction PASS APPLI2 -VTS-06 1. Bathymetry maps are corrected according to the modelled tidal level
Depending on the time and location, the tide level is retieved thanks to FES2014 model. A chart displays the tide level during the sensing day and the accurate tide level at the exact time of the satellite image. This tide level is then added to the Water Depth data and so the final Bathymetry is obtained.
7 Averaged bathymetry and
estimate of variability FAIL APPLI2 -VTS-07
This step provides the median value of all the bathymetry results (and associated standard deviation) available over the same area of interest during a period shorter than six months.
It finally appeared that averaging dozens of bathymetry maps is not relevant or reliable enough as each of the maps can be affected by clouds, sun glint or other undesirable effect that lead to erroneous synthetic data. It has been decided not to integrate that module.
Date of execution: 27/09/2018
Validation Report | Research Applications
PUBLIC Page 20 of 41
3.3 Vessel detection & Oil Spill Detection with SAR & Optical
TEST PROCEDURE
VTP Id.: APPLI3-VTP-001 Id of Associated VTS:
APPLI3-VTS-01
APPLI3-VTS-02
APPLI3-VTS-03
APPLI3-VTS-04
APPLI3-VTS-05
APPLI3-VTS-06
APPLI3-VTS-07
APPLI3-VTS-08
APPLI3-VTS-09
APPLI3-VTS-10
APPLI3-VTS-11
Functionality to be Tested: The good performance of the Vessel and Oil Spill Detection application through a python environment
Required Test Environment: Co-ReSyF online platform available, with a valid user. No special input needed, Co-ReSyF catalogue is sufficient. Overview of the test procedure: Test if each step of the vessel detection and oil spill processing chain is working as expected.. Successful requirements of the application are outlined for each test in the comment section. For the Oil spill detection algorithm some of the steps in the processing chain were already tested and accepted in the work developed in another project. Those steps are marked as PASS. Detailed description of the test procedure, including how to observe and verify the results:
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 21 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
1 Pre-processing analysis PASS APPLI3-VTS-01
All necessary pre-processing steps are executed within the application
Raw SAR image is successfully calibrated, despeckled, georeferenced and land is masked out. The user is able to define the conditions for the pre-processing analysis through the portal.
Sentinel 1 – RAW Image
Ellipsoid Correction - GG
Land Mask + Calibration + Speckle-Filter
Validation Report | Research Applications
PUBLIC Page 22 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
2 Select Area of Interest (AOI) PASS APPLI3-VTS-02 Pre-processed image is cropped to meet user defined AOI.
Input required – User defined AOI. Image is successfully cropped to geographical specifications
FORMAT: 'POLYGON ((minEst minNorth, minEst maxNorth, maxEst maxNorth, maxEst minNorth, minEst minNorth))‘
Validation Report | Research Applications
PUBLIC Page 23 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
3 Application of ship detection algorithm
PASS APPLI3-VTS-03
Vessels detections outputted as GeoTiff, which includes detection parameters & metadata
Vessel detections successfully outputted internally as a GeoTiff. The user is able to define the minimum and maximum target size through the portal.
4 Merging image tiles UT APPLI3-VTS-04
Generation of original AOI image from all image tiles used for parallel processing
Currently untested. Changes were done to the research application and the current version does not separate image into tiles.
Validation Report | Research Applications
PUBLIC Page 24 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
5 Shapefile generation PASS APPL13-VTS-05
Generation of shapefile (.shp) with the date and time of the SAR image acquisition.
The user is able to download an output shapefile through the portal.
6 All UT APPLI3-VTS-06 GeoTiff output and AIS match-up output
Currently untested because no AIS data in included in the platform. AIS match up to be confirmed. AIS data access and application to be confirmed.
7 Pre-processing analysis (optical)
UT APPLI3-VTS-07
Optical images are atmospherically corrected and the land /clouds are masked out
Currently untested in the Co-ReSyF platform. The platform will not provide an algorithm for oil spill detection in optical imagery.
Validation Report | Research Applications
PUBLIC Page 25 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
8 Application of oil spill detection algorithm (SAR)
PASS APPLI3-VTS-08
Detection of dark features on the sea surface and classify as oil spills or look-alikes
Tested in the Co-ReSyF platform but still requires to be tested in the Co-ReSyF platform. See examples of oil spill detected in image below.
Validation Report | Research Applications
PUBLIC Page 26 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
9 Application of oil spill detection algorithm (optical) UT APPLI3-VTS-09
Separation of oil spills from water or other ocean features is performed on pixel-based spectral signatures
Currently untested in the Co-ReSyF platform. The platform will not provide an algorithm for oil spill detection in optical imagery
10 Auxiliary data shapefile generation PASS APPLI3-VTS-10
Query based on satellite image (SAR and optical) date/time acquisition (allowing user-defined temporal window) generates list of auxiliary data that are suitable for match-up pairs
Tested in the scope of the SenSyF in a cloud environment similar to the Co-ReSyF platform but still requires to be tested in the Co-ReSyF platform (no AIS data available in the platform). See of detected oil spill shapefile/kml example below
Validation Report | Research Applications
PUBLIC Page 27 of 41
Step Nb.
Description Result VTS Verified P/F Criteria Comments
11 All UT APPLI3-VTS-11
Output data format ((i) AOI SAR and optical images in GeoTIFF format, (ii) binary (merged) result over the AOI in shapefile format (SAR), (iii) binary (merged) result over the AOI in shapefile format (optical), (iv) auxiliary match-up dates/times shapefile)
Currently only tested for SAR imagery in the Co-ReSyF platform. See example images below of shapefile and GeoTiff oil spill detection.
Date of execution: 09/11/2017
3.4 Time series processing for hyper temporal optical data analysis
TEST PROCEDURE
VTP Id.:
APPLI4-VTP Id of Associated VTS:
APPLI4-VTS-01
APPLI4-VTS-02
APPLI4-VTS-03
APPLI4-VTS-04
APPLI4-VTS-05
APPLI4-VTS-06
APPLI4-VTS-07
Functionality to be Tested: Operations of the application and comparative success to desktop-based tools.
Required Test Environment: Testing the various requirements involves the use of desktop software (ArcMap 10.0) to cross-compare desktop and platform outputs, ensuring values and spatial locations at times and in output datasets are the same.
Validation Report | Research Applications
PUBLIC Page 28 of 41
Overview of the test procedure: The whole processing chain has been run on a desktop-compiled datacube of 5-day interval SST data for 2011. The output dataset is prepared and ready to be compared to an output dataset derived using the Co-ReSyF system, using the same input SST datafiles.
The confirmation of statistical patterns (using truly random numbers) for the process to build upon is reported on in the ATBD.
The defined processing chain from datacube to Ocean-surface Heterogeneity Map is detailed here, with the verification steps required that it is operating. The validation and product characterisation efforts have been detailed in the ATBD v2.3.
Note that as of November 4th, 2018, the full suite of hypertemporal component modules have not yet been integrated into the Co-ReSyF platform. Integration is ongoing at time of writing. Once the process (and component modules) has been integrated into the WINGS development platform, the verification tests below can be implemented. As such, all tests are currently marked as UT (Untested) in the results column. Furthermore, due to the focus on testing the process in the WINGS environment, and the revised process itself (see ATB V2.3), a number of requirements have been rendered redundant (indicated by N/A (No longer Applicable) in the results section. Progress towards achieving the test has been outlined in the comments section. An updated version of section will be submitted in due course before the final review.
Detailed description of the test procedure, including how to observe and verify the results:
Step Nb.
Description Result Requirement(s) Verified
P/F Criteria Comments
1 Ingestion into the system of a 3-d datacube consisting of cleaned level 4 data to be processed.
UT APPLI4-VTS-01
Updated
Multi-layer dataset successfully uploaded into the WINGS application development portal.
Currently untested due to research application not yet being integrated into Platform.
Progress made:
WINGS development portal for testing uploaded modules and dataset upload functionalities completed. The Classification modules is also compiled, and awaiting upload into the development area, upon which time the 3D dataset can be uploaded and used to test whether the upload has been completely successful
Validation Report | Research Applications
PUBLIC Page 29 of 41
Step Nb.
Description Result Requirement(s) Verified
P/F Criteria Comments
2 Verification that the data cube generated, contains the correct data images, in the correct sequence.
N/A APPLI4-VTS-02 Independent desktop verification that each image has successfully been integrated into the data cube, in the correct order, with no loss of the required data values for further processing.
No longer applicable - Step has been bypassed by enabling upload of user-built datacubes into the WINGS environment.
3 Verification that the randomised image data, is generated within the statistical parameter range constraints sourced from the data cube
N/A APPLI4-VTS-03 Independent desktop verification that each image’s values lie within the range allowed by the statistical constraints.
No longer applicable – Research outlined in the ATBD has shown that the classification algorithm (in particular the separability analysis steps) are not operable on classifications of truly randomised data. If the process works, it is because the data contain real patterns.
4 Independent Verification that the 91 cluster outputs extracted from the data cube by the Co-ReSyF system are equivalent to those extracted by the same methodology operating on a desktop computer
UT APPLI4-VTS-04 Independent comparison of like versus like cluster images (e.g. image of platform-derived 10 clusters, versus image of desktop-derived 10 clusters), and verification that output images are equal.
Currently untested due to research application not yet being integrated into Platform.
Progress made:
ISODATA Clustering code written.
The module has been written for ISODATA and is being enhanced to produce a signature file containing the statistics of each cluster for each cluster output. Following this enhancement, the module can be uploaded into the WINGS environment and tested.
Validation Report | Research Applications
PUBLIC Page 30 of 41
Step Nb.
Description Result Requirement(s) Verified
P/F Criteria Comments
5 Independent Verification that for each of the 91 cluster output image files extracted from the data cube by the Co-ReSyF system, the boundaries between clusters have been successfully captured by the boundary extraction algorithm, and integrated into boundary extraction output image file
UT APPLI4-VTS-05 Independent verification that boundary value pixels in the extracted boundary output image files, are spatially located in inter-cluster boundary areas in the classification output cluster image files.
Currently untested due to research application not yet being integrated into Platform.
Progress made:
The procedure is clearly defined for desktop applications.
Raster to polygon module written and in the process of being adapter for upload.
polygon to polyline module written and in the process of being adapter for upload.
Polyline to raster converter module written and in the process of being adapter for upload.
Awaiting a polyline field attribute manipulation module t be written and integrated into the platform alongside the above modules before the components can be tested.
Validation Report | Research Applications
PUBLIC Page 31 of 41
Step Nb.
Description Result Requirement(s) Verified
P/F Criteria Comments
6 Verification that the raster calculator functionality is operating properly
UT APPLI4-VTS-06 (i) Determine whether the boundary file generated using the platform, and that generated on the desktop PC are equivalent.
(ii) Determine whether the land/ocean boundary areas have the highest value.
Currently untested due to required module not yet being integrated into Platform.
Progress made re. (i):
Final output boundary file has been generated from the desktop-generated datacube of level 4 SST data and is on file to be used to verify platform-generated outputs.
Progress made re. (ii):
Final output boundary file has been generated from the desktop-generated datacube of level 4 SST data. This exhibits areas of high boundary probability, including highest boundary probability areas along all coastlines.
This boundary dataset is on file to be used to verify platform-generated outputs.
Awaiting the composition and platform integration of a raster calculation module generate comparative datasets and complete tests.
Validation Report | Research Applications
PUBLIC Page 32 of 41
Step Nb.
Description Result Requirement(s) Verified
P/F Criteria Comments
7 Verification that the statistical analyses are being conducted successfully by the system
N/A APPLI4-VTS-07 Verification that the same statistical conclusions are reached from the same data being processed using the two different system pathways (platform and desktop PC).
No Longer Applicable – the WINGS component of Co-ReSyF will produce OHMA maps for the user to download and analyse. Furthermore the required statistical analyses to validate the produced OHMA maps rely on underway data from multiple sources which are not yet integrated into the Co-ReSyF platform.
Progress made:
Statistical analysis procedure has beenexplored and tested, and is now outlined in the ATBD.
See step 3 for the Randomised comparison validation, now described in the ATBD.
Date of execution: Estimated completion date, December 2018.
3.5 Ocean and Coastal altimetry
TEST PROCEDURE
VTP Id.: APPLI5-VTP-001 Id of Associated VTS:
APPLI5- VTS-01
APPLI5- VTS-02
APPLI5- VTS-03
APPLI5- VTS-04
APPLI5- VTS-05
APPLI5- VTS-06
APPLI5- VTS-07
APPLI5- VTS-08
Functionality to be Tested: Test the good performance of the Ocean & Coastal Altimetry Application
Required Test Environment: Co-ReSyF online platform available, with a valid user account. No special input needed, Co-ReSyF catalogue is sufficient.
Validation Report | Research Applications
PUBLIC Page 33 of 41
Overview of the test procedure: The test aims to assess the whole processing chain that delivers SWH, SSHA and Sigma0. This test assumes that input data
to the processor are present in the catalogue.
Detailed description of the test procedure, including how to observe and verify the results: Step Nb.
Description Result VTS Verified P/F Criteria Comments
1
Test the application as a whole. The aim is to check if the application is running smoothly and delivering the required results
PASS APPLI5- VTS-01
The outputs of all processing steps shall be along-track CF-compliant NetCDF files, loadable by most common NetCDF readers (for instance Ferret)
The application3 for jason1, jason2 and jason3 altimetry mission was tested on the geoportal. It ran as expected and delivered the results in NetCDF format.
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 34 of 41
2 User specifies mission, pass list, cycle list or time limits, latitude limits
PASS (partially)
APPLI5- VTS-02
Selection module locates and plots/lists the correct list of all SGDR data to process
At the geoportal, user can choose the mission, time limits and latitude limits. [Note that the choice of selecting data by cycle or pass is not yet available and will have to be tested at a later date]. The test was done successfully and the geoportal was able to define and plot all SGDRs available within a geographical box or polygon with a certain time limits as per picture below:
3 User specifies region from predefined list and time limits
PASS (partially)
APPLI5- VTS-03
Selection module locates and plots/lists correct list of all SGDR data to process
As for step no. 0, but polygon/region selection not yet enabled so it will have to be tested at a later stage
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 35 of 41
4 User selects a few passes from list
PASS APPLI5- VTS-04
Module correctly passes to processor the sub-selection of SGDR data to process
The user then is able to create a data package and select the passes that are to be processed by the processor
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 36 of 41
5
Retracking of example data from different missions with appropriate algorithm
PASS APPLI5- VTS-05
Retracked quantities (range, SWH, sigma0) for each mission and algorithm compare favourably (i.e. have equivalent or less noise) with original data
The retracked data compare well with the improved original SGDR data (see SSH, SWH plot).
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 37 of 41
6 Different combinations of range corrections specified by the user
PASS APPLI5- VTS-06 Corrections are properly applied according to user selection
The user is able to choose from a set of improved corrections before running the processor or keep the default corrections. This functionality is working well within the geoportal.
Step Nb.
Description Result VTS Verified P/F Criteria Comments
Validation Report | Research Applications
PUBLIC Page 38 of 41
7 Output with different degrees of richness, as specified by the user
FAIL APPLI5- VTS-07
NetCDF output is generated at different levels of complexity (i.e. including different combinations of variables) according to user specification
There are still some issues in reading the along-track data provided in the NetCDF format by the Geoportal. Once these are sorted then the processor can be evolved to give the user flexibility in choosing the richness of the outcome. This will have to be tested again at a later stage
8 Validation of output reprocessed SSHA against selected tide gauges
FAIL APPLI5- VTS-07
1. Number valid measurements recovered in the coastal zone increases when using reprocessed data w.r.t. using standard data.
2. Correlation between time series (altimetry vs tide gauges) does not decrease when using reprocessed data w.r.t. using standard data.
There is still need to project the output onto reference tracks (i.e. co-locate the data) so time series analysis can be performed. And also more mission to be integrated within the Geoportal. At this moment only jason-1, jason-2 and Jason-3 are integrated .
Date of execution: 07/11/2017
Validation Report | Research Applications
PUBLIC Page 39 of 41
3.6 Application synergy: SAR/Optical Bathymetry cross-validation and
fusion
During the project, it appeared that a synergy between SAR and Optical Bathymetry was highly
suitable and would have been an interesting demonstration for Co-Resyf. But the schedule was
too short and needed the SAR Bathymetry, the Optical Bathymetry and Workflow manager to
be ready and integrated at least 6 months before the end of the project, in order to have the
time to set up and deploy a prototype. For that reason, building this synergy was set aside but it
is to be kept in mind for later development. However, during this cooperation, a deep reflexion
about what can be done has been carried out and the algorithm has been defined and is for sure
a strong basis for later research.
Validation Report | Research Applications
PUBLIC Page 40 of 41
4 References Co-ReSyF. (2016a). GRANT AGREEMENT-687289. European Commission, Research Executive
Agency.
Co-ReSyF (2016b). System Requirements Document – Tools, issue 1.0. European
Commission, Research Executive Agency.
Deimos (2016a). VERIFICATION AND VALIDATION PROCEDURE, EDG-CMS-PRO16-E, Issue
2.0. ELECNOR DEIMOS CORPORATE MANAGEMENT SYSTEM.
Co-ReSyF (2016c). Verification and Validation Plan - Tools, issue 1.1. European Commission,
Research Executive Agency.
Deimos (2016b). NON-CONFORMANCE MANAGEMENT PROCEDURE, EDG-CMS-PRO12-E,
Issue 3.0. ELECNOR DEIMOS CORPORATE MANAGEMENT SYSTEM.
Validation Report | Research Applications
PUBLIC Page 41 of 41
END OF DOCUMENT