Sentinel HubCloud DetectorDr. Anže Zupanc

makes extraction of valuable information from satellite imagery easy

EOPatch

Cloud Detection

EOTask

EOWorkflow

build complex value-extraction pipelines

sentinelhub-py – Sentinel Hub Python SDK

WMS/WCS

Requests

Standard and

Custom Layers

Multi-temporal

Requests

Cloud Coverage

Filtering

Download to /

Read from Disk

Sentinel Hub Ecosystem

2

Sentinel Hub Services

Cloud Detector Requirements

Works on any Area Of Interest

Works on different zoom levels

Makes fast predictions (can be used in real time)

Has global coverage (provides good results anywhere in the world)

3

pixel-based CD

global training dataset

fast model

pixel-based CD

Pixel-based mono-temporal Cloud Detector I

Training dataset options at the time

o Hollstein et al. manually labeled dataset

+ high quality

+ global coverage throughout a year

- correlated pixels; effective size small

o MAJA multi-temporal Cloud Detector

- Noise in labels (classification mistakes, buffered regions)

+ global coverage throughout a year

+ unlimited sample size

Geographic location of tiles labeled by Hollstein et al.

Geographic location of processed tiles available at theia.cnes.fr in 2017.

Tra

inin

g d

atas

etT

est

dat

aset

4

Pixel-based mono-temporal Cloud Detector II

Training dataset

o 1000 randomly sampled pixels from each of 14,140 Sentinel-2 tiles with MAJA cloud masks

o 596 unique tiles from 77 different countries from Europe, Asia, Africa, North and South Americas, and Oceania

Final Model

o 10 bands as input: B01, B02, B04, B05, B08, B8A, B09, B10, B11, B12

o LightGBM (gradient boosting framework using tree-based algorithm)

o Model hyper-parameters tuned on random and independent subsample of the training sample

5

Pixel-based mono-temporal Cloud Detector III

o Visual assessment

o detector underperforms on bare land areas (sand, mountains and alike)

o Augment training dataset with misclassified pixels from handpicked scenes

Bad agreement between MAJA and our CD

Good agreement between MAJA and our CD

6

Pixel-based mono-temporal Cloud Detector IV

Final evaluation of the Cloud Detector performed with Hollstein et al. dataset

o With post-processing of the cloud mask (convolution + dilation)

Fmask - Python implementation (v 0.4.5) Sen2Cor v 2.4.0

Sentinel Hub’s CD (s2cloudless) has high cloud detection rate and lower misclassification rate of land and snow as clouds compared to other popular cloud detection algorithms

7

Multi-temporal Cloud Detector Imono-t CDRGB Sentinel 2

t +

3 d

ays

tt

+ 1

0 d

ays

o Mono-temporal CD still has problems with false detections of bright (bare) areas as clouds

o Problematic since often the same area is misclassified and hence filtered from further analysis (i.e. land cover classification, …)

o Difficult to circumvent with mono-temporal approach

o False positives are easier to identify with multi-temporal approach

o Comes for free with eo-learn

o Use similar approach to MAJA

o Check for correlations in time between cloudy pixel and its neighborhood in frame t and frames before and after in time

8

Multi-temporal Cloud Detector II

t + 3 days t + 10 days

t

t + 13 days t + 23 days

t - 7 dayst – 21 days

mono-t CD

multi-t CD

Multi-temporal cloud detector successfully corrects for mistakes made by the mono-temporal cloud detector.

9

Multi-temporal Cloud Detector III

Final evaluation of the Cloud Detector performed with part of the Hollstein et al. dataset

o The other part used in training of the multi-t CD

o With post-processing applied as in the case of mono-t CD

o Multi-t CD will be included as an EOTask in eo-learn

Multi-temporal cloud detector at the moment misses Cirrus clouds too often. We believe that this can be fixed (and will be) in future versions of the algorithm by augmenting the model with additional information.Similarly, we expect better snow/cloud separation by adding extra info.

10

Comments to CMIX Protocol

o Proposed validation datasets may have been used in the development of algorithms (as training dataset, to optimize hyperparameters, …)

o Part of the Hollstein dataset has been used for the development of multi-temporal CD

o All participants need to disclose if validation samples have been used in any other way apart from validation

o New validation dataset (similar to Hollstein’s) could be easily produced

o Sinergise can offer the usage of ClassificationApp for this purpose

o We propose that CMIX validation samples are published together with the results

o Future algorithm developers can make one-to-one comparison with algorithms compared in this workshop

11

Conclusions

o Sinergise is entering this workshop with two Cloud Detection Algorithms

o Mono-temporal Cloud detector s2cloudless

o Multi-temporal Cloud detector (to be available within eo-learn)

o Algorithms are (will be) freely available (open-sourced)

o s2cloudless

o eo-learn

o Use them, provide feedback, contributions welcome

o Looking forward to be part of this activity

12

The EO Research Team at Sinergise

o Matej Aleksandrov

o Matej Batič

o Matic Lubej

o Devis Peressutti

o Anže Zupanc

13

Backup

14

Classification App

15

Mono-t vs. Multi-t Cloud Detection

Large difference in number of cloud free observations between mono- and multi-temporal cloud detectors seen only in the problematic area that produces false positives in mono-temporal approach. In the rest of the scene the difference is small.

#valid observations 2017 (multi-t)#valid observations 2017 (mono-t) difference in #valid observations

16