1 ESA SNAP HACKATHON · 10/16/2015 · Java API Docs Python (cont.) Java to (C)Python type conversions: 1st SNAP Hackathon, ESRIN, 15+16 Oct 2015 Java Python java.lang.String str

1ST ESA SNAP HACKATHON

SNAP Architecture

SNAP Development Team

ESA ESRIN, 15 + 16 October 2015

SNAP Architecture

• Dynamic, module-based architecture, with various extension points and extensions

• Install and update of extension modules

• Use from Java and Python programs, extend by Java and Python plugins

• Write a SNAP plugin, use it in all toolboxes

• High-level architecture comprises two subsystems– SNAP Engine, the core and command-line interface

– SNAP Desktop, the graphical user interface

SNAP Architecture

1st SNAP Hackathon, ESRIN, 15+16 Oct 2015

SNAP Engine

Java SE 8 Platform

NetBeans RCP

SNAP Desktop

Sentinel-3 Toolbox (S3TBX)Sentinel-2 Toolbox (S2TBX)Sentinel-1 Toolbox (S1TBX)

Python

GeoTools JAI NetCDF …

Any combination of toolboxes add-ons is allowed, even none, as SNAP Desktop is a already a useful stand-alone application for EO data exploitation.

Programminglanguage layer

3rd-party library layer

SNAP layer

Graph Processing Framework

• Majority of SNAP „functions“ are implemented asoperators

• Each operator can be invokedfrom SNAP Desktop and fromthe command line

• Processing chains („graphs“)are configured in XML files

• Graphical Graph Builder

• Graph Processing Tool (gpt) forexecuting of graphs (chains)>gpt -help

Generic Functions and Tools

• Applicable for all toolboxes and wide range of sensors

• Raster data and vector data tools• Visualisation

– Multi-layer displays, layer editors– Image, mask, shapes overlays– Colour management, fast navigation

• Data Analysis– Various statistics and plot types– Spectrum display (optical)

• Data processing– Reprojection, Collocation, Mosaicing– Level-3 processor– Graph processing, ürocessing graph

builder


SNAP Basic Concept


Reader API Writer API Operator API

GeoTIFFWriter Impl.

v1.0.1

Gap-Filling Alg. Impl.

v2.1

SST ECV Reader Impl.

v1.2

Product Data Model

RCP

NetBeans Platform

Gap-Filling GUI Impl.

v2.2

SNAP DesktopSNAP Engine

Basic Concepts

• SNAP has been designed as a EO raster data visualisation, analysis and processing system

• However vector data is supported through an OGC compatible model (GeoTools)

• In SNAP, almost all EO data processing is done through a pull-processing approach:– If some raster data is required, such as for display or

processing, it is obtained from some raster data source– The source can be a product reader that reads from a file,

or a raster data operator that computes its output from other raster data

– This way, raster data read/computation requests are made from one raster data node to another forming directed acyclic imaging graphs


Basic Concepts – Product Data Model

• Product : – has a file-name, type-name– has a scene raster size– has a GeoCoding– has a model coordinate reference system (CRS)– has any number of RasterDataNodes– has any number of VectorDataNodes– has MetadataElements

• RasterDataNode:– has a name, unit, description– has a raster data type and size– provides raster data – can be a TiePointGrid, Band, VirtualBand, FilteredBand, Mask


Basic Concepts – Product Data Model

• TiePointGrid: – a grid that is up-sampled by linear interpol. to a larger raster size

• Band:– receives its raster data on-demand from a ProductReader or an

Operator or a MultiLevelImage

• VirtualBand: – a band which computes its raster data on demand from an band

maths expression

• FilteredBand: – a band which computes its raster data on demand from filter

kernels

• Mask: – a band with 1-byte Boolean pixels which can be used as ROI


Basic Concepts - Behaviour

• ProductWriter: 1. writes header once (structure + metadata) 2. writes raster data to output on demand3. requests raster data from sources

• Operator: 1. One target product, N source products2. computes raster data for target on demand3. requests raster data from sources

• ProductReader: 1. reads product header once (structure + metadata) 2. reads raster data regions from input on demand


Programming Language

• SNAP is programmed entirely in Java. Why? Java is cool and beautiful.Java is also very fast (as C), secure, and truly platform independent.

• So we have a SNAP Java API

• SNAP now supports also Python. Why? Python is cool and beatiful, and most of you use it. However, for raster data processing plain Python is very, very slow. Unless you use numpy and the like.

• Therefore SNAP supports two Python implementations– CPython 2.7, 3.3, 3.4 to use with numpy etc ( snappy)– Jython 2.7 to develop also SNAP GUI extensions

• There is no dedicated SNAP Python API• Instead we reuse the SNAP Java API from Python


SNAP API Use Cases


User’s code calls back into SNAP libraries via

SNAP API

User’sJavaProgram

SNAP Engine

- Product- Band- TiePointGrid- …

- ProductIO- GPF- …

In-built or user’snative Java Extension

In-built or user’snative Pythonextension

In-built or user’sexecutableextension

User’sPythonProgram

User’s program calls into SNAP libraries via SNAP API

e.g. ProductIO.readProduct(..)

SNAP invokes external user tool with selected ECV file, e.g. a MATLAB program, then ingests generated

outputs: stand-alone tools adapter

SNAP calls into user’s code via extension

point API (e.g. Operator)

Use case #2: SNAP

extension

Use case #1: Toolbox reuse

Native Rextension

Development Environment

• SNAP 2.0 beta 8, the installed version

• Python coding

– Python 2.7, 3.3 or 3.4

– git to clone code from github.com/senbox-org, especially snap-examples

– PyCharm, probably the best Python IDE

• Java coding

– JDK 1.8 update 60

– IntelliJ IDEA – for sure the best Java IDE

– Maven – to compile and package Java sources


https://github.com/senbox-org

Java API Docs

• Java API Docs

– SNAP Engine http://step.esa.int/docs/v2.0/apidoc/engine/

– SNAP Desktophttp://step.esa.int/docs/v2.0/apidoc/desktop/

• Start Python coding

– by looking at the examples

– by looking up classes used in the Java API Docs

• How to interpret as Python developer?


http://step.esa.int/docs/v2.0/apidoc/engine/

http://step.esa.int/docs/v2.0/apidoc/desktop/

Java API Docs - ProductIO


Java API Docs Python code

• From ProductIO:

• From Product:


from snappy import ProductIO

p = ProductIO.readProduct(‘/eodata/test.nc‘)

b13 = p.getBand(‘radiance_13’)

Java API Docs Python (cont.)

Java to (C)Python type conversions:


Java Pythonjava.lang.String str

boolean bool

byte, short, int, long int

float, double float

byte[], short[], int[], long[], float[], double[]

Python buffer protocol,Python sequence protocol e.g. pass to numpy array constructors

null value None value

any java.lang.Object Wrapped Java object

http://jpy.readthedocs.org/en/latest/reference.html

http://jpy.readthedocs.org/en/latest/reference.html

Product Data Model - Product

uuid: String

fileLocation: URI

startTime: UTC

stopTime: UTC

Product

RasterDataNode

VectorDataNode

*

*

*

Product Data Model – Metadata

MetadataElement

value: Object

MetadataAttribute

name: String

Node

0..1

metadata

0..*

children

attributes

0..*

Product Data Model – Node Groups

getNodeCount(): int

getNode(index: int): T

iterator(): Iterator<T>

addNode(node: T)

insertNode(i: int, node: T)

removeNode(node: T)

Group<T extends Node>

Node

0..1

parent

0..n

children

Product Data Model –Node Observation

addNodeListener(l: NodeListener)removeNodeListener(l: NodeListener)

fireNodeChanged()fireNodeDataChanged()...

Node

nodeChanged(e: NodeChangeEvent)nodeDataChanged(e: NodeChangeEvent)nodesAdded(e: NodeChangeEvent)nodesRemoved(e: NodeChangeEvent)

NodeListener

0..*nodeListeners

nodeChanged(e: NodeChangeEvent)

nodeDataChanged(e: NodeChangeEvent)

nodesAdded(e: NodeChangeEvent)

nodesRemoved(e: NodeChangeEvent)

ProductExplorerNodeListener

ProductExplorerController

Product Data Model

(SNAP-Engine)RCP (SNAP Desktop)

register listener implementation

Product Data Model – Vector Data

VectorDataNode

SimpleFeatureType

SimpleFeature

Node

Geometry

Polyline

Point

Polygon

MultiPolygon

MultiGeometry

featureType

1

0..*

features

Product Data Model - Raster Data

getGeoPhysicalImage(): MultiLevelImage

getSourceImage(): MultiLevelImage

setSourceImage(img: MultiLevelImage)

createSourceImage(): MultiLevelImage

readPixels(x:int, y:int, w:int, h:int, data:int[])

readPixels(x:int, y:int, w:int, h:int, data:float[])

readPixels(x:int, y:int, w:int, h:int, data:double[])

writePixels(x:int, y:int, w:int, h:int, data:int[])

writePixels(x:int, y:int, w:int, h:int, data:float[])

writePixels(x:int, y:int, w:int, h:int, data:double[])

rasterWidth: int

rasterHeight: int

RasterDataNode

getLevelCount(): intgetLevelImage(level:int): RenderedImage

MultiLevelImage

getTileWidth(): intgetTileHeight(): intgetWidth(): intgetHeight(): intgetData(): RastergetData(r: Rectangle): RastergetTile(tileX:int, tileY:int): Raster

RenderedImage

provide Band

TiePointGrid VirtualBand

Mask

FilteredBand

0..1validPixelMask

1sourceImage

0..* ancillaryBands

role: String

Used to represent

uncertainty information.

Used to reflect missing-data, e.g. given by a no-

data-value or quality flag combintation, etc.

Product Data Model –Flags and Index Coding

rasterWidth: int

rasterHeight: int

rasterType: int

RasterDataNode

Node SampleCoding

FlagCoding

IndexCoding

MetadataElement

flagCoding

0..1

0..1

indexCoding

Product Data Model – Geo Coding

rasterWidth: int

rasterHeight: int

rasterType: int

RasterDataNodegetGeoCrs(): CRS

getMapCrs(): CRS

getImageCrs(): CRS

getPixelPos(g: GeoPos): PixelPos

getGeoPos(p: PixelPos): GeoPos

GeoCoding

0..1

geoCoding

TiePointGeoCoding

GcpGeoCoding

PixelGeoCoding

CrsGeoCoding

Data „Pull“ Processing

FileReadOperator:

Operator

<CDM-Instance-1>:

Product

operator

SubsetOperator:

Operator

<CDM-Instance-2>:

Product

operator

targetProducttargetProduct

AggregOperator:

Operator

<CDM-Instance-3>:

Product

operator

targetProduct

SstCciProductReader:

ProductReader

reader

GapFillOperator:

Operator

<CDM-Instance-4>:

Product

operator

targetProduct

sourceProduct sourceProduct sourceProduct

AnimGifWriter:

ProductWriter

convert and write ECV data

Documents

1 ESA SNAP HACKATHON · 10/16/2015 · Java API Docs Python (cont.) Java to (C)Python type conversions: 1st SNAP Hackathon, ESRIN, 15+16 Oct 2015 Java Python java.lang.String str