Sciamachy features and usage with Sciamachy features and usage with respect to end-usersrespect to end-users

The typical fate of retrieval people dealing with large The typical fate of retrieval people dealing with large datasets…datasets…

C. Frankenberg, SRON team, IUP Heidelberg teamC. Frankenberg, SRON team, IUP Heidelberg team

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 2

SCIAMACHY on SCIAMACHY on ENVISATENVISAT, a brief introduction, a brief introduction

SCIAMACHY

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 5

SCIAMACHY data viewer (1 orbit =300Mb)SCIAMACHY data viewer (1 orbit =300Mb)

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 6

Scientific question in my case:Scientific question in my case:Retrieval of CHRetrieval of CH44 and CO and CO22

Spectra vertical column densities of CO2 and CH4 xVMR(CH4)

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 7

CHCH44 VMR August through November 2003 VMR August through November 2003

Frankenberg et al., Assessing methane emissions from global space borne observations, Science 2005

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 8

Issues related to ADAGUCIssues related to ADAGUC

• SCIAMACHY data access, 5Gb/day SCIAMACHY data access, 5Gb/day direct download from the Netherlands SCIAMACHY data centerdirect download from the Netherlands SCIAMACHY data center

• Data access, binary PDS fileData access, binary PDS file• No library available at that timeNo library available at that time• Official reading tool not useful for nearly operational Official reading tool not useful for nearly operational

retrievalsretrievals• Own C/C++ access routine was writtenOwn C/C++ access routine was written• Complex code structure, retrieval and data access are Complex code structure, retrieval and data access are

difficult to separatedifficult to separate

Too instrument specific to be of general interest in ADAGUCToo instrument specific to be of general interest in ADAGUC

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 9

Issues related to ADAGUCIssues related to ADAGUC

• General procedure:General procedure:1) Level 1 PDS File:1) Level 1 PDS File:

Geographic entity (usually a 60*120km rectangle) Geographic entity (usually a 60*120km rectangle) comprises spectra and numerous auxiliary datasetscomprises spectra and numerous auxiliary datasets

2) Retrieval via own C++ code, results stored in so called level 2 file2) Retrieval via own C++ code, results stored in so called level 2 file

3) 3) Level2 File (own format, so far ASCII) Level2 File (own format, so far ASCII)Geographic entity comprises eg Geographic entity comprises eg CHCH44 total column and additional parameters such as cloud total column and additional parameters such as cloud

cover, albedo, fit error, etc.cover, albedo, fit error, etc.

4) Generating gridded plots of the level 2 files depending on filter 4) Generating gridded plots of the level 2 files depending on filter criteria (eg. CloudTopHeight < 1km, fitError < 2%)criteria (eg. CloudTopHeight < 1km, fitError < 2%)

5) Compare data (raw and gridded) with other datasets (eg. Model 5) Compare data (raw and gridded) with other datasets (eg. Model output, retrievals of other groups, other satellite sensors)output, retrievals of other groups, other satellite sensors)

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 10

What is of general interest?What is of general interest?

• Points 3-5: Points 3-5:

3) Output file generation (file format, no standards!)3) Output file generation (file format, no standards!)

4) Gridding and plotting data based on predefined selection criteria4) Gridding and plotting data based on predefined selection criteria

5) Comparing datasets5) Comparing datasets

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 11

Output file generationOutput file generation

• Why ASCII?Why ASCII?• Human readableHuman readable• Easiest exchange between different groups (preferred format Easiest exchange between different groups (preferred format

for the comparison between SRON, IUP Bremen, IUP for the comparison between SRON, IUP Bremen, IUP Heidelberg)Heidelberg)

• Variety of linux tools available for processing, most notably Variety of linux tools available for processing, most notably awkawk

• Drawbacks…Drawbacks…• Slow access, big files, files not self-describingSlow access, big files, files not self-describing

• Why didn’t I use HDF/netCDF/GIS format?Why didn’t I use HDF/netCDF/GIS format?• Lazy (additional work, new skills necessary)Lazy (additional work, new skills necessary)• Awk tools not availableAwk tools not available

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 12

Gridding, projections, plottingGridding, projections, plotting

• What did I use?What did I use?• Admittedly very simple methods, lat/lon box gridding with Admittedly very simple methods, lat/lon box gridding with

own routines, IDL plotting/projection routinesown routines, IDL plotting/projection routines

• What would be nice?What would be nice?• Better gridding options (eg weighting by the overlapping Better gridding options (eg weighting by the overlapping

area)area)• Data conversion tools for easier access to tools such as GMT Data conversion tools for easier access to tools such as GMT

(Generic mapping tool)(Generic mapping tool)

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 13

Comparing datasetsComparing datasets

• a headachea headache• Even within SCIA: different pixel sizesEven within SCIA: different pixel sizes

comparing different species needs averaging to the lowest comparing different species needs averaging to the lowest resolution, how to do the averaging?resolution, how to do the averaging?

• Processing a lot of files is slow due to the ASCII formatProcessing a lot of files is slow due to the ASCII format

• Data exchangeData exchange• In my case only within the atmospheric community, so no In my case only within the atmospheric community, so no

direct problems as people were experienced with the direct problems as people were experienced with the formats, ASCII no problem anyway (but slow and large)formats, ASCII no problem anyway (but slow and large)

• What is needed for the GIS community, level 2 and/or level 3 What is needed for the GIS community, level 2 and/or level 3 (gridded) data?(gridded) data?

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 14

What I find ideal…What I find ideal…

• Results stored in a relational database management system Results stored in a relational database management system (RDBMS) with extracting routines of subsets to HDF, netCDF, (RDBMS) with extracting routines of subsets to HDF, netCDF, ASCIIASCII

• Why? Database systems are meant for large datasets and Why? Database systems are meant for large datasets and complex queries to derive subsetscomplex queries to derive subsets

• Simple example in SQL languageSimple example in SQL languageselect avg(CH4) from results where latitude>50 and latitude select avg(CH4) from results where latitude>50 and latitude <51 … and albedo>0.2 and cloudCover<0.05<51 … and albedo>0.2 and cloudCover<0.05

• FAST due to indexing (tested with a test database with 5 million FAST due to indexing (tested with a test database with 5 million entries, one query takes no time)!entries, one query takes no time)!

• Selection criteria easy (no awk necessary)Selection criteria easy (no awk necessary)

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 15

Even better: Spatial SQLEven better: Spatial SQL

• Spatial SQL: Spatial extension of the database systems (eg. Spatial SQL: Spatial extension of the database systems (eg. Points, polygons, etc)Points, polygons, etc)

• Example syntax (Postgres):Example syntax (Postgres):

SELECT ch4_total_column FROM results WHERE SELECT ch4_total_column FROM results WHERE distance( center_point, GeomFromText( 'POINT(10.0 20.0)', -distance( center_point, GeomFromText( 'POINT(10.0 20.0)', -1 ) ) < 1001 ) ) < 100

• Dumpers to eg “shape files” available:Dumpers to eg “shape files” available:

pgsql2shp [<options>] <database> <query>pgsql2shp [<options>] <database> <query>

• Direct connection to data viewers such as QGIS possibleDirect connection to data viewers such as QGIS possible

• Web interface to the interactive plotting tool mapserver Web interface to the interactive plotting tool mapserver

ADAGUC meeting, KNMI, De Bilt, 03/04 October 2006 16

What takes most of the time?What takes most of the time?

• SCIA data format SCIA data format Esp. level2 files for validations are far too complex and frustrate Esp. level2 files for validations are far too complex and frustrate peoplepeople

• Data filtering Data filtering plotting plotting interpreting interpreting change filters change filters and and so forthso forth

An interactive data viewer would be great (such as in GIS, click An interactive data viewer would be great (such as in GIS, click on the point and you get additional information)on the point and you get additional information)

Lots of time for discussionLots of time for discussion

Website for spatial RDBMS:Website for spatial RDBMS:

www.postgis.orgwww.postgis.org