CEOS System Engineering Toolset (CSET)
CSET is a Software Framework + Suite of Tools (Apps) that leverages a Common Architecture, Unified Data Model, Common User Experience (App continuity)
Data Management and Data Integration Toolso Space Data Management System (SDMS) - Extensible Data Management framework
Analysis Tools, Data Visualization & Presentation Toolso Rapid Acquisition Tool - Determines when satellites may get acquisitions of regions/locationso Coverage Analyzer - Determines number of actual acquisitions with given cloudiness thresholdo COVE Google Earth visualization tool for satellites - Displays potential and coincidence paths for Earth
Observing Satellites. Data presentation, reporting, and charts
Informational Tools: Mission Instrument Browser, Data Policy portal, Labs…
History: CEOS Visualization Environment (COVE) started in 2009– Web-based suite of tools for searching, analyzing and visualizing actual & potential
satellite sensor coverage– Philosophy: Simple, intuitive, easily accessible– 2014 COVE Stats: ~2500 users, 20K page views, broadly utilized by countries worldwide
Space Data Management System (SDMS)
1
2
3
CSET Application Portal
Analysis &Visualization
Apps
Data Management(SDMS) Apps
www.ceos-tools.orgLive in late November 2014
Space Data Management System (SDMS)
SDMS is a flexible Data Integration & Data Management Framework – Key is the ability to quickly create applications (e.g., country/organization specific applications):
– Common Architecture & Data model, Security model, Project Model– Search & Data Discovery Capability
o Can pull in and search archives by metadata, criteriao Search the entire federated database
– Analysis & Tool Execution Capabilityo Tool Integration framework (Java/JS based tool wrapping)o Existing and custom tools to analyze data, e.g.:
GDAL, Google Earth Engine, Open Foriso Flexibility to work with different workflows/use-cases (mix-and-match)o GIS Tools
– Data Visualization and Presentation Capability– Big Data Framework (Hadoop – Local Big Data Cluster)
• Working toward model for low bandwidth connections– Data housed, fetched, and processed remotely– Tiered solution for different bandwidth levels
• SDMS is not the cure all…
SDMS Framework
Columbia
GFOI
FAO
CommunityCapabilities
SDMS Interface
2
1
3
Global SDMS Data
& Tools
User-specificSDMS Data
& Tools(Analysis Case)
Context specificWork Area
4 Search & Filter
Project (Analysis)… Data… Tools… Output
SDMS Tool Execution (Script Editor)
2
3
User-specificData & Tools
ScriptEditor
4ExecutionConsole
>> Initializing…
1Global
SDMS Data & Tools
SDMS Tool Execution (Terminal Interface)
2
1
3
Global SDMS Data
& Tools
User-specificData & Tools
Terminal Interface
GFOI SDMS Prototype
SDMS Integration - Google Earth Engine
Google Earth Engine Demonstrator:– GEE is powerful satellite data acquisition and processing system– SDMS has JavaScript wrappers to the GEE API (small subset)
o Allows heavy-lifting to be done at Google data centerso Can download result subsets at user specified bounds and resolutions
– Benefits of integration:o Data: Provides additional source of data (SDMS Unified Data Model)o Tools: Additional processing capabilitieso Workflow: Can be a source of data that is then processed by other tools
– Current demonstrator prototypeo Allows fetching of Landsat 7,8 band TIF files and
combined thumbnail datao Focus is on cloud cover removal
SDMS Status
• CSET / SDMS development has been ongoing over the last year • Our initial release is a prototype/demonstrator (Amazon servers)• Alpha release to a small user group in December 2014• Looking for feedback, hoping CEOS SEO tools continue to be of
use to the broader community• SDMS Live Demonstration
1. SDMS Interface Prototype – Concept of Operations
2. Google Earth Engine Prototype – L8 Cloud removal
3. GFOI Prototype - Kenya
POC: Brian Killough, Ph.D. [email protected] Brian Williams, Ph.D [email protected] Sanjay Gowda, Ph.D. [email protected]
10
Data Services Prototypes ... Kenya Demo
SEO is developing an online version of a generic data services tool for future testing.
A demo of this tool is being worked with FAO to test processing and product development for one Kenya scene.
DEMO Details Location: Kenya (Mt. Kenya National Park) Dataset: Landsat 7, (Path-168, Row-60) Step #1: LEDAPS Processing - TOA reflectance, cloud
screening (ACCA), Atmospheric Correction to SR Step #2: BRDF Correction - Use MODIS data? Step #3: Cloud/Shadow Removal - Tag pixels with a value
or color for statistics and viewing Step #4: SLC Gap Removal - Tag pixels with a value or
color for statistics and viewing Step #5: Forest Classifier (unsupervised, or MODIS
Vegetation Continuous Fields (VCF), or Random Forest) Final Product: Forest Map with Statistics (Clouds, SLC
Gaps, Forest, Non-Forest)