View
283
Download
1
Embed Size (px)
Citation preview
AusCover Earth Observation Services& Data Cubes
Matt Paget – AusCover data and systems lead coordinator & AGDC team leaderCSIRO Land and Water, Canberra
IntroductionTERN / AusCover
• TERN’s Remote Sensing Data Facility• Coordinated infrastructure for management and access to EO products from
Universities, CSIRO, Geoscience Australia (GA) and Bureau or Meteorology (BoM)
Australian Geoscience Data Cube (AGDC)• Partnership between GA, CSIRO and National Computation Infrastructure (NCI)• Version 2 is presently in development, due June 2016
Committee on Earth Observation Satellites (CEOS)• AGDC forked as a collaboration between CEOS Systems Engineering Office (NASA-
SEO), CSIRO and GA• Implementations for Kenya and Colombia
Copernicus Data Hub• Australia and south Asia mirror of Europe’s Sentinel satellite series and services• Partnership between Federal and State Govt agencies and departments and
AARNET/RDS/NCI
Two PathwaysPresent
Data producers / custodians• Internal infrastructure, downloads etc• Storage and services close to
custodians
Open data access• Low level programmable data services• Interoperability with related and
cross-discipline activities
Aggregation• Promote open data formats and
consistent metadata• Metadata services and data portals
Emerging
Data producers / custodians• Large central data stores• Coordinated data and resource
management between custodians
Open data access• Analytics facility close to data• Optimised storage units with API
access
Aggregation• Mature algorithms and exploratory
data analysis• Community engagement in
validation and methods
AusCoverwww.auscover.org.au
(in transition)
Remote sensing data delivery backbone
• Interoperability formats and services
• Storage and infrastructure
Data production network• Central coordination and regional
nodes• National biophysical data products
Ground validation and characterisation• Data collection and processing
protocols • Field campaigns with airborne
hyperspectral and lidar
AusCover data systemDistributed network of storage and data access services• Extensible nationwide system• Data providers upload to their own
storage
Open standards, systems and services• Non-proprietary formats, OGC
Direct data access• Programmable and Human• No license click-throughs
Metadata and data licensing• Aligned with TERN, ANZLIC and
ANDS/RDA
Central coordination• Website, Metadata, Services• TERN coordination
Emerging EO informaticsEstimated storage and processing needs and opportunities for the next decade• Increasing volume: spatial, temporal and spectral resolution and number of sensors• Increasing velocity: near real time applications and automation• Increasing variety: best practice processing and integration of disparate sources
AGDC and CEOS Data Cubes
Dice… …and Stack
Version 1• GA’s Unlocking Landsat Archive• Tiling, Indexing, Geotiff, HPC workflow• Water Observations from Space
• Collaboration between GA, CSIRO and NCI• Uptake by CEOS and Kenya Cube development
Version 2• Analysis Ready Data• Relax requirement for a common grid• Modular design• Multiple platforms & environments
• Partnership and Governance• Transition Plan (archives, products & users)• Greater open collaboration with the community• Functional requirements due June 2016
www.datacube.org.augithub.com/data-cube/agdc-v2
github.com/ceos-seo
Storage Query and Access Module
Task Execution
N-Dimensional Array Interface
DataIndex Store Data
Multi-Dimensional Storage Unit
Business Processes(Data Management, Access Controls, Job Management, Etc.)
Integrating Tool
Forest Economics App N
Reporting Visualization
Application / GIS Mobile App
Data CubeInfrastructure
Data & Application Platform
UI & Application Layer
Virtual LabPlatform
Analytical Task
Definition
Data Acquisition& Inflow Analysis Ready DataObservations
Data Cube Notional Architecture
Data Collection 1 Data Collection 2
DataIndex Store Data
Multi-Dimensional Storage Unit
Authentication, Authorisation,
Accounting
Discovery Web Services
Ingester (ARD and Result)
SummaryNCRIS and TERN
• Investment has enabled open data access across Australian institutions• Major contribution to, and application of, open source standards and
services
International links• CEOS WGISS Future Data Architectures (NASA, USGS, ESA, JAXA)• Google Earth Engine and Amazon Web Services• Unprecedented Australian Govt and research collaboration through
Copernicus Data Hub
Implications and opportunities• Reducing the need for expert EO know-how• Enabling best available data and analytics from the research sector for
applications and operations• Building innovative and common infrastructures that adapt to massive
data volumes and emerging storage and compute environments
Contact details
AusCover www.auscover.org.au (in transition)Alex Held, CSIRO Canberra [email protected] Phinn, UQ & JRSRP [email protected] Paget, CSIRO Canberra [email protected] Scarth, UQ & DSITI [email protected]
AGDC www.datacube.org.au (in transition)Trevor Dhu, GA [email protected] Woodcock, CSIRO Canberra [email protected] Wyborn, ANU & NCI [email protected] Lymburner, GA [email protected]
CEOS Data Cube www.ceos.orgAlex Held, CSIRO Canberra [email protected] Killough, NASA-SEO [email protected]
Australia’s Regional Copernicus Data Access/Analysis HubAn integrated ‘Team Australia’ approach toSupport government information requirements.
Support the broader objective of enhancing access to satellite Earth observation data by
research, industry and civil society.
Facilitate collaboration between Australians, Europeans and inhabitants of the South-East
Asia-South Pacific region in exploitation of Earth observation data.
Benefits for: Australia, the region, EC/ESA/EUMETSAT, and the global
satellite EO community.
PartnersConsortium CollaboratorsSupporting
All Sentinel Products For
Partnerships(pay-per-use)
Australia’s Regional Copernicus Data Access/Analysis Hub
Master Repository (NCI)
ESAIntHub
EUMETCASTTerrestrial
Operational ProductsDeep National Archives(Government Systems)
GEANT/AARNET
Original Format Data Files(HSM Storage)
Analysis-Ready Data(Lustre/spinning disk)
Cloud and HPC(Petascale)
Agreed Standards InteroperableServices(e.g. Thredds)
Regional
Portal(a la ‘S
ciHub’)
Industry Exploitation
R&D for:Research/AcademiaCivil SocietyGovernmentIndustryAustralia and Region
Delivery for:AustraliaRegion