The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) one step at a time Steve Hankin (PMEL),

The Unified Access Framework (UAF)The Unified Access Framework (UAF)Building NOAA’s Global Earth Observation Building NOAA’s Global Earth Observation

Integrated Data Environment (GEO-IDE) Integrated Data Environment (GEO-IDE) one step at a timeone step at a time

Steve Hankin (PMEL), Kevin Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), Lewis O’Brien (PMEL/JISAO), Lewis McCulloch (HQ/TPIO) and the McCulloch (HQ/TPIO) and the

NOAA UAF teamNOAA UAF team

[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

Dec. 2010Dec. 2010 22

Enterprise-wide integration of dataEnterprise-wide integration of datais a very hard problem! is a very hard problem!

(i.e. adopting shared approaches (i.e. adopting shared approachesto achieve data interoperability) to achieve data interoperability)

NOAA/UAF-gridNOAA/UAF-grid

Dec. 2010Dec. 2010 33NOAA/UAF-gridNOAA/UAF-grid

Why?Why?

Different fields have different Different fields have different concepts of ‘data’concepts of ‘data’

… … and have developed solutions that and have developed solutions that make sense make sense to them.to them.

Getting people (and organizations) Getting people (and organizations) to change habits is difficult!to change habits is difficult!

Dec. 2010Dec. 2010 44

NOAA-worldNOAA-world


weather forecast weather forecast (time critical)(time critical)

fisheries management fisheries management (regulatory concerns)(regulatory concerns)

nautical chartingnautical chartingclimate, ocean, atmosphere researchclimate, ocean, atmosphere research

… … the list goes on … the list goes on …

NOAA management has been reluctant to invest NOAA management has been reluctant to invest funds in Agency-wide data integrationfunds in Agency-wide data integration

Dec. 2010Dec. 2010 55

The accepted approach:The accepted approach:build a “system of systems”build a “system of systems”

““Wrap” existing systems with loosely Wrap” existing systems with loosely coupled, standardized servicescoupled, standardized services

a Service Oriented Architecturea Service Oriented Architecture


Dec. 2010Dec. 2010 66

Global Earth ObservationGlobal Earth Observation

Integrated Data EnvironmentIntegrated Data Environment

GEO-IDE Con-ops GEO-IDE Con-ops outlines such a planoutlines such a plan

2005-06, 66 pages2005-06, 66 pages(by “DMIT”)(by “DMIT”)


Dec. 2010Dec. 2010 77

Seed fundingSeed funding ٨٨ available last yearavailable last year

Question:Question:How do you approach a very big How do you approach a very big problem with a very small amount of problem with a very small amount of money?money?

finallyfinally


Dec. 2010Dec. 2010 88

1.1. Generate use casesGenerate use cases

2.2. Define requirementsDefine requirements

3.3. Write a Concept of OperationsWrite a Concept of Operations

4.4. … … and an Implementation Planand an Implementation Plan

5.5. Assemble volunteer teams to implement Assemble volunteer teams to implement

NOT !NOT !


Tried and true approach …

Dec. 2010Dec. 2010 99

An alternative (‘agile’) approachAn alternative (‘agile’) approach

Don't Solve ProblemsDon't Solve Problems-- Copy Success -- Copy Success

""Switch: How to Change Things When Change Is HardSwitch: How to Change Things When Change Is Hard", ", Chip and Dan Heath (psychologists), 2010Chip and Dan Heath (psychologists), 2010


What “success” did UAF chose to copy?What “success” did UAF chose to copy?

Year 1 focused on gridded datasets.Year 1 focused on gridded datasets.

Dec. 2010Dec. 2010 1010

Projects: (too many to name)Projects: (too many to name)

DataDataformats:formats:

netCDF GRIB HDF

ServiceServicestack:stack:

netCDF-CF-DAP-THREDDS-WMS

Applications:Applications: Matlab ArcGIS Ferret

GrADS Google Earth IDV LAS ERDDAP …

Users: (too many to name)Users: (too many to name)

…

Dec. 2010Dec. 2010 1111

UAFUAF(*)(*)-grid-grid(initial focus: gridded data)(initial focus: gridded data)

– netCDF netCDF – CF CF – OPeNDAP OPeNDAP – THREDDS THREDDS – WMS & WCSWMS & WCS

(*) Unified Access (*) Unified Access FrameworkFramework

---- file format/API/data file format/API/data modelmodel

---- metadata conventionsmetadata conventions

---- network transportnetwork transport

---- network topologynetwork topology

---- OGC rendering and OGC rendering and transporttransport


THREDDS “network topology”THREDDS “network topology”a tree defined as distributed XMLa tree defined as distributed XML

Dec. 2010Dec. 2010 NOAA/UAF-gridNOAA/UAF-grid 1212

NO

MAD

S

GEO-IDE/UAF

NOAA NOAA Affiliated

NMFSOAR NWS NESDIS

NO

DC

NG

DC

GFD

L

PMEL

AOM

LO

CO

PFEG

ND

BC

ESRL

Coas

twat

ch

IOOS National Partners

IOOS Regional Partners

NAV

O

AOO

S

NAN

OO

S

CEN

COO

S SCCO

OS

PACI

OO

SG

LOS

NER

ACO

OS

MAC

OO

RA SECO

ORA

CARI

COO

S GCO

OS

Dec. 2010Dec. 2010 1313

Who is using this approach?Who is using this approach?

ModelersModelers– IPCC, GFDL, NCAR, … IPCC, GFDL, NCAR, …

Satellite programsSatellite programs– GHRSST, PathFinder, CoastWatch, …GHRSST, PathFinder, CoastWatch, …

NCEP weather and ocean forecastsNCEP weather and ocean forecasts– GRIB files served via NOMADSGRIB files served via NOMADS

Coastal (“HF”) radarCoastal (“HF”) radar

A growing list of observations programsA growing list of observations programs– Argo, OceanSites, …Argo, OceanSites, …

Pending adoption by OGCPending adoption by OGC


Dec. 2010Dec. 2010 1414

Some rough spotsSome rough spots

compliance with CF conventions is inconsistentcompliance with CF conventions is inconsistent

individual files often not aggregated into logical individual files often not aggregated into logical datasetsdatasets

metadata are often in need of enhancementmetadata are often in need of enhancement


Dec. 2010Dec. 2010 1515

… … but excellent repair tools exist:but excellent repair tools exist:NcML and “virtual files”NcML and “virtual files”

e.g. Improve CF compliance by adding “standard_name” e.g. Improve CF compliance by adding “standard_name” attribute to GRIB dataattribute to GRIB data

<variable name="vorticity"><variable name="vorticity"> <attribute <attribute name="standard_name" name="standard_name" value="atmosphere_absolute_vorticity“ value="atmosphere_absolute_vorticity“ /> /></variable></variable>

The file, itself, is untouched. But access to the file through the The file, itself, is untouched. But access to the file through the services is made to conform to standards.services is made to conform to standards.


Dec. 2010Dec. 2010 1616

NcML for aggregationNcML for aggregation

e.g. Aggregate three 1-year files of the same (say) model run e.g. Aggregate three 1-year files of the same (say) model run

<aggregation type="joinExisting“<aggregation type="joinExisting“ dimName="TimeAxis"> dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/>

</aggregation></aggregation>

The files are untouched, but services see a single logical file.The files are untouched, but services see a single logical file.


Dec. 2010Dec. 2010 1717

The emerging UAFThe emerging UAF

The UAF THREDDS catalog:The UAF THREDDS catalog:

~500 “clean” datasets~500 “clean” datasets.. growing as the team aggregates .. growing as the team aggregates and configures using ncMLand configures using ncML– team members become coachesteam members become coaches

lasting solutions must be implemented lasting solutions must be implemented at the data provider sitesat the data provider sites

The UAF TeamThe UAF Team– 8-10 active participants8-10 active participants– self-selected (volunteers)self-selected (volunteers)

1 year into the project …1 year into the project …


Dec. 2010Dec. 2010 1818

How to reach users?How to reach users?

Through Through theirtheir preferred tools preferred tools (without downloading files)(without downloading files)


Dec. 2010Dec. 2010 1919

Model 1: UMASS-ECOMModel 1: UMASS-ECOM Model 2: UMAINE-POMModel 2: UMAINE-POM Data: SST 2008-Sep-08 07:32Data: SST 2008-Sep-08 07:32

Desktop access in MatlabDesktop access in MatlabNOAA/UAF-gridNOAA/UAF-grid

Dec. 2010Dec. 2010 2020

Access in ArcGISAccess in ArcGISusing the Environmental Data using the Environmental Data

Connector (EDC)Connector (EDC)NOAA/UAF-gridNOAA/UAF-grid

Dec. 2010Dec. 2010 2121

Desktop access in FerretDesktop access in Ferret


Dec. 2010Dec. 2010 2222

Desktop access in GrADSDesktop access in GrADS


Dec. 2010Dec. 2010 2323

Desktop access in IDVDesktop access in IDV


Web browserWeb browser examples … examples …

Dec. 2010Dec. 2010 2424

Take differences

Google Earth

Scripts

Live Access Server Live Access Server (LAS)(LAS)


Dec. 2010Dec. 2010 2626

WMS can be WMS can be used by many used by many

GIS applicationsGIS applications


Dec. 2010Dec. 2010 2727

Browser access using ERDDAP Browser access using ERDDAP ……and from there …and from there …

Dec. 2010Dec. 2010 2828

.mat.mat

REST URL access to data subsetsREST URL access to data subsetsin several simple formatsin several simple formats

(accessible through home-grown scripting of many types)(accessible through home-grown scripting of many types)

.kml

.mat

.nc


Dec. 2010Dec. 2010 2929

How can users find the datasets?How can users find the datasets?(less mature)(less mature)

1.1. Web crawlersWeb crawlers• extract metadata from files (and elsewhere)extract metadata from files (and elsewhere)• ingest into a data basesingest into a data bases

and from there:and from there:

2.2. formal metadata records generatedformal metadata records generated (ISO 19115) (ISO 19115)

3.3. data discovery portals harnesseddata discovery portals harnessed(e.g. RAMADDA, GeoNetwork, GI-Cat, …)(e.g. RAMADDA, GeoNetwork, GI-Cat, …)

4.4. metadata made available for harvesting metadata made available for harvesting – GCMD, Geospatial One-Stop, GEOSS, data.gov GCMD, Geospatial One-Stop, GEOSS, data.gov



UAF home page

Instructions for end users: how to access data through their favorite applications

UAF is experimenting UAF is experimenting to see how to see how documentation may documentation may be shared by other be shared by other projects…projects…


• netCDF Java Tools• LAS• ERDDAP• GODIVA

From the THREDDS catalog “viewers” may also be called up


File-internal metadata accessible directly from THREDDS catalogs

Dec. 2010Dec. 2010 3333

The coming year’s target:Integrating “1-d grids” (observations)

Collections of time series & profiles in netCDF-CF format are already on-line


Dec. 2010Dec. 2010 3434

UAF team met just 3 weeks ago (Seattle)


Preliminary plans:

Some initial target collections of in situ obs:• Argo, OceanSites, ICOADS• stream gauge data from USGS• atmospheric profilers (NCDC)• …

Common denominators for accessing the collections:• UAF year-1 techniques allow 1-at-a-time access• ERDDAP tables (ASCII and OPeNDAP Sequence)

• a gateway for IOOS/SOS, RDBMS, DAPPER, …

Year 1 metadata techniques largely applicable to obs

Still largely volunteer-powered

Dec. 2010Dec. 2010 3636

Thank youThank you


Documents

The Unified Access Framework (UAF) Building NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) one step at a time Steve Hankin (PMEL),