1

US IOOS Regional Data Management:

The NANOOS NW Perspective

Emilio Mayorga, Ph.D.NANOOS Data ManagerApplied Physics LaboratoryUniversity of Washington

Seattle, WA

MEOPAR Ocean Data Management Expert Forum. Montreal, 2015 Nov 18-19

http://www.nanoos.org

2

US IOOS Regional Data Management:

The NANOOS NW Perspective

Emilio Mayorga, Ph.D.NANOOS Data ManagerApplied Physics LaboratoryUniversity of Washington

Seattle, WA

MEOPAR Ocean Data Management Expert Forum. Montreal, 2015 Nov 18-19

http://www.nanoos.org

Putting data to use for local to (bi)national needs:Data partnerships, support, enablement, dissemination

3

NANOOS: NW Association of Networked Ocean Observing Systems

Supported observing assets (~ 2013)

Matching Regional Priorities to Regional

Observing Capability & Strategy

4

Reg. Associations

Engage, Disseminate

DIVERSE LOCAL STAKEHOLDERS

Assure, Expose

CONSISTENT NATIONAL CAPABILITY

Leverage and Link

5

NANOOS Governing Council Members to date… (Mar. 2010)

http://www.nanoos.org/about_nanoos/members.php

6

IOOS vs. Regions ?

• IOOS (National)– Standards-based data servicesStandards-based data services– Enable seamless data integration at national scaleEnable seamless data integration at national scale– Assess and ensure system maturity, robustnessAssess and ensure system maturity, robustness

• NANOOS (Regional Associations)– Serve regional usersServe regional users– Consume federal (national) dataConsume federal (national) data– Facilitate integration of local data sets, data streamsFacilitate integration of local data sets, data streams– Expose and distribute local data nationally, interoperablyExpose and distribute local data nationally, interoperably– Redistribute data regionally, easy accessRedistribute data regionally, easy access– Support community of users and providersSupport community of users and providers

• Ongoing engagement– Reconciling national and regional views, prioritiesReconciling national and regional views, priorities

7

NANOOS Visualization System (NVS):Data Integration with Customized Views

http://nvs.nanoos.org

8

NVS Data Explorer

http://nvs.nanoos.org

9

NVS Data Explorer

http://nvs.nanoos.org

A key objective for NANOOS is to provide data and user-defined products to a diverse group of stakeholders in a timely fashion, and at spatial and temporal scales appropriate for their needs. To this end, NANOOS developed the NVS, which aggregates, displays and serves meteorological and oceanographic data, derived from buoys, gliders, tide gauges, HF Radar, meteorological stations and satellites, as well as model forecast information in such a way that it presents end users with a rich, informative and user friendly experience.

Strong focuson usability,user experience

10

Other NVS Apps

11

Other NVS Apps

12

NVS Information System

DS: Data SourceDAC: Data Assembly CenterNH: NVS data HarvesterMD: MetadataUI: User Interface (web)

(note: diagram from 2010)

13

NVS Data Harvesting: Ingesting data from providers

DS: Data SourceDAC: Data Assembly CenterNH: NVS data HarvesterMD: MetadataUI: User Interface (web)

• Accommodate local providers’ constraints

• Harvesting frequency: 5 min - 3 days

• Ingest widely diverse set of data sources

• Extensive use of metadata from NVS DB

(note: diagram from 2010)

14

NVS Data Harvesting: Ingesting data from providers

DS: Data SourceDAC: Data Assembly CenterNH: NVS data HarvesterMD: MetadataUI: User Interface (web)

• Accommodate local providers’ constraints

• Harvesting frequency: 5 min - 3 days

• Ingest widely diverse set of data sources

• Extensive use of metadata from NVS DB

Data Services

(note: diagram from 2010)

15

NVS Metadata: Enabling consistency across all providers

• Vocabulary translation between diverse data sources & NVS unified information

– Date-time encoding & timezone, location, asset identifier and name, platform type, observation depth representation, measurement names, units, etc

• Widely used across all of NVS.

• Reusable.

(note: from 2010)

16

NVS Metadata: Enabling consistency across all providers

• Vocabulary translation between diverse data sources & NVS unified information

– Date-time encoding & timezone, location, asset identifier and name, platform type, observation depth representation, measurement names, units, etc

• Widely used across all of NVS.

• Reusable.

(note: from 2010)

From Canada: Environment Canada VENUS Vancouver Island University

17

Data Provider Partnerships: PNW Shellfish aquaculture community

•PCSGA•Pacific Shellfish Institute•Taylor Shellfish•Whiskey Creek Hatchery•Penn Cove Shellfish•Stillaguamish Tribe•Vancouver Island University•(WA Dept. of Health)

18

Data Provider Partnerships: PNW Shellfish aquaculture community

•PCSGA•Pacific Shellfish Institute•Taylor Shellfish•Whiskey Creek Hatchery•Penn Cove Shellfish•Stillaguamish Tribe•Vancouver Island University•(WA Dept. of Health)

19

Data Provider Partnerships: PNW Shellfish aquaculture community

•PCSGA•Pacific Shellfish Institute•Taylor Shellfish•Whiskey Creek Hatchery•Penn Cove Shellfish•Stillaguamish Tribe•Vancouver Island University•(WA Dept. of Health)

20

Data Provider Partnerships: PNW Shellfish aquaculture community

•PCSGA•Pacific Shellfish Institute•Taylor Shellfish•Whiskey Creek Hatchery•Penn Cove Shellfish•Stillaguamish Tribe•Vancouver Island University•(WA Dept. of Health)

21

NVS Shellfish Growers Web App

22

NVS Shellfish Growers Web App

23

2015 Partnership with Washington Department of Health

24

Local observation providers:Relationship, Benefits

• Incentives, motivations (esp. when no funding is involved)– Online access to their data (RA tools as Online access to their data (RA tools as

supplemental or main data access)supplemental or main data access)– Wider data dissemination and recognitionWider data dissemination and recognition– Provider data available in broader context, with Provider data available in broader context, with

similar data from other providerssimilar data from other providers– Regional mechanism for platform status-update Regional mechanism for platform status-update

broadcastingbroadcasting– Support for national networks with minimal Support for national networks with minimal

overheadoverhead• Challenges

– Everyone is stretched thinEveryone is stretched thin– Building closer ties, support, feedbackBuilding closer ties, support, feedback

25

• Minimal requirements for participation– Make data files or services available to NANOOS, in Make data files or services available to NANOOS, in any any

accessible format (even emailed zipped MS Access files!)accessible format (even emailed zipped MS Access files!)

– NANOOS handles standardization / match-upsNANOOS handles standardization / match-ups

– Respond to queries about status, updates, etcRespond to queries about status, updates, etc

• Distributed team, closer to range of users– Leverage distributed expertiseLeverage distributed expertise

– Bridges to NVS from local providers or specialized data Bridges to NVS from local providers or specialized data (forecast models, remote sensing, climatologies)(forecast models, remote sensing, climatologies)

– Enable greater productivity for group as a whole, Enable greater productivity for group as a whole, diversity of feedback and perspectivesdiversity of feedback and perspectives

Approach also applied to models

Local observation providers:Making it work

26

Web Services Ecosystem

• From NVS MySQL Database (mainly near-real-time) to:

– Light-weight JSON services > [NVS Web Apps], [Mobile Apps (data, Light-weight JSON services > [NVS Web Apps], [Mobile Apps (data, plots, UI elements)]plots, UI elements)]

– PostgreSQL > IOOS 52North SOSPostgreSQL > IOOS 52North SOS

– PostgreSQL > “Situational Awareness” (GeoServer OGC WMS, WFS)PostgreSQL > “Situational Awareness” (GeoServer OGC WMS, WFS)

– NDBC data pushNDBC data push

Models, remote sensing, temporal map data products

– Overlay map tile serviceOverlay map tile service

– THREDDS / OPeNDAP / WMSTHREDDS / OPeNDAP / WMS

Long time series and NCEI archiving (under construction)

– NetCDF > [THREDDS (> ERDDAP?) > NVS, users], [NCEI archives]NetCDF > [THREDDS (> ERDDAP?) > NVS, users], [NCEI archives]

Map layers

– GeoServer (OGC WMS, WFS, KML, etc)GeoServer (OGC WMS, WFS, KML, etc)

● HF Radar Network

● Glider data (NetCDF)

– IOOS Glider DACIOOS Glider DAC

– NDBCNDBC

– THREDDS THREDDS (under construction)(under construction)

27

Benefits: Leverage NANOOS Data and Analysis Tools

VIU added Nov 14

28

Benefits: Leverage NANOOS Data and Analysis Tools

VIU added Nov 14

29

Benefits: Integration into other Regional efforts: ERMA PNW / Encyclopedia of Puget Sound

30

Benefits: West Coast Ocean AcidificationMonitoring, with WCGA, C-CAN, IOOS, RA's

31

Benefits: West Coast Ocean AcidificationMonitoring, with WCGA, C-CAN, IOOS, RA's

32

Benefits: National Data Services and Integration:Vancouver Island University Deep Bay in US IOOS Catalog (currently offline)

33

Data Management And Communication: Regional Components of a National Network

• Local/regional presence in close interactions with local stakeholders and potential data providers

• Strongly complementary partnership• Formal, close linkages to national coordinating network (IOOS)• Nimble: Reacting to changing needs, emerging opportunities,

informal partnerships• Neutral party• Pooled diverse, technical expertise, with ready access to

national Regional Association peers for technical consultation, best practices

• Interoperable, open data and web service standards, coupled with exposure of data to wide community of users through user-friendly applications

• Value of incremental but steady progress

34

Thank You