UWG 2013 Meeting Publish-Subscribe (Datacasting)

UWG 2013 MeetingPublish-Subscribe (Datacasting)

Goal PO.DAAC to publish availability of our data

Why? So users can subscribe and no longer have to

crawl FTP

Goal

High Level Functions: Publishing

Consumers

Archive

Ingest

Web Services & Publishing

Direct Data Access

High-Level Access Tools

Web Portal Visualization

Inventory

Information Providers

Data In Information In Data OutInformation Out

Data Providers

Datacasting Extension of RSS Work funded by NASA ACCESS 2005, 2009 Data providers publishes availability of new

data granules via XML feed Data consumers subscribes to feed to learn

about recently available data granules

Solution

Datacasting enables Down select items in feed based on multiple criteria

Temporal Spatial Other metadata

Automatically download selected data files Mash up across different types of feeds

Combine newspaper articles, scientific reports, event analyses, data, etc. in one place

And much more

By the way…

Publishing DatacastingRSS

Consumers

Archive

Ingest

Web Services & Publishing

Direct Data Access

High-Level Access Tools

Web Portal Visualization

Inventory

Information Providers

Data In Information In Data OutInformation Out

Data Providers

Data

cast

ingR

SS

Publish-Subscribe Sequence

DatacastingRSS<rss xmlns:datacasting="http://datacasting.jpl.nasa.gov/datacasting" xmlns:georss="http://www.georss.org/georss" xmlns:gml="http://www.opengis.net/gml" version="2.0"> <channel> <title>MetOp-A ASCAT Level 2 25.0 km Ocean Surface Wind Vectors</title> <link>http://podaac-test.jpl.nasa.gov/dataset/ASCAT-L2-25km</link> <description>This dataset contains operational near-real-time Level 2 ocean surface wind vector retrievals...</description> <pubDate>Wed, 27 Feb 2013 07:49:44 GMT</pubDate> ... <datacasting:channelUID>PODAAC-ASOP2-25X01</datacasting:channelUID> <datacasting:dataSource>METOP-A/ASCAT</datacasting:dataSource> <item> <title>ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc</title> <link>http://podaac-test.jpl.nasa.gov/dataset/ASCAT-L2-25km</link> <description>ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc</description> <enclosure length="847714" type="application/x-netcdf" url="ftp://podaac-ftp.jpl.nasa.gov/allData/ascat/preview/L2/25km/2013/057/ascat_20130226_190001_metopa_32990_eps_o_250_2101_ovw.l2.nc.gz" /> <pubDate>Wed, 27 Feb 2013 07:49:44 GMT</pubDate> ... <georss:where> <gml:Envelope> <gml:lowerCorner>-89.37 0.02</gml:lowerCorner> <gml:upperCorner>89.26 359.99</gml:upperCorner> </gml:Envelope> </georss:where> <datacasting:acquisitionStartDate>Tue, 26 Feb 2013 19:00:01 GMT</datacasting:acquisitionStartDate> <datacasting:acquisitionEndDate>Tue, 26 Feb 2013 20:41:59 GMT</datacasting:acquisitionEndDate> </item> ... <item> ... </item> </channel></rss>

Link to DatacastingRSS Feed

Client Tool: Scripts

Scripts in Python, Perl, MATLAB, etc. can be written to download data granules as they are made available

Sample MATLAB script for downloading data granules:xml = xmlread('http://podaac-test.jpl.nasa.gov/ws/metadata/granule/?format=datacasting&datasetId=PODAAC-ASOP2-25X01');

enclosures = xml.getElementsByTagName('enclosure');

for i = 0:enclosures.getLength-1 url = char(enclosures.item(i).getAttribute('url')); [pathstr, name, ext] = fileparts(url); filename = [name, ext]; if exist(filename, 'file') disp(['File exists: ' filename]); else urlwrite(url,filename); disp(['Downloaded: ' filename]); endend

exit;

Perl tool for subscribing to feeds and downloading and maintaining files on disk

Client Tool:Data::Downloader

% dado config init --filename ./ascat-l2-25km.conf% dado feeds refresh% dado files download

Terminal

Client Tool:Datacasting Feed Reader

Subscribe to DatacastingRSSFilter feed

items Download dataCreate mash up