Satella: Tool for Peer to Peer Landsat

Data Sharing

Bruno Margerin (SSAI)

John Owens and Dr. Chris Justice (UMd)

RationaleHow can scientists share data easily ? we need a way let data users know - who has what data. we need a tool to access and provide data. we want to minimize the burden from those sharing data

We need to reduce the costs of managing NASA data

We need to promote reduce obstacles to increased use of NASA data

The internet provides a mean for electronic data exploration and delivery

Can we take advantage of IT currently being developed in the private sector and apply these advances to our science data needs ?

NASA Data Needs: An exampleNASA has purchased large volumes of Landsat data to support the science community Landsat Pathfinder (Landsat 5) Regional Data Buys Earthsat Data Buy 1990 Landsat 7 Data Purchases for R and A (LCLUC) Landsat 7 Earthsat Data Buy Data 2000

These data are held by individual PI’s – most of whom are willing and able to share data with other scientistsOnce purchased by NASA, these Landsat data can be shared openly for science use We need a way to facilitate data sharing and reduce costs of dataWe need to promote free and open sharing of science dataWe need to move the burden of data management away from centralized systems to more distributed systemsOther opportunities – MODIS Data, ASTER Data, Landsat Derived Products, Model Output

Requirements for the System

A unified method of access to heterogeneous, distributed archives

Better utilization of Internet bandwidth, storage

Low management and maintenance

Easy to install and use

Highly scalable performance

The Satella Concept

A scalable, robust system for data search and distribution over a large collection of heterogeneous providers

A means for individual scientists to obtain and share satellite data

Satella Conceptualization

A distributed, self-organizing network based on peer-to-peer topology

Integrating the data available in individual archives

Through a freely available client software for search and transfer of the data

Satella: Peer 2 Peer SystemPrivileges Local Peer Interactions.Replicates information towards ConsumerDiscover and Adapt dynamically to new information sources.Build complex services by combining simple ones.Increase performance as system ages and more peer participates. Protect data by replicating informationBuild reliability from interchangeable peers

Based on JXTA P2P Protocols and Implementation

Open Network Programming platform for P2P ComputingSet of Protocols for interoperating (XML based)Language and network Agnostic Technology.Open Source Technology: http://www.jxta.orgVirtual network overlayGeneralize the P2P problemSolve key technical challenge (security, stability, scalability ...)Interoperable, Platform independance, ubiquity136,000 Downloads, 40+ Projects, 7500 Members (4/25/2001 – 12/8/2001)

From http://www.jxta.org/community

Software Architecture 1/2Prototype Service software allowing sharing and search for Landsat 7 ETM datasets.

2 Set of Service: Root Peer Service:

Maintains and provides secure access to Satella Peer Group. Provide initial topology information of Net Peer Group and Satella Peer

Group Provide Relay and Rendez Vous capability for Satella Peer Group and

Net Peer Group. Data Peer Service:

Provide P2P File Sharing capability Automated Landsat 7 Metadata creation Search and Download of files based on their metadata.

Fully Open Source

N-Pier Architecture

UDDI/WSDL/SOAP Web Service Ready Architecture.

Software Architecture 2/2

Data Peer Service

Root Peer Service

XML XML

Oth

er P

2P D

ata

& R

oo

t S

vc

Comm. ObjectsRMI Interface

Tomcat Web Server

MVC: Turbine + VelocityRMI + Web Based GUI

RMI

Web Browser

HTTP

Interact

Data Peer Service

Root Peer Service

XML XML

Oth

er P

2P S

vc

Comm. ObjectsWSDL/SOAP Interface

Other Web Service

SOAP

Associate GUIInteract

UD

DI

Satella Development to Date

Phase I:Internal Proof of concept and feasibility. AWT/SWING based GUI Search on File Name Only Basic, download capability. Partially Multi-Threaded.

Phase II: N-Pier Architecture Service Based Architecture 3- Pier Model View Controller (MVC) Web Based GUI Automatic Landsat 7 Metadata Generation Improved Download capability. Search on Landsat 7 Metadata (in progress).

Satella Caveats

JXTA is one competing web services platform (there are other competitors e.g. Microsoft Hailstorm), will it be the eventual winner? Mitigated since Other platforms are less mature especially in the P2P Architecture.Distributed archive depends on participation by potential users/contributors. Success of Napster/Gnutella music file sharing networks illustrate people willing to give as well as take.Programmatic guidance could encourage PIs and providers to play an active role. Satella would not replace the need for a long-term archiving of satellite data but will share the load and reduce the cost of data distribution

Satella: The Way Forward

Future Improvements:Continued improvements for Satella

network performance and reliabilityEvolution to a UDDI/WSDL Web ServiceEnhanced Semantic, distributed searchDownloading of a single file from multiple

available peersOther improvement dictated by the user

community.

Satella project Contributions to the Open Source Community

Involvement (Committer or Contributor) on 9 JXTA projects

Provided tutorial for the creation of a Secure Peer Group on JXTA’s Java implementation build 34c (10-12-2001)

Provided segment “Creating a Secure Peer Group” to “Project JXTA: Java[tm] Programmers Guide”.

In collaboration with Robin Johnson, Steer the architectureof the development of Metadata Support into CMS (Content Management System), JXTA File Sharing engine.

A look at the futureThe costs of data distribution are becoming unaffordable – DAAC’s are struggling to manage the large data volumes - new and innovative solutions are needed to improve data access and reduce costs to NASA and the PI – we need to invest in new approaches and long term solutionsData distribution is low compared to the size of the archives and the potential user community – we need to stimulate data useNASA Science requires regional time-series analysis i.e. large volume data sets giving an increased burden to existing data systems already proving to be unaffordableThe NASA Earthsat Data Buys provide an important asset for the global change research community – need to ensure their widest possible use – extensive data sharing is now feasibleFuture Landsat Continuity Missions will hopefully allow free and open sharing of 30m dataThe user community can and is willing to play an increasing role in data management – we need to develop sustainable ways of doing businessIn the context of the NEWDISS program NASA needs to support innovative use of advances in information technology to solve its data management challenges

Credits:John Owens and Dr. Chris Justice From University of Maryland, Dept of Geography.

Bernard Traversat and Juan C. Soto From Sun Microsystems

All participating JXTA members.

Satella Phase I: SnapShots

Contact Information

Bruno Margerin: bruno@turbodog.gsfc.nasa.gov

John Owens: jowens@umd.edu

Dr. Chris Justice: justice@hermes.geog.umd.edu

Web Site (to come): http://satella.geog.umd.edu