1

Connecting data and services with Web Services

(SECAA/SPDF CDAWeb/SSCweb)Robert M. Candey (Code 632)

Bernard T. Harris (Code 583)

Reine A. Chimiak (Code 583)

David B. Han (Code 586)

Rita C. Johnson (QSS, Code 630)

Colin A. Klipsch (QSS, Code 630)

Tamara J. Kovalick (QSS, Code 630)

Howard A. Leckner (QSS, Code 630)

Michael H. Liu (Raytheon ITSS, Code 630)

Robert E. McGuire (Code 632)

NASA Goddard Space Flight Center, Greenbelt MD 20771

2

Outline

• What we’ve done• Why scientists should care• How to use web services (example calls from IDL, Java,

Javascript and Perl)• How to setup your own web service• Lessons we are learning (and concerns)• Client development

3

SECAA/SPDF providing services today

• Accessible via FTP, HTML browser, and now web services APIs and new clients that use the web services<http://spdf.gsfc.nasa.gov/>

• Space physics data access - Coordinated Data Analysis Web (CDAWeb) and Coordinated Data Analysis System (CDAS)

• World-wide data finding across public FTP sites and in CDAWeb (Public data file finder)– Return URLs to files matching a range of time in many FTP and some

HTTP data sites (filenaming format based on strptime % strings)

– Example XML (required tags in bold) <dataset ID="ace_mfi_h0_16sec_cdf_cdaweb" nssdc_ID="None” serviceprovider_ID="ac_h0_mfi">

<access filenaming="ac_h0_mfi_%Y%m%d_%Q.cdf" protocol="ftp" subdividedby="%Y" timerange_start="1997-09-02 00:00:12" timerange_stop="2004-03-11 23:59:46"> <URL>ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/mfi_h0</URL> </access> </dataset>

• Orbit locations - Satellite Situation Center Web (SSCweb)• Data Format Translation web service (DTWS)• Radbelts (OMNIweb) solar and magnetospheric indices • More to come

4

What are web services?• Services Oriented Architecture (vs. client-server)• Distributed software to software communication,

analogous to older technologies such as RPC, DCOM, CORBA, RMI

• No HTML or human interaction required• Cross-platform and language-independent• Enables others to develop tools and services

leveraging core logic and science data and orbit information

• Everyone can use their own clients/tools

5

Lower level components of VOs as a set of interoperable services

• Strung together in many combinations• Form an integrated system that is much greater than the sum of its parts• Easily extendable• Open to other systems and external applications by using standard

distributed Application Program Interfaces (APIs)• Based on XML and Simple Object Access Protocol [SOAP] standards

and/or HTTP calling interface• Tie together existing data holdings• Standardize and simplify their use• Enable much enhanced interoperability and data analysis.

6

How to use our web services:Example IDL program

• Uses IDL 6.0 IDL-Java bridge feature to call the SSCWeb services

<http://sscweb.gsfc.nasa.gov/WebServices/RsiIdlExample.html>• Get list of satellites on SSCweb and then get orbit locations for 1 day

of Polar in GSE coordinates• Configure IDL-Java bridge provided with the IDL software• Add JAXRPC jar files and sscWebServices-client.jar to CLASSPATH• IDL> .run SSCexample.pro

7

SSCexample.prorpcStub = OBJ_NEW('IDLJavaObject$Static$JAVAX_XML_RPC_STUB', 'javax.xml.rpc.Stub')rpcStub -> getProperty, ENDPOINT_ADDRESS_PROPERTY=endpointAddressProperty sscWsImpl = OBJ_NEW('IDLJavaObject$SSCWS_IMPL', 'gov.nasa.gsfc.spdf.ssc.SSCWS_Impl')ssc = sscWsImpl -> getSatelliteSituationCenterPort()ssc -> _setProperty, endpointAddressProperty, 'http://sscweb.gsfc.nasa.gov/WS/jaxrpc'sats = ssc -> getAllSatellites()

gregorianCalendar=OBJ_NEW('IDLJavaObject$Static$GREGORIANCALENDAR','java.util.GregorianCalendar')calendar = gregorianCalendar -> getInstance()calendar -> getProperty, DATE=calendarDatePropertyendDate = calendar -> getTime()calendar -> add, calendarDateProperty, -1startDate = calendar -> getTime()coordinateSystem = OBJ_NEW('IDLJavaObject$Static$COORDINATESYSTEM',

'gov.nasa.gsfc.spdf.ssc.CoordinateSystem')coordinateSystem -> getProperty, GSE=coordinateSystemGsePropertysats = ['polar']locInfo = ssc -> getSatelliteLocation(sats, coordinateSystemGseProperty -> toString(), 3L, startDate, endDate)locInfoBeginTime = locInfo[0] -> getBeginTime() locInfoEndTime = locInfo[0] -> getEndTime() coords = locInfo[0] -> getCoordinates()plots, coords[*, 0], coords[*, 1], coords[*, 2]OBJ_DESTROY, rpcStub, sscWsImpl, gregorianCalendar, coordinateSystemEND

8

How to setup your own web service:SECAA implementations

• Server Software– Apache HTTPD– Apache Jakarta Tomcat 5 (triggers on “/WS/” in URL)– Java Runtime Environment (JRE) 1.4.x– Sun JAX-RPC 1.1 runtime libraries (could replace with Java Axis)– Apache Jakarta Commons/Net 1.1.0 library (for FTP access)– CDAS and SSC Web application implementing web services– CDAS and SSC core (existing) software– Services described in WSDL files

• Development Software– Sun Java Web Services Developers Pack (JWSDP) 1.3– Sun Java Developers Kit (JDK) 1.4.2

• Java Client Software– Application– Java Runtime Environment (JRE) 1.4.x– Sun JAX-RPC 1.1 runtime libraries– CDAS/SSC Web service client libraries

9

WSClient

ApacheHTTPD

IDL

our WS Java code

CDASmetadatadatabase

non-CDAWeb FTP/HTTP site

metadata (XML)

ApacheTomcat

CDAWeb Server

RemoteComputer

FTPServer

RemoteComputer

HTTPServer

RemoteComputer

SO

AP

req

uest

/re

spon

se

Get

dir

Getdir

Create

exec

uteH

TT

P G

ET

data

/res

pons

e

CDASdata file/

plot/listing

Readcreate

Get datafile

read

read

get file

10

Lessons we are learning (and concerns)• Our services require underlying standards, although still manual input for each

dataset stored in master CDFs• Many firewalls limit ports available and may impose timeout on sessions (had to

modify Ethernet stack to send Keep-alive packets)• Web service standards are evolving, subject to interpretation, and

implementations may choose which options to support– Different data bind options (RPC-encoded, RPC-literal, document-literal)– Different communication model options (synchronous RPC, asynchronous messaging)– Handling of null Date values (pass by reference [Java] or value [.NET])– Support for passing multi-dimension arrays (ASP .NET)– Handling of empty arrays

• Our Web services were initially implemented before Web Services Interoperability (WS-I) Basic Profile 1.0 and use RPC-encoded binding style. We’ll be changing to document-literal to be compliant with WS-I and .NET.

• Need to be flexible and rewrite code but stay stable for clients (how to update API without breaking existing clients: add new methods?, add whole new API?)

• Difficult error handling (error codes) between applications• Need careful attention to how existing services are done• Created duplicate functionality that we will have to merge software to remove (Perl

interface to call web services)

11

Tie together with clients• SECAA-provided or external clients (own customer of web services and

dependent on service stability)• HTML web page access always for simple access• TIPSOD (Tool for Interactive Plotting, Sonification and 3-D Orbit Display)• CDAS Java client of getting CDAWeb data listings, plots, files (via

CDAWlib) and URLs to files on other systems (via FTP) (functions to be merged soon):

– Original CDAS Client (no FTP support)

• <http://cdaweb/~rchimiak/cdasClient>

– CDAS FTP Client (no CDAWlib support)

• <http://cdaweb/~rchimiak/cdasFtpClient>

• Easy client install (WebStart)• Translator page• VSPO also using CDAS Web services

12

Call to action

• Cooperative identification among major services and clients to aid in understanding usage and various reporting questions: Set user-agent string to identify calling routine/institution in all web services calls

• Try out our services and give us feedback

13

Joint effort of the NASA GSFCSun-Earth Connections Active Archive (SECAA)

in the Space Physics Data Facility (SPDF) and National Space Science Data Center (NSSDC)

See <http://spdf.gsfc.nasa.gov/> for links to these tools and services

or contact Robert.M.Candey@nasa.gov

14

Distributed components of a space physics virtual observatory

• Work together via standard interfaces and metadata agreements to form a system

• Appears as a single super-instrument providing geophysical measurements and models across time and space

• Enables researchers to easily and seamlessly analyze data from many more sources than possible before

15

Tie together services

One scenario:• Search bowshock event database• Query indices database for times of no sun spots• Correlate with spacecraft on same field lines• Search for data files• Translate files into preferred format• Plot

16

Simple Object Access Protocol (SOAP)

• An XML-based protocol for exchanging information in a distributed environment

• Defines a way to perform remote procedure calls (RPCs) - method name and a list of parameters

• HTTP is the most popular option for data transport (since it is simple, stable, and widely deployed), but not mandatory

• SOAP messages masqueraded as HTTP messages can pass through some firewalls - can be good or bad

• Supported in C, C++, Java, C#, Perl, Python, and JavaScript

17

Web Services Definition Language (WSDL)

• XML description of web service purpose, calling methods and location• WSDL file specifying web service can describe both SOAP and CGI

interfaces• CGI calls are lighter and less opaque but not as powerful as SOAP

calls. See REST discussions by Mark Baker <http://webservices.devchannel.org/article.pl?sid=02/12/10/1849249>, Paul Prescod <http://www.prescod.net/> and <http://www.xml.com/lpt/a/2002/02/06/rest.html>

• Example CDAWeb CGI call for plots of the lat, long and altitude variables for NOAA5/6 datasets for 1 day:

<http://cdaweb.gsfc.nasa.gov/cgi-bin/cdaweb/eval3.cgi?dataset=NOAA05_H0_SEM+NOAA06_H0_SEM&start=1980%2F11%2F17+03%3A00%3A00&stop=1980%2F11%2F18+03%3A00%3A00&auth=hello&index=radiation&action=plot&var=NOAA05_H0_SEM+Lat&var=NOAA05_H0_SEM+Lon&var=NOAA05_H0_SEM+Alt&var=NOAA06_H0_SEM+Lat&var=NOAA06_H0_SEM+Lon&var=NOAA06_H0_SEM+Alt

18

Coordinated Data Analysis Web (CDAWeb)

• HTML user interface supports graphics, digital listings, file retrieval and merged/subsetted CDF creation, and simultaneous multi-mission, multi-instrument selection and comparison of science data among a wide range of current and past space missions and ground-based facilities

• Now comprises >600 datasets and >1M files of science data• Very popular with 165k user sessions, 94k plots, 622k FTP

requests, 67k ASCII listings, 586 CDF create requests, 2882 file retrieval requests in FY2004

19

Coordinated Data Analysis System (CDAS) Web services• Simultaneous multi-mission, multi-instrument

selection and comparison of science data among a wide range of current space missions from CDAWeb

• Distributed programming interface so you can add access to many space science datasets, listings and plots to your science application

• Higher level web services planned for the near future

20

CDAWeb data available through

• FTP file access, including software and documents

• Web browser

• Web services in addition

• Latter two provide data listings, plots, original data files and combined files

21

Data File Finder• Using just several pieces of metadata, we can return URLs to files

matching a range of time in many FTP and some HTTP data sites• Filenaming format based on strptime % strings• Example XML (required tags in bold) <dataset ID="ace_mfi_h0_16sec_cdf_cdaweb" nssdc_ID="None” serviceprovider_ID="ac_h0_mfi">

<access filenaming="ac_h0_mfi_%Y%m%d_%Q.cdf" protocol="ftp" subdividedby="%Y" timerange_start="1997-09-02 00:00:12" timerange_stop="2004-03-11 23:59:46"> <URL>ftp://cdaweb.gsfc.nasa.gov/pub/istp/ace/mfi_h0</URL> </access> <mission_group ID="ACE" serviceprovider_ID="ACE"><description short="Advanced Composition Explorer"/> </mission_group> <observatory ID="ACE" nssdc_ID="None" serviceprovider_ID="AC"> <description short="Advanced Composition Explorer"/> </observatory><instrument ID="MAG" nssdc_ID="None" serviceprovider_ID="MAG"><description short="ACE Magnetic Field Instrument"/> </instrument><data_producer affiliation="Bartol Research Institute" name="N. Ness" title="None"/><other_info> <link URL="http://www.srl.caltech.edu/ACE/ASC/level2/index.html" title="the ACE Science Center Level 2 Data website">Release notes and other info available at </link> </other_info> <spase:product_ID>ace_mfi_h0_16sec_cdf_spase</spase:product_ID><instrument_type ID="Magnetic Fields (space)" serviceprovider_ID="Magnetic Fields (space)"/><description short="H0 - ACE Magnetic Field 16-Second Level 2 Data - N. Ness (Bartol Research Institute)"/> </dataset>

22

What is Common Data Format (CDF)?

• Self-describing platform-independent scientific data format (object oriented like HDF5)

• Most widely used format in Space Physics– ISTP, Stereo, TIMAS

• Available on all major computer platforms• Interfaces supported: C, Fortran, Java, and Perl• CDF FAQ <http://nssdc.gsfc.nasa.

gov/cdf/html/FAQ.html>

23

Software for CDF

• IDL, MATLAB, IBM DX, FlexPro, Application Visualization System• Goddard-developed Web-based systems leverage CDF power• CDAWlib <http://spdf.gsfc.nasa.gov/CDAWlib.html>

– Powerful set of IDL routines, including reading/writing CDFs– Available at <ftp://cdaweb.gsfc.nasa.gov/pub/CDAWlib/>

• Standard CDF tools– CDFcompare, CDFconvert, CDFedit, CDFexport, CDFstats,

SkeletonTable, SkeletonCDF distributed as part of the standard CDF distribution package

– SKTeditor metadata editor applying the ISTP Guidelines

– MakeCDF converts ASCII and binary custom formats into CDF

24

CDF Markup Language (CDFML)

• XML Markup Language describing Common Data Format (CDF) data and metadata

• CDFML file (XML file) is an ASCII representation of a CDF file

• Used as a mechanism for data interoperability with other science data formats (since translation between data formats via XSL is easy)

25

Data Translation Web Service (DTWS)• Translation between one or more local or remote files into

other data formats• Formats: CDF, netCDF, HDF4, HDF5, FITS, • Interoperability with other data formats using XML: CDF

Markup Language (CDFML), FITSML, XDF Version of CDFML (CDFX-ML), HDF in XML, Native Data Format (NDF) in XML, ASCII (data object mapping via Extensible Style Language (XSL))

• Input: Files to be translated, translation mode• Output: Direct or e-mail notification with URL links to

translated files• <http://nssdc.gsfc.nasa.gov/cdf/html/dtws.html>

26

Satellite Situation Center (SSCweb)• Browser-based service for geocentric spacecraft

locations with geophysical regions and mappings along magnetic field lines

• Supports multiple coordinate systems• Extensive business logic and comprehensive list of

active and past space physics missions• Supports complex multi-satellite and ground

conjunction queries• Outputs listings and graphs• However, SSCweb currently limited:

– Static, 2D GIF images rendered by a browser– Not easy to access from other applications

27

Tool for Interactive Plotting, Sonification and 3-D Orbit Display (TIPSOD)

• Shows spacecraft moving in time and 3D space as an interactive movie, while user changes orientation and scale (zoom and rotation)

• Interfaces with the new flexible SSC XML Web Services (expanding on operational system)

• Exposes relationships between spacecraft visually (in 3D) and temporally

• Easy to download and use (Web Start downloads or updates TIPSOD client, Java and Java 3D with a few clicks from the browser)

• Runs on Windows, Unix, Linux, Mac OS X (requires Java3D)• Additional functions planned, such as sonification and

connections to CDAWeb science data and logic

28

Future and Improved Web Services• Adding more I/O formats to CDAWeb (FITS, HDF, netCDF, IDL

savesets)• Smart client issues (WebStart)• Other web services

– OMNI_near_earth field and plasma data and indices

– ATMOweb ionospheric data

– ModelWeb

– Orbit_regions and cross-spacecraft queries

– COHO_heliospheric data

– NMC_metadata (NSSDC Master Catalog)

– SPASE_dictionary (Space Physics Archive Search and Exchange)

• Community-wide client project? plugins? (NOAA plot tool)• Grid support now that Globus 3 is OGSA/web services compatible

29

Examples of calling web services• Example IDL program (IDL 6.0 IDL-Java bridge)• Example Java client• Example JavaScript• Example Perl script• SSC Web Services <http://sscweb.gsfc.nasa.

gov/WebServices/> • Same techniques used to call CDAS Web

services <http://cdaweb.gsfc.nasa.gov/WebServices/>

30

Example Java clientpackage gov.nasa.gsfc.spdf.ssc.test;import java.io.*;import java.util.Date;import java.util.Calendar;import java.util.GregorianCalendar;import java.text.SimpleDateFormat;import javax.xml.rpc.Stub;import gov.nasa.gsfc.spdf.ssc.*;public class WsExample { public static void main(String args[]) throws Exception { System.setProperty("http.agent", "WsExample (" + System.getProperty("os.name") + " " + System.getProperty("os.arch") + ")"); Stub stub = (Stub)(new SSCWS_Impl().getSatelliteSituationCenterPort()); stub._setProperty(Stub.ENDPOINT_ADDRESS_PROPERTY, args[0]); SatelliteSituationCenter ssc = (SatelliteSituationCenter)stub; String[][] satellites = ssc.getAllSatellites(); for (int i = 0; i < satellites.length; i++) System.out.println(" " + satellites[i][0] + " " + satellites[i][1]); Calendar calendar = GregorianCalendar.getInstance(); Date endDate = calendar.getTime(); calendar.add(Calendar.DATE, -1); Date startDate = calendar.getTime(); String[] sats = new String[] {"polar"}; SatelliteLocation[] locInfo = ssc.getSatelliteLocation(sats, CoordinateSystem.GSE.toString(), 3,

startDate, endDate); double[][] coords = locInfo[0].getCoordinates(); for (int j = 0; j < coords.length; j++) { for (int k = 0; k < coords[j].length; k++) System.out.print(" " + coords[j][k]);} } }

31

Example JavaScript (Mozilla-based browsers)

<script type="text/JavaScript">netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead");var call = new SOAPCall();call.transportURI="http://sscweb.gsfc.nasa.gov/WS/jaxrpc";call.encode(0, "getAllSatellites", "http://sscweb.gsfc.nasa.gov/WS/SSCWS", 0, null, 0, null);var response = call.invoke();if (response.fault == null) { var result = response.getParameters(true, {}); for(i = 0; i != result.length; i++) { var param = result[i]; var name = param.name; document.write("result[" + i + "].name = " + name + "<br>"); var value = param.value; for (j = 0; j < value.length; j++) document.write("value[" + j + "] = " + value[j] + "<br>"); }}</script>

32

Example Perl script#!/usr/local/bin/perl -wuse SOAP::Lite;my $ssc = SOAP::Lite -> proxy('http://sscweb.gsfc.nasa.gov/WS/jaxrpc') -> uri('http://sscweb.gsfc.nasa.gov/WS/SSCWS');my $result = $ssc -> getAllSatellites();unless ($result -> fault) { print "getAllSatellites() returned:\n"; my $satellites = $result->valueof('//result'); foreach $satellite (@$satellites) { foreach $name (@$satellite) { print " $name";} } }my $satVar = SOAP::Data->type('string')->name('String_1')->value('polar');$result = $ssc -> getSatelliteResolution($satVar);unless ($result -> fault) { my $value = $result->valueof('//result'); print "getSatelliteResolution() = $value\n";}$result = $ssc -> getSatelliteTime($satVar);unless ($result -> fault) { my $values = $result->valueof('//result'); print "getSatelliteTime() returned:\n"; print " $$values[0] - $$values[1]\n";}

33

What’s important to you?

• What services and features do you want?

• What other web services are being developed? (VHO, VSO, EGSO, STARS)

• ???

34

SECAA Goals

• Ease of use• Open source, modular and reusable software• Access to multi-spacecraft and multi-instrument data• Performance (reasonable response times)• Broad base of users• ???

35

Importance of Standards and Agreements• Wide variety of science data file formats (CDF, netCDF, HDF, FITS,

etc. and many custom formats) impedes science research across instruments and missions by greatly increasing learning and translating time for each dataset

• Custom binary and ASCII formats can be converted into CDF with the makeCDF tool for possible further conversion to other formats

• Optional mapping templates handles dataset-specific translations• These tools and service will increase interoperability and improve

cross-mission dataset transparency• Cooperative identification among major services and clients to aid in

understanding usage and various reporting questions (set user-agent string)

36

Poster specific stuff

Fall 2004 AGU Special Session SF05 - Data and Services for Space and Earth Sciences

(SF05-9402)

37

Importance of SECAA Web

Services for the Integrated SEC

Data Environment

38

AuthorsRobert M. Candey (Code 632)

Reine A. Chimiak (Code 583)

David B. Han (Code 586)

Bernard T. Harris (Code 583)

Rita C. Johnson (QSS, Code 630)

Colin A. Klipsch (QSS, Code 630)

Tamara J. Kovalick (QSS, Code 630)

Howard A. Leckner (QSS, Code 630)

Michael H. Liu (Raytheon ITSS, Code 630)

Robert E. McGuire (Code 632)

NASA Goddard Space Flight Center, Greenbelt MD 20771

39

AbstractThe Space Physics Data Facility (SPDF) at NASA GSFC has developed new and state-of-the-art, distributed programming (web services) interfaces to SPDF’s space science mission services and data. This work is an essential step in integrating these powerful and popular SPDF software systems (and the NASA mission data they now serve) with external science data and model software systems under a NASA Virtual Observatory paradigm for enhancing the scientific return of space physics research.The distributed components of a space physics virtual observatory work together via standard interfaces and metadata agreements to form a system that appears as a single super-instrument providing geophysical measurements and models across time and space, enabling researchers to easily and seamlessly analyze data from many more sources than possible before. We are providing a critical set of these lower level components, leveraging our data format expertise and our existing and very popular science and orbit data web-based services, Coordinated Data Analysis Web [CDAWeb] and Satellite Situation Center Web [SSCweb], by adding web services for orbit location, data finding across FTP sites and in CDAWeb, data file format translation, and display. These services tie together existing data holdings, standardize and simplify their use, and enable much enhanced interoperability and data analysis. A Java3D application, TIPSOD (Tool for Interactive Plotting, Sonification and 3-D Or!bit Display), developed in-house, makes use of these web services.CDAWeb, SSCWeb, CDF tools, and web services are a joint effort of the Sun-Earth Connections (SEC) Active Archive (SECAA) in the NASA GSFC Space Physics Data Facility (SPDF) and the National Space Science Data Center (NSSDC); see <http://spdf.gsfc.nasa.gov/>.

40

Outline• What we’ve done (virtual observatories (VOs))• Why scientists should care• Overview of technologies (web services, public file service, standards

such as CDF, translators, 3D clients, etc.)• SECAA services (importance to evolution of integrated SEC data

environment for effective multi-mission space physics)• How to use (call from IDL, Java, Javascript and Perl)• How to setup your own web service• Emphasize what’s now working (VSPO, Visbard, ???)• Lessons we are learning• Future• Call to action• Demos (Tipsod and CDAS clients)

41

SECAA SOAP implementations• SOAP uses HTTP calls (port 80) to Apache which sees "/WS/" in the

path and calls Tomcat (or Tomcat use own port but lose logging and other features Apache provides). Other URLs are handled by Apache directly

• Tomcat <http://jakarta.apache.org/tomcat/> calls calls Java SOAP servlet to parse the incoming message, call the underlying tool, and martial the SOAP response <http://ws.apache.org/soap/docs/install/tomcat.html>

• Axis <http://ws.apache.org/axis/> called from Tomcat in place of Sun Jax-RPC