66
The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9, 2002 MIDDLEWARE SESSION

The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

Embed Size (px)

Citation preview

Page 1: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

The Mars Odyssey Middleware Architecture

for the Planetary Data System

J. Steven Hughes

Daniel Crichton

Quality Mission Software Workshop (QMSW)

May 7-9, 2002

MIDDLEWARE SESSION

Page 2: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

2

Outline

• An Overview– The Data– The Data Model– The Organization

• The Distribution Problem• A New Approach• The Architecture

– Product Servers– Profile Servers– Middleware

• Conclusion

Page 3: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

An Overview

Page 4: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

4

The Data

• Variety and Volume– 5TB of data from 30 years of exploration– ~700 Data Sets– ~1700 Archive Volumes – CD/DVDs– Camera, Spectrometer, LECP, SAR, RS, …– Images, Time_Series, Spectra, Qubes– Binary and ASCII– Spacecraft and Earth Based– Many data representations– Maintain original bits and emulate as needed

Page 5: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

The Data Model

Level Group/Element Structure_________________________________________

1 spacecraft instrument identification group 2 instrument identification 2 instrument name 2 spacecraft identification 2 instrument type1 instrument description

...1 filter group 2 filter name 2 filter number 2 filter type

...

TargetData Set describehas

produce

InstrumentSpacecraft

Reference

instinfo

instid instname ...insttype scid

OBJECT = INSTRUMENT INSTRUMENT_ID = VISA SCID = VO1 INSTRUMENT_NAME = VISUAL_IMAGING... INSTRUMENT_TYPE = VIDICON_CAMERA ...END_OBJECT

Page 6: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

An Image Label

DATA_SET_ID = "VO1/VO2-M-VIS-5-DIM-V1.0"SPACECRAFT_NAME = {VIKING_ORBITER_1, ...TARGET_NAME = MARSIMAGE_ID = MG88S045^IMAGE = 2SOURCE_IMAGE_ID = {"383B23", "421B23", ...INSTRUMENT_NAME = {VISUAL_IMAGING_SUBSYSTEM ... NOTE = "MARS DIGITAL IMAGE ...

OBJECT = IMAGE LINES = 160 LINE_SAMPLES = 252 SAMPLE_TYPE = UNSIGNED_INTEGER SAMPLE_BITS = 8 SAMPLE_BIT_MASK = 2#11111111# CHECKSUM = 2636242END_OBJECT

Page 7: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

Categories of Meta-Data

Data Represenation•Data Representation

•ITEM_TYPE = VAX_INTEGER•File Attributes

•RECORD_TYPE = FIXED_LENGTH•RECORD_BYTES = 252

•Data Organization•LINES = 160•LINE_SAMPLES = 252•SAMPLE_TYPE = UNSIGNED_INTEGER•SAMPLE_BITS = 8

Catalog •Identification

•DATA_SET_ID = "VO1/VO2-M-VIS-5-DIM-V1.0•IMAGE_ID = MG88S045•INSTRUMENT_NAME = {VISUAL_IMAGING_...

•Observation Context•FILTER_NAME = CLEAR

•Interpretation•MAP_PROJECTION_TYPE =SINUSOIDAL•MAP_RESOLUTION =64<PIXEL/DEG>

Page 8: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

Iowa

Stanford

ATMOSPHERESGISS

ARC

GSFCCENTRAL NODE

JPL

New MexicoState

University

RINGS

AmesResearch

Center

NAIF JPL

PLANETARYPLASMA INTERACTIONS

UCLA

Consortium

of PDS Nodes

GEOSCIENCES

Stanford

MIT

WashingtonUniversity

Brown/Hawaii

GermanyAustria

Wash U

SMALL BODIES

University of Maryland

UofAZ

UofHI

Budapest

PSI

IMAGING

JPL

USGS –Flagstaff, AZ

UCLA

Stanford

Stanford

UV (tbd)

Johns Hopkins

Stanford

The Organization

Page 9: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

The Distribution Problem

Page 10: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

10

Current PDS Data Distribution

• Receive archive copies of data from flight projects

– PDS helps planetary missions to create high quality data archives and to release them in a timely manner

• Validate data for compliance to PDS standards

– PDS assembles, publishes, distributes, and maintains peer-reviewed, documented planetary data sets

• Generate CD-ROMs or DVD-ROMs for archive and distribution

• Distribute data to the community on media; data also available on-line

Page 11: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

11

Current PDS Online Data Access

• Data product search and retrieval– Discipline expertise is local, at the nodes– Data placed online in local repositories– Product catalogs developed for local repositories

• Voyager images (50 attributes)• Galileo images (>100 attributes)• MGS MOC images (41 attributes)

– Custom user interfaces developed for local repositories– Primary customers are discipline oriented

• Data set search– Search for data sets across the entire archive is relatively easy

because of the single high level model

Page 12: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

12

The Problems

• Large volumes of data will not allow the distribution of hard media due to media costs

• There exists no high level global data product search

Page 13: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

13

Drivers for PDS On-Line Data Distribution

• Increased data volumes too much for PDS CD distribution system

– MGS CD costs exceeded $500K for 600GB

– THEMIS alone will produce over 4TB of data

– CD/DVD distribution too costly and cumbersome

• Data will be physically distributed among various sites and sources

• Scientists want online access to all available data

CD/DVD Distribution Costs

0

200

400

600

800

1000

1200

1400

1600

1800

Fiscal 1998 Fiscal 1999 Fiscal 2000 THEMIS only

Distribution

Do

llars

K

Mars Data Only

Non-Mars Data

Page 14: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

14

Mars Exploration Program: Very Large Data Volumes

Growth in Size of PDS Archive

5 10

300

0

50

100

150

200

250

300

350

2001 (Current) 2003 (Mars Odyssey) 2007(MRO)

Year

Terabytes

Archive

Page 15: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

A New Approach

Page 16: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

16

New PDS Data Distribution Approach: Goals

• Online access should be the primary method for data distribution, with improved tools to support it– Find out what data exist

– Select data of interest

– Retrieve data

– Correlate data across instruments, missions, and nodes

• Data should be made publicly available as soon as possible

• Copies on physical media should be available on demand

• Special collections containing data of high interest should be published on physical media

• Copies of complete data sets on cost-effective physical media for long term archives at PDS and NSSDC (minimum 3 copies) will still be required

Page 17: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

17

New PDS Data Distribution Approach: Benefits

• Access for all PDS Mars data across all PDS Nodes

– Design a system to scale to all Mars data

– Begin with Odyssey• Incorporate THEMIS team as a PDS data node

– Ultimately include entire archive

• Benefits of online access

– Reduces distribution costs

– Provides cross-comparability of data across missions

– Moves toward a usable Mars data base

– Allows users who need media to download data

electronically and write CD/DVD on demand

Page 18: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

18

The Plan

• Implement multi-tiered architecture– Clients

– Middleware

– Data and Metadata Services

• Use existing PDS subsystems– Custom User Interfaces– Data repositories– Catalog databases– Remain geographically distributed and locally managed

• Use archive metadata to its full potential

Page 19: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

The Architecture

Page 20: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

20

Information Architecture in a Nutshell

• Profiles describe and provide location for anything of interest– Things of interest

• Data Sets, Data Set Browsers, Data Products, Volumes, Websites, Web Applications, etc

– Written as XML documents– Grouped

• Data Product profiles grouped by data set• All others grouped into single database

– Provide sufficient information to describe and locate resources– Helps determines if the resource can resolve a user query

• Profile Servers serve profiles– Allow search and retrieval of profiles– Distributed for local management and scalability– Access profile databases

• Static profiles stored as XML documents• Dynamic profiles generated from information stored in databases

Page 21: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

21

System Architecture in a Nutshell (cont)

• Product Servers serve Data Products– Data Products are served from data repositories– Input: unique product identifiers– Output: products in the requested formats

• Middleware– Uses XML documents for communication

• Common language and protocols• XML profiles for resource descriptions• XMLQUERY for queries

– Implements message passing protocol for distributed processing

– Web service encapsulation of existing resources• Product servers for data repositories• Profile servers for catalogs

Page 22: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

22

VolumeServer

ProductServers

QueryManager

Single Point of Entry Interface(Data Set View)

Default Data SetBrowsers

In:QueryOut::Data Products

In:QueryOut::Profiles

In:QueryOut::Data Products Data Products

Virtual Volume

OODTMiddleware Virtual Volume

Physical Volume DVDVolume

In:QueryOut::Profiles

PDS-D (Distribution) D01

Custom Data SetBrowsers

In:QueryOut::Data Products

ProfileServers

MARIE,PDS PPI

Data Productsand Metadata

Documents andAncillary FilesPDS CN

GRSPDS GEO

THEMISASU

Radio SciencePDS GEO

SPICEPDS NAIFACCEL

PDS ATMOS

Page 23: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

23

Product Servers

• Product Servers serve archive data products– Provide a standard interface to data product repositories for

the retrieval of data products

• Java server– Generic server receives and returns XML queries– Data source interface extends server– Multiple communication path besides XML

• Scalable Implementation– Performance– Deployment

• Automated ability to query status of product servers across a community

• Automated ability to automatically update software– Less time required to add new product interfaces– Support for a variety of backend data system interfaces

Page 24: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

24

Product Server Requirements

• A product server shall retrieve data products from a data repository

• For retrieval, a product server shall return the data product identified by a unique product identifier. (e.g. volume_id/directory path/ file name/ file extension pointing to the PDS label file.)

• A product server shall return the data product in the specified format. (based on local requirements)

Page 25: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

25

DistributedProductServers

HTTP, IIOP, Java, C++

APIs

THEMISData Set

Data Source InterfaceFor

Dynamically Loaded Query Handlers

Java Server Framework

File system access

GRSData Set

Java Server Framework

Database access

XMLQUERY

Data Source InterfaceFor

Dynamically Loaded Query Handlers

HTTP, IIOP, Java, C++

APIs

XMLQUERY

DistributedDataRepositories

Product Server Architecture

Page 26: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

26

DistributedProductServers

HTMLUser Interface

PDS JAVAUser Interface

PDS C++User Interface

GRSData Set

GRSProduct ServerGeoscience

THEMISData Set

THEMISProduct ServerASU

Middleware

IDLUser Interface

ISYSUser Interface

ENVIUser Interface

RSData Set

RSProduct ServerGeoscience

DistributedClients

Product Server Interfaces

Page 27: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

27

Product Server PDS-D D01 Implementations

• THEMIS - PDS.ASU.PRODSERV.ODY_M_THEMIS to be used by the Atlas

• GRS - PDS.GEO.PRODSERV.ODY_M_GRS to be used by the Atlas• MARIE - DITDOS• SPICE - DITDOS?• ACCEL - PDS.ATMOS.PRODSERV.ODY_M_ACCEL

to be used by a Default Browser• RS - PDS.GEO.PRODSERV.ODY_M_RS to be used by a Default Browser

Page 28: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

28

Product Server Query

THEMISData Set

Product ServerPDS.ASU.PRODSERV.ODY_M_THEMIS

Data Source InterfaceFor

Dynamically Loaded Query Handlers

Java Server Framework

File system access

http://starbrite.jpl.nasa.gov/servlet/jpl.oodt.servlets.Product_Servlet?

keywordQuery={FILE_SPECIFICATION_NAME=/ody_2001/xxx/H0133.DAT

&object= PDS.ASU.PRODSERV.ODY_M_THEMIS

&mimeType=octet & mimeType=*/*}

Client

IIOP://PDS.ASU.PRODSERV.ODY_M_THEMIS Message = XMLQuery [{FILE_SPECIFICATION_NAME=/ody_2001/xxx/H0133.DAT}]

WebServerhttp://starbrite.jpl.nasa.gov/servlet/jpl.oodt.servlets.Product_Servlet

ResultXMLQuery [{FILE_SPECIFICATION_NAME=/…},MIME(BASE64(ZIP(H0133.DAT))]

Page 29: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

29

XML Query Example (1 of 2)

<query> <queryAttributes> <queryId>1.2.3.4…</queryId> <queryTitle>Query Example</queryTitle> <queryDesc>Query for INSTRUMENT_ID = HEND</queryDesc> <queryType>QUERY</queryType> <queryStatusId>ACTIVE</queryStatusId> <querySecurityType>UNKNOWN</querySecurityType> <queryRevisionNote>2000-05-12 JSH V1.2 Updated for new prof.dtd</queryRevisionNote> <queryDataDictId>1.2.3.5…</queryDataDictId> </queryAttributes> <queryResultModeId>ATTRIBUTE</queryResultModeId> <queryPropogationType>BROADCAST</queryPropogationType> <queryPropogationLevels>N/A</queryPropogationLevels> <queryMaxResults>100</queryMaxResults><queryResults>0</queryResults> <queryKWQString>INSTRUMENT_ID = HEND</queryKWQString>

Page 30: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

30

XML Query Example (2 of 2)

<querySelectSet></querySelectSet> <queryFromSet></queryFromSet> <queryWhereSet> <queryElement> <tokenRole>elemName</tokenRole> <tokenValue>INSTRUMENT_ID</tokenValue> </queryElement> <queryElement> <tokenRole>LITERAL</tokenRole> <tokenValue>HEND</tokenValue> </queryElement> <queryElement> <tokenRole>RELOP</tokenRole> <tokenValue>EQ</tokenValue> </queryElement> </queryWhereSet> <queryResultSet>….</queryResultSet></query>

Page 31: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

31

Profiles

• Profiles describe and provide the location for anything of interest– Things of interest (data and system resources)

• Data Sets, Data Set Browsers, Data Products, Volumes, Websites, Web Applications, etc

– Written as XML documents– Grouped

• Data Product profiles grouped by data set• All others grouped into single database

– Provide sufficient information to determine if the resource can resolve a user query

Page 32: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

32

Profiles (cont)

• The profAttributes (Profile Attributes) section describes the profile.

• The resAttributes (Resource Attributes) section describes the resource using a common set of keywords (Dublin Core)

• The profElement (Profile Element) section describes the resource using domain specific attributes (PDS keywords)

Page 33: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

33

PROFILE DTD

<!ELEMENT profiles (profile*)>

<!ELEMENT profile (profAttributes, resAttributes, profElement*)>

<!ELEMENT profAttributes (profId, profVersion?, profType, profStatusId, profSecurityType?, profParentId?, profChildId*, profRegAuthority?, profRevisionNote*, profDataDictId?)>

<!ELEMENT resAttributes (Identifier, Title?, Format*, Description?, Creator*, Subject*, Publisher*, Contributor*, Date*, Type*, Source*, Language*, Relation*, Coverage*, Rights*, resContext+, resAggregation?, resClass, resLocation*)>

<!ELEMENT profElement (elemId?, elemName, elemDesc?, elemType?, elemUnit?, elemEnumFlag, (elemValue* | (elemMinValue, elemMaxValue)), elemSynonym*, elemObligation?, elemMaxOccurrence?, elemComment?)>

Page 34: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

34

Data Product Profile (ODYSSEY HEND)-<profile> -<profAttributes> <profId>1.3.6.1.4.1.1306.2.104.10018791</profId> <profVersion>null</profVersion> <profType>profile</profType> </profAttributes> -<resAttributes> <Identifier>ODY-M-HEND-EDR-2-V1.0+H0133</Identifier> <Title>Data_Set_Name: ODYSSEY-MARS-HEND-EDR-2-V1.0 +

Product_Id:H0133</Title> <Description>null</Description> <resContext>NASA.PDS</resContext> <resAggregation>null</resAggregation> <resClass>data.granule</resClass> <resLocation>iiop://PDS.ProfServer.GEO.ODY.GRS</resLocation> </resAttributes> -<profElement> <elemName>FILE_SPECIFICATION_NAME</elemName> <elemValue>/ody_2001/xxx/H0133.DAT</elemValue> </profElement> -<profElement> <elemName>INSTRUMENT_ID</elemName> <elemValue>HEND</elemValue> </profElement>

Page 35: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

35

Profile Server

• Profile Servers serve profiles– Allow search and retrieval of profiles– Retrieves from profile databases

• Static profiles stored as XML documents• Dynamic profiles generated from information stored in databases

– Distributed for local management and scalability

Page 36: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

36

Profile Server Requirements

• A profile server shall search and retrieve profiles from a profile database

• For search, a profile server shall allow any profile attribute as a query constraint. Profile attributes include those from the profile element, resource attribute, and profile attribute sections of the profile document.

• For retrieval, a profile server shall return matching profiles. The user can request the complete profile or any subset of the profile.

Page 37: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

37

Profile Server Specialization

• A profile server can be either Type A or Type B

• A type A (static profile) profile server, searches and retrieves profiles stored as XML documents.

• A type B (dynamic profile) profile server stores the profile information in a database (e.g. relational). It formats search results into XML document formats. Type B profile servers are appropriate as interfaces to traditional catalogs.

Page 38: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

38

Profile Server Interfaces

• Accepts an XMLQUERY XML document as input

• Optional servlet to convert keyword/value query to XMLQuery

Page 39: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

39

Object Oriented Data Technology (OODT)Principles

• Location Independence• Information Hiding• Model driven architecture

– Decouple solutions from platform specific technology– Adopted by OMG– Allows data architecture and technology architecture to evolve

independently– The technology architecture supports the data architecture (i.e. it

supports the model)• Scalable, Extensible• Client APIs for locating and accessing distributed data products

Page 40: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

40

OODT Architecture

• Address two key architectural focuses

– Technology Architecture• Basic communication between geographically distributed data systems• Framework for plugging in new data systems

– Data Architecture• Standard method for describing data (profile)• Standard method for exchanging data (XMLQuery)

Page 41: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

41

OODT Technology Architecture

• Implemented in Java

• Data layer implemented with XML

• Uses CORBA for remote object invocation– JacORB

• SSL for data encryption

• Uses a standardized XML DTD (XMLQuery) as the messaging format (See oodt/dtd/xmlquery.dtd)

• Supports a variety of client access methods– Java API– HTTP– C++ API

Page 42: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

42

Conceptual PDS Implementation

Query Server

Node 1Profile Server

Node 2Profile Server

Node 2Product Server

XMLQuery

XMLQuery

XMLQuery

XM

LQ

ue

ry

XM

LQ

ue

ry

Node 1 Catalog

Que

ryC

lien

tWeb

se

rver

Plu

gin

s

Web

Ser

ver

Node 1Product Server

XMLQuery

Node 1 Products

Node 2 Catalog

Node 2 Products

Web I/F

Desktop I/F

XMLQuery

XMLQuery

XMLQuery

XMLQuery

Name Server

CentralNode

ClientEnvironment Discipline

Node

Page 43: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

43

About OODT

• Object Oriented Data Technology (OODT) was funded starting in 1998 by the Office of Space Science to address future data system architectures and interoperability needs. It's major focus has been the capture, location, access and exchange of distributed data products.

• OODT has focused on PDS needs as a key customer– Input from the ISAIA project– Specifically addresses distributed heterogeneous data system

interoperability issues– Prototype developed to demonstrate applicability to PDS– PDS-D D01 implementation

Page 44: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

44

About OODT (cont)• OODT has addressed needs across a wide variety of science

disciplines and is working with several groups– National Institutes of Health– National Cancer Institute (Early Detection Research Network)– Seawinds, Quikscat– DSMS standards program (Peter Shames area)– Virtual Pediatric Intensive Care Unit (pilot data sharing architecture

between Children's Hospital LA and Johns Hopkins Medical Institute)

– JPL Institutional Computing and Information Services for enterprise data management services

• Research in simulation and modeling

Page 45: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

45

PDS-D Data Distribution

• PDS-D designers examined three ways to “get to the data:”– Subscription/Notification - data or notification sent to users– Global Product View - product-level search across all data sets in

the PDS– Data Set View - ability to browse and access data within the

context of a data set (or a small number of data sets)

• In D01: Subscription/Notification is implemented as Notification

• Global Product View is difficult– Technical problems: cartographic registration, object model

deficiencies, keywords depending on context– Political problems: CN prohibited from product level search– In D01: No Global Product View

• In D01: Implements the Data Set View for all Odyssey data sets and all PDS 3 compliant archived data sets

Page 46: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

46

Data Set View - Description• A data set search provides a

web-based interface for finding data sets based on data set-related parameters, such as MISSION, TARGET, etc.

• The data set search returns a list of data sets (see drawing).

• For each data set in the list, the user has the following options:– Get information about the data

set– Use a “Data Set Browser” to

find and retrieve data and ancillary files

– Use other resources (e.g., Map-a-Planet) to view the data set

• A product server retrieves the data for the user.

Page 47: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

47

Data Set View – Proof of Concept Screen 1

DRAFT

Page 48: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

48

Data Set View – Proof of Concept Screen 2

Page 49: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

49

Data Set View – Proof of Concept Screen 3

Page 50: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

50

Data Set Browser (Custom) – Screen 1

Page 51: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

51

Data Set Browser

• A web-based user interface that enables the user to find and retrieve data products and ancillary files from a single data set (or a small number of data sets)

• Vary in design, as they are designed for specific data sets.– A very small data set with only a handful of data products may be a simple

HTML framework that directly accesses data directly through a product server

– A very large data set with thousands of data products may offer a sophisticated, multi-tiered, data set search capability, as shown below:

• In D01, there will be a default data set browser automatically generated for every PDS 3-compliant data set in the archive

• The discipline nodes that are the curators of a data set have control over the interface to that data set. And therefore, they may substitute a custom data set browser for the default data set browser.

Page 52: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

52

Scalability

• To add NEW data repository to PDS-D– Enable data product search and retrieval

• Develop catalog• Install product server and profile server• Generate default or develop custom data set browser

– Enable Data Set Search• Add profiles (Data set, Data Set Browser, Ancillary Resources)

• Data space remains partitioned as needed by discipline, expertise, geography, facilities, etc

• System components remain distributed as needed by discipline, expertise, geography, facilities, performance, etc

Page 53: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

53

Scalability (cont)• Number of system component interconnections increases

linearly– One systematic connection required from each component to

middleware– Exponential number of inter-operational connections made

dynamically via message passing

• Since distribution system is built as a light layer on top of the archive system, it will scale as long as the archive system scales

Page 54: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

54

Simple Correlative Searches

• Background (the mysterious stuff)– Resources

• Can be data products, data set browsers, product servers, and profile servers

– Profiles• Profiles describe data products using information from label keywords• Profiles for data sets are the union of the data product profiles• Profiles for resources associated with data sets are equivalent to the data set

profiles (e.g. product server)• Profiles contain sufficient information to determine whether the resource can

resolve an incoming query• Provide sufficient information for on-the-fly user interfaces

– Profile servers• Allow search and retrieval of profiles• Profiles for profile servers are equivalent to the profiles of associated data sets

and product servers

Page 55: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

55

Simple Correlative Searches (cont)

• Example Scenario– Query enters into system– Query is used to identify all profile servers that can resolve the query– Query is broadcasted to the identified profile servers– Resulting data product profiles are collected into solution set– Product ids, links to data product profiles, and links to product servers

serving the products are available for display in user interface

• Assumptions– A set of profile servers profile all data products of interest– A top level profile server contains profiles for all such profile servers

Submit a query that describes what you want and system will respond with descriptions of all matching resources

Page 56: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

Conclusion

Page 57: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

57

Status

• First release scheduled for July 22• Operational release at end of September• After two months of implementation

– Not all data products are designed– Few data products are available– PDS system resource upgrades on track for archive, search,

and retrieval – Middleware installation on track

Page 58: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

58

Lessons Learned

• Technology is not the problem– Integration of the OODT middleware into the PDS will have

little impact on existing PDS subsystems– The PDS can evolve its technology base

• The data model is the real problem

Page 59: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

59

Lessons Learned

• The Importance of a Formal Model• Defines context for user• Defines context for search• Maintains consistency• Supports validation

• The Model Fitting Problem• Meta-data collection involves work• Meta-data must be modified to fit model• Model must evolve

• The Importance of Meta-Data Validation• Critical for consistency• Ensures usefulness of data• Builds user confidence

Page 60: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

Backup

Page 61: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

61

Governing Standards

• PDS Standards Reference • Planetary Science Data Dictionary• W3C Extensible Markup Language (XML) 1.0 (Second Edition)

(Recommendation 6 October 2000)• W3C HTML 4.01 Specification (Recommendation 24 December

1999)• W3C XHTMLT 1.0: The Extensible HyperText Markup

Language (Recommendation 26 January 2000)• ISO/IEC 11179 - Specification and Standardization of Data

Elements, Parts 1-6, ISO/IEC specification, http://www.iso.ch/iso.

• Dublin Core Metadata Initiative. The Dublin Core Element Set Version 1.1, Dublin: DMCI, July 1999.

Page 62: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

62

References• OODT Papers (http://oodt.jpl.nasa.gov/doc/papers)

• “Science Search and Retrieval using XML” by OODT Team. Presented at Second National Conference on Scientific and Technical Data, National Academy of Sciences, Washington D.C. March 2000.

• “A Distributed Component Framework for Science Data Product Interoperability” by OODT Team. 17th Annual International CODATA conference. Baveno, Italy. October 2000.

• “A Component Framework Supporting Peer Services for Space Data Management” by OODT Team, IEEE Space Data SystemsBig Sky Montana, March 2002.

• OODT Presentations (http://oodt.jpl.nasa.gov/doc/presentations)

• JPL Enterprise Data Architecture White Paper (e-mail: [email protected])

• Hughes, J.S., et al. “A Multi-Discipline Metadata Registry for Science Interoperability,” Santa Fe: Open Forum on Metadata Registries, ISO/IEC JTC1/SC32, Data Management and Interchange, 2000.

• Federal CIO Statement on Metadata - http://www.cio.gov/docs/metadata.htm

Page 63: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

63

XML Profile ExampleMDIM Data Set

<profile> <profAttributes> <profId>1.3.6.1.4.1.1306.2.102</profId> <profType>profile</profType> </profAttributes> <resAttributes> <Identifier>VO1/VO2-M-VIS-5-DIM-V1.0</Identifier> <Title>VO1/VO2_MARS_VISUAL_IMAGING_SUBSYSTEM_DIGITAL… <Format>text/html</Format> <resContext>NASA.PDS</resContext> <resClass>data.dataSet</resClass> <resLocation>http://pdsproto.jpl.nasa.gov/catalog/dataset/Resultsds.CFM?… </resAttributes>

Page 64: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

64

XML Profile ExampleMDIM Data Set (cont)

<profElement> <elemName>MISSION_NAME</elemName> <elemType>ENUMERATION</elemType> <elemValue>VIKING</elemValue> </profElement><profElement> <elemName>TARGET_NAME</elemName> <elemType>ENUMERATION</elemType> <elemValue>MARS</elemValue></profElement> <profElement> <elemName>MAXIMUM_LATITUDE</elemName> <elemType>REAL</elemType> <elemMinValue>-87.50000</elemMinValue> <elemMaxValue>90.00000</elemMaxValue> </profElement>

Page 65: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

65

XML Profile ExampleProduct Server for MDIM Data Set

<profile> <profAttributes> <profId>1.3.6.1.4.1.1306.2.54</profId> <profDataDictId>1.3.6.1.4.1.1306.2.10</profDataDictId> </profAttributes> <resAttributes> <Identifier>PDS Product Server</Identifier> <Title>PDS Product Server <Format>image/pds</Format> <Language>en</Language> <resContext>PDS</resContext> <resAggregation>data.granule</resAggregation> <resClass>system.productServer</resClass> <resLocation>iiop://JPL.PDS.Product_Server.MDIM</resLocation> </resAttributes>

Page 66: The Mars Odyssey Middleware Architecture for the Planetary Data System J. Steven Hughes Daniel Crichton Quality Mission Software Workshop (QMSW) May 7-9,

66

XML Profile ExampleProduct Server for MDIM Data Set (cont)

<profElement> <elemName>MISSION_NAME</elemName> <elemType>ENUMERATION</elemType> <elemValue>VIKING</elemValue> </profElement><profElement> <elemName>TARGET_NAME</elemName> <elemType>ENUMERATION</elemType> <elemValue>MARS</elemValue></profElement> <profElement> <elemName>MAXIMUM_LATITUDE</elemName> <elemType>REAL</elemType> <elemMinValue>-87.50000</elemMinValue> <elemMaxValue>90.00000</elemMaxValue> </profElement>

Note: Profile Elements for product server are the same as for the data set.