06/30/20031

Data Product Service (DPS)Packaging and Context

Dan Crichton

Steve Hughes

Ron Joyner

Chris Mattman

Paul Ramirez

Peter Shames

06/30/20032

Acknowledgements

Ed Greenburg

Larry Preheim

Steve Levoe

Tom McVittie

Tim M

Erik Barkley

06/30/20033

Data Product Service (DPS)Packaging and Context

• DPS definition: a service that provides reliable delivery of well documented data products that can be directly related to science observation requests

• DPS mission system context– Functions and high level data flows

• Data packaging specification– Generic approach– Applied to DPS problem– Evolving content during processing

• Prototype & deliverables

• Open issues & next steps

06/30/20034

Data Product Service Functional Elements

Science SequenceDevelopment

Acquisition Plan Development Sequence PlanObservation Requests Obs. Definition Obs. Planned Products Obs. Uplink Files

Science MissionPlanning

Objective-CampaignsInstrumentsObservation sets

Integrated SequenceDevelopment

Sequence Plan Development Sequence Plan Sequence Uplink FilesActivity Requests Activity Definition Activity Planned Products Activity Uplink Files

Spacecraft Activities

Sequence Execution Activity Sequence Uplink Files Sequence Events Product Creation Product Notification Product Transfer

Service Management

Service Scheduling Service Possibilities Service Requests Service Schedule PredictsService Monitoring Receive Reports Capture Data Provide User Views Performance Assessment Commitments Operational Visibility

Tracking Services

Scheduled Activities (General) Service Reports Service Notifications Performance ReportsTelemetry Service Frame Accounting ReportCommand Service Radiation Reports

DSMS File Transfer Svc

Telemetry Service Product Accounting/Reports Product Assembly/DeliveryCommand Service Product Accounting/Reports

DSMS ScienceProduct Service

Science Product File(s) Assembly Received Product Product Accounting Cataloging Data Ancillary Data Reporting Processing History??

Observation Planning

Observation Planning Integrated Obs. Descriptions Observation Templates Product Types Product Type Relationships Product Type Definition Ancillary Specification

Engineering SequenceDevelopment

Spacecraft Plan Development Sequence PlanEngineering Requests Eng. Definition Eng. Planned Products Eng. Uplink Files

InformationInfrastructure

XML SchemasInterface SpecificationsDictionary & Repository svc

06/30/20035

Data Product Service Flows

Science SequenceDevelopment

Acquisition Plan Development Sequence PlanObservation Requests Obs. Definition Obs. Planned Products Obs. Uplink Files

Science MissionPlanning

Objective-CampaignsInstrumentsObservation sets

Integrated SequenceDevelopment

Sequence Plan Development Sequence Plan Sequence Uplink FilesActivity Requests Activity Definition Activity Planned Products Activity Uplink Files

Spacecraft Activities

Sequence Execution Activity Sequence Uplink Files Sequence Events Product Creation Product Notification Product Transfer

Service Management

Service Scheduling Service Possibilities Service Requests Service Schedule PredictsService Monitoring Receive Reports Capture Data Provide User Views Performance Assessment Commitments Operational Visibility

Tracking Services

Scheduled Activities (General) Service Reports Service Notifications Performance ReportsTelemetry Service Frame Accounting ReportCommand Service Radiation Reports

DSMS File Transfer Svc

Telemetry Service Product Accounting/Reports Product Assembly/DeliveryCommand Service Product Accounting/Reports

DSMS ScienceProduct Service

Science Product File(s) Assembly Received Product Product Accounting Cataloging Data Ancillary Data Reporting Processing History??

Observation Planning

Observation Planning Integrated Obs. Descriptions Observation Templates Product Types Product Type Relationships Product Type Definition Ancillary Specification

Engineering SequenceDevelopment

Spacecraft Plan Development Sequence PlanEngineering Requests Eng. Definition Eng. Planned Products Eng. Uplink Files

InformationInfrastructure

XML SchemasInterface SpecificationsDictionary & Repository svc

06/30/20036

XML Packaging forScience/Observation Data Objects*

An end-to-end data product production pipeline requires the packaging of data objects, ancillary data, and documentation to address mission support, science analysis, and data archive requirements. This task proposes the development of a generalized standards specification to:– describe the data objects, ancillary data, and documentation

required to support science planning, mission operations, science analysis, and the packaging and cataloging of an archive quality packaged data product

– prescribe the process used to produce a packaged data product.

* A science data object is defined as any digital data file(s) collected by a science instrument or compiled by a scientist that has science content. A data product is a searchable, retrievable, and orderable package consisting of one or more data objects.

06/30/20037

XML Packaging forScience/Observation Data Objects

• XML will be used to specify, collect, and validate package content

• A simple prototype will demonstrate the ability to create, search, and retrieve data product packages.

06/30/20038

Core Data Product Service

DataProductCreation

DOM

TransactionProduct

MetadataProduct

Accounting

ObservationProduct

Inst. StateS/C StateTraj State Product

Accounting

AMMOS’CFDPService

TransactionProduct

N.B.Multiple instrMultiple modesMultiple productsAll in XML

Anc & Eng Data Spec

MetaData

Specs

Input Data Spec

DPS Process Spec

Data Product Spec

Eng & Anc Data

Observation FileObs tagDPS tag

S/CData

Sources

Standard Repository

Product Retrieval

Product Search

Users

Data Dictionary

06/30/20039

Core Data Product Service(in Context)

DataProductCreation

DOM

TransactionProduct

MetadataProduct

Accounting

ObservationProduct

Inst. StateS/C StateTraj State Product

Accounting

AMMOS’CFDPService

TransactionProductN.B.

Multiple instrMultiple modesMultiple productsAll in XML

Anc & Eng Data Spec

MetaDataSpecs

Input Data Spec

DPS Process Spec

Data Product Spec

Eng & Anc Data

IncludingObservationTag

S/CData

Sources

Operational Repository

Product Retrieval

Product Search

Users

Data Dictionary

Science Campaign SpecInstrument Spec

ObservationTemplate(multi-inst, multi-product,)

ObservationRequests

Observation Set(Sequence)

ExpectedDataProducts Observation

tag

ScienceProcessing

Metadata

Including Observation tag

06/30/200310

Prescriptive XML Package Schema

• Package Specification – Structure/format of package components– Content of package

• data object(s)• data object accounting data• spacecraft, instrument, and target state information

• ancillary data including but not limited to – calibration data

– navigation data

• other useful documentation and metadata

06/30/200311

Product Service XML Schema

• Package Specification – Structure/format of package components– Content of Delivered Science Data Product package

• received telemetry data product/object(s)

• received data product accounting data

• ancillary data including but not limited to – spacecraft, instrument, and target state information from S/C engineering

telemetry– calibration data– navigation data

• cataloging information extracted from observation request metadata – observation type and template– campaign associations

• processing history

• other useful documentation

06/30/200312

Post Processing XML Schema

• Package Specification – Structure/format of package components– Content of package

• Spacecraft and ancillary files (see Product Service XML Schema)• Process Specifications

– Preprocessors– Engineering Analysis– Science Level Processors– Science Analysis and Quality Reporting– Archive Quality Packaging– Subscription, Notification, and Distribution

• Control Specifications– Process monitoring and control– Instrument Commanding– Data Verification

• Metadata providing identification, descriptive, relational, and location information for each package component

06/30/200313

Information Infrastructure

• XML Schema• Data Dictionary• Operational Repository• APIs for program / subsystem integration

• Infrastructure is used by all subsystems for …– Access / location of schema, other resources– Reference to data dictionary for interoperability– Optional storage of instances of XML schema– Optional exchange of data and state information

06/30/200314

Deliverables

• XML schemas• Active data dictionary • Java class library that provides basic get and put

functions for all components of the package• Profile server that provides a standard system level

search interface for packaged products within a standard repository

• Product server that provides a standard system level interface for the retrieval of the identified package from the repository

• Prototype Web interfaces for:– Data Dictionary– Product Service– Monitor/Performance Reports– Schema Display and Population– User Viewpoint Web Page Construction

06/30/200315

Issues / Questions

• Observation: existing huge chasm between uplink and downlink• What are driving scenarios / requirements? Is this just for MRO or does it meet a

broad variety of JPL (and other) mission needs?• Do packages embed raw data or point to it, or both? What about calibration data? • Which parts of this need standardization?• How complete a view of the context do we need, is Nav / predicts included?• Packaging is suitable for more than just science / observation data objects• What is nature of “Observation Tags”, how are they created and tracked, how do

they handle the one-many, many-one problems?• How is processing history handled, are program artifacts archived w/ data,

referenced, or just named?• What kinds of information infrastructure are required, what technologies, how many

instances of what?• Are DPS needs for info infra different from ICIS, SMS, monitoring or other DSMS

needs? • Is “data dictionary” just that or does it relate to an ontology or taxonomy? How is it

created and maintained?• What is relationship between DPS, monitoring, and data accountability? Are they

separate but related?

06/30/200316

Next Steps

• Develop representative science / mission scenarios (DC, SH, EG)– Orbiter– Observatory– Rover

• Evaluate approach vs scenarios (team)

• Ensure handling of typical sequence operations vs MDS goal oriented process (TM, EG)

• Develop User Viewpoint Use Cases, at least … (DC, SH, EG)– Science team– MOS or S/C team– DSMS operations team

• Update prototype plan to match agreed scenarios & Use Cases