06/30/20031
Data Product Service (DPS)Packaging and Context
Dan Crichton
Steve Hughes
Ron Joyner
Chris Mattman
Paul Ramirez
Peter Shames
06/30/20032
Acknowledgements
Ed Greenburg
Larry Preheim
Steve Levoe
Tom McVittie
Tim M
Erik Barkley
06/30/20033
Data Product Service (DPS)Packaging and Context
• DPS definition: a service that provides reliable delivery of well documented data products that can be directly related to science observation requests
• DPS mission system context– Functions and high level data flows
• Data packaging specification– Generic approach– Applied to DPS problem– Evolving content during processing
• Prototype & deliverables
• Open issues & next steps
06/30/20034
Data Product Service Functional Elements
Science SequenceDevelopment
Acquisition Plan Development Sequence PlanObservation Requests Obs. Definition Obs. Planned Products Obs. Uplink Files
Science MissionPlanning
Objective-CampaignsInstrumentsObservation sets
Integrated SequenceDevelopment
Sequence Plan Development Sequence Plan Sequence Uplink FilesActivity Requests Activity Definition Activity Planned Products Activity Uplink Files
Spacecraft Activities
Sequence Execution Activity Sequence Uplink Files Sequence Events Product Creation Product Notification Product Transfer
Service Management
Service Scheduling Service Possibilities Service Requests Service Schedule PredictsService Monitoring Receive Reports Capture Data Provide User Views Performance Assessment Commitments Operational Visibility
Tracking Services
Scheduled Activities (General) Service Reports Service Notifications Performance ReportsTelemetry Service Frame Accounting ReportCommand Service Radiation Reports
DSMS File Transfer Svc
Telemetry Service Product Accounting/Reports Product Assembly/DeliveryCommand Service Product Accounting/Reports
DSMS ScienceProduct Service
Science Product File(s) Assembly Received Product Product Accounting Cataloging Data Ancillary Data Reporting Processing History??
Observation Planning
Observation Planning Integrated Obs. Descriptions Observation Templates Product Types Product Type Relationships Product Type Definition Ancillary Specification
Engineering SequenceDevelopment
Spacecraft Plan Development Sequence PlanEngineering Requests Eng. Definition Eng. Planned Products Eng. Uplink Files
InformationInfrastructure
XML SchemasInterface SpecificationsDictionary & Repository svc
06/30/20035
Data Product Service Flows
Science SequenceDevelopment
Acquisition Plan Development Sequence PlanObservation Requests Obs. Definition Obs. Planned Products Obs. Uplink Files
Science MissionPlanning
Objective-CampaignsInstrumentsObservation sets
Integrated SequenceDevelopment
Sequence Plan Development Sequence Plan Sequence Uplink FilesActivity Requests Activity Definition Activity Planned Products Activity Uplink Files
Spacecraft Activities
Sequence Execution Activity Sequence Uplink Files Sequence Events Product Creation Product Notification Product Transfer
Service Management
Service Scheduling Service Possibilities Service Requests Service Schedule PredictsService Monitoring Receive Reports Capture Data Provide User Views Performance Assessment Commitments Operational Visibility
Tracking Services
Scheduled Activities (General) Service Reports Service Notifications Performance ReportsTelemetry Service Frame Accounting ReportCommand Service Radiation Reports
DSMS File Transfer Svc
Telemetry Service Product Accounting/Reports Product Assembly/DeliveryCommand Service Product Accounting/Reports
DSMS ScienceProduct Service
Science Product File(s) Assembly Received Product Product Accounting Cataloging Data Ancillary Data Reporting Processing History??
Observation Planning
Observation Planning Integrated Obs. Descriptions Observation Templates Product Types Product Type Relationships Product Type Definition Ancillary Specification
Engineering SequenceDevelopment
Spacecraft Plan Development Sequence PlanEngineering Requests Eng. Definition Eng. Planned Products Eng. Uplink Files
InformationInfrastructure
XML SchemasInterface SpecificationsDictionary & Repository svc
06/30/20036
XML Packaging forScience/Observation Data Objects*
An end-to-end data product production pipeline requires the packaging of data objects, ancillary data, and documentation to address mission support, science analysis, and data archive requirements. This task proposes the development of a generalized standards specification to:– describe the data objects, ancillary data, and documentation
required to support science planning, mission operations, science analysis, and the packaging and cataloging of an archive quality packaged data product
– prescribe the process used to produce a packaged data product.
* A science data object is defined as any digital data file(s) collected by a science instrument or compiled by a scientist that has science content. A data product is a searchable, retrievable, and orderable package consisting of one or more data objects.
06/30/20037
XML Packaging forScience/Observation Data Objects
• XML will be used to specify, collect, and validate package content
• A simple prototype will demonstrate the ability to create, search, and retrieve data product packages.
06/30/20038
Core Data Product Service
DataProductCreation
DOM
TransactionProduct
MetadataProduct
Accounting
ObservationProduct
Inst. StateS/C StateTraj State Product
Accounting
AMMOS’CFDPService
TransactionProduct
N.B.Multiple instrMultiple modesMultiple productsAll in XML
Anc & Eng Data Spec
MetaData
Specs
Input Data Spec
DPS Process Spec
Data Product Spec
Eng & Anc Data
Observation FileObs tagDPS tag
S/CData
Sources
Standard Repository
Product Retrieval
Product Search
Users
Data Dictionary
06/30/20039
Core Data Product Service(in Context)
DataProductCreation
DOM
TransactionProduct
MetadataProduct
Accounting
ObservationProduct
Inst. StateS/C StateTraj State Product
Accounting
AMMOS’CFDPService
TransactionProductN.B.
Multiple instrMultiple modesMultiple productsAll in XML
Anc & Eng Data Spec
MetaDataSpecs
Input Data Spec
DPS Process Spec
Data Product Spec
Eng & Anc Data
IncludingObservationTag
S/CData
Sources
Operational Repository
Product Retrieval
Product Search
Users
Data Dictionary
Science Campaign SpecInstrument Spec
ObservationTemplate(multi-inst, multi-product,)
ObservationRequests
Observation Set(Sequence)
ExpectedDataProducts Observation
tag
ScienceProcessing
Metadata
Including Observation tag
06/30/200310
Prescriptive XML Package Schema
• Package Specification – Structure/format of package components– Content of package
• data object(s)• data object accounting data• spacecraft, instrument, and target state information
• ancillary data including but not limited to – calibration data
– navigation data
• other useful documentation and metadata
06/30/200311
Product Service XML Schema
• Package Specification – Structure/format of package components– Content of Delivered Science Data Product package
• received telemetry data product/object(s)
• received data product accounting data
• ancillary data including but not limited to – spacecraft, instrument, and target state information from S/C engineering
telemetry– calibration data– navigation data
• cataloging information extracted from observation request metadata – observation type and template– campaign associations
• processing history
• other useful documentation
06/30/200312
Post Processing XML Schema
• Package Specification – Structure/format of package components– Content of package
• Spacecraft and ancillary files (see Product Service XML Schema)• Process Specifications
– Preprocessors– Engineering Analysis– Science Level Processors– Science Analysis and Quality Reporting– Archive Quality Packaging– Subscription, Notification, and Distribution
• Control Specifications– Process monitoring and control– Instrument Commanding– Data Verification
• Metadata providing identification, descriptive, relational, and location information for each package component
06/30/200313
Information Infrastructure
• XML Schema• Data Dictionary• Operational Repository• APIs for program / subsystem integration
• Infrastructure is used by all subsystems for …– Access / location of schema, other resources– Reference to data dictionary for interoperability– Optional storage of instances of XML schema– Optional exchange of data and state information
06/30/200314
Deliverables
• XML schemas• Active data dictionary • Java class library that provides basic get and put
functions for all components of the package• Profile server that provides a standard system level
search interface for packaged products within a standard repository
• Product server that provides a standard system level interface for the retrieval of the identified package from the repository
• Prototype Web interfaces for:– Data Dictionary– Product Service– Monitor/Performance Reports– Schema Display and Population– User Viewpoint Web Page Construction
06/30/200315
Issues / Questions
• Observation: existing huge chasm between uplink and downlink• What are driving scenarios / requirements? Is this just for MRO or does it meet a
broad variety of JPL (and other) mission needs?• Do packages embed raw data or point to it, or both? What about calibration data? • Which parts of this need standardization?• How complete a view of the context do we need, is Nav / predicts included?• Packaging is suitable for more than just science / observation data objects• What is nature of “Observation Tags”, how are they created and tracked, how do
they handle the one-many, many-one problems?• How is processing history handled, are program artifacts archived w/ data,
referenced, or just named?• What kinds of information infrastructure are required, what technologies, how many
instances of what?• Are DPS needs for info infra different from ICIS, SMS, monitoring or other DSMS
needs? • Is “data dictionary” just that or does it relate to an ontology or taxonomy? How is it
created and maintained?• What is relationship between DPS, monitoring, and data accountability? Are they
separate but related?
06/30/200316
Next Steps
• Develop representative science / mission scenarios (DC, SH, EG)– Orbiter– Observatory– Rover
• Evaluate approach vs scenarios (team)
• Ensure handling of typical sequence operations vs MDS goal oriented process (TM, EG)
• Develop User Viewpoint Use Cases, at least … (DC, SH, EG)– Science team– MOS or S/C team– DSMS operations team
• Update prototype plan to match agreed scenarios & Use Cases