Upload
mark-watson
View
212
Download
0
Embed Size (px)
Citation preview
September 2003 ERPANET - Marburg 2003 View # 1
Data preservationUsing CCSDS and ISO Standards
Denis [email protected]
September 2003 ERPANET - Marburg 2003 View # 2
The problem to solve
• Data is often expensive to produce– Bits must not be lost (preserved medium)– Bits must remain accessible (known syntax)– Bits must remain understandable (known
semantics)
• This presentation focuses on point 2 and 3
September 2003 ERPANET - Marburg 2003 View # 3
The syntactic aspect
• How to master the bit organisation– 2 possible solutions
• Define exchange formats– Better done by user communities (ex : FITS for astronomy)
– Not generic enough
• Define a standard way for describing any data– Was the CNES proposal to CCSDS
– Was entrusted to CNES (definition and tools) by CCSDS
September 2003 ERPANET - Marburg 2003 View # 4
History
• CNES proposed a description language– EAST for data syntax
– CCSDS recommendation since May 1997CCSDS 644.0-B-2 Issue 2 November 2000
– ISO standard since 2000ISO 15889-2000
CCSDS recommendations
September 2003 ERPANET - Marburg 2003 View # 5
EAST principles
• EAST is a standard language– Enhanced Ada SubseT
• inherits from a huge standardisation work
• enhanced with a few features missing for data description
• EAST has not to be mastered to be used• thanks to the tools
Data syntax description
September 2003 ERPANET - Marburg 2003 View # 6
EAST principles (2)
• Assumption : any kind* of digital data (available as an ordered set of bits) can be described as a tree.
• The minimal tree is a rake
*note : optimal for binary data, interesting for raw ASCII format, of no use for XML data
• To describe such a tree, EAST proposes :• composite types for the branches
• elementary types for the leaves
Data syntax description
September 2003 ERPANET - Marburg 2003 View # 7
EAST data types
• Composite types are :• records• arrays• lists
• Elementary types are :• integers and reals• enumerated• characters
Data syntax description
September 2003 ERPANET - Marburg 2003 View # 8
Optionality
• Some branches or leaves of the data tree may be optional.– EAST offers a way to indicate which data
entities are optional and under which condition they exist
– Example : • Housekeeping data exists if telemetry mode is set to
« housekeeping »
Data syntax description
September 2003 ERPANET - Marburg 2003 View # 9
Variable size
• Some repetitions of data entities may have a variable number of elements
– EAST offers a means to indicate how many of those elements are present in the current data
Data syntax description
September 2003 ERPANET - Marburg 2003 View # 10
EAST use conclusion
• All the features described in the previous slides have been sufficient to describe almost every data encountered since 1997
– Exceptions : • variable size not carried by the data
• variable size provided after the variable size field
Data syntax description
September 2003 ERPANET - Marburg 2003 View # 11
History (2)
• CNES has also been in charge of writing the recommendation for data semantics description :– DEDSL (Data Entity Dictionary Specification Language)
– XML/DTD syntax
– CCSDS recommendation since January 2002CCSDS 647.1-B-1 Issue 1 January 2002
– ISO standard since 2003ISO 22643 : 2003
CCSDS recommendations
September 2003 ERPANET - Marburg 2003 View # 12
DEDSL principles
• The data semantics description can be kept as a set of attribute values.
• The recommendation defines standard attributes : name, definition, units….
• The recommendation gives a standard way to define user attributes to be added to the standard ones if needed
Data semantics description
September 2003 ERPANET - Marburg 2003 View # 13
DEDSL principles (2)
• A data description consists in the tree organisation (as seen on the sysntactic aspect) completed by all the fields (nodes or leaves) caracteristics given as attributes values.
• Example of the descripon of a leaf of such a tree description.
September 2003 ERPANET - Marburg 2003 View # 14
OASIS
• OASIS is the tool to describe data (syntax and semantics) in a friendly way (without any requested knowledge of the EAST or DEDSL syntax)
• A dedicated Man-Machine interface offers the requested features (those listed previously) to describe data
A tool to describe data
September 2003 ERPANET - Marburg 2003 View # 15
OASIS (2)
• Example : • a data tree as built and shown by OASIS
A tool to describe data
September 2003 ERPANET - Marburg 2003 View # 16
OASIS (3)
• Once the tree and each of its nodes and leaves described, the descriptions can be produced just by a click (EAST and DEDSL-XML files)
• Once saved, the EAST description can be used by the EAST tools to read and write the described data
• Once saved, the XML dictionary can be published as an html or word document
A tool to describe data
September 2003 ERPANET - Marburg 2003 View # 17
EAST tools• Application Programming Interface (API)
• GENERATOR : for data writing• INTERPRETER : for data reading
– Both available for C, C++, FORTRAN and Ada
• Standalone tools (based on the API) • data_checker : data vs description compliance• ascii_dump : display any data in ascii values• DUW : interactive data generator• other utilities…
Tools for EAST described data
September 2003 ERPANET - Marburg 2003 View # 18
EAST tools (2)
• EAST tools are available on :– PC (DOS, NT, Linux), – HP-UX, – SUN-Solaris, – Dec true64 (OSF1) , – VAX-VMS– other on request….
Tools for EAST described data
September 2003 ERPANET - Marburg 2003 View # 19
Interface Control Documents
• The DEDS-XML OASIS output can be transformed in an html or Word document
• A tool (named SCRIBE) is being developed for that purpose .
Documenting the interfaces
September 2003 ERPANET - Marburg 2003 View # 20
Synthesis
Data dictionaryDEDSL/XML
XML Data
EASTApis
Html or Word document
DCI generation
File descriptionsXML schema
ReadWrite
GenerateCheck
‘RAW’ Data
OASIS tool
File descriptionsEAST
ReadWrite
GenerateCheck
XML applications
September 2003 ERPANET - Marburg 2003 View # 21
Main advantages• Only one consistent reference for syntactic
description used for input/output and for semantic description (documentation for understanding)
• Reusable and reliable generic input/output software available
• User application easier to maintain due to less impact from changes of data formats
• Help in describing data and producing documentation
The project conclusions
September 2003 ERPANET - Marburg 2003 View # 22
EAST/DEDSL-XML in action
• Many projects at CNES– SPOT, SSALTO, SIPAD, HELIOS,
PHARAO…
• ESA projects – AMS, DEBAT
• NASA– Science data rehabilitation
Use in other projects
September 2003 ERPANET - Marburg 2003 View # 23
EAST/DEDSL-XML feed back
• PROs : • Easier sub-system integration
– description accuracy (even after long time!!!)
– early data simulation
– assisted data checking
• Easier ICD evolutions– full top-down process
• Helpful in heterogeneous platforms context– automatic conversions
• Helpful in multi-missions context
Projects feedback
September 2003 ERPANET - Marburg 2003 View # 24
EAST/DEDSL-XML feed back (2)
• CONs :• Performances to be again improved
– reading in time constrained context
– writing in any case
• User type sharing to be added– to facilitate evolutions
• Documentation production to be improved– project customisation
– add as much filters as possible to allow adaptation of the document to its target
Projects feedback
September 2003 ERPANET - Marburg 2003 View # 25
Current actions
• Documentation tool in progress• generic tool to transform XML-DED in Word or html documents
• 2nd prototype delivered
• Main lacks taken into account by the DEBAT project (ESA/ESRIN)
• Support to NASA• tests on data migration - good first results
• Support to XML schema production
On going activity
September 2003 ERPANET - Marburg 2003 View # 26
CNES policy for the EAST tools
• OASIS and EAST tools are freely available
• They can be downloaded from a CNES server
• Tools are maintained by a CNES team and a sub-contractor team (CS-SI)
• Policy : if CNES stops founding the maintenance the source code would be made available
Information
September 2003 ERPANET - Marburg 2003 View # 27
Links• For any support request :
• [email protected] (e-mail address)
• For information and documentation• east.cnes.fr (web server)
• To download the software :• logiciels.cnes.fr (web server)
• To get the standards• http://www.ccsds.org (web server)
– then under publications, recommendations and reports, information access and interchange systems.
Information