C. Huc/CNES, D. Boucon/CNES-SILOGIC Producer-Archive Interface Specification

C. Huc/CNES, D. Boucon/CNES-SILOGIC

Producer-Archive Interface SpecificationProducer-Archive Interface Specification

CCSDS Meeting, 05/12/05 Producer-Archive Interface Specification 2

Main changesMain changes

Main changes between V01 and V02

• Documents sent on Descriptor Definition (end May 2005) and SIPs (July 2005).

• Concepts have changed:

• suppression of Slips, Generic Descriptor Model.

• Implementation parts (sections 4 and 5)

• Completed and rewritten document.


ContentsContents

Introduction

• Objectives• Main steps• General framework• Example

General principles for implementing the Producer-Archive Interface

• Process description• Plan of Objects to be Transferred (POT)• SIPs• Sequencing constraints

Formal Phase Specification Transfer Phase and Validation Phase Specification Managing modifications Tools

Conclusion


Provide a standard method to • formally define the digital information objects to be transferred by a

Producer to an Archive

• effectively transfer these objects in the form of SIPs

• validate the SIPs

IntroductionIntroductionObjectives (reminder)Objectives (reminder)


Preliminary Phase

Formal Definition Phase

Transfer Phase

ValidationPhase

IntroductionIntroductionGeneral framework: PAIMAS PhasesGeneral framework: PAIMAS Phases

Initial contacts, feasibility study, draft of the SIPs, Preliminary Agreement

Precise definition of the digital objects, SIP design, Submission Agreement

Actual transfer of the SIPsActual validation processing ofthe SIPs by the Archive and Required follow-up action with the Producer

•Organisational sub-phase•Formal definition sub-phase Complete and precise definition of the data to be transferred and the way they are organised in the form of a SIP Complementary points (contractual and legal aspects, …)•Submission Agreement

• implementing tests sub-phase• performing the transfer sub-phase (technical means –XFDU …)


How can we define an object (and have a sufficiently precise description without creating the objects)?

How can we define the organisation between the objects (« Plan of Objects to be Transferred » )?

How can we define a complete packaging description for an object and a group of objects?

How can we adapt the general framework to the domain specificities and project needs?

How can we specify the constraints (granularity and sequencing) applying to these objects?

How can we validate the transferred objects?

How can we have a unique visual representation of the objects to transfer and their constraints?

IntroductionIntroductionGeneral framework: needs to meet (reminder)General framework: needs to meet (reminder)


IntroductionIntroductionExampleExample

WIND

WAVES Experiment

WIND WAVES TNR L2 Data set

WIND WAVES TNR L2 Data (1..N)

Complementary Data Objects

EAST Description (1) Document (1)

Space Physics domain

•WIND mission•WAVES experiment •WIND WAVES TNR L2 data set•Complementary Data Objects:

EAST syntactic descriptor Mission description document


ContentsContents

Introduction


General principles for implementing the Producer-Archive Interface• Process description• Plan of Objects to be Transferred (POT)• SIPs• Sequencing constraints


Conclusion


General principlesGeneral principlesProcess, from Object description to SIP Model instancesProcess, from Object description to SIP Model instances

Object content description

SIP constraints

SIP content description

TimeFormal Definition Phase Transfer and validation Phases

POT

SIP Models

Sequencing constraints

Submission Agreement

Descriptor instances

SIP Model instances

Descriptor Models

Digital Objects

Data

Rules

01000010001Models and POT creation


General PrinciplesGeneral PrinciplesProcess, from SIP creation to SIP acceptanceProcess, from SIP creation to SIP acceptance

Transfer management

Anomaly management

Time

SIP reception

SIP validation

Producer

SIP creation

SIP transfer

Archive

ControlOk

Yes AIPs No

Transfer and validation Phases

Reception acknowledgement

Anomaly acknowledgement

Data SIP Model instances

01000010001


Plan of Objects to be Transferred (POT)

Gives a complete and overall view of all digital objects to be transferred: described by Descriptors.

Descriptor : This is an information unit for describing a set of characteristics for a given Data Object.

A Descriptor may come in the form of a model; in this case, it is used to identify the object category which can then be described as well as the attributes for this description.

The Descriptor may have the state of an instance; in this case, it describes an object instance and supplies the attribute values defined in the Descriptor model.

– POT creation: dictionary for project Descriptor Models, SIP Models, Descriptor instances

– POT constraint: All the Objects defined in a Descriptor must be delivered together in the same SIP.

– POT representation

General principlesGeneral principlesPlan of Objects to be TransferredPlan of Objects to be Transferred


Process

– Generic Descriptor Model Structure

– Specialisation Phase

– Instantiation Phase of the Project Descriptor Models

General principlesGeneral principlesPlan of Objects to be Transferred: DictionaryPlan of Objects to be Transferred: Dictionary


Descriptor Model, attributes for (see section 3.5.2 page 16):

– Descriptor identification: identifying the type of object described.

– Content description: describing the object content.

– Relationships within the POT: defining the relations between the object

described and the other Model objects.

– Object content: describes all the Data and Metadata included in this Descriptor.

– Specialisation section.

– SIP Model reference.

General principlesGeneral principlesPlan of Objects to be Transferred: the DescriptorsPlan of Objects to be Transferred: the Descriptors

CNES


Identification (M, 1..1)

– descriptor_type (M, 1..1): defines the type of Object Model and is constant for a given type of Descriptor.

– descriptor_ID (M, 1..1): Descriptor Identifier, used to identify the nodes in the POT.

– object_occurrence (M, 1..1): number of Data Objects described by this descriptor_ID. This Descriptor instance may describe a unique Object (this is often the case for a Data Collection, or a syntactical description of a Data Collection), or a representative of a category containing a certain number of Objects. This number may not be known ahead of time (value 1..n).

– version (M, 1..1).


CNES


Content description (M, 1..1)

– title (M, 1..1): extensive name of the Object.

– content (M, 1..1): explanatory text specifying the content and main characteristics of the described Object.

– size (O, 0..1): estimated volume of an object and unit.

– Non definite attribute (O, 0..1): specialization attribute (to be defined if necessary). The name and form of this attribute depends on the future implementation.


CNES


Relationships within the POT (M, 1..1)

– parent_collection (M, 1..1): identifier of Collection Descriptor to which this object belongs.

– association (O, 0..N): this is used when the Descriptor describes a Complementary Data Object or a Complementary Collection (transversal links).

• related_descriptor_ID (M, 1..1) : Descriptor identifier for the Data Object Collection or Data Object to which this Object refers

• relation_with_DO (O, 0..1): type of relation between the Object in

question and the related_descriptor_ID.


CNES


Object content (O, 0..N). – content_ID (O, 0..1): the Data section describes the Data Object itself,

while the Metadata section describes the Metadata attached to the Data. – data (O, 0..1):

• data_ID (M, 1..1): Data Identifier. • data_content (O, 0..1): Explanatory text specifying the content and

main characteristics of the described Data• data_format (O, 0..1): Format of objects in the collection (PNG,

PDF, CDF, Flat Binary, Flat ASCII, …)– Metadata (O, 0..1)

• metadata_ID (M, 1..1): Metadata Identifier. • metadata_content (O, 0..1): Explanatory text specifying the

content and main characteristics of the described Metadata.• metadata_format (O, 0..1): Used to specify the metadata standard

to be applied.• metadata_schema (O, 0..1): Used to specify the ‘catalogue’ data

characterising the object: for instance in the form of an XML document specified by an XML Schema.


CNES


Non definite section: optional section in which can be included any attributes to specialise the Descriptor.

SIP Model reference: SIP Model Identifier. For a collection of collections, there may not be a corresponding delivery during the Transfer Phase, and hence no applicable SIP Model: the attribute value is then NONE.


CNES


Addition of an attribute by using the: – “Non definite attribute” (if related to information description).

– “Non definite section”.

Elimination of an optional attribute.

Modification of existing attributes: – by occurrence restriction, by modification of status attribute (e.g. optional

changed in mandatory).

General principlesGeneral principlesPlan of Objects to be Transferred: specialisationPlan of Objects to be Transferred: specialisation

Generic DescriptorModel Domain Descriptor Models Producer-Archive Project Descriptor Models

CNES


General principlesGeneral principlesPlan of Objects to be Transferred: instancesPlan of Objects to be Transferred: instances

Most crucial task during the Formal Definition Phase definition of the actual objects to be transferred:

– Identifying all of the project data Collections,– Organising these Collections in a hierarchical tree structure,– Defining a Descriptor instance for each of the Collections,– Defining a Data Object Descriptor for the objects in each Collection,

– …

Building the POT

CNES


General principlesGeneral principlesPlan of Objects to be Transferred: instancesPlan of Objects to be Transferred: instances

The SIPTemplate describes the organisation of the Unit Package

This part contains everything that is known about the object at the time of the Formal Definition Phase (characteristics of the object and its relations to other objects)

CNES


General principlesGeneral principlesPlan of Objects to be Transferred: examplePlan of Objects to be Transferred: example

ROOT

WIND_WAVES

WA_TNR_L2_COMETADATA_STANDARD=DIF

WA_TNR_L2_DO (N)METADATA_MODEL=CDPP_DO.xsl

WIND_WAVES_CC

EAST_DESCRIPTION (1) WAVES_DOCUMENTATION (1)

Object Descriptor ID

SPACE_PHYSICS_SIPWind_waves_tnr_l2_aaammjj.datWind_waves_tnr_l2_aaammjj.xml

EAST_SIPWind_waves_tnr_l2.east

EXPERIMENT_DESCRIPTION_SIPWind_waves_22000_0387.pdf

SPACE_PHYSICS_COLLECTION_SIPWind_waves_tnr.dif

N

Files to be transferred


ContentsContents

Introduction




Conclusion


Coherent group of Digital Objects in a package to be transferred (objects of the same or different types).

– May contain pointer towards the Digital Objects.

Defined during the Formal Definition Phase:

– Rules for grouping objects.

– SIP categories for the project

General principlesGeneral principlesSIPsSIPs


General principlesGeneral principlesLinks between SIP and DescriptorLinks between SIP and Descriptor

Descriptor Model

Attribute = descriptor_ID…Attribute = SIP Model ID

descriptor_ID = A…SIP Model ID = sipC

Instantiated Descriptor

descriptor_ID = B…SIP Model ID = sipC

SIP Model ID = sipC…Descriptor_ID = A

Data

________________Descriptor_ID = B

Data

SIP Model ID = sipC….Attribute = descriptor_ID…

Delivered SIP

SIP Model

Formal Definition Phase Transfer Phase

01000010001

01000010001


General principlesGeneral principlesSIP ModelsSIP Models

Generic SIP Model Domain SIP Models Producer-Archive Project SIP Models

•Process for the development of the Project SIP Models


General principlesGeneral principlesSIP ModelsSIP Models

Producer IDProject IDSip Model IDSIP Model location

descriptor_IDDescriptor locationlast_objectupdated_object

Other SIP attributes

Digital Object

Each sub-packageis associated to1 Descriptor instance

SIP structure (see 3.6.3 page 22):

Global information (M, 1..1)Information associated with each embedded object (M, 1..N)Other information (M, 1..1)Digital Object (O,0..N)


A given digital object may have to be transmitted before or after another.

Constraint group: set of objects related to each other by a set of dependent sequencing constraints.

General principlesGeneral principlesSequencing ConstraintsSequencing Constraints

Attribute name Meaning Value syntax Occurrence

TIME_CONSTRAINT_GROUP define the groups identifier 0..1

SERIAL_NUMBER_IN_CONSTRAINT define the constraints within the same group

integer 0..1


General principlesGeneral principlesSequencing ConstraintsSequencing Constraints

SIP1: Collection WIND_WAVES_TNR_L2SIP2: Complementary Objects EAST file and document: is delivered before any delivery of SIP3.SIP3: Data Objects

Example of SIP3 instance

Attribute Value

SIP_MODEL_ID SIP3

SIP_CONTENT WIND_WAVES_TNR_L2_DATA

TIME_CONSTRAINT_GROUP Group_1

SERIAL_NUMBER_IN_CONSTRAINT 2

Example of SIP2 instance

Attribute Value

SIP_MODEL_ID SIP2

DESCRIPTOR_ID EAST_DESCRIPTION

DESCRIPTOR_ID DOCUMENTATION

TIME_CONSTRAINT_GROUP Group_1

SERIAL_NUMBER_IN_CONSTRAINT 1

1

2


ContentsContents

Introduction




Conclusion


Formal Phase SpecificationFormal Phase SpecificationGlobal viewGlobal view

project_ID


Data dictionary creation

Anomaly management

POT creation

Sequencing constraints

SIP templates creation

Producer

descriptor_IDdescriptor_typeobject_occurrencetitleparent_collectioncontent_IDdata_IDmetadata_ID…

sip_template_IDtime_constraint_groupserial_number_in_constraintdescriptor_ID (1..n)

anomaly_IDdate…

Archive

POT

message

Validation

SIP templates

Ingest Base


Formal Phase SpecificationFormal Phase SpecificationPOT creationPOT creation

Proposed implementation based on XML Schema and XML files.

Process

– Generic Descriptor Model: complete enough to cover the needs for the description of different object categories (Data Objects or

Complementary Data Objects), flexible enough to adapt to the specialisations required.


Data Dictionary

POT

Validation


For

mal

Ph

ase

Spe

cifi

cati

onF

orm

al P

has

e S

peci

fica

tion

Gen

eric

Des

crip

tor

Mod

el I

mpl

emen

tati

onG

ener

ic D

escr

ipto

r M

odel

Im

plem

enta

tion


Formal Phase SpecificationFormal Phase SpecificationGeneric Descriptor SpecialisationGeneric Descriptor Specialisation

The generic Descriptor can be specialized by:

– suppression of optional elements ;

– modification of occurrence numbers (for example 10..15 instead of 1..n);

– addition of new elements (or sequence of elements) by the use of the groups « other » or « any »;

– definition of a list (enumerate) for the elements « descriptor_type » or « sip_template_ID »;

– definition of patterns for identifiers (imposed nomenclature) or restriction of string length.


Formal Phase SpecificationFormal Phase SpecificationDescriptor instances and POT creationDescriptor instances and POT creation

XML files Based on the previous specialised schemas Instantiated in several times during the Formal Definition Phase Use of an XML editor or forms

Base of the information contained in the Descriptor instances Validation (XML schemas, coherence of the nodes, …)

POT visualisation (mock-up)


Formal Phase SpecificationFormal Phase SpecificationSIP implementationSIP implementation

based on XFDU

Generic SIP Model Generic SIP Model Specialization

SIP templates implementation


Formal Phase SpecificationFormal Phase SpecificationSIP implementationSIP implementation

List of the specific SIP elements in an XFDU:

•Global information: sip_ID;producer_ID;project_ID;sip_template_ID;sip_template_location.

•Information associated with each embedded object:descriptor_ID;descriptor_location;last_object;updated_object.


ContentsContents

Introduction




Conclusion


Transfer/Validation Phase SpecificationTransfer/Validation Phase SpecificationFunctionsFunctions

SIP instances

Transmission sessions

SIP reception and validation

Transfer Management

Anomaly management


Transfer/Validation Phase SpecificationTransfer/Validation Phase SpecificationGlobal viewGlobal view

project_ID

SIPs

Transmission session

Anomaly management

SIP reception

SIP validation

Transfer management

Producer

session_IDproducer_IDproject_IDtransfer_datesip_listtransfer_repository

session_IDsip_listtransfer_date

descriptor_IDstatusoccurrence_numbernumber_validated_objects

sip_template_IDtime_constraint_groupserial_number_in_constraintdescriptor_ID (1..n)content_ID (0..n)data_ID (0..n)metadata_ID (0..n)

session_IDanomaly_IDerror_leveldate…

Archive

POT

log

message

Ingest Base


ContentsContents

Introduction


General principles for implementing the Producer-Archive Interface

• Process description• Plan of Objects to be Transferred (POT)• SIPs• Sequencing constraints


Conclusion


Managing modificationsManaging modifications

POT modification– Impact on the data to be delivered

– Technical impact on the current functions and tools

– Possible re-negociation of the Submission Agreement

Object modification– « Updated » element

– The new object replaces the older one


ToolsTools

Formal Definition Phase– Tool for creating specialised dictionaries from the generic Model.

– Tool for generating Descriptor instances.

– Tool for creating and validating the POT.

Transfer and Validation Phases– Tool for generating SIP instances (prototype to create XFDU instances ?).

– Tool for the transfer validation and management:

• Validate the content of each received SIP according to the initial validation plan (object expected and conforms to the associated Descriptor).

• Respect of the sequencing constraints.

• Monitoring project progress for the Producer and for the Archive.

• Propose a follow-up graphical view by using, if possible, the POT graphical view.


How can we define an object (and have a sufficiently precise description without creating the objects)?– Descriptor and attributes

How can we define the organisation between the objects (« Plan of Objects to be Transferred « )?– Descriptors and relation definitions between objects

How can we define a complete packaging description for an object and a group of objects?– SIPs

How can we adapt the general framework to the domain specificities and project needs?– Specialisation process

How can we specify the constraints (granularity and sequencing) applying to these objects?– TIME_CONSTRAINT_GROUP, SERIAL_NUMBER_IN_CONSTRAINT attributes

How can we validate the transferred objects?– Validation process

How can we have a unique visual representation of the objects to deliver and their constraints? – See the mock-up

ConclusionConclusionNeeds to meet: some answersNeeds to meet: some answers


ConclusionConclusion

Reach an Agreement on– Terminology,– Concepts (Descriptors, SIPs, sequencing constraints),– Process,– XFDU attributes,– Implementation (Descriptor schema, SIP schema, …)
