CD Sender Software Design Description - CTBTO

IDC/CD Sender/SDD 5. October 2006

English only

CD Sender Software Design Description

This document defines the CD Sender software design description. The software design will be used as the basis for implementation.

Summary

CD Sender is a software system that can send continuous seismoacoustic data received from IMS stations (or from a data centre forwarding data from IMS stations). The software is able to send connection requests and establishes a connection with a data consumer. Multiple simultaneous connections can be established. For each connection, the software is able to send specific data channels and re-sign modified frames. A number of output files can also be generated.

IDC/CD Sender/SDD Page 2

5. October 2006

Document History

Version Date Author Description

0.1 23 January 2003 Stephen Lloyd Initial template.

0.2 4 February 2003 Johannes Schreitl

First issue.


Changes following from comments from John Coyne and Stephen Lloyd.


Changes following from comments from John Coyne and Stephen Lloyd. Adding of shutdown and startup scenarios, removal of history features.

0.5 05 November 2003 Stephen Lloyd Conversion to new template and other updates to bring document up to date at start of new CD Sender contract with Siemens.

0.6 01 March 2004 Gerald Klinkl Convert external graphics to inline drawings

0.7 07 January 2005 Gerald Klinkl Document update

0.8 27 January 2005 Gerald Klinkl Add John’s comments

0.9 15 November 2005 Gerald Klinkl Document update

1.0 5 October 2006 Gerald Klinkl Document update


5. October 2006

Contents

1. Scope............................................................................................................................................................. 5 1.1. Identification .............................................................................................................. 5 1.2. System overview ........................................................................................................ 5 1.3. Document overview ................................................................................................... 5

2. External Interfaces ......................................................................................................................................... 5 2.1. Libraries ..................................................................................................................... 5

2.1.1. Public Libraries .................................................................................................. 5 2.1.2. IDC Libraries...................................................................................................... 5

2.2. Rationale..................................................................................................................... 5 2.3. General Implementation............................................................................................. 5

2.3.1. Requirements...................................................................................................... 5 2.3.2. Design decisions................................................................................................. 5 2.3.3. External Interfaces.............................................................................................. 5

3. Processing entities......................................................................................................................................... 5 3.1. Main processing entity ............................................................................................... 5

3.1.1. Dependencies ..................................................................................................... 5 3.1.2. Overview ............................................................................................................ 5 3.1.3. Design Decisions................................................................................................ 5

3.2. Schedule processing entity ......................................................................................... 5 3.2.1. Overview ............................................................................................................ 5 3.2.2. Dependencies ..................................................................................................... 5 3.2.3. Design decisions................................................................................................. 5

3.3. Worker processing entity ........................................................................................... 5 3.3.1. Overview ............................................................................................................ 5 3.3.2. Restart................................................................................................................. 5 3.3.3. Design decisions................................................................................................. 5

4. Interface entities............................................................................................................................................. 5 4.1. Configuration interface entity .................................................................................... 5

4.1.1. Overview ............................................................................................................ 5 4.1.2. General ............................................................................................................... 5 4.1.3. Operating mode parameters ............................................................................... 5 4.1.4. Persistence parameters ....................................................................................... 5 4.1.5. Processing parameters ........................................................................................ 5 4.1.6. Input file parameters........................................................................................... 5 4.1.7. Output file parameters........................................................................................ 5 4.1.8. Connection parameters....................................................................................... 5 4.1.9. Logging parameters............................................................................................ 5

4.2. Command interface entity .......................................................................................... 5 4.2.1. Overview ............................................................................................................ 5 4.2.2. Dependencies ..................................................................................................... 5 4.2.3. Design decisions................................................................................................. 5

4.3. File finder interface entity .......................................................................................... 5 4.4. File reader interface entity.......................................................................................... 5

4.4.1. Overview ............................................................................................................ 5 4.4.2. Dependencies ..................................................................................................... 5 4.4.3. Design decisions................................................................................................. 5

4.5. File writer interface entity .......................................................................................... 5 4.5.1. Overview ............................................................................................................ 5 4.5.2. Dependencies ..................................................................................................... 5


5. October 2006

4.6. Sender interface entity................................................................................................ 5 4.6.1. Overview ............................................................................................................ 5 4.6.2. Dependencies ..................................................................................................... 5 4.6.3. Design decisions................................................................................................. 5

Appendix I Additional requirements ................................................................................................................... 5 Terminology............................................................................................................................................................. 5

Glossary.................................................................................................................................. 5 Abbreviations ......................................................................................................................... 5

References .............................................................................................................................................................. 5


5. October 2006

1. SCOPE

1.1. Identification

This document applies to the CD Sender version 1.1.

1.2. System overview

CD Sender is a software system that can send continuous seismoacoustic data received from IMS stations (or from a data centre forwarding data from IMS stations). The data will be sent to data consumers according to the formats and protocols defined in IDC (2002), Formats and Protocols CD1.0 and IDC (2002), Formats and Protocols CD1.1.

The software is able to send connection requests and establishes a connection with a data consumer. Multiple simultaneous connections can be established.

For each connection, the software is able to send specific data channels and re-sign modified frames. A number of output files can also be generated.

The primary user of the software will be the International Data Centre (IDC) Division and the International Monitoring System (IMS) Division at the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO). In addition, it is expected that the software will be used at National Data Centres (NDCs). The software is available to State Signatories.

The software will continue to develop in the future. This is required in order to follow expected advances in the formats and protocols (for example, the addition of command and control messages).

1.3. Document overview

This document defines the CD Sender software design. The software design will be used as the basis for implementation. The design is based upon the software requirements described in IDC (2003), CD Sender Software Requirements Specification.

This document is mainly intended for developers, maintainers and documentation writers. It is also of interest to project management, requirements analysts, quality assurance staff and user representatives.

The design is described in terms of a set of connected entities. An entity is an element (component) that is structurally and functionally distinct from other elements and that is separately named and referenced. Entities may be sub-systems, data stores, modules, programs, processes, or object classes. Entities may be nested or form hierarchies.

Each entity is described in terms of requirements and design decisions.

Each mandatory, testable requirement is stated using the word shall. Therefore, each shall in this document should be traceable to a documented test. Each mandatory, non-testable requirement is stated using the word will. Each recommended requirement is stated using the word should. A permissible course of action is stated using the word may. This convention is used in ISO/IEC (1995), ISO/IEC12207.

Each mandatory, design decision is stated using the word shall or will. Each design recommendation is stated using the word should. A permissible course of action is stated using the word may. This convention is used in ISO/IEC (1995), ISO/IEC12207.


5. October 2006

This document is compliant with IDC (2003) Software Design Description Template, IDC (2002), Software Documentation Framework and the CTBTO/PTS (2002), Editorial Manual.


5. October 2006

2. EXTERNAL INTERFACES

All external interfaces visible to the user (command line interface, configuration options, input files and output files) are described in IDC (2004), CD Tools Software User Guide.

2.1.Libraries

2.1.1. Public Libraries

Table 1. Public Libraries

Type Description

Standard C Required for all C programs and used automatically when a C program is linked.

POSIX threads Required for the POSIX threads of CD Receiver (-lpthread).

Socket Used for the network communication of CD Receiver (-lsocket and -lnsl; included in the standard C library on Linux).

Mathematic Used for simple arithmetic by CD Receiver (-lm; most of the functions are included in the standard C library on Linux).

Crypto Used for authentication verification. CD Receiver uses a library coming with the openssl package (-lcrypto).

LDAP Used to retrieve certificates from an LDAP directory server. CD Receiver uses a library coming with the openldap package (-lopenldap).

2.1.2. IDC Libraries

Table 2. IDC Libraries

Type Description

cancomp Used for Canadian compression/uncompression and is part of the CD-Tools distribution.

paridc Used for reading and processing parameter files and is part of the CD-Tools distribution.

idcsyslog Used logging to syslog and for debug messages and is part of the CD-Tools distribution.

cd Used for decoding of CD frames, signing and verifying frame data and is part of the CD-Tools distribution

2.2. Rationale

To take advantage of concurrency within the OS and the hardware the CD Sender has been designed as a multithreaded implementation. The multithreading approach guarantees full use of multiprocessor systems and exploits the high level of inherent parallelism.

2.3. General Implementation

The sender has been written in ANSI C and has been designed to run on UNIX and Linux systems.


5. October 2006

2.3.1. Requirements

The requirements for the CD Sender are defined in IDC 2004, CD Sender Software Requirement Specification.

2.3.2. Design decisions

The CD Sender is designed as a multithreaded application with a main thread, a scheduler thread, an optional command thread, an optional certificate thread, and one or more handling threads. The main thread is responsible for initialisation and handling of signals, the scheduler thread checks for new data and informs the handling threads about new data, and the handling threads manage the connections to the data consumers. Threads are synchronized using mutex locks. The application is designed to avoid using locks where possible using a design which does not require locking. Another approach to avoid mutex locks used in the CD Sender is the use of ‘atomic operations’. An ‘atomic operation’ is an operation which can be executed as a single CPU statement and needs therefore no mutex lock. Changing a pointer value is an example for an ‘atomic operation’. If the critical value cannot be changed by an ‘atomic operation’ (e. g. modifying a string value), the critical value is transformed to an atomic type (e. g. a pointer alternating pointing to two strings). An example where this transformation could be used is a file path. The file path is a string used by all handling threads. It cannot be change directly, because one or more threads may use the string while it is being modified. This would lead to unexpected results. But if all threads use a pointer to a string and the new value is copied into another string and then the pointer value is changed, it is to use for all threads. Another method to avoid mutex locks in the CD Receiver is to design the code to have only one writer (thread) for a particular state of an object. An example for this method is the per station certificate cache. It is initialized by the main thread (during pre-load, no other threads started) or by the scheduler thread (when the production line is initialised) and modified by the certificate thread (only if the cache has the proper state). Functions of common interest are designed as library functions and are implemented in a separate library, libCD. This library is used also from CD Receiver and CD2W. The CD Sender will require a Concise List of Frames (clf) file for each binary data file to send and an index file which contains information where the Concise List of Frames files are located to be able to send CD frames. This decision was made to avoid re-parsing of binary files, because the sender has to know size, offset and nominal time of frames in the binary files to sort the frames according to the current sending mode. No protocol transformation from CD-1.0 format into CD1.1 format or vice versa will be supported.

Deleted: Receiver

Deleted: listener

Deleted: during first

Deleted: connection request


5. October 2006

2.3.3. External Interfaces

The CD sender uses the following external interfaces:

Optional

File system

Configuration

Data Station

wfdisc records

Certificates

write

concise listof frames

Syslog

text represen-tation offrames

received

Leased Lines,Internet, etc.

CD1.0CD 1.1

SystemMessages

CD Receiver

Required

CD Frames

Index File

write

Main

CD Sender

Monitor

read

Configuration

read

write read/write

Syslog

Data Consumer

Leased Lines,Internet, etc.

CD 1.0CD 1.1

Receiver - SenderOverview

Internal and External Interfaces

write read

Sch

edul

e

??

Wor

ker

Internalreference

list

write

concise list offrames received

write

Productionline list

DataConsumer

File

sentCD frames

text represen-tation of

sent/receivedframes

Figure 1: CD Receiver – CD Sender Overview Internal and External Interface


5. October 2006

3. PROCESSING ENTITIES

3.1. Main processing entity

The main thread of the sender is primarily concerned with system configuration, starting the scheduler, and the optional command and certificate threads.

3.1.1. Dependencies

Figure 2: Thread Overview Thread Model

3.1.2. Overview

3.1.3. Design Decisions

3.1.3.1. Process flow Main Thread

Worker Thread 1

Scheduler

Worker Thread 2

Main

Certificate

Worker Thread 3

Command


5. October 2006

Figure 3: Process Flow Main Thread

3.1.3.2. States of Main Thread

shutdown

Configure Sender

Start Command Thread

Start Certificate Thread

Start Schedule Thread

Wait for user termination

Wait until all worker threads are stopped

Shutdown all threads

Start

CD- Sender Main Thread Process Flow


5. October 2006

Figure 4: States of Main Thread

3.2.Schedule processing entity

3.2.1. Overview

The scheduler thread of the sender is spawned by the main thread and can run in two modes: fixed interval mode or continuous mode. The scheduler thread spawns the worker threads. The number of spawned worker threads depends on defined entries in the Production Line File.

3.2.2. Dependencies

Please see Figure 2 on page 5.


CD Sender will support manually configured and dynamic created production lines. A manual configured production line is a production line with a defined data provider and defined data consumer. A dynamic production line does not have a defined data provider. It only have a defined data consumer. For each manually configured production line, the scheduler thread will spawn a worker thread at startup time For each dynamic production line, the scheduler thread will spawn a worker thread for each data provider found in the index file. The number of worker threads depends on defined relations in the Production Line File. The scheduler thread will read the configured index file periodically and will pass an initial list of clf files to process to each new started worker thread. The scheduler thread will also update a worker threads clf file list if a new clf file to process is available. The clf file list is implemented as a linked list of structures and a modification of the clf file list must be protected by a mutex lock. This is caused by the fact that the list has to be modified by the worker thread when a clf entry is removed and by the scheduler thread when a clf entry is added.

Initialize

Wait for termination

request or signal to shutdown

Wait for all threads to shutdown

User termination request

All threads to shutdown or second user termination request received

Start application


5. October 2006

The schedule thread can run in several modes: (a) Continuous mode

After startup the scheduler thread searches for frame summary (clf) files in the CD-Receiver generated index files which are created before look back scheduler time interval defined hours backwards from the current start time, generates a list of clf files, and notifies the related worker threads and passes the appropriate clf files to each worker thread. The name of index file depends on the well known port of the related receiver, for more details please see IDC 2004, CD Tools SUG. The search path for the index file depends on the configured file path . Each sleep time scheduler defined seconds, the scheduler thread looks into the index file to see if new clf files are available. The related worker threads will be notified of any newly available clf file.

(b) Fixed Internal Mode In this mode the schedule thread searches in the index files for frame summary (clf) files created after start time and before end time, generates a list of clf files, notifies the related worker threads and passes the appropriate clf files to each worker thread. Then the scheduler thread waits until all worker threads have sent their data and are being stopped. Start time and end time are passed as command line parameters to the sender. If no end time parameter is provided, the current date and time is used as end time.

(c) Auxiliary Station Mode In this mode the schedule thread will behave like in the Continuous Mode, but will not start worker threads immediatelly. Instead it will put all productions lines into stopped state and wait for a “data request” command. The scheduler thread will start the worker thread only when a “data request” command is received. The worker thread will send the requested data, puts the production line into stopped state again and shut down after the requested data have been sent. The schedule thread will start the worker thread again when a new “data request” command arrives. As long as the requested data is found in the clf files defined by current time and lookBackTime , the same data may be requested more than one time. No restart files are written in Auxiliary Station Mode.

Formatted


5. October 2006

3.2.3.1. Process Flow of Scheduler Thread

Figure 5: Process Flow of Scheduler Thread

Start

Shut down

sleep

Search in index files for data files to send

Configure Thread

mode

Data files found

Write file names into buffer of working

threads

Shutdown processing

Shut down

Fixed Interval mode

No

Continuous/ Auxiliary station mode

Spawn worker Threads

Write file names into buffer of

working threads

Data available

Shut down processing

Look in current index file for data

files to send

Write file names into buffer of

working threads

Shut down

no

Termination request

For each relation data provider vs. Data consumer one worker thread exist; Not in auxiliary station mode

Index is written by CD- Receiver

Wait for worker threads finished

Search in index files for data files to send

For each relation data provider vs. data consumer one worker thread exist

Spawn worker Thread(s) if necessary

In auxiliary station mode is the station into

STOPPED state when no

data request is pending


5. October 2006

3.2.3.2. States of Scheduler Thread

Figure 6: States of Scheduler Thread

Initialize

Processing shutdown

Search for data files

Start worker threads

Inform worker thread

Wait for new data

Wait for worker threads finished

Start application

Continuous mode

Termination request

New .clf file available

Check new .clf file available

Data files found

Processing shutdown

Termination request or all data are sent

Fixed interval mode


5. October 2006

3.2.3.3. Process Flow of Scheduler Thread Start Up in Continuous Mode

Figure 7: Scheduler Thread Start Up in Continuous Mode

Spawned and initialized by main thread

Search in index files updated by CD

Receiver for .clf files using look back

parameter

spawn worker thread

Read production line file

For each production line

Create a station object

Create a reference list of found clf files

End for each production line

sleep

Check index file for new clf files


Shutdown

Update reference list of clf files


wait for all worker threads to shut down

No

Yes

spawn worker thread if necessary


5. October 2006

3.2.3.4. Process Flow of Scheduler Thread Start Up in Auxiliary Station Mode

Figure 8: Scheduler Thread Start Up in Auxiliary Station Mode

3.2.3.5. Process Flow of Scheduler Thread Start Up Fixed Interval Mode




parameter




Create a reference list of found clf files


sleep

Check index file for new clf files


Shutdown

Update reference list of clf files


No

Yes

spawn worker thread Put station in stopped

state

Data request pending


No

Formatted: Bullets andNumbering

Formatted: Bullets and

Numbering


5. October 2006

Figure 9: Scheduler Thread Start Up Fixed Interval Mode

A clf file list is never shared between different worker threads, regardless if worker threads use the same data provider. This approach simplifies the design, because it requires only a 1:1 lock of the clf file list. On the other hand there is some performance cost, because each clf file must be parsed in each worker thread.




parameter

spawn worker thread




Create a list of found clf files



Deleted: reference


5. October 2006

Figure 10: Data flow scheduler to worker threads

scheduler

Worker Worker

Index file

Internal reference list

clf clf

clf file list

Internal reference list

clf file list



5. October 2006

3.2.3.6. Process Flow of Scheduler Thread Shut Down

If a shutdown notification is reached by the scheduler thread, the scheduler thread will wait until all worker threads have been terminated.

Figure 11: Scheduler Thread Shut Down

Write log

exit

Notification for shutdown

received



5. October 2006

3.3. Worker processing entity

The worker thread is spawned by the scheduler thread. For each Production Line a worker thread is spawned. Each worker thread manages a single connection to a data consumer. In continuous mode a worker thread remains active until the sender is shut down and is notified by the scheduler thread if new clf files are available for processing.

3.3.1. Overview

In the worker thread there are several functions implemented.

(a) Parse the CD Receiver generated clf files for data frames to send and store them in an internal reference list.

(b) Filter data sub channels by defined rules and assemble, if necessary, new frames containing only sub channels which match the filter condition.

(c) Maintain the connection to a data consumer. (d) Send CD data to data consumer. (e) Logging of sent and received frames. (f) Compute the correct frame to send after a re-start

3.3.1.1. Internal Reference List

The internal reference list is a linked list of frame descriptions stored in memory. The list is either sorted by nominal time or unsorted depending on processing mode. Information related to the current state of the internal reference list will be stored to a restart file every reference list store interval seconds. The name of the restart file reflects the relation between data provider and data consumer: .<data_provider_name>_<data_consumer_name>.irl Example:

.WRA_BRNDC.irl

The restart file will contain information about the current processing mode (FIFO, LIFO or as arrived), the last clf file parsed, the last frame sent, and a list of unacknowledged CD-1.1 frames. Please see IDC 2004, CD Tools SUG, for more details concerning the format of the restart file. This list is used for keeping track of data to be sent to the data consumer. The scheduler thread provides a list of clf files for the worker thread at startup time. The worker thread parses the clf files to find Data and Data Format frames to send and uses information from the restart file to computes the correct restart point (the first frame to send) and adds unsent or not acknowledged frames to the internal reference list. The last step is to sort the internal reference list according the current processing mode. Entries in the internal reference list are removed when the restart file is updated or when the internal reference list is compacted. Only entries which have state acknowledged are removed from the internal reference list. Changing the state to acknowledged is done when:

Formatted

Formatted

Deleted: The worker computes the correct restart point (the first unset frame) using information from the restart file Then the

Deleted: those


5. October 2006

o the state of the frame is sent, but not acknowledged and the internal reference list is compacted (only CD-1.0 connections)

o an AckNack Frame is received which acknowledges the frame (only CD-1.1, not implemented yet)

o the maximum retry count is reached

Sent, not acknowledged

Not sent

Not sent, Retry count > 0

Not sent, requested

Acknow-ledged

Frame sent

Frame sent

Data Request Command

Frame sent

Reconnect/ Restart

Max. retry count reached, IRL compacted or frame acknowledged

Figure 12: Frame states

The internal reference list is periodically updated in continuous mode.

3.3.2. Restart

To prevent loss of data when the connection is closed, the worker thread will write all unacknowledged frames to the restart file. For CD-1.0 connections, the last N frames will always be treaded as unacknowledged. For CD-1.1 connections, frames are acknowledged by AckNack frames sent by the peer CD receiver (not implemented yet). For N a value of 10 seems to be reasonable (to change it search for NBR_RESENT_FRAMES in the source code). This behavior can cause sending previous sent frames when the sender performs a restart. The worker thread uses the information in the restart file to determine which frames have to be sent when the sender restarts. If no restart file is available the sender will start to send data received since the begin of the current day. If the restart information is too old, all frames found in the clf file list are sent. No restart information is written when a “data request” command is being processed.


Numbering


Deleted: the complete frame has been sent (CD-1.0) or when the sent frame has been acknowledged (CD-1.1).

Deleted: go

Deleted: back in the internal reference list before writing or updating the information in the restart file


5. October 2006

3.3.2.1.1. Differences between continuous mode, fixed interval mode and auxiliary mode

Differences between continuous mode, fixed interval mode and auxiliary mode are:

• there is no update of the clf file list (and therefore no update of the internal reference list) in fixed interval mode.

• the worker thread will shutdown in fixed interval mode if the internal reference list becomes empty, i.e. after all specified data have been sent.

• the worker thread will shutdown in auxiliary mode if all requested data have been sent, but will started again if a new data request command is received.

3.3.2.1.1.1. FIFO processing mode

In this mode the reference list is sorted in ascending frame nominal time order. The oldest frames will be sent first.

3.3.2.1.1.2. LIFO processing mode

In this mode the reference list is sorted in descending frame nominal time order. The newest frames will be sent first.

3.3.2.1.1.3.As arrived processing mode

In this mode the reference list is not sorted. The frames will be sent in the same order as they are received.


A worker thread will not terminate if the connection has been closed or cannot established. A worker thread will terminate when a disconnect or stop command is received. Update of the clf list must be protected by a mutex lock.

3.3.3.1. Process Flows



Numbering


Numbering


Numbering


Numbering


Numbering

Deleted: and

Deleted: The two d

Deleted: and


5. October 2006

Figure 13: Process Flow of Worker Thread (overview)

Start

Block A Process restart file

have clf file list

fixed interval mode

No

Yes

No Yes

have clf file to process

have frames to send

Yes

Shut down

fixed interval mode

Yes

Shut down

Save internal reference list

No

Shutdown pending

Block D Connect if

unconnected, send frames and process

received data

Yes

(re-)parse clf file and update reference list

No

Block C Wait with timeout for

received data if connected

No

Block B Check for shutdown and wait until data to send is available if

unconnected

Shutdown pending


5. October 2006

The first action when a worker thread starts is to check for and process the restart file. If a restart file exists, an attempt is made to build an internal reference list containing the same unsent frames as before shutdown of the production line. If the shutdown happened before the (configured) look back time of the scheduler thread, the internal reference list will not contain frames received before the look back time.

Figure 14: Process Restart File (Block A)

After processing the restart file, the worker thread checks if a shutdown is pending. If a shutdown is pending, the worker thread terminates. Otherwise it checks if the connection to

parse clf files until restart

point reached

Read restart information

Restart file valid

Search restart point in clf list

Restart point found

Log warning

Processing mode

skip clf files until restart

point reached

parse restart clf file

parse remaining clf

files

Yes

Yes

Yes

No

No

No

FIFO, LIFO As arrived

Restart file exists

Block A

Deleted: To avoid performance problems when saving huge internal reference list, especially during restarts, the restart file will not contain the same information as stored in the internal reference list. Instead it contains information of the last processing mode, the last parsed clf file, the last sent frame and a list of unacknowledged frames (only CD-1.1)


5. October 2006

the data consumer is established, if a clf file is queued to be processed or if the current clf file has grown since the last parsing. If there is nothing to do, the worker thread waits sleep time worker thread before it checks for new data.

Have data to send

connected

Shutdown pending

No

Yes

No

wait

No

Yes

Yes

Figure 15: Wait until data to send is available (Block B)

If data to send is available, all clf files found in the clf file list will be parsed. If only the current clf file has grown, all the new lines will be parsed. Frames to send found in the clf file will be added to the internal reference list. New elements will always be inserted as first element. If the internal reference list does not contain unsent frames after parsing of the clf file and the connection is established, the worker thread waits sleep time work for received data. If data are received, the Sender tries to read a complete frame and continues with Block B.

Block B

Deleted: until new data arrives.

Deleted: the current

Deleted: (re-)


5. October 2006

Figure 16: Wait with timeout for received data (Block C)

If the internal reference list contains unsent frames after parsing of the clf file, an attempt is made to establish a connection, if necessary, and frames, marked as unsent are sent. After each frame sent the worker thread checks for and processes received data. If the connection cannot be established, the worker thread waits sleep time retry, checks for a pending shutdown and tries again to establish the connection.

connected

Wait for incoming data

or timeout

Data received

Process frame

Yes

Yes

No

No

Block C


5. October 2006

Figure 17: Connect, send data and checks for received data (Block D)

If the connection is established and data to send are available, CD Sender sorts the internal reference list, if necessary, and starts sending frames beginning with the first element in the internal reference list (Block E). Before a frame is sent, CD Sender checks for incoming data and reads a single frame when incoming data are pending. Then the first element on the internal reference list is sent. On CD-1.0 connections, a check is made if it is necessary to send a Data Format Flush Alert Frame and a Data Format Frame before a Data Frame is sent. The sent frame will be marked as sent (CD-1.0) or sent, not acknowledged (CD-1.0) and the sender checks if reference list store interval is reached. In that case it will compact the internal reference list and update the restart file. Then the sender checks for new data arrived in the current clf file (or in a new clf file if a file switch has occurred in the meantime). If no user shutdown request is pending, the next check for incoming data is done and the next frame

connected try to

connect

retryWait

Block E Send frame and

check for and process received

data

yes yes

Irl empty

connected

yes

Pending shutdown

no

no

no

Block D


5. October 2006

is sent. This is repeated until no more frames to send are on the internal reference list, or a user shutdown request is pending or the connection is closed by the remote site.

Sort IRL if necessary

Have more frames to send

Send DFF

Check for and process

incoming data

Send DF

Compress IRL and update

restart file

yes

no

Must send DFF

N data frames sent

no

yes

no

yes

yes

no

Connection closed, shutdown request

Shutdown pending or connection closed

Check for new arrived

data and update IRL


5. October 2006

Figure 18: Send frame and check for and process received data (Block E)

3.3.3.2. States of Worker Thread

Figure 19: States of Worker Thread

3.3.3.3. Process Flow of Worker Thread Shut Down

When the worker thread gets a notification that a shutdown request is pending, it

• sends an Alert frame and closes the connection when it is connected; • updates the restart file when data have been sent; • clear the internal reference and exits

The parsing of a clf file is not stopped when a user termination request is pending. The pending user termination request is served when the parsing of the clf file is completed.

Initialize Wait for new data

Start application

user Termination

All data sent in fixed internal mode, requested data sent in auxiliary mode or shutdown request

Check clf file

Send frame

Processing received frames

Processing shutdown

Data sent in continuous mode

Update reference list

Determine data available to send

Frame received


Numbering


Numbering

Deleted: and the clf file list


5. October 2006

Figure 20: Worker thread shutdown processing

3.4. Certificate processing entity

The certificate processing entity is the same as used for CD Receiver and is described in IDC 2004, CD Receiver Software Design Description.

connected

Update restart file

Shutdown notification

Clear IRL and clf list

Shut down

yes

Send alert frame and disconnect

no


5. October 2006

INTERFACE ENTITIES

3.5.Configuration interface entity

3.5.1. Overview

The CD Sender is using several text files to import configuration data. If any configuration file is not found, a warning will written to stderr and the software will stop.

3.5.2. General

This chapter gives only a rough overview of the interface entities. You will find a detailed description of the command line interface and the format of the configuration files in IDC 2004, CD Tools SUG. The CD Sender will use libCD functions for reading and parsing common used files like the configuration file, the station file and the key preload file. File parsing functions used by the CD Sender, which are of common interest will be designed as library functions. Examples for such functions are parsing of the index file and parsing of clf files.

3.5.2.1. Command Line Interface

The CD Sender accepts one mandatory argument, which is the name of configuration file and two optional arguments, which are start time and end time for fixed mode processing. This configuration file contains all configuration parameter and references to additional configuration files. (e.g. name of Production Line File) The parameters of the configuration file are described below. If the sender is running typing ctrl-c can stop it. If no configuration file is specified an error message will written to stderr and the software will stop. If in the configuration file some parameters are missing, a warning will written to stderr and syslog, and default values will be used.

3.5.2.2. Additional Mandatory Configuration Files

Additional mandatory configuration files are: (a) Data Consumer File (b) Production Line File If either of these files are missing or are invalid, the CD sender software will write an error message to stdout, syslog and stop. In the Data Consumer File there is at least one configuration of Data Consumer required. Also in Production Line File there is at least one configuration of a Production Line required. If these minimum requirements are not available an error message will written to stderr, syslog and the software will stop.

3.5.3. Operating mode parameters


5. October 2006

3.5.3.1. Design decisions

The operating parameters that may be specified in the configuration file are described in the following table:

Table 3. Operating mode parameters

Parameter Description

Sleep time scheduler For scheduler thread The number of seconds to sleep while waiting for new data (continuous mode only).

Sleep time worker For worker thread The number of seconds to sleep while waiting for new data (continuous mode only).

Processing mode Processing sequence of arrived data (FIFO,LIFO, as arrived)

Look back time scheduler Time in hours, the scheduler thread looks for data files after startup which are maximal older of the scheduler’s startup time when a restart file exists. Otherwise only data files created at the current day are sent.

3.5.4. Persistence parameters


The persistence parameters that may be specified in the configuration file are described in the following table:

Table 4. Persistence parameters


Reference list store interval Number of sent frames after the restart file should be updated and the internal reference list compacted

Reference list store path Path to store area of internal reference list

3.5.5. Processing parameters


The processing parameters that may be specified in the configuration file are described in the following table:

Table 5. Processing parameters


Processing mode Processing sequence of arrived data (FIFO,LIFO, as arrived)


5. October 2006

3.5.6. Input file parameters


3.5.6.1.1. Data Consumer File

The Data Consumer File contains a list of valid data consumer names, IP-addresses and well known ports for which connections will be established and processed.

3.5.6.1.2. Production Line File

The Production Line File contains a list of valid station names and specifies which data are sent to data consumers. Station data can be extracted and sent to one or multiple data consumers (semicolons are delimiters). Each consumer must be defined in ‘Data Consumer File’. For each relation Data Provider vs. Data Consumer, one line is required. The Scheduler thread will spawn a worker thread for each Data Provider/Data Consumer pair. Each line in the Production Line File has the following fields: (a) station name (b) data consumer name or wildcard ‘*’ for passive data forwarding (c) extraction of site rule (regular expression only) (d) extraction of channel (regular expression only) (e) station name of data consumer (not implemented yet)


The input file parameters that may be specified in the configuration file are described in the following table:

Table 6. Input file parameters


Index file path The operating directory where the index files by the CD Receiver will be written.

port Well known port of the related CD-Receiver written index file. This value is part of the name of index file. A defined port is mandatory.

Station file Path and name of file containing names, IP address and station names which are allowed to connected to the command port of the CD Sender.

Data Consumer file Path and name of file containing the names, IP address and CD Receiver’s well known port.

Production Line file Path and name of file containing the description of which data has to be sent from each station to each data consumer and which pattern are defined to extract the proper set of sites and channels.

3.5.7. Output file parameters



Numbering

Formatted

ERROR: syntaxerrorOFFENDING COMMAND: --nostringval--

STACK:

38 7294 1

Documents

CD Sender Software Design Description - CTBTO