Angewandte Wissenschaft Software und Technologie (AWST) GmbH, Mariahilfer Str. 47/3/1, A-1060 Vienna, Austria

Telephone: +43-1-5861314 – Facsimile: +43-1-5861314-22 – Email Address: ertl@awst.at

DisclaimerThe views expressed on this poster are those of the

author and do not necessarily reflect the view of the

CTBTO

SnT2015 Poster No. T3.3-P14 IDCDACS: IDC’s Distributed Application Control System ported to Open TechnologiesMartin Ertl1, Alexander Boresch1, Ján Kianička1, Alexander Sudakov2, Elena Tomuta2

(1) Angewandte Wissenschaft Software und Technologie (AWST) GmbH, Vienna, Austria

(2) Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO), Vienna, Austria

Background

The Distributed Application Control System (DACS) is the

backbone of the automatic processing of seismic, hydro-

acoustic and infrasound (SHI) waveform data at the IDC. It

drives the execution of processing applications by organizing

data into time intervals, processing steps into pipelines, and

using message queues for task scheduling.

Objective

Because licensed software hampers free distribution and

use of IDC software to and by National Data Centres, IDC

aimed at eliminating the dependency on the proprietary Tuxedo

middleware.

Accomplishments

We redesigned the existing system and implemented the new

IDCDACS based on an open-source messaging solution in

combination with existing in-house IDC libraries and a custom-

developed application framework, which together replace

Tuxedo in a robust, reliable and scalable way.

Open Technologies

Our solution utilizes the RabbitMQ high availability message

broker and the Advanced Message Queuing Protocol (AMQP),

an open industry standard and wire-level protocol mandating

that senders and recipients can interoperate irrespective of

their specific implementation.

IDCDACS is flexibly configurable to control different

processing applications and pipeline-like processing workflows,

i.e. it is by design not limited to SHI data processing, which is

the primary use case at the IDC.

Development Approach

The new IDCDACS was implemented using the Scrum agile

development methodology, aligned with evolving requirements

and priorities. The first major Release is running on the IDC

development LAN. Transition to Testbed and Operations

LANs is in progress. Further development will improve

usability, extend features, and lift current limitations.

Message Queue for Task Scheduling

Figure 2: Task Queue decouples Publishing and Consuming Clients. Yields a reliable and robust

system. Permits parallel processing, for load balancing and scaling up.

RabbitMQ Message Broker

Consuming Client 1

Publishing Client

Data Monitor, Tortoise

publish Task Queuerouting Consuming Client 2

Consuming Client N

Consumer 1

Consumer 2

Consumer N

Direct Exchange

Round-robbin dispatch with prefetch=1

consume

consume

consumeConnection + Channel Binding

Connection + Channel

Toirtoise, Database Service

Processing Pipelines and Message Flow

Figure 3: DACS processes (blue) run as daemons and listen on Message Queues (purple).

Scheduler wakes up Data Monitor. Data Monitor checks conditions (data availability, status

information, time, etc.) and creates Time Intervals. Tortoise executes Processing Application (aqua),

reports status to Interval Table, and publishes to next Task Queue. Each Time Interval passes through

all Steps until reaching Done Queue. In case of an error it will be short-circuited into Failed Queue.

Deployment and Distribution

Figure 4: 1 Database, 1 Message Broker, 1-N Processing Servers for

DACS Server Programs and Processing Applications.

ProcessingServer 2

Oracle DatabaseAccount (User)

RabbitMQ Message BrokerVirtual Host

ProcessingServer 1 Operator Workstations

ProcessingServer N

. . .

Software Application Framework

Figure 1: Building blocks for common functionality like main control

loop, exceptions, resource management, transactional messaging,

client-side failover for high-availability cluster.

Analyzable and Maintainable Code

Figure 5: Object-oriented programming style.

/* Create and setup the application object */

app = Application.new(NULL, argc, argv);

/* Create the shell object */

shell = plain_shell_builder(1, "ticron_rabbit", app);

/* Create resource objects */

dbconn = DbConn.new(0, "db_connection", shell);

mbconn = MbConn.new(0, "mb_connection", shell);

mbconsumer = MbConsumer.new(0, "mb_consumer", shell, mbconn);

/* Create the message processing state machine */

machine = StateMachine.new(10, "processor", shell);

aesi

r

std

tim

e

par

idc

tab

le

cbase

gdi

ibase

rabbitmq

third party

idcdacs

domain layer

idcd

acsl

og

idcp

ipel

ine

idcr

ab

bit

idcstate

technical services layer / application framework

tis_

rab

bit

tise

g_

rab

bit

tin

_ra

bb

it

ticr

on

_ra

bb

it

tort

ois

e

db

_ra

bb

it

application layer

dbconn

idcstate::resource

mbconn

mbconsumer

msgstore

idcstate::activity

mbreader

message_builder

msgwriter

genmsg_builder

intvlmsg_builder

message

cmdmsg

sqlmsg

intvlmsg

idcstate::state

state

pseudostate

activity

state_machine

resource

application_signal

event

state_impl

application

shell