InvenireAude.com Copyright (C) 2015 Invenire Aude Ltd.

Practical Guide to

Data Processors

Data Processors

Data processors can be configured to act as:

Data transformation nodes, using PASCAL-like script language, Gateways and bridges (e.g. HTTP/JSON and Queues/XML), SQL Database interfaces using the data mapping script extension.

You can configure and use the Data Processors as single threaded programs but you can define many logic implementations and run them in parallel as threads.

You can choose the transaction support from the three available modes: auto-commit, single phase (independent) commits, distributed two phase commit with XA when the supported coordination software is used.

And last but not least, one can find the Data Processors as a very helpful command line admin's tool.

Architecture

Concept

The Data Processor is designed to process the incoming messages calling desired action accordingly.

1. Begin remote transactions (optional),2. Receive message,3. Parse content (if configured) and call prerequisit actions (if defined).4. Invoke the configured logic.5. Commit or rollback transaction (in case of an exception).

The program can interact with databases and send messages according to the requirements.

First Usage

export IAS_LANG_XSD=${your_project}/data/customer.xsd

File operations:

ias_qs_processor ­i file:customer.xml ­o file:stdout ­l fwd

ias_qs_processor ­i file:cust.xml ­o file:stdout_fmt ­l fwd

ias_qs_processor ­i file:c1.xml ­o file:somedir/c2.xml ­l fwd

­l fwd vs ­l fwd! Turns the parsing off.

ias_qs_processor ­i file:customer.xml \   ­o file:stdout_fmt?format=JSON ­l fwd

Directories

ias_qs_processor ­i dir:data/xml ­o file:output ­l fwd

The above command will display subsequent messages on the terminal output. In order to direct the messages into series of files use the dir protocol in output specification.

ias_qs_processor ­i dir:data/xml \­o 'dir:output/${ID}.${COUNT}.out' ­l fwd

The parameter is a directory name (output in this case) and the file name pattern. 

Timer

The next command will produce two messages with a one second (1000ms) delay.

ias_qs_processor ­N 2 ­i timer:clock?timeout=1000 \    ­o file:stdout ­l fwd

Queue Operations

ias_qs_processor ­i shm://sample/Q.TEST \­o file:stdout ­l fwd

By default, the messages (and files) are read in the shared browser mode.

ias_qs_processor ­i shm://sample/Q.TEST?mode=input \­o file:stdout ­l fwd

Note that if the transport layer does not support browsing (e.g. HTTP/FCGI) omitting the modifier yields in an error.

By default, if there are no messages available, the reading ends immediately. You can specify the timeout parameter (a number of milliseconds or -1 for an infinite wait).

ias_qs_processor \ ­i shm://sample/Q.TEST?mode=input?timeout=10000 \ ­o file:stdout ­l fwd

Network Operations 1

When the receiver is going to be a server, one can start it with the command:

ias_qs_processor  ­i nets://localhost:10000/?mode=input  ­o file:stdout_fmt ­l fwd

and invoke a client with the matching address specified on its output:

ias_qs_processor ­i file:data/xml/customer.xml ­o netc://127.0.0.1:10000/ ­l fwd

Network Operations 2

The data flow in the other direction is also possible. 

Server:

ias_qs_processor ­o nets://localhost:10000/?mode=input  ­i file:data/xml/customer.xml ­l fwd

Client:

ias_qs_processor ­i netc://127.0.0.1:10000/?mode=input ­o file:stdout_fmt ­l fwd

Modes

Typical operation patterns are:

● Producer,● Consumer,● Requester,● Responder,● Publisher,● Subscriber,● Browser,● Shared Worker Browser (default!)

Producer and Consumer

Requester and Responder

Publish-Subscribe

The application calls the send() and receive() procedures and is not aware of such a configuration.

Not all transports support this pattern.

Implementing a Service

In this case API allows to perform send() and receive() calls on both i/o descriptors.

One can specify the same or different names for the requester input and output.

Browser

autoRecycle specifies if the processing should resume after end of data condition occurs.

● never Browser is not reset when no message is available.

● next Browser will be reset where no message is available but the exception is thrown this time. The next call will grab a new message.

● immediate Browser will be reset where no message is available and the program attempts to read a message immediately.

Shared Browser

This time, the dedicated thread reads messages on behalf of all logic threads. This pattern allows to take full advantage of parallel processing.

Built-in Actions

Typical operation patterns are:

● Forward.● Execute.● Input data type match execute.● Attribute based selection execute.● Job Daemon.● Synchronous proxy.● Asynchronous proxy.

● Message preview.● Message extraction.● Queue System statistics producer.● Writing custom logic.● Java Script execute.● Simplistic Process Server.

Considerations

Performance Notes

Options:

 ­T <no. of threads>

 ­C <commit count>

 ­N <no. of messages to process> ­O <no. of messages to skip>

Parsing

export IAS_LANG_XSD=${your_project}/data/customer.xsd

Note the exclamation mark right after the logic name:

­l fwd vs ­l fwd! Turns the parsing off.

Command Line Hacks

Data Format Change

 ias_qs_processor ­i mqm://MQTEST/Q1?mode=input \   ­o mqm://MQTEST/Q2?format=JSON ­l fwd

 

Working with Meta Attributes

 ias_qs_processor ­i mqm://MQTEST/Q1?mode=input \   ­o 'mqm://MQTEST/Q2?*Folder.ATTR1=VALUE' ­l fwd

The '*' attributes provided in URL are considered as message meta attributes and will be seen in RFH2 or message properties according to the application or IBM WebSphere MQ version.

 

Protocol Gateway

Example Queue Setup

DEF QL(GATEWAY.IN)DEF QL(SERVICE.IN)DEF QA(GATEWAY.OUT) TARGQ(SERVICE.IN)

Echo Service

If you do not have a service for this example just start the following command to setup the echo program.

ias_qs_processor \ ­i 'mqm://MQTEST/SERVICE.IN?mode=responder&timeout=­1' \  ­o dummy: ­l fwd!

 

Gateway Synchronous Proxy

ias_qs_processor \  ­i 'srvhttp://localhost:50000/mode=input&responderName=input' \  ­o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=20000' \  ­l proxy! ­m server

Note the srvhttp service and the proxy logic. 

Gateway Asynchronous Proxy

ias_qs_processor \  ­i 'asrvhttp://localhost:50000/mode=input&responderName=input' \  ­o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=­1' \  ­l aproxy!

Note the asrvhttp service and the aproxy logic.

Testing

curl ­X POST ­d "{ 'key': 'value1234' }" \     ­H "Content­Type: application/json" \        http://localhost:50000/

You can use the HTTP client of your choice.

Modes

processor Each processor thread opens a session and processes incoming messages until the end conditions are met. The threads are not restarted and program terminates when all threads terminate. This is a default mode.

server The session processing is as in the processor mode. However when the thread encounters the end of data condition, the session is recreated. This mode is useful for the synchronous network servers and when one wants to automatically release resources after some period of inactivity.

Transaction Control

NONTXN No transaction call are executed.

TXN After each iteration the commit (or rollback) API call will be executed on success (error).

XA No transaction control calls are executed and it is assumed that proper XA resource manger is configured.

Configuration Files

  ias_qs_processor ­f <configuration_file>

Configuration (XML or JSON)

- input and output specification (Queues, HTTP(S), Files)

- data sources (data bases)

- logic(s) configuration

You can split the configuration into separate definitions of inputs, outputs or/and datasources.

ias_qs_processor ­i inputs.xml ­o outputs.xml ­d ds.xml

Environment Specific Settings

Writing Data Processor Configuration

Follow the XSD files from the IAS-QsystemLib project. You can write in XML or JSON.

File name Contents

qsystem.service.xsd  QSystem service data schema.qsystem.workers.xsd  QSystem main processor schema.qsystem.workers.logic.xsd  QSystem processor logic schema.qsystem.workers.ds.xsd  QSystem data sources (SQL) schema.qsystem.workers.io.xsd  QSystem input/output (messaging) schema.qsystem.workers.txm.xsd  QSystem XA/TX schema.qsystem.workers.spec.xsd  QSystem combined specification schema.qsystem.workers.cache.xsd  QSystem cache module schema.

Input Configuration

<inputSpec> <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output"> <connection alias="qs.esb"/> <destination>CIF.INQ.IN</destination> <txnMode>NONTXN</txnMode> <timeout>-1</timeout> </inputs> </inputSpec>

The above configuration creates a server.

The responderName field creates corresponding output to send replies.

The connection details can be specified here or are referenced via an alias name from the registry file (as shown above).

Output Configuration

The above configuration creates a client.

The inputDestination field specified where the underlying protocol expects the replies .

<outputSpec> <outputs xsi:type="io:RequesterOutput" outputName="output"> <connection alias="qs.esb"/> <destination>WWW.INQ.OUT</destination> <timeout>2000</timeout> <attrUpdates name="FMT" value="XML" override="true"/> <inputDestination>WWW.INQ.IN</inputDestination> </outputs> </outputSpec>

Exit Configuration

The above configuration specifies that the cb::ValidateCustomer program is to be called.The exit program parameters are matched against the input data type.

Available error actions are: IGNORE_ALL, IGNORE_PROGRAM, IGNORE_NONE.

<inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output" > <connection> <protocol>mqm</protocol> <host>MQTSTBRK</host> </connection> <destination>DB.INQ.IN</destination> <txnMode>TXN</txnMode> <timeout>-1</timeout> <attrUpdates name="DB" value="@timestamp@" override="true"/>

<onSend> <program>cb::ValidateCustomer</program> <errorAction>IGNORE_ALL</errorAction> </onSend>

</inputs>

Data Source

<datasourceSpec> <datasources name="ds.mcif"> <connection> <protocol>db2</protocol> <user>user</user> <password>password</password> <location>CIFDB</location> </connection> <txnMode>TXN</txnMode> </datasources> </datasourceSpec>

This example provides all configuration without referencing the registry.

Logic – Explicit Execute

In this scenario the processor will start two threads.Note how “input” and “output”, “error” i/o destinations are specified.

<logicSpec> <logics xsi:type="logic:ExplicitExecute" error="brk.error" instances="2"> <load>esb::broker::inq::routeInput</load> </logics> <logics xsi:type="logic:ExplicitExecute"

input="CIF" output="output" error="brk.error"> <load>esb::broker::inq::routeOutput</load> </logics> </logicSpec>

Logic – Mach Execute

<logicSpec> <logics xsi:type="logic:MatchExecute"> <load>mcif::svc::party::getParties</load> <load>mcif::svc::role::getContextRoles</load> <load>mcif::svc::role::getPartyProducts</load> <load>mcif::svc::comm::getCommunicators</load> </logics> </logicSpec>

In this scenario the processor will find programs that match the input data.

Logic – Input and Output

Each logic has at least the following defaults:- input- output- error (optional)

Above are used by the predefined logic implementations and the script execution framework.

You can specify many input and output definitions in the configuration file. Then you can either assign them as defaults forthe desired logic or use them explicitly via std::send() or std::receive()Script calls.

Basics of Scripting

Environment

Data model:IAS_LANG_XSD=/home/ias/data/xsd/model.xsd

Script Sources:IAS_LANG_SRC_DIRS=/home/ias/lang

First Script

 vi /home/ias/lang/update_msgs.y

The Eclipse source highlighting plugin is available.

See the Script Language for more details (for example on writing procedures accessing databases and files and more).

Processing Data

Testing, browsing and displaying result on the screen:

ias_qs_processor ­i mqm://TEST/Q1 \ ­o file:stdout_fmt ­l exe update

Processing, moving and saving changed content in the output queue:

ias_qs_processor ­i mqm://TEST/Q1?mode=input \ ­o mqm://TEST/Q2 ­l exe update

Learn More

http://www.invenireaude.com/content/books/QSystemProc/index.html?show=chapters/ref/index.html