Messaging Architectures with NoSQL Databases as Message Stores

MESSAGING ARCHITECTURES WITH NOSQL

DATABASES AS MESSAGE STORES

JavaOne 2012 Conference (#JavaOne)

Srini Penchikala (@srinip)

10.4.12

GOALS OF THIS PRESENTATION

NoSQL databases to store messages with focus on EIP

Patterns

Target Audience: Integration Architects and Developers

Redis, Oracle NoSQL, and MongoDB databases

Demos (Java application)

Not a NoSQL or Messaging intro presentation

Format:

50 min presentation & demos

10 min for Q&A

2

WHO AM I

Software Architect

Co-author of Spring Roo in Action book

Editor at InfoQ.com

Background:

Financial services industry

Regulatory compliance (PCI & FISMA)

J2EE architecture model

Scrum for software development

Agile (v./and) Architecture 3

QUICK POLL

Who is using EIP Patterns?

Spring Integration?

NoSQL?

Spring Data?

4

TODAY’S DISCUSSION

Enterprise Integration Patterns

NoSQL databases

NoSQL as message stores

MongoDB / Redis / Oracle NoSQL Database

Spring Data

Spring Integration framework

Sample Java application

Best Practices

Conclusions 5

MESSAGING

Advantages

Limitations

Message Loss

Performance

Security

Availability

Design Considerations*

6

*Main focus of this presentation

MESSAGING PATTERNS

Message Store

Claim Check

Queue Channel

Aggregator

Resequencer

Control Bus*

Wire Tap*

7

URL: http://www.eaipatterns.com

*System monitoring and management

MESSAGE STORE

Store Messages in a persistent

store (e.g. Database)

Why Message Store?

Performance

Availability

8

http://www.eaipatterns.com/MessageStore.html

MESSAGE STORE USE CASES

Message buffering

Aggregator

Resequencer

Queue Channel

Message loss prevention

Claim Check

9

CLAIM CHECK

Store message data in a

persistent store and pass a

Claim Check to subsequent

components.

These components can use

Claim Check to retrieve the

stored information.

10

http://www.eaipatterns.com/StoreInLibrary.html

PERSISTENCE

Key Value Data Store

Redis

Oracle NoSQL

Document Database

MongoDB

GemFire

JDBC

In-Memory

11

NOSQL DATABASES

Redis

Redis-based MessageStore

Publish-Subscribe Messaging adapters

Oracle NoSQL Database

Distributed key-value database

ACID transactions

Administration via Web console or CLI

Use Cases: Online Display Advertising & Mobile Social Gaming

MongoDB

Document database

12

http://www.redis.io, http://www.mongodb.org

http://www.oracle.com/technetwork/products/nosqldb/overview/index.html

SPRING INTEGRATION

Extension of the Spring framework

Implements Enterprise Integration Patterns (EIP)

Lightweight messaging

Integration with external systems via declarative adapters

Separation of concerns

13

*http://www.springsource.org/spring-integration

SPRING DATA

Spring applications that use new data access technologies

Non-relational databases

Map-reduce frameworks

Current Implementations:

Big Data (Apache Hadoop)

Data-Grid (GemFire)

Key Value Stores (Redis)

Document Stores (MongoDB)

Graph Databases (Neo4j)

Column Stores (Hbase)

14

*http://www.springsource.org/spring-data

SPRING ROO

Rapid Java application development

Nice integration for Spring Integration and Spring Data

frameworks

15

*http://www.springsource.org/spring-data

MESSAGE STORE “STRATEGY”

Based on Strategy design pattern

MessageStore strategy

Used by QueueChannel and ClaimCheck patterns

MessageGroupStore strategy

Used by Aggregator and Resequencer patterns

Implementations

SimpleMessageStore (In-memory, Map-based)

JdbcMessageStore (uses a relational database via JDBC)

NoSQL Data Store

16

MESSAGE STORES

17

Message Store

Simple (In-Memory)* NoSQL JDBC

Redis* MongoDB* Oracle NoSQL* GemFire

*Covered in the sample application demos

SAMPLE APPLICATION

Oracle NoSQL Database

Redis

Spring Integration

Spring Data

JUnit

18

*Code will be posted on Github

DEMO APPLICATION - USE CASE 1 (CLAIM CHECK)

19

Message Header

Enricher

Message

Store

(In-Memory)

Claim

Check (In)

Queue

Channel Transformer

Claim

Check (Out)

Message

Store

(MongoDB)

MESSAGE STORE DEMO 1

In Memory Message Store

MongoDB

20

DEMO APPLICATION - USE CASE 2 (AGGREGATOR)

21

Message Aggregator

Message

Store

(Redis)

Queue

Channel Transformer

Message

Store (Oracle

NoSQL)

ARCHITECTURE DIAGRAM

22

Source:

http://static.springsource.org/spring-integration/docs/latest-ga/reference/html/samples.html

MESSAGE STORE DEMO 2

Redis Demo

23

CUSTOM MESSAGE STORE IMPLEMENTATION

MessageStore interface

CustomOracleNoSqlMessageStore class

24

MESSAGE STORE DEMO 3

Oracle NoSQL Message Store

25

MESSAGE STORE LIMITATIONS

When using JdbcMessageStore, only Serializable data is

persisted by default

Different instance of some Spring Bean is returned upon

deserialization for headers that represent certain types of

data

Temporary Reply Channel issue

26

CONCLUSIONS

Message Persistence

Message Store pattern

Claim Check

NoSQL and Messaging

“One Size Fits All” fits nothing

27

RESOURCES - BOOKS

28

Enterprise Integration Patterns

(http://eaipatterns.com) Spring Integration in Action

(http://www.manning.com/fisher)

Spring Roo in Action

(http://www.manning.com/rimple)

RESOURCES

Enterprise Integration Patterns (http://www.eaipatterns.com)

Spring Integration (http://static.springsource.org/spring-integration/reference/html)

Spring Integration Support for Redis (http://static.springsource.org/spring-integration/reference/html/redis.html)

Oracle NoSQL Database (http://www.oracle.com/technetwork/products/nosqldb/overview/index.html)

http://blog.codecentric.de/en/2012/04/spring-data-redis/

Message Store (http://www.eaipatterns.com/MessageStore.html)

Redis (http://redis.io/topics/pubsub)

MongoDB

Spring Data

Spring Data Redis

Sample Application on Github 29

THANK YOU

Contact Information

http://www.infoq.com/author/Srini-Penchikala

srinipenchikala@gmail.com

@srinip

http://srinip2007.blogspot.com

Spring Roo in Action Book

Questions?

30

BONUS SLIDES

Bonus Slides

31

MESSAGING GATEWAYS

JMS

AMQP

File

FTP/SFTP

HTTP (REST)

WS (SOAP/POX)

JDBC

Twitter

MongoDB

Redis 32

CONTROL BUS

Administer a messaging

system distributed across

multiple platforms

Uses same messaging

mechanism used by

application data, but uses

separate channels to

transmit data for the

management of message

flow components

33

http://www.eaipatterns.com/StoreInLibrary.html

WIRE TAP

Inspects messages that

travel on a point-to-point

channel

Recipient List into the

channel that publishes

each incoming message to

the main channel and a

secondary channel

34

http://www.eaipatterns.com/WireTap.html

XML CONFIGURATION OF MESSAGE STORE

<bean id="redisMessageStore"

class="org.springframework.integration.redis.store.RedisMess

ageStore">

<constructor-arg ref="redisConnectionFactory"/>

</bean>

<int:channel id="somePersistentQueueChannel">

<int:queue message-store="redisMessageStore"/> <int:channel>

<int:aggregator input-channel="inputChannel" output-

channel="outputChannel" message-store="redisMessageStore"/>

35

MONITORING

Control Bus

JMX

36

MONITORING SCREENSHOT

Screenshot

37

MONITORING (DEMO 4)

DEMO

38