Upload
ceposta
View
1.177
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
CamelOne 2013 June 10-11 2013
Boston, MA
Apache ActiveMQ Internals
Christian Posta Senior Consultant and Architect
June 10, 2013
1
CamelOne 2013
Cam
elO
ne
2
Why are You Here?
• Use ActiveMQ?
• What is ActiveMQ?
• Dug into the code?
• Interested in contributing back?
• Understand configuration better
CamelOne 2013
Cam
elO
ne
3
Goals of Presentation • Give back to community
• Deliver understanding about architecture
• Brief intro to ActiveMQ
• Overview of architecture
• Dive into major subcomponents
CamelOne 2013
Cam
elO
ne
4
Who am I?
• Senior Consultant and Architect at Red Hat (formerly FuseSource)
• Committer on ActiveMQ
• Author: Essential Camel Components DZone Refcard
• Christian Posta
• Blog: http://christianposta.com/blog
• Email: [email protected]
• Twitter: @christianposta
CamelOne 2013
Cam
elO
ne
5
What is ActiveMQ?
• OpenSource messaging broker
• Apache v2 licensed
• Very active community
• Used at top companies
• 5.8.0 latest release (in JBoss A-MQ 6.0) ���5.9.0 on its way
CamelOne 2013
Cam
elO
ne
6
Features • High performance
• High availability
• Light-weight
• Multi-protocol
• JMS compliant
• Supported in production
CamelOne 2013
Cam
elO
ne
7
When to use ActiveMQ?
• Asynchronous communication
• Reliable message passing
• Loose coupling
• Heterogeneous integration
• Real-time data
CamelOne 2013
Cam
elO
ne
8
This is ActiveMQ…
CamelOne 2013
Cam
elO
ne
9
How it’s built
• Java project (supports Java 7!)
• Maven project
• Divided into client/broker and feature modules
• Unit tests in activemq-unit-tests
• mvn clean package –Dtest=false
CamelOne 2013
Cam
elO
ne
10
ActiveMQ Source
• SVN at Apache: https://svn.apache.org/repos/asf/activemq/trunk/
• Git at Apache: ���http://git.apache.org
• Github: ���https://github.com/apache/activemq
• Use git-svn and point directly to Apache SVN!
CamelOne 2013
Cam
elO
ne
11
Major Subcomponents
• Transports and transport connectors
• Broker core
• Persistence adapters
• Network of brokers / clustering
CamelOne 2013
Cam
elO
ne
12
Architecture
Broker CoreTransport
Transport
TransportJMX
NetworkConnectors
Security
Store
Adapter
CamelOne 2013
Cam
elO
ne
13
Transports���and ���
Transport Connectors
CamelOne 2013
Cam
elO
ne
14
Transport Connector
• Let data in or out
• Think of as “SocketServer” (which it is J)
• Broker side
• Highly configurable through URI
• TransportConnector vs Transport
CamelOne 2013
Cam
elO
ne
15
Example Configuration
• Broker side: ������<transportConnector uri=“tcp://0.0.0.0:61616” />
• Broker side tuning: ������<transportConnector uri="tcp://0.0.0.0:0?transport.option=true”/>
• Client side: ������ConnectionFactory("tcp://0.0.0.0:61616?soTimeout=xx”)
CamelOne 2013
Cam
elO
ne
16
Out of the box
• TCP, NIO
• UDP
• SSL, SSL+NIO
• VM
• HTTP
• WebSockets
CamelOne 2013
Cam
elO
ne
17
Wire Formats
• Part of transport
• Encode/marshall
• Decode/unmarshall
• Creates Java objects
Wire
Broker
Transport
WireFormat
Socket
CamelOne 2013
Cam
elO
ne
18
Wire Formats
• AMQP 1.0
• OpenWire v1- v10
• STOMP 1.0, 1.1, 1.2
• MQTT 3.1
CamelOne 2013
Cam
elO
ne
19
Filters, Listeners
• TransportFilter – decorator pattern to add specific functionality, implements Transport, TransportListener
• TransportListener – callback for staged handling of incoming frame objects, on the way to broker
• Examples: ���Stomp, inactivity heartbeat, negotiation, logging, mutex, etc
CamelOne 2013
Cam
elO
ne
20
Transports Architecture
Wire Broker
TransportListener
TransportFilterTransport
WireFormat
TransportFilter
TransportFilter
TransportListener
TransportListener
TransportListener
CamelOne 2013
Cam
elO
ne
21
Broker Core
CamelOne 2013
Cam
elO
ne
22
Broker Core
• Regions, layers of brokers
• Memory management
• Message cursors
• Threading
CamelOne 2013
Cam
elO
ne
23
Region Broker
• Core of the broker
• Create consumers
• Create producers
• Create destinations
• Send messages to correct destination
RegionBroker
Router
Topic Region TempTopic Region
Queue Region
TempQueue Region
ConnectionStates Destinations ClientIDs
CamelOne 2013
Cam
elO
ne
24
Broker Filters
• Decorate RegionBroker
• How internal features like advisories and scheduling are implemented
• Basis for broker plugins
• http://activemq.apache.org/interceptors.html
• http://activemq.apache.org/developing-plugins.html
CamelOne 2013
Cam
elO
ne
25
Broker Filters SchedulerBroker
TransactionBroker
AdvisoryBroker
RegionBroker
Router
Send message
Add Consumer
Add Connection
CamelOne 2013
Cam
elO
ne
26
System Usage
• Allocate resources for the broker
• Persistent, Non-Persistent, in Memory Cache
<systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="64 mb"/> </memoryUsage> <storeUsage> <storeUsage limit="100 gb"/> </storeUsage> <tempUsage> <tempUsage limit="50 gb"/> </tempUsage> </systemUsage> </systemUsage>
CamelOne 2013
Cam
elO
ne
27
Memory Usage JVM Heap Space
Messages (MemoryUsage)
Queue Queue Queue Queue
ActiveMQ Objects
Index Cache Threads
16 MB 16 MB 16 MB
256 MB
CamelOne 2013
Cam
elO
ne
28
Message Cursors
• Buffers for tracking position
• Consumer Subscriptions
• VMPendingMessageCursor
• FilePendingMessageCursor
• Store Cursors + PrefetchSubscriptions
CamelOne 2013
Cam
elO
ne
29
Store Cursors
http://activemq.apache.org/message-cursors.html
CamelOne 2013
Cam
elO
ne
30
Store Cursors
http://activemq.apache.org/message-cursors.html
CamelOne 2013
Cam
elO
ne
31
Threading
• Producer connections
• Dispatching within destinations
• Consumer connections
• Store write threads
• Ancillary broker tasks (expiration, producer flow control, scheduled messages, etc)
CamelOne 2013
Cam
elO
ne
32
Threads
TCP
TCP
Queue Dispatch Thread
Consumer Transport Thread
Producer Transport Thread
Queue
Consumer Connection
Subscription
Producer Connection
CamelOne 2013
Cam
elO
ne
33
TCP
TCP
Consumer Transport Thread
Producer Transport Thread
Queue
Consumer Connection
Subscription
Producer Connection
<policyEntry queue=“>” optimizedDispatched=“true” >
CamelOne 2013
Cam
elO
ne
34
TCP
TCP
Producer Transport Thread
Queue
Consumer Connection
Subscription
Producer Connection
<policyEntry queue=“>” optimizedDispatched=“true” >
tcp://host:port?jms.dispatchAsync=false
CamelOne 2013
Cam
elO
ne
35
Persistence Adapter
CamelOne 2013
Cam
elO
ne
36
Persistence Adapters
• Add reliability to message delivery
• Optimized databases
• Pluggable implementations
• Out of the box:
• KahaDB
• LevelDB
• JDBC
CamelOne 2013
Cam
elO
ne
37
Persistence Adapter
Locker
Queue Store Topic Store
Journal
Index Redo
File System / RDBMS
CamelOne 2013
Cam
elO
ne
38
Lockers
• Used for master election
• SharedFileLocker
• DatabaseLocker, LeaseDatabaseLocker
• ZooKeeper based for replicated LevelDB ���new in upcoming 5.9.0, not available yet
• Your own!
CamelOne 2013
Cam
elO
ne
39
Message Stores
• Topic vs Queue stores
• Difference use cases
• Storage vs broadcast
• Relatively cheap
Topic StoreTopic Store
Topic Store
Journal
Index Redo
File System
Queue StoreQueue Store
Queue Store
CamelOne 2013
Cam
elO
ne
40
Store implementation
• Journal
• Index
• Redo/write ahead log
Index Journal
Redo Log
X X X X
CamelOne 2013
Cam
elO
ne
41
Network of Brokers
CamelOne 2013
Cam
elO
ne
42
Network of Brokers
• Connect brokers together
• Achieve scale
• Store and forward
Broker
Message
Broker
CamelOne 2013
Cam
elO
ne
43
Networking components
• Network Connector
• Discovery Agents
• Bridge
• Advisory messages/static destination filters
• Proxy Consumers
CamelOne 2013
Cam
elO
ne
44
Advisory Messages
• Advisory Topics
• Internal events���ActiveMQ.Advisory.Connection���ActiveMQ.Advisory.Consumer
• Event driven
• Replay upon first connect
• Necessary for Dynamic Demand Forwarding
CamelOne 2013
Cam
elO
ne
45
Dynamic Demand Forwarding
Broker
Message
Broker
Producer
Message
Consumer
Message
establish demand
forward message
CamelOne 2013
Cam
elO
ne
46
Bridging in Detail
Broker A
Broker B
Forward Msg
AdvisoriesQueue
CQueue
Producer Consumer
Bridge
C
CamelOne 2013
Cam
elO
ne
47
Wrap Up • Checkout the ActiveMQ wiki: ���
http://activemq.apache.org
• Mailing list: ���[email protected]
• Book: ActiveMQ in Action���http://www.manning.com/snyder/
• Blog���http://christianposta.com/blog
CamelOne 2013
Cam
elO
ne
48
Questions?