44
AMQP Messaging Foundations Jeremy Deane http://jeremydeane.net

AMQP Messaging Foundations

Embed Size (px)

Citation preview

AMQPMessagingFoundations

JeremyDeanehttp://jeremydeane.net

Agenda

v EnvironmentSetup

v HelloBunny!

v AMQPFundamentals

v Exercises

v RabbitMQ Architecture

v RabbitMQ Security&Operations

v AMQPIntegrations

v Exercises

AdditionalEnvironmentSetup§ DownloadandinstallRabbitMQ (3.6.2+)

§MacStandalone orHomebrew§WindowsStandalone

§ Add$RABBITMQ_HOME/sbin toPATH

§ Commands(WindowsasaService)§ Start:rabbitmq-server -detached§ Status:rabbitmqctl status

§ Stop:rabbitmqctl stop

§ InstallManagementPlugin

§ VerifyManagementConsole(guest/guest):

rabbitmq-plugins enable rabbitmq_management

http://localhost:15672/

Mac&LinuxO/S– Update/etc/hosts

127.0.0.1 <<name of your laptop>>

EnvironmentSetup§ Download(Clone)CodeFromGithub

§ TestHarness-https://github.com/jtdeane/amqp-client§ AMQPConsumer- https://github.com/jtdeane/amqp-consumer§ AMQPProducer- https://github.com/jtdeane/amqp-producer§ AMQPComplexEventProcessor(CEP)- https://github.com/jtdeane/amqp-cep

Hello Bunny!

1. StartRabbitMQ

2. Buildandimportamqp-client intoIDE

3. SendAMQPMessage– RunasJUnit Test

4. ViewExchange,Queue,andMessageinManagementConsole

5. ReceiveAMQPMessage– RunasJUnit Test

mvn clean install

mvn test -Dtest=ProducerFunctionalTest

mvn test -Dtest=ConsumerFunctionalTest

http://localhost:15672/#/exchanges/%2F/test.direct

http://localhost:15672/#/queues/%2F/magic.greetings

amqp-client/src/test/java/ws/cogito/magic/messaging/ProducerFunctionalTest.javaamqp-client/src/test/java/ws/cogito/magic/messaging/ConsumerFunctionalTest.java

MessagingFoundations

P2PHidden Costs

BottomLineSOAbyMarcRix

WebservicesAwebservicedoesNOT trulydecoupletheconsumerandprovider

P2PIntegrationsThecostofmaintainingP2Pintegrationsincreasesexponentiallyasthenumberoftheconnectionsincreases

#ofconnections

$

gain

value

cost

loss

Enterprise IntegrationPatterns*

*A.K.A. MessageExchangePatterns(MEP)

Publish Subscribe Domain (Topics)

Point-to-Point Domain (Queues)

Enterprise IntegrationPatterns

Channel Message FIlterSquareTriangle ChannelSquare SquareSquare

GuaranteedMessageDelivery

FilterMessages

AsynchronousRequestReply

Message OrientedArchitecture(MOA)

MessageSender Receiver

Broker

TopicQueue

createsprocesse

s

Delivers

Connects/ to Connects/ to

Hosted//by

AdvancedMessageQueuingProtocol(AMQP)

AdvancedMessagingQueueing Protocol(AMQP)

RabbitMQ implement0.9.1ofAMQPStandard

AMQPMessageMessage

Properties{Standard or Broker Defined}

Headers{Custom}

Payload{Stream of bytes}

Property Purpose

ContentType ThecontenttypeofthemessageasaMediaType.

ContentEncoding Theencodingto andfrombytes(UTF-8)

RoutingKey UsedbyanExchangeformessagerouting

DeliveryMode Persistent(2)ornon-persistent(1)

Reply To FormessagingforwardingorRequest-ReplyPattern

Expiration Time-to-Live (TTL)inMilliseconds

Priority MovemessageupQueue(defaultis0)

CustomHeaderExample:TrackingID

AMQPVirtualHosts, Exchanges&RoutingKeys§ VirtualHostsprovidelogicalcontainersforExchanges&Queues

§ ReferredtoasVHost§ DefaultVirtualHost(/)– Donotdelete!§ SuggestedNamingScheme:<owner>.<domain || purpose>.vhost

§ ExchangesroutemessagesbasedonaRoutingKey§ DirectExchange:Point-to-Point(P2P)§ TopicExchange: Publish-Subscribe§ Fanout Exchange: Broadcast

§DefaultExchanges(AMQP)– Donotdelete!§ SuggestedNamingScheme:<owner>.<domain || purpose>.<Exchange Type>

§ RoutingKeysrepresentahierarchicalnamespace§ ws.cogito.magic.sales.orders§ ws.cogito.magic.finance.bills

Message Broker

Virtual Host

Publisher ExchangeMessage

AMQPQueues andBindingKeys§ QueuesareassociatedwithExchangesviaBindingKeys

§ Durable orTransient§ SuggestedNamingScheme:<owner>.<hierarchy>.<entities>

§ acme.magic.finance.invoices

§ acme.magic.sales.orders

§ BindingKeysoftenmatchQueueNamesexactlyforDirectExchanges§ acme.magic.finance.invoices

§ BindingKeysoftenusespecialcharactersforTopicExchanges§ * (star)cansubstituteforexactlyoneword§ # (hash)cansubstituteforzeroormorewords

§ acme.magic.finance.*§ acme.magic.#

RabbitMQ SimulatorDemo

http://tryrabbitmq.com/

AMQP Connections,ChannelsandAcknowledgements§ Connections arelonglivedTCPconnectionswhileChannelsmultiplexedinteractionswithinaConnection.

§MessageAcknowledgementsareautomaticbydefaultbutcanbemanual

§Messagescanalsoberejected orre-queued orsenttoDeadLetterQueue(DLQ)

AMQPDeclarations

ConflictingDeclarationswillfail!

AMQPExercises

AMQPExercisesSetup1. StartRabbitMQ

2. CreateUser(client/client)– AdminTag

3. CreateVirtualHost(VHost)– cogito

4. AssociateUserclient toVHost cogito&/

5. Buildandbootamqp-client

6. ViewdeclaredExchanges

http://localhost:15672/#/users

http://localhost:15672/#/vhosts

http://localhost:15672/#/vhosts/cogito

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

http://localhost:15672/#/exchanges/cogito/magic.directhttp://localhost:15672/#/exchanges/cogito/magic.topic

Code:amqp-client/src/main/java/ws/cogito/magic/utilities/AmqpDeclarationsConfiguration.java

AMQPExercisesSetup1. CreateQueuemagic.tests *

2. CreateQueueDirect&TopicBindings**

3. CreateUsers)– AdminTag• consumer/consumer• producer/producer• evaluator/evaluator

4. AssociateUserstoVHost cogito&/

http://localhost:15672/#/queues

Exchange: magic.direct, Queue & Routing Key: magic.tests

Exchange: magic.topicTopic: magic.testing.eventsRouting Key: magic.testing.#

Knuffle-Bunny-Cautionary*Shortenedfortesting,shouldbews.cogito.magic….etc**Normallythereshouldbeonequeueperconsumer

AMQPSender

Note:normallythereshouldbeonequeueperconsumer;thisisjustfortesting

http://tryrabbitmq.com/

AMQPSender1. IssueHTTPRequestviacurlorRESTclient(e.g.Postman)- Direct

2. ViewmessageinQueueandthenpurgeQueue

3. IssueHTTPRequestviacurlorRESTclient(e.g.Postman)- Topic

4. ViewmessageinQueueandthenpurgeQueue

5. ViewCode

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1125, "customer":"Houdini", "item":"Cards", "amount":6}' http://localhost:9999/amqp/magic.direct/magic.tests

http://localhost:15672/#/queues/cogito/magic.tests

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1126, "customer":"Teller", "item":"Dice", "amount":4}' http://localhost:9999/amqp/magic.topic/magic.testing.foo

http://localhost:15672/#/queues/cogito/magic.tests

/amqp-client/src/main/java/ws/cogito/magic/utilities/web/AmqpController.java

AMQPListener&Subscriber

http://tryrabbitmq.com/

1. StartRabbitMQ

2. Buildandbootamqp-consumer

3. ViewQueuesandBindings

4. ViewDeclarationCode

5. Bootamqp-client

AMQP Listener

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

http://localhost:15672/#/queues/cogito/magic.eventshttp://localhost:15672/#/queues/cogito/magic.sales.orders

/amqp-consumer/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

6. SendOrderMessageviaamqp-client

7. Viewamqp-client andamqp-consumer consolelogs

6. ViewOrderListenerCode

7. Stopamqp-consumerandsendanotherOrderMessage

8. ViewmessageviaManagementConsole

9. Bootamqp-consumerandviewconsolelogs

AMQP Listener

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1127, "customer":"Blain", "item":"Marbles", "amount":2}' http://localhost:9999/amqp/magic.direct/magic.sales.orders

/amqp-consumer/src/main/java/ws/cogito/magic/messaging/OrdersMessageListener.java/amqp-consumer/src/main/java/ws/cogito/magic/MessageLogging.java

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1128, "customer":"Blackstone", "item":"Scarf", "amount":4}' http://localhost:9999/amqp/magic.direct/magic.sales.orders

1. SendEventMessageviaamqp-client

2. Viewamqp-client andamqp-consumer consolelogs

6. ViewEventListenerCode

7. SendanotherEventMessageviaamqp-client

8. Viewamqp-client andamqp-consumer consolelogs

AMQP Subscriber

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":21, "location":"Las Vegas", "title":"Penn and Teller", "date":"2016-12-31"}' http://localhost:9999/amqp/magic.topic/magic.events.shows

/amqp-consumer/src/main/java/ws/cogito/magic/messaging/EventsMessageListener.java

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":22, "location":"Clearwater", "title":"ArchConf", "date":"2016-12-12"}' http://localhost:9999/amqp/magic.topic/magic.events.conferences

RabbitMQ Architecture

RabbitMQ HighAvailabilityTopologyRabbitMQ Server

Message BrokerVirtual Host

RabbitMQ Server

Message BrokerVirtual Host

Mirrored Queues

Exchange Exchange

Queue A

Queue B

Queue A

Queue B

§ RabbitMQ ServersareClustered – SharedSecret– Erlang Cookie

§ RabbitMQ HAPolicy:rabbitmqctl set_policy ha-all "" '{"ha-mode":"all"}'

§ ClientsconnecttosetofaddressesorLoadBalancercanproxyClients(e.g.HAProxy)

RabbitMQ Federated Topologies

RabbitMQ Server RabbitMQ Server

Finance Cluster

RabbitMQ Server

RabbitMQ Server RabbitMQ Server

Sales Cluster

RabbitMQ Server

Federation Plugin

Shovel Plugin

§ FederationPlugin– Both“upstream”and“downstream”Consumersreceivemessage

§ ShovelPlugin– ForwardsMessagefrom“upstream”Queuetoremote“downstream”Exchange

RabbitMQ Security§ UsershaveVHost Read,Write,andConfigurepermissions

§ LDAPPlugin – Authentication-Authorization

§ TransportLevelSecurity(TLS)– Port5671§ TLSConnectionFactoryGist

§EncryptionatRest(Mnesia DB&Logs)– e.g.VormetricRabbitMQ Server

Message BrokerVirtual Host

DB

User Security

Logs

Transport Level Security

Encryption at Rest

RabbitMQ Operations§WebConsole

§ HTTPRESTAPI (guest/guest)– ListoutVirtualHosts

§ NewRelic &Zabbix Plugins

§ CommandLineInterface(CLI)&Tool

1. DownloadCommandLineToolin$RABBITMQ_HOME/sbin

2. Mayneedtoupdatepermissionsonfile

3. ListoutVirtualHosts

http://localhost:15672/cli/

chmod 777 rabbitmqadmin

rabbitmqadmin list vhosts

http://localhost:15672/api/vhosts

AdvancedAMQPPatterns

AMQPComplexEventProcessing

http://tryrabbitmq.com/

1

3

2

AMQP– ComplexEventProcessing1. StartRabbitMQ

2. Buildandbootamqp-cep

3. ViewQueuesandBindings

4. ViewDeclarationCode

5. Bootamqp-consumer &amqp-client

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

http://localhost:15672/#/queues/cogito/magic.evaluationshttp://localhost:15672/#/queues/cogito/magic.alerts

/amqp-cep/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

AMQP– ComplexEventProcessing6. PublishEventviacurlorRESTclient(e.g.Postman)

7. Viewamqp-client, amqp-consumer, amqp-cep consolelogs

8. ViewtheEvaluationMessageListener

9. PublishEventviacurlorRESTclient(e.g.Postman)

10. Viewamqp-client, amqp-consumer, amqp-cep consolelogs• Notehowanalertisnotraisedbecauseofthelocation

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":21, "location":"Las Vegas", "title":"Penn and Teller", "date":"2016-12-31"}' http://localhost:9999/amqp/magic.topic/magic.events.shows

/amqp-cep/src/main/java/ws/cogito/magic/messaging/EventsMessageListener.java

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"event":22, "location":"Clearwater", "title":"ArchConf", "date":"2016-12-12"}' http://localhost:9999/amqp/magic.topic/magic.events.conferences

AMQP– WhynotSpringIntegrationsorApacheCamel?@Bean

public IntegrationFlow amqpFlow() {return IntegrationFlows

.from(Amqp.inboundGateway(this.rabbitConnectionFactory, queue())).transform("hello "::concat).transform(String.class, String::toUpperCase)

.get();}

@Beanpublic IntegrationFlow amqpOutboundFlow() {

return IntegrationFlows.from("amqpOutboundInput")

.handle(Amqp.outboundAdapter(this.amqpTemplate).routingKeyExpression("headers.routingKey"))

.get();

context.addRoutes(new RouteBuilder() {public void configure() {

from("rabbitmq://localhost?exchange=direct_logs&exchangeType=direct&bindingKey=ERROR")

.setBody().simple("ERROR: ${in.body}")

.to("stream:out");

from("rabbitmq://localhost?exchange=direct_logs&exchangeType=direct&bindingKey=WARN").setBody().simple("WARN: ${in.body}")

.to("stream:out");

}

});

SpringIntegrations– Immature- poorlydocumentedDSL

ApacheCamel– ImmatureAMQP&RabbitMQ Components

AMQPAsynchronousRequest-Reply– TempQueue

http://tryrabbitmq.com/

1

2

3

StartswithPOSThttp://localhost:9000/priority/orders

4

AMQP– Asynchronous Request-Reply – TempQueue1. StartRabbitMQ

2. Buildandbootamqp-producer

3. ViewQueueandBinding

4. ViewDeclarationCode

5. Bootamqp-consumer

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

http://localhost:15672/#/queues/cogito/magic.priority.orders

/amqp-producer/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

AMQP– Asynchronous Request-Reply – TempQueue6. IssuePriorityOrderRequestviacurlorRESTclient(e.g.Postman)

7. ViewthePriorityOrderController:processOrder

8. Viewamqp-consumer andamqp-producer consolelogs• NotetheTrackingIDandReplyTo Queue

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1129, "customer":"Penn", "item":"Rabbit", "amount":6}' http://localhost:9000/priority/orders

/amqp-consumer/src/main/java/ws/cogito/magic/web/PriorityController.java

“Beware of time-outs & downtime”

AMQPAsynchronousRequest-Reply– CorrelationID

http://tryrabbitmq.com/StartswithPOSThttp://localhost:9000/priority/vip/orders

12

3

4

AMQP– Asynchronous Request-Reply – CorrelationID1. StartRabbitMQ

2. Buildandbootamqp-producer

3. ViewQueueandBinding

4. ViewDeclarationCode

5. Bootamqp-consumer

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

http://localhost:15672/#/queues/cogito/magic.priority.orders

/amqp-producer/src/main/java/ws/cogito/magic/AmqpDeclarationsConfiguration.java

mvn clean installmvn spring-boot:run -Drun.arguments="-Xmx256m,-Xms128m"

AMQP– Asynchronous Request-Reply– CorrelationID1. StartRabbitMQ

2. Buildandbootamqp-producer and amqp-consumer

3. IssueVIPOrderRequestviacurlorRESTclient(e.g.Postman)

4. ViewthePriorityOrderController:processOrderVip

5. View amqp-consumer andconsolelogsamqp-producer• NotetheTrackingID,CorrelationID,andReplyTo Queue

6. ViewthePriorityMessageListener

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"order":1130, "customer":"Angel", "item":"quarters", "amount":6}' http://localhost:9000/priority/vip/orders

/amqp-consumer/src/main/java/ws/cogito/magic/web/PriorityController.java

/amqp-producer/src/main/java/ws/cogito/magic/messaging/PriorityMessageListener.java

Questions&FeedbackQuestions&Feedback

MyContactinformation:

[email protected]://jeremydeane.net

https://github.com/jtdeane/amqp-client

https://github.com/jtdeane/amqp-consumer

https://github.com/jtdeane/amqp-producer

https://github.com/jtdeane/amqp-cep