Advanced Messaging Scenarios with Azure Service Bus MessagingDan Rosanova M374

Azure Service Bus Messaging is the most sophisticated cloud messaging platform in the world

What is “Messaging”

It’s not this…But it’s not all that different

Messaging is the decoupling of systems and services via asynchronous communication

Why would anyone use messaging?Increase availabilityProtect components from each other and outside forcesProvide elasticityOvercome temporal couplingLimit the scope and impact of change propagationSimplify programming models

What are common tasks people use messaging for?Order processingWorkflowIntegrationAsynchronous programmingEventual consistency

A brief history of messaging

Teknekron Informatio

n Bus

MQ Series

MSMQ

TIBCO

BizTalk

RabbitMQ

ServiceBus

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

1980s 2015

Where is messaging used

Inter-Bank transfersOrder fulfillment & tradingSettlementPricing updates and quotes

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Where is messaging used

Customer information systemsBilling integrationSmart meter & gridCommand and control

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Where is messaging used

Order processingDelivery notificationsInventory managementRouting

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Where is messaging used

EnrollmentClaims processingRisk modelingRegulatory filing

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Where is messaging used

EnrollmentClaims processingRisk modelingRegulatory filing

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Where is messaging used

Medial recordsCare quality metricsPatient monitoring & diagnostics

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Where is messaging used

You really don’t want to knowHow do you think the NSA collects all that data?

Financial Services

LogisticsUtilities & Telecom

Insurance Healthcare Government

Two runtime models have emerged

Google Cloud Pub/Sub

Amazon SQS/SWF

PubNub

IronMQ

CloudAMQP

RackspaceMicrosoft

Dedicated Capacity

Multitenant Platform

Two pricing models have emerged

Google Cloud Pub/Sub

Amazon SQS/SWF

PubNub

IronMQ

CloudAMQP

RackspaceMicrosoft

Dedicated Capacity

Multitenant Platform

Pay p

er

use

Fixe

d F

ee

Microsoft

1.5 Trillion

Messages delivered every month with Event Hubs

>250 Billion

Message operations onAzure Service Bus

Messaging (Standard) per month

1.5 MillionMessage Queues and Topics in Production

>10 PBMonthly Data Volume

85Microsoft’s Cloud

backbone is connected to over 85

Global Internet Exchanges, enabling

high speed, optimized

connectivity to the world.

4 timesMicrosoft’s North

America optical fiber network, can wrap around the earth 4

times

Azure Messaging

>30,000Daily Active Service Bus

Namespaces

>250 TBDaily Data Volume

Common messaging patternsLoad Leveling

Load Balancing

Filtering

Partitioning

Scheduled Delivery

Fan Out

Routing

Claim Checks

Workflow

Partitioning

Taps

Logging

Dead Letter

Sequence

Correlation

Expiration

Features of Service Bus MessagingScheduled delivery

Poison message handling

ForwardTo

Defer

Sessions

Batching

Auto-delete on idle

OnMessage

Duplicate detection

Filters

Actions

Transactions

How do we bridge this in practice?

Features => Patterns

Basic Queuing

Creating a queuevar nsm = NamespaceManager.CreateFromConnectionString(

connectionString);

if(nsm.QueueExists(basicQueue) == false) nsm.CreateQueue(basicQueue);

Sending to a queuevar qc = QueueClient.CreateFromConnectionString(connectionString, basicQueue);

//your codevar order = CreateOrder();

var msg = new BrokeredMessage(order);qc.Send(msg);

Don’t be a baby…

Crawl Walk Run

await qc.SendAsync(msg);qc.Send (msg);

Nothing is more permanent than a temporary solution

Reading from a queueThere are a few ways to do thisThere is only one really easy way

Reading from a queueqc.OnMessage((message) =>{ var myOrder = message.GetBody<Order>(); Console.WriteLine("Received Order {0}", myOrder.CustomerName); message.Complete();});

Want more optionsvar options = new OnMessageOptions() { MaxConcurrentCalls = 5, AutoComplete = true};

Message header dataMetadata about the message

msg.Properties.Add("myproperty", "some value"); msg.Properties.Add("total", 1255.25);

Ever seen this?

Submit

To avoid multiple charges only click this button once

Get with the times!

Make operationsidempotent

Duplicate detectionnsm.CreateQueue(new QueueDescription(“DuplicateQueue”){ RequiresDuplicateDetection = true, DuplicateDetectionHistoryTimeWindow = new

TimeSpan(0, 0, 30)});

msg.MessageId = order.Id;

Things that can go wrong when receivingDownstream system is unavailableData format has changed and serializer throws

var myOrder = message.GetBody<Order>();

Oops

Good idea to use a property to define type / version

Lifecycle of a message

Reader Complete

DeliveryCount++

DLQ

DeliveryCount > 10

Things to know about dead letteringDesigned to handle poisoned messagesOn by defaultDefault setting of 10 deliveries (set at queue level)YOU MUST EMPTY THE DEADLETTER… OR ELSECan be explicitly called message.DeadLetter("Bad format", "Some description");

I repeatYou must empty the dead letter queue

Really it’s just another queueWith a special path – but please don’t hard code the path, use the client

var dlq = QueueClient.CreateFromConnectionString(

"connectionstring", QueueClient.FormatDeadLetterPath("myqueue"));

Temporal decouplingOne of the original goals of messaging

Sender Receiver

Thinking about the futureScheduled delivery

msg.ScheduledEnqueueTimeUtc = DateTime.Now.AddMinutes(20); qc.Send(msg);

Reflecting on your lifeSet TimeToLive before sending a messageThis will expire messages i.e. delete them

msg.TimeToLive = new TimeSpan(1, 0, 0);

Or set at the entity queue level (code or portal) nsm.CreateQueue(new QueueDescription(“myqueue”){ DefaultMessageTimeToLive = new TimeSpan(1, 0, 0)});

What can you do with this?Limit Order

Good ‘Till Cancelled (GTC)Day Order

You can also schedule a finite window for actions to happen – like complete or roll back

What if you can’t be bothered to process something right now?Defer: the feature we all use in our daily lives

deferList.Add(msg.SequenceNumber); msg.Defer();

qc.Receive(sequenceNumber);

Sometime when you’re good and ready

If you lose the sequence number of a message there is no way to read it

WARNING

Ordered messaging & convoysSome things need to happen in order – like updatesOrdering can be difficult for asynchronous systemsJust ask any NodeJS developerOr any BizTalk developer

What is so hard about order?Just process the first one first and the second one second

Sender Receiver

Receiver

What is so hard about order?Just process the first one first and the second one second

Sender Receiver

How can you have order AND competing consumers?Use Sessions

var sessionId = Guid.NewGuid().ToString(); msg.SessionId = sessionId; qc.Send(msg); msg2.SessionId = sessionId; qc.Send(msg2);

easy

Receiving ordered messagesvar session = qc.AcceptMessageSession();session.OnMessage((message) =>{ var myOrder = message.GetBody<Order>(); Console.WriteLine("Received Order {0}", myOrder.CustomerName); message.Complete();}, options);//later onsession.Complete();

But wait there’s more!public interface IMessageSessionHandler{ void OnCloseSession(MessageSession session); void OnMessage(MessageSession session,

BrokeredMessage message); void OnSessionLost(Exception exception);}

qc.RegisterSessionHandler(typeof(MySessionHandler));

Sometimes the messages aren’t enoughYou may want to keep track of what you’ve done with the messages

Stream state = session.GetState();

session.SetState(state);

In sumCompeting consumers are necessary for scale, availability, and throughputStrict ordering is still possible even with competing consumersService Bus gives you all the tools you need to do this

And if that didn’t sell you….Ever hear of something called Eventual Consistiencyhttps://github.com/affandar/durabletask

What about bi-directional communication?A simple approach

System 1 System 2

Request Q

Reponse Q

msg.ReplyTo = "replyqueue";

The request-response queue approach has some limitationsThere is no direct correlation between sender and receiverIf you want that you may need a lot of queuesOr use properties on your ownOr…

Use sessions!What don’t these things do?

System 1 System 2WorkQueue

Request-Response: A more elegant approachReqestor var sessionId = Guid.NewGuid().ToString(); msg.ReplyToSession = sessionId; qc.Send(msg); qc.AcceptMessageSession(sessionId,

new TimeSpan(0, 5, 0));

Responder msg.SessionId = inmsg.ReplyToSession; qc.Send(msg);

Take a deep breathRelaxWe’re just getting started!

$100$1500

Publish-Subscribe

System 1

System 3

Topic

Subscription

System 2Subscription

Filter:total > 1000

Filter:total <= 1000

var td = nsm.CreateTopic("topicname");var sd = new SubscriptionDescription("topicname", "bigorders");var sc = SubscriptionClient.Create("topicname", "bigorders");nsm.CreateSubscription(sd);var rd = new RuleDescription("default", new SqlFilter("total > 1000"));sc.AddRule(rd);

What can you do with Topics/SubscriptionsMessage partitioning (segmenting)Fan outInitiate multiple downstream steps (overlapping rules)All the same things you can with queuesChange the metadata as the message goes moves with ActionsOne thing to know – dead lettering happens per subscription

Actions on Rules to set headersvar rd = new RuleDescription("default", new SqlFilter("NOT EXISTS(stage)")) { Action = new SqlRuleAction("set stage = 1") };

Actions best friendForwardToForwards all messages for a subscription (or queue) to another entity (topic or queue)

sd.ForwardTo = "stage1topic";

Let that sink in for a minute…

Building rich topologies with pub-subVery expressive and powerfulDon’t go too crazy

Forwarding deadlettersAvailable for every deadletter

sd.ForwardDeadLetteredMessagesTo = "subdeadletter";

Transaction with entitiesWorks with only one entity in scope

using (var scope = new System.Transactions.TransactionScope()){ qc.Send(new BrokeredMessage(order1)); qc.Send(new BrokeredMessage(order2)); scope.Complete();}

AutoDeleteOnIdleGreat featureDoes what it sounds likeCan be used to clean up entities that exist for transient periods or volatile lifetimes nsm.CreateQueue(new QueueDescription(“queue”){

AutoDeleteOnIdle = new TimeSpan(24,0,0)});

Claim check patternYou want to send something really big

Sender

Receiver

The Cleaner(Victor)

Q&A

Key takeawaysYes – Service Bus Messaging is THE MOST SOPHISTICATED messaging platform in the cloudThere’s a lot we didn’t coverCheckout how affordable this service isGo back and review yesterday’s session about availability

You decide where this ride goes!

Related Ignite NZ Sessions

http://blogs.msdn.com/b/servicebus/

Find me later at… Closing drinks Fri 3:00-4:30pm @DanRosanova

Resources

TechNet & MSDN FlashSubscribe to our fortnightly newsletter

http://aka.ms/technetnz http://aka.ms/msdnnz

http://aka.ms/ch9nz

Microsoft Virtual AcademyFree Online Learning

http://aka.ms/mva

Sessions on Demand

Complete your session evaluation now and be in to win!

© 2015 Microsoft Corporation. All rights reserved.Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or

other countries.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.