Real Time Data IntegrationDavide Mauridmauri@solidq.com@mauridb

Davide Mauri

• Microsoft SQL Server MVP

• Works with SQL Server from 6.5, on BI from 2003

• Specialized in Data Solution Architecture, Database Design, Performance Tuning, High-Performance Data Warehousing, BI, Big Data

• President of UGISS (Italian SQL Server UG)

• Regular Speaker @ SQL Server events

• Consulting & Training, Mentor @ SolidQ

• E-mail: dmauri@solidq.com

• Twitter: @mauridb

• Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

Agenda

• Scenario

• The Actors• Service Broker

• Integration Service

• The Playground• On-Premise

• Azure

• The Solution• Action!

Scenario

Scenario

• From a real world solution (an insurance company)

• Each time a new distress call is received (to recover a damaged car for example) a lot of other process needs to be executed• Dispatch tow truck

• Dispatch doctor (if needed)

• Send distress call info to CRM

• Send distress call info to ERP

• Send distress call info to DWH/BI for real time dashboard

Scenario

• The customer was using TIBCO to do (near) real-time data integration

• While it is a very powerful product, for them• It was a complex one (No real knowledge in-house)• Expensive• (Probably) not used in the correct way

• Performance problems + Always needed an external consultant

• They feel more confident with Microsoft Technologies

Scenario

• Is a (near) real-time integration solution achievable with the Microsoft Data Platform

• Using common and well known solutions• SQL Server

• Integration Services

• Easy to be managed, maintined and deployed

• Maybe also cloud-aware

The Actors

Integration Services

• (Very well know) ETL Application / Framework

• Very fast (especially when moving big amounts of data)• Very easy to use

• Allows very complex transformation if needed• Can also calls external applications / services

• Can be easily extended using .NET

• Unfortunately has a high «spin-up» time

Integration Services

• Execution Unit• Package

• Within a Package• Control Flow & Data Flow

• Control Flow => Tasks

• Data Flow => Transformations

Service Broker

• (Not so well-known) Transaction Queuing Engine• With SQL Server• Between SQL Servers• Available even in the Express edition

• 100% Asychronous, No polling at all• Notification system to execute function when a message arrives

• Use specific T-SQL commands• But still, is T-SQL

Service Broker

• Elements:• Message Types

• Contracts

• Services

• Queues

• Routes

• Activators

• Commands:• BEGIN DIALOG / SEND / RECEIVE / END DIALOG

• WAITFOR

The Playground

On-Premise

• After having created the Service Broker infrastructure…• If you are in a domain it is really as simple as configuring Service Broker

and Database Security

• Otherwise you can just authenticate Brokers using Certificates

• From a networking perspective• Open used Service Broker ports

• Nothing else, really.

Cloud

• It’s possible to use Service Broker to update a SQL Server VM on Azure• Of course Firewall and routing must be properly configured or…

• …a Point-To-Site VPN can do the magic

• It works very well to do NRT updates to SQL Server VM used, for example, to show Dashboards, Mobile BI or to use it a «dispatching» center to remote clients.

The Solution

Architecture

• Drawing time

Implementation Notes

• The trick is to create a package that runs and never stops• So we won’t pay for high spin-up times

• Use «WAIT FOR RECEIVE» to get messages in the queue• No polling -> No wasted CPU cycles

• That’s it • Again, simplicity rules!

DemoFrom Service Broker basics to Full Solution

Questions?

Thanks!

• If you want to rate this session on my SpeakerScore page:

• www.speakerscore.com

• Feedback Key: 6MLL

Demo Material

• Can be found here• http://1drv.ms/1Av5mb5

• Everything is release under the Creative Common Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) • http://creativecommons.org/licenses/by-nc-sa/4.0/