Expo Sponsors
Event Sponsors
Expo Light Sponsors
IoT for the BI professional
David L. Bojsen - Principal Architect
What to expect
∠ Level 200 session∠ Which basically means PowerPoint and talking
∠ Enthusiastic approach to (I)IoT∠ Want to spread the IoT-flu
What is IoT
IoT”The Internet of Things (IoT) is the network of physical devices,
vehicles, home appliances and other items embedded withelectronics, software, sensors, actuators, and connectivity
which enables these objects to connect and exchange
Data”
Source: en.wikipedia.org/wiki/Internet_of_things
What is IIoT
IIoT”The term Industrial Internet of things (IIoT) is often
encountered in the manufacturing industries, referring to the industrial subset of the IoT”
Source: en.wikipedia.org/wiki/Internet_of_things
Internet of Things opportunity
Examples of IIoT scenarios
Auto
What does that mean to us?
Auto
But not your typical datasource∠ IoT sensor data typically consists of time base measurements∠ Storing, indexing, querying, analyzing, visualizing can be challenging∠ Especially considering “IoT scale”
Devices
Storage
IoT Architecture
IoT Solutions Have a Common Pattern
ActionsThings Insights
A more realistic view…
ActionsThings Insights
Field Gateway
Device Connectivity & Management
Analytics & Operationalized Insights
Presentation & Business Connectivity
Dev
ices
RTOS
, Lin
ux, W
indo
ws,
Andr
oid,
iOS
Protocol Adaptation
Cold Path AnalyticsAzure HDInsight, AzureML
Azure Data Lake, Data Lake Analytics
Hot Path Business LogicService Fabric
Azure Functions
Field Gateway
Protocol Adaptation
Cloud GatewayIoT Hub
Hot Path AnalyticsAzure Stream Analytics
Azure HDInsightAzure Time Series Insights
Power BI
App Service
Logic Apps
Web Apps
Mobile Apps
Notification Hubs
BizTalk Services
• Bi-directional device-cloud • Telemetry
ingestion
• Command & Control• Device Registry
& Identity• Device Mgmt• HTTP, AMQP,
MQTT
High level IoT Architecture
Field Gateway
Device Connectivity & Management
Analytics & Operationalized Insights
Presentation & Business Connectivity
Dev
ices
RTOS
, Lin
ux, W
indo
ws,
Andr
oid,
iOS
Protocol Adaptation
Cold Path AnalyticsAzure HDInsight, AzureML
Azure Data Lake, Data Lake Analytics
Hot Path Business LogicService Fabric
Azure Functions
Field Gateway
Protocol Adaptation
Cloud GatewayIoT Hub
Hot Path AnalyticsAzure Stream Analytics
Azure HDInsightAzure Time Series Insights
Power BI
App Service
Logic Apps
Web Apps
Mobile Apps
Notification Hubs
BizTalk Services
• Bi-directional device-cloud • Telemetry
ingestion
• Command & Control• Device Registry
& Identity• Device Mgmt• HTTP, AMQP,
MQTT
Lets focus on the stuff we already know
Hot Path Analysis
Hot path analysis
∠ Azure StreamAnalytics
Query Language Declarative queries in a SQL like language
Easy to author and deploy
Temporal semanticsAll operators respect, and some use, the temporal properties of events
Built-in operators and functionsFilters, projections, joins, windowed (temporal) aggregates, text and date manipulation
∠ SELECT
∠ FROM
∠ WHERE
∠ GROUP BY
∠ HAVING
∠ CASE WHEN THEN ELSE
∠ INNER/LEFT OUTER JOIN
∠ UNION
∠ CROSS/OUTER APPLY
∠ CAST
∠ INTO
∠ ORDER BY ASC, DSC
SAQL – Language & Library
• WITH• PARTITION B• OVER
• DateName, DatePart, Day• Month, Year, DateDiff• DateTimeFromParts, DateAdd
• TumblingWindow• HoppingWindow• SlidingWindow
• SUM• COUNT• AVG• MIN• MAX• STDEV• STDEVP• VAR• VARP• TopOne
• Len, Concat, CharIndex• Substring, Lower• Upper, PatIndex
• Lag IsFirst,Last• CollectTop
• ABS, CEILING, EXP, FLOOR• POWER, SIGN, SQUARE• SQRT
SELECT TimeZone, COUNT(*) AS Count FROM TwitterStream TIMESTAMP BY CreatedAtGROUP BY TimeZone, TumblingWindow(second,10)
Tell me the count of tweets per time zone every 10 seconds
1 5 4 26 8 6 5
Time(secs)
1 5 4 26
8 6
A 10-second Tumbling Window
3 6 1
5 3 6 1
SELECT Topic, COUNT(*) AS TotalTweetsFROM TwitterStream TIMESTAMP BY CreatedAtGROUP BY Topic, HoppingWindow(second, 10 , 5)
Every 5 seconds give me the count of tweets over the last 10 seconds
1 5 4 26 8 7
A 10-second Hopping Window with a 5-second “Hop”
4 26
8 6
5 3 6 1
1 5 4 26
8 6 5 3
6 15 3
SELECT Topic, COUNT(*) FROM TwitterStreamTIMESTAMP BY CreatedAtGROUP BY Topic, SlidingWindow(second, 10)HAVING COUNT(*) > 10
Give me the count of tweets for all topics which are tweeted more than 10 times in the last 10 seconds
1 5
A 10-second Sliding Window
8
8
51
9
51 9
1
Hot path analysis
∠ Azure Stream Analytics∠ Azure Time Series Insight
Azure Time Series Insights
∠ IoT scale time-series data store
∠ Schema-less store, just send data
∠ Easy IoT Hub connection∠ Store, query and visualize
billions of events
∠ Simple and fast navigation
Hot path analysis
∠ Azure Stream Analytics∠ Azure Time Series Insight∠ Azure HDInsight
Big Data in Azure
Data Lake StoreHyper-scale Storage optimized for analytics
IaaS Hadoop
Hadoop distros on Azure VMs
Blob Storage
HDInsight
Azure-managed Hadoop clusters
HadoopSparkHBaseStorm
Data LakeAnalytics
Big Data as a Service
Cold Path Analysis
Cold path analysis
∠ Azure Data Lake Store
Petabyte size files and Trillions of objects
Scalable throughput for massively parallel analytics
HDFS for the Cloud
Always encrypted, Role-based Security & Auditing
Enterprise-grade Support
Azure Data Lake StoreA No limits Data Lake that powers Big Data Analytics
The first cloud Data Lake for enterprises that is secure, massively scalable and built to the open HDFS standard. With no limits to the size of data and the ability to run massively parallel analytics, you can now unlock value from all your unstructured, semi-structured and structured data.
Cold path analysis
∠ Azure Data Lake Store∠ Azure Data Lake Analytics
Start in seconds, Scale instantly, Pay per job
Develop massively parallel programs with simplicity
Debug and Optimize your Big Data programs with ease
Virtualize your analytics
Enterprise-grade Support and Security
Azure Data Lake AnalyticsMassively parallel, extensible, analytics made simpleThe first cloud analytics service where you can easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python and .Net over petabytes of data. With no infrastructure to manage, process data on demand, scale instantly, and only pay per job.
DECLARE @endDate DateTime = DateTime.Now;DECLARE @startDate DateTime = @endDate.AddDays(-7);
@orders = EXTRACT
OrderId int, Customer string, Date DateTime, Amount float
FROM "/input/orders.txt"USING Extractors.Tsv();
@orders = SELECT * FROM ordersWHERE Date >= startDate AND Date <= endDate;
@orders = SELECT * FROM ordersWHERE Customer.Contains(“Contoso”);
OUTPUT @ordersTO "/output/output.txt"USING Outputters.Tsv();
U-SQL Example
DECLARE constant values
C# Expressions
RowSets enable dataflow programming
EXTRACT performs schema on read for files
OUTPUT for writing files
Built-in handling for CSV & TSV
Whole-script optimization
REFERENCE ASSEMBLY [ExtPython];
DECLARE @myScript = @"def get_mentions(tweet):
return ';'.join( ( w[1:] for w in tweet.split() if w[0]=='@' ) )
def usqlml_main(df):del df['time']del df['author']df['mentions'] = df.tweet.apply(get_mentions)del df['tweet']return df
";
@t = SELECT * FROM
(VALUES("D1","T1","A1","@foo Hello World @bar"),("D2","T2","A2","@baz Hello World @beer")
) AS D( date, time, author, tweet );
@m =REDUCE @t ON datePRODUCE date string, mentions stringUSING new Extension.Python.Reducer(pyScript:@myScript);
Python Extensions
Use U-SQL to create a massively distributed program.
Executing Python code across many nodes.
Using standard libraries such as numpy and pandas.
Cold path analysis
∠ Azure Data Lake Store∠ Azure Data Lake Analytics∠ Azure Machine Learning
NotebooksIDEs
Azure M achine Learning W orkbench
VS Code Tools for AI
Experim entation and M odel M anagem ent
Services
AZURE MACHINE LEARNING SERVICES
SparkSQL Server Virtual m achines GPUsContainer services
SQL ServerM achine Learning Server
O N - P R E M I S E S
E D G EAzure IoT Edge
TRAIN & DEPLOY OPTIONS
A Z U R E
Azure Machine Learning
Cold path analysis
∠ Azure Data Lake Store∠ Azure Data Lake Analytics∠ Azure Machine Learning∠ And good old SQL
SQL ServerThe platform of choice
Linux
101001010010
{ } T-SQL
Java
C/C++
C#/VB.NET
PHP
Node.js
Python
Ruby
Any data Any application
Anywhere Choice of platform
Field Gateway
Device Connectivity & Management
Analytics & Operationalized Insights
Presentation & Business Connectivity
Dev
ices
RTOS
, Lin
ux, W
indo
ws,
Andr
oid,
iOS
Protocol Adaptation
Cold Path AnalyticsAzure HDInsight, AzureML
Azure Data Lake, Data Lake Analytics
Hot Path Business LogicService Fabric
Azure Functions
Field Gateway
Protocol Adaptation
Cloud GatewayIoT Hub
Hot Path AnalyticsAzure Stream Analytics
Azure HDInsightAzure Time Series Insights
Power BI
App Service
Logic Apps
Web Apps
Mobile Apps
Notification Hubs
BizTalk Services
• Bi-directional device-cloud • Telemetry
ingestion
• Command & Control• Device Registry
& Identity• Device Mgmt• HTTP, AMQP,
MQTT
High level IoT Architecture
Field
Gateway
Device
Connectivity & Management
Analytics &
Operationalized Insights
Presentation &
Business Connectivity
De
vic
es
RTO
S, Li
nu
x, W
ind
ow
s, A
nd
roid
, iO
S
Protocol
Adaptation
Cold Path AnalyticsAzure HDInsight, AzureML
Azure Data Lake, Data Lake Analytics
Hot Path Business LogicService Fabric
Azure Functions
Field
Gateway
Protocol
Adaptation
Cloud GatewayIoT Hub
Hot Path AnalyticsAzure Stream Analytics
Azure HDInsight
Azure Time Series Insights
Power BI
App
Service
Logic
Apps
Web
Apps Mobile
Apps
Notification
Hubs
BizTalk
Services
• Bi-directional
device-cloud
• Telemetry ingestion
• Command & Control
• Device Registry
& Identity
• Device Mgmt
• HTTP, AMQP,
MQTT
But what good is it without data from devices…
Connect & ControlAzure IoT Hub
Azure IoT Hub Device Management
IoT Hub
Device twin Device twin
Software
FirmwareConfiguration
Azure IoT Hub Device Management
IoT Hub
Device twin Device twin
DesiredReported
TagsMethods
Desired
Reported
Methods
JobsSchedule and broadcast Device twin changes across large fleets
Azure IoT Hub Message RoutesDe
vices
IoT Hub
IoT Solution
Routing Logic
Event Processing
Queue
StorageQueue
AlarmQueue
Remember email before inbox rules?
Device Connectivity
Field G atew ay
CoAP, A llJoyn, O PC
Custom Protocol G atew ay
(C loud Service, VM )VPN /ExpressRoute
O PC, H TTP, CoAP
Fie ld G atew ay*
CoAP, A llJoyn, O PC
IoT H ub
Custom Protocol G atew ay
(C loud Service, VM )
AM Q P, M Q TT, H TTPS
Custom Protocols
Data Path
Optional solution component
Azure IoT solution component
DeviceIoT C lient
DeviceIoT C lient
Device
IoT C lientDevice
Device
Device
AM Q P, M Q TT, H TTPS
* Field Gateway is currently a concept that may represent Microsoft, 3rd party or custom capabilities at a hardware or software level
Industrial protocols
∠ Ethernet/IP∠ Modbus∠ OPC UA∠ OPC DA
∠ Profibus∠ BACNet∠ KNX
∠ Zigbee∠ ProfiNetAnd many, many more
Source: en.wikipedia.org/wiki/List_of_automation_protocols
Things
IoT Pattern
Insights ActionsCloud Gateway
Things
IoT Pattern + Edge
Insights ActionsInsights ActionsCloud Gateway
Azure Machine LearningS p a r k
S Q L S e r v e r
V ir t u a l m a c h in e s G P U s
C o n t a in e r s e r v ic e s
N o t e b o o k sID E s
A z u r e M a c h in e L e a r n in g W o r k b e n c h
S Q L S e r v e r
M a c h in e L e a r n in g S e r v e r
ON-PREMI SES
EDGE COMPUTI NGA z u r e Io T E d g e
E x p e r im e n t a t io n a n d M o d e l M a n a g e m e n t
A Z U R E MA C H I N E L E A R N I N G S E R V I C E S
T R A I N & D E P L O Y O P T I O N S
AZURE
Field Gateway
Device Connectivity & Management
Analytics & Operationalized Insights
Presentation & Business Connectivity
Dev
ices
RTOS
, Lin
ux, W
indo
ws,
Andr
oid,
iOS
Protocol Adaptation
Cold Path AnalyticsAzure HDInsight, AzureML
Azure Data Lake, Data Lake Analytics
Hot Path Business LogicService Fabric
Azure Functions
Field Gateway
Protocol Adaptation
Cloud GatewayIoT Hub
Hot Path AnalyticsAzure Stream Analytics
Azure HDInsightAzure Time Series Insights
Power BI
App Service
Logic Apps
Web Apps
Mobile Apps
Notification Hubs
BizTalk Services
• Bi-directional device-cloud • Telemetry
ingestion
• Command & Control• Device Registry
& Identity• Device Mgmt• HTTP, AMQP,
MQTT
High level IoT Architecture
Field Gateway
Device Connectivity & Management
Analytics & Operationalized Insights
Presentation & Business Connectivity
Devi
ces
RTOS
, Linu
x, W
indow
s, An
droi
d, iO
S
Protocol Adaptation
Cold Path AnalyticsAzure HDInsight, AzureML
Azure Data Lake, Data Lake Analytics
Hot Path Business LogicService Fabric
Azure Functions
Field Gateway
Protocol Adaptation
Cloud GatewayIoT Hub
Hot Path AnalyticsAzure Stream Analytics
Azure HDInsightAzure Time Series Insights
Power BI
App Service
Logic Apps
Web Apps
Mobile Apps
Notification Hubs
BizTalk Services
• Bi-directional device-cloud • Telemetry ingestion
• Command & Control• Device Registry
& Identity• Device Mgmt• HTTP, AMQP,
MQTT
Who (or what) are we building this for…
Questions ?
AZURE REGIONS
50Announced Azure regions world wideHyper-Scale Capacity
Get started building IoT solutions today
AZURE IOT SUITE REGIONS
14Azure IoT Suite regions
Thank youfor your
time