Samedi SQL Québec - La plateforme data de Azure

Azure Data OverviewEric Moreau, MVPwww.emoreau.com – [email protected]

Microsoft Azure

http://www.emoreau.com/

mailto:[email protected]

SQL

Database

SQL Server

VMDocumentDB

Search HDInsightAdditional

databases

Agenda – Part 1

Agenda – Part 2

Blobs Files

Queues Tables

Agenda – Part 3 – Announced in May 2015

Data Lake

Azure SQL

Data

Warehouse

SQL

Database

SQL Server

VMDocumentDB


databases

Microsoft Azure Data Platform

Run SQL on VM

Run any SQL product on cloud VM

Support for SQL Server, Oracle, MySql

Ready to go VM images available in Gallery

Persistent storage using attached disk in blob storage (Azure Files)

Has all the benefits and powers of VMs combined with the full features of a SQL Engine

SQL

Database

SQL Server

VMDocumentDB


databases


A Server is not a machine

TDS = Tabular Data Stream

SQL Server database technology “as a Service”

Fully Managed (PaaS)

Enterprise-ready with automatic support for HA, DR, Backups, replication and more

SQL Database – The Basics

Scale out with ElasticScale

Built-in regional database replicas for additional protection

Uptime SLA of 99.99% (4.38 minutes / month)

SQL Database – The Basics

ElasticScale

• New portal

• Some features in preview

• Full Text Search

• Column Store Index

• Users in contained database (+row-level security)

• Parallel queries (Premium only)

• CLR Integration

• T-SQL window functions (over clause)

• Change tracking

SQL Database – v12

Service

Tier

Performance

Level

Common App

Pattern

Performance Business Continuity

Max DB

Size

Transaction Perf.

Objective

DTU PITR DR / GEO-Rep

Basic Basic Small DB, SQL opp 2 GB Reliability / Hr. 5 7 Days DB Copy +

Manual Export

Standard S0

S1

S2

Wrkgp/cloud app,

multiple concurrent

operations

250 GB Reliability / Min. 10

20

50

14 Days DB Copy +

Manual Export

Premium P1

P2

P3

Mission Critical, High

volume, Many

concurrent Users

500 GB Reliability / sec. 100

200

800

35 Days Active Geo-

replication

Selecting the right SQL Database edition

This information is subject to change over time.

Blended Measure

Relative

SQL Database SQL IaaS

Why (at least) two offerings of SQL in Azure?

SQL Database (PaaS) SQL in a VM (IaaS)

SQL

Database

SQL Server

VMDocumentDB


databases


Fully managed, scalable JSON document database service

Fully Managed NoSQL Document-based Database ServiceAutomatic and consistent indexing of all properties

Tunable High PerformanceBacked by write-optimized SSD

Easy to scale

Native Support for JSON and JavaScript

Designed cloud first

A document storeCollections

Request

Document 1

{

"name": "John",

"country": "Canada",

"age": 43,

"lastUse": "March 4, 2014"

}

{

"name": "Eva",

"country": "Germany",

"age": 25

}

Document 2

{

"name": "Lou",

"country": "Australia",

"age": 51,

"firstUse": "May 8, 2013"

}

Document 3

{

"docCount": 3,

"last": "May 1, 2014"

}

Document 4

{…}

DocumentDB

Application

DocumentDB OverviewA NoSQL document database-as-a-service, fully managed by Microsoft Azure.

For cloud-designed apps when query over schema-free data; reliable and predictable performance; and rapid

development are key. First of its kind database service to offer native support for JavaScript, SQL query and

transactions over schema-free JSON documents.

Perfect for cloud architects and developers who need an enterprise-ready NoSQL document database.

Release TimingPublic Preview: August 21, 2014GA: H2 FY15

Proven @ Scale

Data store powering production Microsoft services.

Scale tested with databases over 10TBs of JSON document

data from Office OneNote API service

Rapidly evolving data models and query patterns based on

3rd party applications

Serving over 1 million active users

Operating consistently with 99.95% availability

First Party Use Cases

Store, roam and query user generated data across

applications

Extensible data models for 3rd party integration

Social activity and awareness feeds with views and filtering

ProgrammabilityBroad language and platform support

REST/HTTP APIs

C#, Node.js, JavaScript and Python SDKs

for Public Preview

SDKs open sourced through GitHub

C# Client SDK

Supports gateway and direct connectivity

Async APIs for all operations

HTTP and TCP transports available

POCOs, inherited document types and dynamics

LINQ provider for query execution

client = new DocumentClient(new Uri(endpointUrl), authKey);

Database database = await client.CreateDatabaseAsync(newDatabase{Id = “myNewDatabase”});

DocumentCollection collection = awaitclient.CreateDocumentCollectionAsync(database.SelfLink, newDocumentCollection{Id = “Pages"});

Page page = new Page

{

Id = "Sample",

Title = "About Paris",

Language = new Language { Name = "English" },

Author = new Author { Name = "Don” },

Content = "Don's document in DocDB is a valid JSON document",

PageViews = 10000,

Topics = new Topic[] {

new Topic { Title = "History" },

new Topic { Title = "Places to see" }}

};

ResourceResponse<Document> resp = awaitclient.CreateDocumentAsync(collection.DocumentsLink, page);

SQL

Database

SQL Server

VMDocumentDB


databases


Data size

Access

Updates

Structure

Integrity

Scaling

Relational DB vs. Hadoop

SQL

DatabaseSQL on IaaS DocumentDB

Search HDInsightMySQL,

Oracle


Additional Database options in Azure

Oracle VMs are supported in Azure.

MySQL is offered from the partner ClearDB.

Agenda – Part 2 - Storage

Blobs Files

Queues Tables

Microsoft Azure

Storage Blob

Two Types of Blobs Under the Hood


Block Blob:


Page Blob:

Blob Storage Concepts

Microsoft Azure

Storage Files

Azure Files – Usage

Share data across VMs and applications

Share settings throughout services

Dev/Test/Debug

Azure Files

Shared Network File Storage for Azure

Availability, durability, scalability are managed automatically

Supports two interfaces: SMB and REST

“I wish I could go to storage and provision a cloud drive, giving it a namespace, and that drive would then be UNC-addressable by the OSes.”

“I need two VM's running with a shared drive. One will write to the drive, the other will read [it].”

“Hi, I have two VM's in Microsoft Azure. All I want to do is set up a file share between them. Is this possible?”

“Is it possible to share a secondary disk between different VM instances?”

Azure Files – Customer Quotes

Microsoft Azure

Storage Queue

Producers Consumers

Queue C1

C2

P1

P2

4 3 12

Queue-based Load Levelling Pattern

Queue length reflects how well the backend processing nodes are doing.

Decouples the application.

Flexibility of efficient resource usage within an application.

Absorb traffic bursts and reduce the impact of individual component failures.

Why use a Queue?

Messages are ordered but not guaranteed FIFO.

Message will be processed at least once.

Message may be processed more than once.

.DequeueCount increases every time.

-> Processing must be idempotent.

Queue Considerations

Messages are stored up to 7 days

Event buses are built on top of queues

Queue Considerations

Microsoft Azure

Storage Table

Table Storage Details

Table Storage Concepts

No Fixed Schema

Querying

Table Storage Details

Table Storage Details – Entity Properties

A key/value store

Tables

Partition

B

Partition

A

Property

Entity

2B

Property Type

Data

Property Name

String intString Date

Name LastUseCountry Age

String intString

Name Country Age

String intString Date

Name FirstUseCountry Age

Row key

1

2

1

2Partition key

A

A

B

Bint Date

LastCount

2B

Azure Storage Tables

Application

To recap Azure Storage

TablesScalable service for non-relational data

Blob StorageStorage of files, VHDs, mp4s, pngs, etc…

FilesSMB File Sharing

QueuesLow latency message processing

Azure Data Lake

Azure Data Lake

Azure Data Lake

Microsoft Azure

SQL Data Warehouse

Azure SQL Data Warehouse

Azure SQL Data Warehouse

SQL Database

(PaaS)

SQL Server

VM (IaaS)DocumentDB


Oracle

Microsoft Azure Data Platform - Recap

Relational

database in the

cloud (PaaS)

SQL Server

VM (IaaS)DocumentDB


Oracle


Relational

database in the

cloud (PaaS)

Full relational

database in the

cloud (IaaS)DocumentDB


Oracle


Relational

database in the

cloud (PaaS)

Full relational

database in the

cloud (IaaS)

NoSQL – Json

documents


Oracle


Relational

database in the

cloud (PaaS)

Full relational

database in the

cloud (IaaS)

NoSQL – Json

documents

Search Big DataMySQL,

Oracle


Relational

database in the

cloud (PaaS)

Full relational

database in the

cloud (IaaS)

NoSQL – Json

documents

Search Big Data

Any other

database in the

cloud

(IaaS/PaaS)


Blobs Files

Queues Tables


Huge storage

Block = Streaming

Page = Random accessFiles

Queues Tables


Huge storage

Block = Streaming

Page = Random access

Shared access

between

services

Queues Tables


Huge storage

Block = Streaming


Shared access

between

services

Asynchronous

processes Tables


Huge storage

Block = Streaming


Shared access

between

services

Asynchronous

processes

NoSQL –

Key/Value pair


Data Lake

Azure SQL

Data

Warehouse


fully featured RDBMS

transactional processing

rich query

managed as a service

elastic scale

internet accessible http/rest

schema-free data model

arbitrary data formats


Azure Data OverviewEric Moreau, MVPwww.emoreau.com – [email protected]

Microsoft Azure

http://www.emoreau.com/

mailto:[email protected]

Technology

Samedi SQL Québec - La plateforme data de Azure