Building a scalable database solution

with SQL Azure

About me

• Shy Engelberg, CTO @

• Email : Shy@Valinor.co.il• Phone : 054-771-711-5• Twitter : @ShyEngelberg

Agenda• SQL Azure Database• Intro• Technical details• Pricing tiers and performance• Business continuity

• Elastic Database client library• Shard management• data-dependent routing• cross-shard query capabilities

DB

Microsoft Azure SQL Database

Microsoft Azure SQL Database• SQL Database is a relational database service in the cloud based on

the market-leading Microsoft SQL Server engine, with mission-critical capabilities.

SQL Database delivers predictable performance, scalability, business continuity, data protection, and near-zero administration to cloud developers and solution architects.

You can focus on rapid app development and accelerating your time to market, rather than managing virtual machines and infrastructure.

Microsoft Azure SQL Database

• Because it’s based on the SQL Server engine, Azure SQL Database supports existing SQL Server tools, libraries and APIs, which makes it easier for you to move and extend to the cloud.( Using the same TDS interface as regular SQL database)

Microsoft Azure SQL Database• Based on SQL server technology. (A subset of the product)

• Fully managed service (Microsoft manage it completely!)

• We Don't to think and worry about: VMs Resources (IO, CPU, Memory) Installations, upgrades, patches Services Files placement Transaction logAvailability solutions

Microsoft Azure SQL Database – Server• The basic unit in Azure that holds DBs is a “server”

• A server is a logical entity used for logical grouping of DBs and security bounding.It looks and smell like a server but it's basically only a TDS proxy endpoint.(For the application it looks like a server (instance))

• When creating it, you are required to provide a unique server name- (xxxx.database.windows.net).

• The server Initially contains only a master database.

Microsoft Azure SQL Database –DBMicrosoft Azure SQL Database has the following properties:

• Name – of the DB• Service Tier (Performance Level)• Max Size • Collation – for all the tables are columns.• “Server” – in which group to place the DB

Microsoft Azure SQL Database – service tiersAzure SQL Database provides multiple service tiers to handle different types of workloads.

• Performance level – measured by “DTU” (a combination of CPU, IO and log usage)• Max size• Point-in-time restore retention• Parallel connections• Size – does not include the log file size

You can change service tiers at any time with zero downtime to your application. (Scale up)

Microsoft Azure SQL Database – service tiersBasic, Standard, and Premium service tiers all have an uptime SLA of 99.99%

• Basic: Best suited for a small size database, supporting typically one single active operation at a given time. (predictable performance hour over hour)

• Standard: The go-to option for most cloud applications, supporting multiple concurrent queries. (predictable performance minute over minute)

• Premium: Designed for high transactional volume, supporting a large number of concurrent users and requiring the highest level of business continuity capabilities. (predictable performance second over second)

Microsoft Azure SQL Database – service tiers• The Database Transaction Unit (DTU) is the unit of measure in SQL

Database that represents the relative power of databases based on a real-world measure: the database transaction. We took a set of operations that are typical for an online transaction processing (OLTP) request, and then measured how many transactions could be completed per second under fully loaded conditions (that’s the short version, you can read the gory details in the Benchmark overview).

• A Basic database has 5 DTUs, which means it can complete 5 transactions per second, while a Premium P11 database has 1750 DTUs.

Microsoft Azure SQL Database – service tiers

Microsoft Azure SQL Database

• Create and manage Azure SQL database

DEMO

Microsoft Azure SQL Database – Connection• The Microsoft Azure SQL Database service is only available with TCP port

1433. (ensure your firewall allows outgoing TCP communication on TCP port 1433)

• The protocol is TDS (Tabular Data Stream) protocol over TCP/IP.

• All connections are encrypted - SSL is required - if encryption is not defined in the connection string, connection will fail.

• Use firewall rules to connect from outside Microsoft data center

Microsoft Azure SQL Database – Connection• Connect using the server name : xxxx.database.windows.net

• Connection Security is managed by SQL Database Firewall, an IP Address-based access control for SQL Database

Rules can be defined at the server level No IP authorized by default Option to disable/enable access from

applications hosted in Windows Azure:

Azure SQL Database

• Firewall and connection

DEMO

Microsoft Azure SQL Database – Business continuity

• Let’s start from the bottom line:

• SLA is built- in 99.9%. In opposed to traditional Always-On and failover cluster which requires a lot of work and money, this solution is cheap and out-of-the-box.

Microsoft Azure SQL Database – Business continuity

• Business continuity can be affected by one or more of the following three major categories of issues:

Failure of individual servers, devices or network connectivity (the disk that my data is on died)

Corruption, unwanted modification or deletion of data(someone dropped a table or deleted a row)

Widespread loss of datacenter facilities(Amsterdam is gone)

Microsoft Azure SQL Database – Business continuity

• Protection from Failure of Individual Servers and Devices - High Availability through Infrastructure Redundancy:

Maintaining multiple (3) copies of all data in different physical nodes located across fully independent physical sub-systems such as server racks and network routers.

Microsoft Azure SQL Database – Business continuity

Point-in-time restore, Programmatic “oops recovery” of data deletion or alteration:• Auto backups, transactional logs every 5 min• Backups in Azure Storage and geo-replicated • Backups retention policy:

Basic, up to 7 daysStandard, up to 14 daysPremium, up to 35 days

Microsoft Azure SQL database - Scaling

Microsoft Azure SQL Database – Elastic tools• Elastic Database features enables you to use the virtually unlimited

database resources of Azure SQL Database to create solutions for transactional workloads, and especially Software as a Service (SaaS) applications.

• Enabling this functionality is facilitated by features such as the elastic database client library and split-merge tool, known together as elastic database tools, which simplify building applications that rely on sharding.

Microsoft Azure SQL Database – Elastic tools• Elastic Database tools:

simplify development and management of sharded database solutions. The tools are: the Elastic Database client library and the Elastic Database split-merge tool.

Microsoft Azure SQL Database – Elastic tools• Elastic Database pools (preview): A pool is a collection of databases to

which you can add or remove databases at any time. The databases in the pool share a fixed amount of resources (DTUs). • Elastic Database jobs (preview): Use jobs to manage large numbers of

Azure SQL databases. Easily perform administrative operations such as schema changes, credentials management, reference data updates and more• Elastic Database query (preview): Enables you to run a Transact-SQL

query that spans multiple databases. This enables connection to reporting tools such as Excel, PowerBI, Tableau, etc.

Microsoft Azure SQL database – sharding“A database shard is a horizontal partition of data in a database.”

• Sharding is a technique to distribute large amounts of identically-structured data across a number of independent databases. It is especially popular with cloud developers who are creating Software as a Service (SAAS) offerings for end customers or businesses. These end customers are often referred to as “Tenants”. • Sharding works best when every transaction in an application can be

restricted to a single value of a sharding key. That ensures that all transactions will be local to a specific database.

Microsoft Azure SQL database – scaling out

Elastic Database client library• The Elastic Database client library helps you easily develop sharded

applications using hundreds—or even thousands—of Azure SQL databases hosted on Microsoft Azure• Elastic Database client library is now available as open source

software on GitHub.• Elastic database client library supports:

Shard managementData-dependent routingMulti-shard queries (MSQ)

Elastic Database library – Shard Map Management

• To manage a collection of shards, a special database called the "shard map manager" is created. It maintains information allowing an application to connect to the correct database based upon the value of the sharding key.• Shard map management is the ability for an application to manage various

metadata about its shards.• Developers can use this functionality to register databases as shards,

describe mappings of individual sharding keys or key ranges to those databases.• Without the elastic database client library, you would need to spend a lot of

time writing the management code when implementing sharding.

Elastic Database library – Shard Map Management

• use the ShardMapManager class, found in the Elastic Database client library to manage shard maps.• Elastic Scale support the following .Net Framework types as sharding

keys:integerlongguidbyte[]datetimetimespandatetimeoffset

Elastic Database library – Shard Map Management

• Shard maps can be constructed using lists of individual sharding key values, or they can be constructed using ranges of sharding key values. • The data managed by a ShardMapManager instance is kept in three

places: Global Shard Map (GSM): You specify a database to serve as the repository

for all of its shard maps and mappings.Local Shard Map (LSM): Every database that you specify to be a shard is

modified to contain shard map information.Application cache

Elastic Database library – Data dependent routing

• Data dependent routing is the ability to use the data in a query to route the request to an appropriate database. This is a fundamental pattern when working with sharded databases.• Each specific query or transaction in an application using data

dependent routing is restricted to accessing a single database per request.• the Shard Map Manager opens connections to the correct databases

when needed, based on the data in the shard map and the value of the sharding key that is the target of the application’s request.

Elastic Database library – Data dependent routing

• The ShardMap.OpenConnectionForKey method returns an ADO.Net connection ready for issuing commands to the appropriate database based on the value of the key parameter.

• The OpenConnectionForKey method returns a new already-open connection to the correct database. Connections utilized in this way still take full advantage of ADO.Net connection pooling.

Elastic Database library – Multi-shard querying

• Multi-shard querying is used for tasks such as data collection/reporting that require running a query that stretches across several shards.• The main entry point into multi-shard querying is the

MultiShardConnection class.• myShardMap.GetShards() method retrieves all shards from the shard

map and provides an easy way to run a query across all relevant databases. • The collection of shards for a multi-shard query can be refined further

by performing a LINQ query over the collection returned from the call to myShardMap.GetShards()

Elastic Database library

Working with Elastic database client library

DEMO

Questions?Thank you