Nivo 300

Data Storage in

Windows Azure

Blaž Lipušček,

Diventic d.o.o.

Nivo 300

Cloud Computing

• Software as a Service

• Platform as a Service

• Infrastructure as a Service

Computing capability delivered as a utility through

Internet standards and protocols.

Nivo 300

Windows Azure Fabric

• Fabric Controller is the brain of the entire ecosystem

• Highly redundant hardware and software environment

• It can‘t ever fail!

Nivo 300

Windows Azure Storage Options

• Instance local storage is not persistent:

• Usefull for cache or temp files!

• Blob service:

• Store large text or binary data

• Files can be served via CDN

• Table service

• Azure Drive:

• NTFS, can be used with I/O

• Can connect only one instance at once

Nivo 300

Windows Azure Blob Service

• REST API

Nivo 300

Store Files to Blob service

• WindowsAzure.StorageClient (REST API)

Nivo 300

Grant Access to Files

• Shared Access Signatures

• Read

• Write

• Delete

• List

https://mycontainer.blob.core.windows.net/files/image.jpg?se=2011-05-

18T12%3A50%3A38Z&sr=b&sp=r&sig=EUtAbKQ%2F%2FYY7k1fJtzE6hWt%2FjadF

AqxDYh5tAPK%2FeIw%3D

Nivo 300

Grant Access to Files:

Shared Access Signatures

• Remember to use UTC on DateTime

• Azure hides resources with invalid Shared Access Signatures

• DeleteIfExists will never fail

• Make sure URL is properly encoded

Nivo 300

Windows Azure Table Storage

• http://myaccount.table.core.windows.net

Nivo 300

Windows Azure Table Storage

• Requires major changes to application data logic

• No Schema, NO-SQL

• More scalable than SQL Azure

• Can handle large datasets

• Relatively cheap

Nivo 300

Windows Azure Tables

• Table names must conform to these rules:

^[A-Za-z][A-Za-z0-9]{2,62}$

• An entity can have up to 255 properties

• The combined size of all data in an entity's cannot exceed 1 MB

• 64KB limitation per property

• An entity always has the following system properties:

• PartitionKey, RowKey,Timestamp property

Nivo 300

Windows Azure Tables

• Not allowed in PartitionKey and RowKey values: /, \, #, ?

• The partition and row key is a string value up to 1 KB in size.

• You must include the PartitionKey and RowKey property in every

insert, update, and delete operation.

Nivo 300

Windows Azure Tables

• REST service via HTTP(S):

• CREATE:

http://myaccount.table.core.windows.net/Tables

• LIST:

http://myaccount.table.core.windows.net/Tables

• GET, UPDATE, DELETE:

http://myaccount.table.core.windows.net/MyTable()

• SINGLE TABLE:

http://myaccount.table.core.windows.net/Tables('MyTable')

Nivo 300

Query Timeout and Pagination

• Query can return up to 1,000 rows at a time:

• x-ms-continuation-NextTableName

• x-ms-continuation-NextPartitionKey

• x-ms-continuation-NextRowKey

• Construct new query using continuation tokens:

• NextTableName, NextPartitionKey , NextRowKey

• 30 seconds for scheduling and processing

Nivo 300

Query Timeout and Pagination

Date: Mon, 13 Feb 2012 20:11:08 GMT Content-Type: application/atom+xml;charset=utf-8 Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0 Cache-Control: no-cache x-ms-request-id: f9b2cd09-4dec-4570-b06d-4fa30179a58e x-ms-version: 2011-08-18 x-ms-continuation-NextPartitionKey: 1!8!U21pdGg- x-ms-continuation-NextRowKey: 1!12!QmVuMTg5OA-- Content-Length: 880298

http://myaccount.table.core.windows.net/Customers? NextPartitionKey=1!8!U21pdGg-& NextRowKey=1!12!QmVuMTg5OA--

Nivo 300

Query Options

Operator URI expression

Equal eq

GreaterThan gt

GreaterThanOrEqual ge

LessThan lt

LessThanOrEqual le

NotEqual ne

Operator URI expression

And and

Not not

Or or

Nivo 300

Windows Azure Table Storage

PartitionKey RowKey Timestamp Property Collection Name Type Value

Jeff C_Jeff DateTime Kind String “Customer” Name String “Jeffrey” City String “Seattle”

Jeff O_Cereal DateTime Kind String “Order” Item String “Cereal” Quantity Int32 1

Jeff O_Milk DateTime Kind String “Order” Item String “Milk” Quantity Int32 2

Paul C_Paul DateTime Kind String “Customer” Name String “Paul” City String “New York”

Nivo 300

Windows Azure Table Storage

Date Type Description

String A UTF-16-encoded string of characters up to 64 KB in size.

Byte array An array of bytes up to 64 KB in size.

Guid A 128-bit globally unique identifier (can be null).

DateTime A 64-bit value expressed as Coordinated Universal Time (UTC) ranging from 12:00 midnight, January 1, 1601 A.D. (C.E.) to December 31, 9999 (can be null).

Int32 A 32-bit signed integer (can be null)

Int64 A 64-bit signed integer (can be null).

Double A 64-bit floating point value (can be null)

Boolean A Boolean value (can be null)

Nivo 300

DataServiceContext

• It is not stateless (like WCF Data Services)

• Any instance members are not guaranteed to be thread safe

Nivo 300

Table Storage: Good to know!

• DataServiceContext is not stateless!

• Create it when you need to perform multiple transactions

• Set timeout by using the Timeout property

• Query using PartitionKey and RowKey

• Only this two parameters are indexed.

• Order records by latest:

• DateTime.MaxValue.Ticks - date.ToUniversalTime().Ticks

Nivo 300

Retrieve a Single Entity • Fastest query

• Handle 404 Not Found

Nivo 300

Retrieve Using Prefix

Nivo 300

Entity Group Transactions

• Limited to a single partition in a single table

• context.SaveChanges(

SaveChangesOptions.Batch);

PartitionKey RowKey ID Kind Amount Title ... Timestamp

7635 7635 7635 ExpenseItem MyExpense 15.5...30:00

7635 7635_8215 8215 ExpenseItemRow 100 15.5...30:05

7635 7635_8216 8216 ExpenseItemRow 50 15.5...30:10

Nivo 300

Parallel Querying for Full Table Scans

• Fetch smaller list of partition keys (eg. Categories)

• Execute multiple queries in parallel:

• serviceContext.FoodTable.AsTableServiceQuery() .Where(c =>

c.PartitionKey == _category && c.Color == "Green");

• Parallel LINQ

Nivo 300

Use Command Pattern

TableCommand (abstract)

Insert Update Delete

Nivo 300

TableCommand abstract class

Nivo 300

Select Command

Nivo 300

Update Command

Nivo 300

Update Command

Nivo 300

Fat Entities

• Instead of going for 64KB limitation per property > single 1MB

limitation for the whole entity.

• Table provider also abstracts away all subtle retry policies:

• Handles OperationTimedOut error

• 4MB restriction of the Table Storage for transactions

• http://code.google.com/p/lokad-cloud/wiki/FatEntities

Nivo 300

Fat Entities

Nivo 300

Table Storage Performance

Optimization

Use when table and entity name are differenct. 8-15ms for deserializing each entity received in a query.

Payload delay. Overhead with POST and PUT requests.

Nivo 300

Windows Azure Storage Analytics

• Track, Analyze, and Debug usage of Windows Azure Storage

• Logs: Provide trace of executed requests for Blobs, Tables and

Queues

• Metrics: Provide summary of key capacity and request statistics for

Blobs, Tables and Queues

• Data stored inside your account

• Turn it on with Get Storage Service Properties API call

Nivo 300

Windows Azure Storage Analytics

• Transaction data and capacity data is stored in the following tables:

• $MetricsCapacityBlob

• $MetricsTransactionsBlob

• $MetricsTransactionsTable

• $MetricsTransactionsQueue

Nivo 300

Logs

• Loggin Authenticated Requests:

• Successful requests

• Failed requests, including timeout, throttling, network, authorization ...

• Requests using a SAS, including failed and successful requests

• Requests to analytics data

• Loggin Anonymous Requests:

• Successful requests

• Server errors

• Timeout errors for both client and server

• Failed GET requests with error code 304 (Not Modified)

Nivo 300

Metrics

• Request information: Provides hourly aggregates of number of

requests, average server side latency, average E2E latency, average

bandwidth, total successful requests and total number of failures

and more.

• Capacity information: Provides daily statistics for the space

consumed by the service, number of containers and number of

objects that are stored in the service.

Note, this is currently only provided for the Windows Azure Blob service.

Nivo 300

Get/Set Storage Service Properties Element Name Description

Version Required. The version of Storage Analytics to configure.

Delete Required. Applies only to logging configuration. Indicates whether all delete requests should be logged.

Read Required. Applies only to logging configuration. Indicates whether all read requests should be logged.

Write Required. Applies only to logging configuration. Indicates whether all write requests should be logged.

Enabled Required. Indicates whether metrics is enabled for the storage service.

IncludeAPIs Required only if metrics is enabled. Applies only to metrics configuration. Indicates whether metrics should generate summary statistics for called API operations.

RetentionPolicy/Enabled Required. Indicates whether a retention policy is enabled for the storage service.

RetentionPolicy/Days Required only if a retention policy is enabled. Indicates the number of days that metrics or logging data should be retained. All data older than this value will be deleted. The minimum value you can specify is 1; the largest value is 365 (one year).

Nivo 300

Windows Azure Storage

Analytics Tools

• Cerebrata WAS Configuration Utility https://www.cerebrata.com/Downloads/SACU/SACU.application

• Sample Tool http://storageanalytics.cloudapp.net

• Cerebrata Cloud Storage Studio

• Cerebrata Diagnostics Manager

Nivo 300

Connecting to SQL Azure

• Connecting to SQL Azure only requires change in configuration

• No support for Windows Authentication

• Encrypt credentials

Nivo 300

Connecting to SQL Azure

• A logged-in session that has been idle for 30 minutes will be

terminated automatically.

Open connections late and close early.

Nivo 300

SQL Azure Feature Limitations

• Common Language Runtime (CLR) and CLR User-Defined Types

• Integrated Full-Text Search

• SQL Server 2008 R2 Features (SQL Server Utility,

PowerShell Provider, Master Data Services)

• SQL Server Replications, Backup and Restore

CREATE DATABASE destination_database_name

AS COPY OF

[source_server_name.]source_database_name

Nivo 300

Troubleshooting SQL Azure

• Service Unavailable or Does Not Exist

• Check if you are unable to resolve the name

• Firewall blocking port 1433 used by SQL Azure

• Using proxy server not configured properly

• General Network Errors

• Connection is idle for an extended period of time.

• Connection consumes an excessive amount of resources

• Connection holds onto a transaction for an extended period of time

• If the server is too busy

https://www.windowsazure.com/en-us/support/service-dashboard/

Nivo 300

SQL Azure vs Table Storage

Feature SQL Azure Common Benefit(s) Azure Table Storage

Select semantics Cross-table queries Primary key-based lookup Single key lookup (by

partition)

Performance and scale

High performance via multiple indices, normalized data structures.

Automatic mass scale by partition and consistent performance even at large scale

User experience

Well-known management tools and traditional database design

Familiar high-level developer experience

Direct serialization; simplified design model by removing relational model

Storage style Traditional relational design model

Data storage for all types of data

Type variability in a single table

Cost factors No transaction cost, pay by database size

Network traffic cost outside of same datacenter

No space overhead cost, pay for what is used

Data loading and sync

Synchronizing between local and cloud-based stores; data easily moved in and out.

Nivo 300

Windows Azure Cost structure

In the context of the cost-oriented architecture, various decisions will impact monthly operational expenses.

• Compute charges

• Storage charges

• Charges for Data Transfer

• Add-on service charges

Compute

Storage

Transfer

Services

Nivo 300

Windows Azure Pricing Windows Azure Capability Charge

Server Usage Small: $0.12 /service-hour

Medium: $0.24/service-hour

Large: $0.48/service-hour

XLarge: $0.96/service-hour

Windows Azure Blobs and

Tables

$0.14/GB

Transactions $0.01/10K transactions

SQL Azure: Web Edition $9.99/month (1GB RDBMS)

SQL Azure: Business Edition $99.99/month (10GB RDBMS)

Windows Azure $0.15/100K message operations

Nivo 300

VPRAŠANJA?

Po zaključku predavanja, prosimo, izpolnite vprašalnik.

Vprašalniki bodo poslani na vaš e-naslov, dostopni pa

bodo tudi preko profila na spletnem portalu konference

www.ntk.si.

Najlepša hvala!

Blaž Lipušček, Diventic d.o.o.

blaz.lipuscek@diventic.si