Nivo 300
Data Storage in
Windows Azure
Blaž Lipušček,
Diventic d.o.o.
Nivo 300
Cloud Computing
• Software as a Service
• Platform as a Service
• Infrastructure as a Service
Computing capability delivered as a utility through
Internet standards and protocols.
Nivo 300
Windows Azure Fabric
• Fabric Controller is the brain of the entire ecosystem
• Highly redundant hardware and software environment
• It can‘t ever fail!
Nivo 300
Windows Azure Storage Options
• Instance local storage is not persistent:
• Usefull for cache or temp files!
• Blob service:
• Store large text or binary data
• Files can be served via CDN
• Table service
• Azure Drive:
• NTFS, can be used with I/O
• Can connect only one instance at once
Nivo 300
Windows Azure Blob Service
• REST API
Nivo 300
Store Files to Blob service
• WindowsAzure.StorageClient (REST API)
Nivo 300
Grant Access to Files
• Shared Access Signatures
• Read
• Write
• Delete
• List
https://mycontainer.blob.core.windows.net/files/image.jpg?se=2011-05-
18T12%3A50%3A38Z&sr=b&sp=r&sig=EUtAbKQ%2F%2FYY7k1fJtzE6hWt%2FjadF
AqxDYh5tAPK%2FeIw%3D
Nivo 300
Grant Access to Files:
Shared Access Signatures
• Remember to use UTC on DateTime
• Azure hides resources with invalid Shared Access Signatures
• DeleteIfExists will never fail
• Make sure URL is properly encoded
Nivo 300
Windows Azure Table Storage
• http://myaccount.table.core.windows.net
Nivo 300
Windows Azure Table Storage
• Requires major changes to application data logic
• No Schema, NO-SQL
• More scalable than SQL Azure
• Can handle large datasets
• Relatively cheap
Nivo 300
Windows Azure Tables
• Table names must conform to these rules:
^[A-Za-z][A-Za-z0-9]{2,62}$
• An entity can have up to 255 properties
• The combined size of all data in an entity's cannot exceed 1 MB
• 64KB limitation per property
• An entity always has the following system properties:
• PartitionKey, RowKey,Timestamp property
Nivo 300
Windows Azure Tables
• Not allowed in PartitionKey and RowKey values: /, \, #, ?
• The partition and row key is a string value up to 1 KB in size.
• You must include the PartitionKey and RowKey property in every
insert, update, and delete operation.
Nivo 300
Windows Azure Tables
• REST service via HTTP(S):
• CREATE:
http://myaccount.table.core.windows.net/Tables
• LIST:
http://myaccount.table.core.windows.net/Tables
• GET, UPDATE, DELETE:
http://myaccount.table.core.windows.net/MyTable()
• SINGLE TABLE:
http://myaccount.table.core.windows.net/Tables('MyTable')
Nivo 300
Query Timeout and Pagination
• Query can return up to 1,000 rows at a time:
• x-ms-continuation-NextTableName
• x-ms-continuation-NextPartitionKey
• x-ms-continuation-NextRowKey
• Construct new query using continuation tokens:
• NextTableName, NextPartitionKey , NextRowKey
• 30 seconds for scheduling and processing
Nivo 300
Query Timeout and Pagination
Date: Mon, 13 Feb 2012 20:11:08 GMT Content-Type: application/atom+xml;charset=utf-8 Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0 Cache-Control: no-cache x-ms-request-id: f9b2cd09-4dec-4570-b06d-4fa30179a58e x-ms-version: 2011-08-18 x-ms-continuation-NextPartitionKey: 1!8!U21pdGg- x-ms-continuation-NextRowKey: 1!12!QmVuMTg5OA-- Content-Length: 880298
http://myaccount.table.core.windows.net/Customers? NextPartitionKey=1!8!U21pdGg-& NextRowKey=1!12!QmVuMTg5OA--
Nivo 300
Query Options
Operator URI expression
Equal eq
GreaterThan gt
GreaterThanOrEqual ge
LessThan lt
LessThanOrEqual le
NotEqual ne
Operator URI expression
And and
Not not
Or or
Nivo 300
Windows Azure Table Storage
PartitionKey RowKey Timestamp Property Collection Name Type Value
Jeff C_Jeff DateTime Kind String “Customer” Name String “Jeffrey” City String “Seattle”
Jeff O_Cereal DateTime Kind String “Order” Item String “Cereal” Quantity Int32 1
Jeff O_Milk DateTime Kind String “Order” Item String “Milk” Quantity Int32 2
Paul C_Paul DateTime Kind String “Customer” Name String “Paul” City String “New York”
Nivo 300
Windows Azure Table Storage
Date Type Description
String A UTF-16-encoded string of characters up to 64 KB in size.
Byte array An array of bytes up to 64 KB in size.
Guid A 128-bit globally unique identifier (can be null).
DateTime A 64-bit value expressed as Coordinated Universal Time (UTC) ranging from 12:00 midnight, January 1, 1601 A.D. (C.E.) to December 31, 9999 (can be null).
Int32 A 32-bit signed integer (can be null)
Int64 A 64-bit signed integer (can be null).
Double A 64-bit floating point value (can be null)
Boolean A Boolean value (can be null)
Nivo 300
DataServiceContext
• It is not stateless (like WCF Data Services)
• Any instance members are not guaranteed to be thread safe
Nivo 300
Table Storage: Good to know!
• DataServiceContext is not stateless!
• Create it when you need to perform multiple transactions
• Set timeout by using the Timeout property
• Query using PartitionKey and RowKey
• Only this two parameters are indexed.
• Order records by latest:
• DateTime.MaxValue.Ticks - date.ToUniversalTime().Ticks
Nivo 300
Retrieve a Single Entity • Fastest query
• Handle 404 Not Found
Nivo 300
Retrieve Using Prefix
Nivo 300
Entity Group Transactions
• Limited to a single partition in a single table
• context.SaveChanges(
SaveChangesOptions.Batch);
PartitionKey RowKey ID Kind Amount Title ... Timestamp
7635 7635 7635 ExpenseItem MyExpense 15.5...30:00
7635 7635_8215 8215 ExpenseItemRow 100 15.5...30:05
7635 7635_8216 8216 ExpenseItemRow 50 15.5...30:10
Nivo 300
Parallel Querying for Full Table Scans
• Fetch smaller list of partition keys (eg. Categories)
• Execute multiple queries in parallel:
• serviceContext.FoodTable.AsTableServiceQuery() .Where(c =>
c.PartitionKey == _category && c.Color == "Green");
• Parallel LINQ
Nivo 300
Use Command Pattern
TableCommand (abstract)
Insert Update Delete
Nivo 300
TableCommand abstract class
Nivo 300
Select Command
Nivo 300
Update Command
Nivo 300
Update Command
Nivo 300
Fat Entities
• Instead of going for 64KB limitation per property > single 1MB
limitation for the whole entity.
• Table provider also abstracts away all subtle retry policies:
• Handles OperationTimedOut error
• 4MB restriction of the Table Storage for transactions
• http://code.google.com/p/lokad-cloud/wiki/FatEntities
Nivo 300
Fat Entities
Nivo 300
Table Storage Performance
Optimization
Use when table and entity name are differenct. 8-15ms for deserializing each entity received in a query.
Payload delay. Overhead with POST and PUT requests.
Nivo 300
Windows Azure Storage Analytics
• Track, Analyze, and Debug usage of Windows Azure Storage
• Logs: Provide trace of executed requests for Blobs, Tables and
Queues
• Metrics: Provide summary of key capacity and request statistics for
Blobs, Tables and Queues
• Data stored inside your account
• Turn it on with Get Storage Service Properties API call
Nivo 300
Windows Azure Storage Analytics
• Transaction data and capacity data is stored in the following tables:
• $MetricsCapacityBlob
• $MetricsTransactionsBlob
• $MetricsTransactionsTable
• $MetricsTransactionsQueue
Nivo 300
Logs
• Loggin Authenticated Requests:
• Successful requests
• Failed requests, including timeout, throttling, network, authorization ...
• Requests using a SAS, including failed and successful requests
• Requests to analytics data
• Loggin Anonymous Requests:
• Successful requests
• Server errors
• Timeout errors for both client and server
• Failed GET requests with error code 304 (Not Modified)
Nivo 300
Metrics
• Request information: Provides hourly aggregates of number of
requests, average server side latency, average E2E latency, average
bandwidth, total successful requests and total number of failures
and more.
• Capacity information: Provides daily statistics for the space
consumed by the service, number of containers and number of
objects that are stored in the service.
Note, this is currently only provided for the Windows Azure Blob service.
Nivo 300
Get/Set Storage Service Properties Element Name Description
Version Required. The version of Storage Analytics to configure.
Delete Required. Applies only to logging configuration. Indicates whether all delete requests should be logged.
Read Required. Applies only to logging configuration. Indicates whether all read requests should be logged.
Write Required. Applies only to logging configuration. Indicates whether all write requests should be logged.
Enabled Required. Indicates whether metrics is enabled for the storage service.
IncludeAPIs Required only if metrics is enabled. Applies only to metrics configuration. Indicates whether metrics should generate summary statistics for called API operations.
RetentionPolicy/Enabled Required. Indicates whether a retention policy is enabled for the storage service.
RetentionPolicy/Days Required only if a retention policy is enabled. Indicates the number of days that metrics or logging data should be retained. All data older than this value will be deleted. The minimum value you can specify is 1; the largest value is 365 (one year).
Nivo 300
Windows Azure Storage
Analytics Tools
• Cerebrata WAS Configuration Utility https://www.cerebrata.com/Downloads/SACU/SACU.application
• Sample Tool http://storageanalytics.cloudapp.net
• Cerebrata Cloud Storage Studio
• Cerebrata Diagnostics Manager
Nivo 300
Connecting to SQL Azure
• Connecting to SQL Azure only requires change in configuration
• No support for Windows Authentication
• Encrypt credentials
Nivo 300
Connecting to SQL Azure
• A logged-in session that has been idle for 30 minutes will be
terminated automatically.
Open connections late and close early.
Nivo 300
SQL Azure Feature Limitations
• Common Language Runtime (CLR) and CLR User-Defined Types
• Integrated Full-Text Search
• SQL Server 2008 R2 Features (SQL Server Utility,
PowerShell Provider, Master Data Services)
• SQL Server Replications, Backup and Restore
CREATE DATABASE destination_database_name
AS COPY OF
[source_server_name.]source_database_name
Nivo 300
Troubleshooting SQL Azure
• Service Unavailable or Does Not Exist
• Check if you are unable to resolve the name
• Firewall blocking port 1433 used by SQL Azure
• Using proxy server not configured properly
• General Network Errors
• Connection is idle for an extended period of time.
• Connection consumes an excessive amount of resources
• Connection holds onto a transaction for an extended period of time
• If the server is too busy
https://www.windowsazure.com/en-us/support/service-dashboard/
Nivo 300
SQL Azure vs Table Storage
Feature SQL Azure Common Benefit(s) Azure Table Storage
Select semantics Cross-table queries Primary key-based lookup Single key lookup (by
partition)
Performance and scale
High performance via multiple indices, normalized data structures.
Automatic mass scale by partition and consistent performance even at large scale
User experience
Well-known management tools and traditional database design
Familiar high-level developer experience
Direct serialization; simplified design model by removing relational model
Storage style Traditional relational design model
Data storage for all types of data
Type variability in a single table
Cost factors No transaction cost, pay by database size
Network traffic cost outside of same datacenter
No space overhead cost, pay for what is used
Data loading and sync
Synchronizing between local and cloud-based stores; data easily moved in and out.
Nivo 300
Windows Azure Cost structure
In the context of the cost-oriented architecture, various decisions will impact monthly operational expenses.
• Compute charges
• Storage charges
• Charges for Data Transfer
• Add-on service charges
Compute
Storage
Transfer
Services
Nivo 300
Windows Azure Pricing Windows Azure Capability Charge
Server Usage Small: $0.12 /service-hour
Medium: $0.24/service-hour
Large: $0.48/service-hour
XLarge: $0.96/service-hour
Windows Azure Blobs and
Tables
$0.14/GB
Transactions $0.01/10K transactions
SQL Azure: Web Edition $9.99/month (1GB RDBMS)
SQL Azure: Business Edition $99.99/month (10GB RDBMS)
Windows Azure $0.15/100K message operations
Nivo 300
VPRAŠANJA?
Po zaključku predavanja, prosimo, izpolnite vprašalnik.
Vprašalniki bodo poslani na vaš e-naslov, dostopni pa
bodo tudi preko profila na spletnem portalu konference
www.ntk.si.
Najlepša hvala!
Blaž Lipušček, Diventic d.o.o.