Design Considerations For Storing With Windows Azure

Windows Azure PlatformWindows Azure Platform

Design considerations for storing data in the cloud with Windows Azure

Eric NelsonMicrosoft UK

Blog: http://geekswithblogs.net/iupdateableTwitter: http://twitter.com/ericnel and http://twitter.com/ukmsdnPodcast: http://bit.ly/msdnpodcast Newsletter: http://msdn.microsoft.com/en-gb/flash

Slides, links and background “diary” posts can be found on my blog

http://geekswithblogs.net/iupdateable

http://twitter.com/ericnel

http://twitter.com/ukmsdn

http://bit.ly/msdnpodcast

http://msdn.microsoft.com/en-gb/flash

Windows Azure Platform

2


• Windows Azure Platform 101• Storage in the Cloud• Blobs• Tables• Relational• Queues

• Lesson learned

Agenda


WINDOWS AZURE PLATFORM 101Just in case you had something better to do over the last 18months


3 Important Services• Windows Azure– Compute and

Storage• SQL Azure– Storage

• .NET Services– Connecting

3 Critical Concepts• Computation– Web and Worker

• Storage– Table, Blob,

Relational• Messaging– Queues, Service Bus

Windows Azure PlatformA simple site

Web TierBrowserRequest

B/L Tier Database

Response

“Wow! What a great site!”

Windows Azure PlatformUnder load

Web Tier B/L Tier Database

Browser

Browser

Browser

Browser

Browser

“Server Busy”

Windows Azure PlatformUnder load

Web Tier B/L Tier Database

Browser

“Timeout”

Browser

Browser

Browser

Browser

Windows Azure PlatformSolve using on-premise

Browser

Browser

Browser

Browser

Browser

B/L Tier

B/L Tier

B/L Tier

N L B

Database

p1 p2 p3

Web Tier

Web Tier

Web Tier

N L B

Windows Azure PlatformHowever…

Browser B/L Tier

B/L Tier

B/L Tier

N L B

Database

p1 p2 p3

Web Tier

Web Tier

Web Tier

N L B

“Not so great now…”

“Hmmm... Most of this stuff is sitting idle...”

“That took a lot of work - and money!”

Windows Azure PlatformSolve using the Cloud aka Windows Azure Platform

Web Role

AzureStorag

e

Browser

Browser

Browser

Browser

Browser

Web Role

Web Role

N L B

Worker Role

Worker RoleN

L B

p1 p2 p3

You don’t see this bit



Worker Role

or…Maybe you do

Windows Azure PlatformSolve using the Cloud aka Windows Azure Platform

Web Role

AzureStorag

e

Browser

Browser

Browser

Browser

Browser

Web Role

Web Role

N L B

Worker Role

Worker RoleN

L B

p1 p2 p3




Worker Role

Ok, you

definitely do

SQLAzure


DEMO: WINDOWS AZURE PORTAL


STORAGE IN THE CLOUD…

Windows Azure Storage and SQL Azure


BLOBS, TABLES, RELATIONAL




16


Blobs stored in Containers 1 or more Containers per account

• Scoping is at container level• …/Container/blobpath

Blobs• Capacity 50GB in CTP• Metadata, accessed independently

Private or Public container access

Blobs

Windows Azure PlatformPut a Blob

Client Azure Blob

Storage

REST API

PutBlob PUT http://account.blob.core.windows./net/containername/blobname

Blob Container

http://account.blob.core.windows.net/containername/blobname

Windows Azure PlatformGet a Blob

Client Azure Blob

Storage

REST API

GetBlob GET http://account.blob.core.windows./net/containername/blobname

Blob Container


Windows Azure PlatformGet part of a Blob

Client Azure Blob

Storage

REST API

GetBlob GET http://account.blob.core.windows./net/containername/blobnameRange: bytes=329300 - 730000

Blob Container


Windows Azure PlatformPut a LARGE Blob

Client Azure Blob

Storage

REST API

PutBlock(blobname, blockid7, data)PutBlockList(blobname, blockid1, …, blockidN)

Blob Container


PutBlock(blobname, blockid1, data)




22


Provides structured storage• Massively scalable tables (TBs of data)• Self scaling• Highly available• Durable

Familiar and easy-to-use API, layered• .NET classes and LINQ• ADO.NET Data Services – .NET 3.5 SP1• REST – with any platform or language

Introduction to Tables


23


No join No group by No order by “No Schema”

Not a Relational Database


24


Table• A Table is a set of Entities (rows)• An Entity is a set of Properties (columns)

Entity• Two “key” properties form unique ID• PartitionKey – enables scale• RowKey – uniquely ID within a partition

Data Model


25


Getting all of dunnry’s blog posts is fast• Single partition

Getting all posts after 2008-03-27 is slow• Traverse all partitions

Key Example – Blog PostsPartition Key

AuthorRow Key

PermalinkProperty 3ChangedOn

Property 4Title

smarx i-love-tables 2009-07-04 I Love Tables!

smarx tables-are-awesome 2009-07-12 Tables are

Awesome!

dunnry cheetos 2008-03-27 I Love Cheetos

dunnry blogging-again 2009-07-15 Finally Blogging

Again

dunnry phluffyfotos 2008-04-09 PhluffyFotos!

Partition 1

Partition 2

Windows Azure PlatformQuery a Table

Worker Role

AzureTable

Storage

http://account.table.core.windows.net

REST: GET http://account.table.core.windows.net/Customer?$filter=%20PartitionKey%20eq%20value LINQ:var customers = from o in context.CreateQuery<customer>(“Customer”) where o.PartitionKey == value select o;


27


Tradeoff between locality and scalability Considerations

• Entity group transactions• Query efficiency• Scalability• Flexible Partitioning

Choosing a Partition Key


28


Pick potential keys (common query filters) Order keys by importance If needed, include an additional unique key Use two most important keys as PK, RK Consider concatenating to form keys

A Method of Choosing Keys


29


Non-key queries are scans Improve performance by scoping

• Usually by partition key• But what about by table?

3 tables• Top 1,000 popular items• Top 10,000 popular items• Everything

Now arbitrary “top 1,000” queries are fast Better locality than clever partition keys

Write many is one approach


DEMO: WINDOWS AZURE STORAGE


31


Azure tables are *not* a relational database• Requires a mind shift• Azure tables scale• 3 - 9s availability• Azure tables support exactly one key• PartitionKey + RowKey

• Case Matters• No foreign keys• No referential integrity• No stored procedures

Lessons LearnedAzure Storage


32


Azure Storage Client Library• No longer just a “sample”

Azure storage is available via REST• Not limited to Azure hosted apps• Not limited to Microsoft platform or tools• Getting the signature correct is the hard part

Lessons LearnedAzure Storage


33


REST is *not* TDS• Be prepared to parse• LINQ and XML classes help• Sometimes, string parsing is the best choice

Azure storage names are picky• So are Azure key values• It’s possible to create an entity in a table and not be able to

update or delete it

Lessons LearnedAzure Storage - RESTful


34


Often better to pull back more than you need vs. multiple roundtrips

LINQ on results in memory is fast & flexible• foreach works well too

Sort and cache tables on the web tier

Lessons LearnedAzure Storage – Roundtrips are expensive


35


Different Entity types in the same table E.g. PK = CustomerId

• Customer, Order and OrderDetails in the same table

Lessons LearnedAzure Storage – Entity Group Transactions

PK RK

Cust1 Cust1 Name=“MSFT” Desc=“...”

Cust1 Order1 Date=“25/12/09”

Cust1 OrdeDetail1 Product=12 Quantity=6

Cust1 OrderDetail2 Product=13 Quantity=1

Cust2 Cust2 Name=“IBM” Desc=“...”




SQL Azure (July 2009)aka SQL Data Services

aka SQL Server Data Services

Windows Azure PlatformOn Premise Programming Model

SQL Server

Client RDBMS

TDSData

This is what we do on-premise...

Windows Azure PlatformSame for the cloud?

SQL Server

Client RDBMS

TDSData

So, is this is what we would like to do in the cloud...

Windows Azure PlatformSQL Azure can do this

SQL Azure

Client RDBMS

TDSData

Windows Azure PlatformSQL Azure can also do this

SQL Azure

Browser RDBMS

TDSWeb Role

HTTP

Windows Azure PlatformAnd this!

SQL Azure

Browser RDBMS

TDS

Web RoleHTTP

Worker Role

Queue

Windows Azure PlatformWhich means you can easily migrate from this

Browser RDBMS

Web TierHTTP

Bus. Logic

“The Data Center”

SQL Server

TDS

Windows Azure PlatformTo this… Windows Azure and SQL Azure

SQL Azure

Browser RDBMS

TDS

Web RoleHTTP

Worker Role

Queue

“The Cloud”


DEMO: SQL AZURE


46


From the database “down” it’s just SQL Server• Well, almost …• Many tools don’t work today

• System catalog is different

Above the database is taken care of for you• You can’t really change anything

Lessons LearnedSQL Azure


47


Tooling• SSMS partially works – “good enough”• Can not create connection using Visual Studio designer• Other tools may work better• No BCP (currently)

DDL• Must be a clustered index on every table• No physical file placement• No indexed views• No “not for replication” constraint allowed• No Extended properties• Some index options missing (e.g. allow_row_locks, sort_in_tempdb ..)• No set ansi_nulls on



48


Types• No spatial or hierarchy id• No Text/images support.

• Use nvarchar(max)• XML datatype and schema allowed but no XML index or schema

collection.

Security• No integrated security



49


Development• No CLR• Local temp tables are allowed • Global temp tables are not allowed• Cannot alter database inside a connection• No UDDT’s• No ROWGUIDCOL column property



50


SQL Server is very familiar• SQL Azure *is* SQL Server in the cloud

Windows Azure Storage is…very different Make the right choice

• Understand Azure storage• Understand SQL Azure• Understand they are totally different

You can use both

Lessons LearnedSQL Azure vs Windows Azure Tables


51


SQL Azure is not always the best storage option SQL Azure costs more

• Delivers a *lot* more functionality

SQL Azure is more limited on scale

Lessons Learned SQL Azure vs Windows Azure Tables


52


Can be done• Many 10GB databases

Not fun

Lessons Learned SQL Azure and Sharding


QUEUES


54


Simple asynchronous dispatch queue• Create and delete queues

Message:• Retrieved at least once• Max size 8kb• Operations:• Enqueue• Dequeue• RemoveMessage

Queues

Windows Azure PlatformUsing the Cloud for Communications

ClientREST

Azure Queue

http://app.queue.core.windows.net/


Client

RESTAzure Queue


Client

Company 1

Company 2


Client

RESTAzure Queue


Client

Company 1

Company 2x


Client

RESTAzure Queue


Client

Company 1

Company 2

Web Role


IN SUMMARY


Windows Azure Platform Benefits

Windows Azure• High Level of Abstraction

– Hardware– Server OS– Network Infrastructure– Web Server

• Availability– Automated Service Management

• Scalability– Instance & Partitions

• Developer Experience– Familiar Developer Tools

SQL Azure• Higher Level of Abstraction

– Hardware– Server OS– Network Infrastructure– Database Server

• Availability– Automated Database

Management & Replication

• Scalability– Databases Partitioning

• Developer Experience– Familiar SQL Environment


Resources

• Slides, links and more– http://geekswithblogs.net/iupdateable– Azure Training Kit (August update)

• www.azure.com– Sign up, links to resources etc

• http://www.azureadvantage.co.uk/ – Rapid provisioning of Windows Azure

http://geekswithblogs.net/iupdateable

http://www.azure.com/

http://www.azureadvantage.co.uk/

Technology

Design Considerations For Storing With Windows Azure