Windows Azure Storage

Anton Boyko.NET developer

Agenda

Windows Azure Storage

Blob Storage

Drives

Tables

Windows Azure Storage

Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Windows Azure Storage AccountUser specified globally unique account name

North Central USNorthern Europe

Western Europe East Asia

South East Asia

US Europe Asia

Can choose geo-location to host storage account:

South Central US

West US East US

Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes

Can co-locate storage account with compute accountExplicitly or using affinity groups

Accounts have two independent 512 bit shared secret keys

100 TBs per account

New Features

Geo-ReplicationStorage AnalyticsLogs: Provide trace of executed requests for your storage accountsMetrics: Provide summary of key capacity and request statistics for Blobs, Tables, and Queues

Storage in the Development FabricProvides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above

http://msdn.microsoft.com/en-us/gg433135

The Storage Client API

In this presentation we’ll cover the underlying

RESTful APICan call these from any HTTP cliente.g. Flash, Silverlight, etc…

Client API from SDK Microsoft.WindowsAzure.StorageClientProvides a strongly typed wrapper around REST services

Storage Security

Windows Azure Storage provides simple security for calls to storage serviceHTTPS endpointDigitally sign requests for privileged operations

Two 512bit symmetric keys per storage accountCan be regenerated independently

More granular security via Shared Access Signatures

Windows Azure Storage Abstractions

TablesStructured storage. A table is a set of entities; an entity is

a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Blob Storage

Blob Storage Concepts

BlobContainerAccount

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/ Blocks

contoso

PIC01.JPG

Block/Page

Block/Page

PIC02.JPG

images

VID1.AVIvideos

Blob Details

Main Web Service

Operations

PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob

Blob Details

Associate Metadata with Blob

Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)

Metadata is <name, value> pairs, up to 8KB per blob

Either as part of PutBlob or independently

Blob Details

Blob always accessed by

name

Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg

Blob ContainersMultiple Containers per AccountSpecial $root container

Blob ContainerA container holds a set of blobsSet access policies at the container level Associate Metadata with ContainerList the blobs in a containerIncluding Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ?

Blobs ThroughputEffectively in Partition of 1Target of 60MB/s per Blob

GET http://.../products?comp=list&prefix=Tents&delimiter=/

<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>

Enumerating Blobs

GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)

http://adventureworks.blob.core.windows.net/ Products/Bikes/SuperDuperCycle.jpg Products/Bikes/FastBike.jpg Products/Canoes/Whitewater.jpg Products/Canoes/Flatwater.jpg Products/Canoes/Hybrid.jpg Products/Tents/PalaceTent.jpg Products/Tents/ShedTent.jpg

http://.../products?comp=list&prefix=Canoes&maxresults=2

&marker=MarkerValue

<Blob>Canoes/Hybrid.jpg</Blob>

Pagination

Large lists of Blobs can

be paginatedEither set maxresults or;Exceed default value for maxresults (5000)

http://.../products?comp=list&prefix=Canoes&maxresults=2

<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>

Two Types of Blobs Under the Hood

Block BlobTargeted at streaming workloads

Each blob consists of a sequence of blocksEach block is identified by a Block ID

Size limit 200GB per blob

Optimistic Concurrency via Etags

Page BlobTargeted at random read/write workloads

Each blob consists of an array of pages Each page is identified by its offset from the start of the blob

Size limit 1TB per blob

Optimistic or Pessimistic (locking) concurrency via leases

TheBlob.wmv

Uploading a Block Blob

Uploading a large blob

10 GB Movie

Blo

ck I

d 1

Blo

ck I

d 2

Blo

ck I

d 3

Blo

ck I

d N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

THE BLOB

Windows AzureStorage

Page Blob – Random Read/Write

Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytesRandom Access Operations

PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:

[0,512) , [1536,2560)GetBlob[1000, 2048) returns

All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

512

1024

1536

2048

2560

10 G

B A

dd

ress S

pace

Shared Access Signatures

Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted

Two broad approachesAd-hocPolicy based

Content Delivery Network (CDN)

High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growing

Same experience for users no matter how far they are from the geo-location where the storage account is hosted

Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/

Windows Azure CDN URL: http://<id>.vo.msecnd.net/

Custom Domain Name for CDN: http://cdn.contoso.com/

pic1.jpg

Windows Azure CDN

To Enable CDN:Register for CDN via Dev PortalSet container images to public

pic1.jpg

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/

pic1.jpg

404

TTL Content Delivery Network

Windows Azure Blob Service

EdgeLocation

EdgeLocation

EdgeLocation

Drives

Windows Azure DrivesDurable NTFS volume for Windows Azure InstancesUse existing NTFS APIs to access a network attached durable driveUse System.IO from .NET

BenefitsMove existing apps using NTFS more easily to the cloudDurability and survival of data on instance recycle Drives can be up to 1TB

A Windows Azure Drive is an NTFS VHD Page BlobMounts Page Blob over the network as an NTFS driveLocal cache on instance for read operationsAll flushed and unbuffered writes to drive are made durable to the Page Blob

Drive Details

Operations performed via Drive API not REST CallsOperations on DrivesCreateDriveCreates a new NTFS formatted VHD in Blob storage

MountDrive/UnmountDriveMounts a drive into Instance at new drive letter

Unmounts a drive freeing drive letter

Get Mounted DrivesList mounted drives; underlying blob and drive letter

Snapshot DriveCreate snapshot copy of the drive

VM

How Windows Azure Drives Works Drive is a formatted page blob stored in blob

service

Mount obtains a blob lease

Mount specifies amount of local storage for cache

NTFS flushed/unbuffered writes commit to blob store before returning to app

NTFS reads can be served from local cache or from blob store (cache miss)

DemoBlob

OS

Application

Drive X:

Windows Azure Blob Service

Local Cache

Failover with Drives

Must issue NTFS Flush command

to persist dataUse System.IO.Stream.Flush()

Read/Write Drives protected with leases1 Minute lease expiryMaintained by Windows Azure OS DriverUnmount on RoleEntryPoint.OnStop

On failureLease will timeout after 1 minuteRe-mount drive on new instance

Tables

Table Storage Concepts

EntityTableAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

Table Details

InsertUpdate Merge – Partial update

Replace – Update entire entity

UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction

Create, Query, DeleteTables can have metadata

Not an RDBMS! Table

Entities

Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order

Timestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

Querying

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

?$filter=Last eq ‘Wegner’

Purpose of the PartitionKeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

BikesQuick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Rafts 14ft Super Tourer … 1999

… … … …

SkisFabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

BikesQuick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

Rafts 14ft Super Tourer … 1999

… … … …

SkisFabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

Partitions and Partition Ranges

Server ATable = Products

Server BTable = Products

[Canoes - MaxKey)

Server ATable = Products

[MinKey - Canoes)