Windows Azure Storage

Anton Boyko

Windows Azure Storage

Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Windows Azure Storage AccountUser specified globally unique account name

North Central USNorthern Europe

Western Europe East Asia

South East Asia

US Europe Asia

Can choose geo-location to host storage account:

South Central US

West US East US

Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes

Can co-locate storage account with compute accountExplicitly or using affinity groups

Accounts have two independent 512 bit shared secret keys

100 TBs per account

New Features

Geo-ReplicationStorage AnalyticsLogs: Provide trace of executed requests for your storage accountsMetrics: Provide summary of key capacity and request statistics for Blobs, Tables, and Queues

Storage Security

Windows Azure Storage provides simple security for calls to storage serviceHTTPS endpointDigitally sign requests for privileged operations

Two 512bit symmetric keys per storage accountCan be regenerated independently

More granular security via Shared Access Signatures

Windows Azure Storage Abstractions

TablesStructured storage. A table is a set of entities; an entity is

a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Blob Storage

Blob Storage Concepts

BlobContainerAccount

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/ Blocks

contoso

PIC01.JPG

Block/Page

Block/Page

PIC02.JPG

images

VID1.AVIvideos

Blob Details

Main Web Service

Operations

PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob

Blob Details

Associate Metadata with Blob

Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)

Metadata is <name, value> pairs, up to 8KB per blob

Either as part of PutBlob or independently

Blob Details

Blob always accessed by

name

Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg

Blob ContainersMultiple Containers per AccountSpecial $root container

Blob ContainerA container holds a set of blobsSet access policies at the container level Associate Metadata with ContainerList the blobs in a containerIncluding Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ?

Blobs ThroughputEffectively in Partition of 1Target of 60MB/s per Blob

Two Types of Blobs Under the Hood

Block BlobTargeted at streaming workloads

Each blob consists of a sequence of blocksEach block is identified by a Block ID

Size limit 200GB per blob

Optimistic Concurrency via Etags

Page BlobTargeted at random read/write workloads

Each blob consists of an array of pages Each page is identified by its offset from the start of the blob

Size limit 1TB per blob

Optimistic or Pessimistic (locking) concurrency via leases

TheBlob.wmv

Uploading a Block Blob

Uploading a large blob

10 GB Movie

Blo

ck I

d 1

Blo

ck I

d 2

Blo

ck I

d 3

Blo

ck I

d N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

THE BLOB

Windows AzureStorage

Page Blob – Random Read/Write

Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytesRandom Access Operations

PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:

[0,512) , [1536,2560)GetBlob[1000, 2048) returns

All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

512

1024

1536

2048

2560

10 G

B A

dd

ress S

pace

Shared Access Signatures

Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted

Two broad approachesAd-hocPolicy based

Content Delivery Network (CDN)

High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growing

Same experience for users no matter how far they are from the geo-location where the storage account is hosted

Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/

Windows Azure CDN URL: http://<id>.vo.msecnd.net/

Custom Domain Name for CDN: http://cdn.contoso.com/

pic1.jpg

Windows Azure CDN

To Enable CDN:Register for CDN via Dev PortalSet container images to public

pic1.jpg

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/

pic1.jpg

404

TTL Content Delivery Network

Windows Azure Blob Service

EdgeLocation

EdgeLocation

EdgeLocation

Drives

Windows Azure DrivesDurable NTFS volume for Windows Azure InstancesUse existing NTFS APIs to access a network attached durable driveUse System.IO from .NET

BenefitsMove existing apps using NTFS more easily to the cloudDurability and survival of data on instance recycle Drives can be up to 1TB

A Windows Azure Drive is an NTFS VHD Page BlobMounts Page Blob over the network as an NTFS driveLocal cache on instance for read operationsAll flushed and unbuffered writes to drive are made durable to the Page Blob

Drive Details

Operations performed via Drive API not REST CallsOperations on DrivesCreateDriveCreates a new NTFS formatted VHD in Blob storage

MountDrive/UnmountDriveMounts a drive into Instance at new drive letter

Unmounts a drive freeing drive letter

Get Mounted DrivesList mounted drives; underlying blob and drive letter

Snapshot DriveCreate snapshot copy of the drive

VM

How Windows Azure Drives Works Drive is a formatted page blob stored in blob

service

Mount obtains a blob lease

Mount specifies amount of local storage for cache

NTFS flushed/unbuffered writes commit to blob store before returning to app

NTFS reads can be served from local cache or from blob store (cache miss)

DemoBlob

OS

Application

Drive X:

Windows Azure Blob Service

Local Cache

Failover with Drives

Must issue NTFS Flush command

to persist dataUse System.IO.Stream.Flush()

Read/Write Drives protected with leases1 Minute lease expiryMaintained by Windows Azure OS DriverUnmount on RoleEntryPoint.OnStop

On failureLease will timeout after 1 minuteRe-mount drive on new instance

Q & A