Embed Size (px)
DESCRIPTION
Jose has been working during the past 10+ months on a startup project that runs on MVC4 and the Microsoft Azure Platform, making heavy use of Azure tables and blobs for data storage. Come along and hear about how it all works, tips, recommended tools, lessons learned, and pitfalls. It will be full of live demos! Time permitting, he will be sharing his experiences on building a online-service using the Microsoft .NET web stack.
Citation preview
A candidate online service must• Runs “almost” by itself• Could be operated from
anywhere• Moving bits instead of atoms• Cheap to run• Clear revenue model• Be able to build a quick MVP
Inspiration
Architecture
Web
UI
XXXXX
XXXXXXX
XXX
XXX XX
XXXXX
XXXXXXX
XXX
XXX XX
XXXXX
XXXXXXX
XXX
XXX XX
Admin
Editors
Biz
Log
ic
Dat
a A
cces
sA
dapt
ers
End Users
Azure Web Role
Portals
Azure Table Storage
Azure BLOBs
Documents
Static Content
Jobs
Logs
Accounts
Payment Gateway(PayPal)
SMTP Gateway
(SendGrid)
Customer Support(UsersVoice)
Our Manifesto• No external funding!• No need to quit our day jobs• No rush• Commitment/discipline to finish it, BUT• No fear of failure• No over-engineer it – not solving issues we do not have yet
• Every tech decision tries to implement a functionality/solve a problem we currently have
• Being open to learn new things (tech/non-tech)!
Technologies/Tools Involved• MVC 4 – Razor• The 3 portals in 1 Single Azure Web Role• Azure Tables and Blobs for Storage• Bootstrap for layout and Responsive UI• Source Code/Task Lists/Build on Azure TFS• GeoIP using MaxMind• AutoMapper (Model <-> DTOs)• SendGrid for SMTP• UserVoice for HelpDesk/Feedback• Glimpse and MiniProfiler for profiling• PayPal as payment gateway• Google Recaptcha
Why Azure?
• CHEAPER!• Reliable-enough• Integrated to VS and .NET• High performance• New datacenters nearby (AUS)• Catching up quickly!
BTW, What is the cloud?A pay-as-you-go approach to computing where you control the provisioning of computing resources
Applicationbuilding blocks
StorageBig data
Caching
CDN
Database
Identity
Media
Messaging
Networking
Traffic
Cloud service
s
Web sites
Virtual machine
sIaaS
PaaS
PaaS
Execution Models
TablesStructured storage. A table is a set of entities; an entity is
a set of properties.
QueuesReliable storage and delivery of messages for an application.
BlobsSimple named files along with metadata for the file.
DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.
Azure SQLRelational SQL Server engine. Clustered, fully managed. SQL Reporting Support
DB
How do you get charged?• Per Endpoint (PaaS)• Per the size of the instance• Per running instance• Per VM and its size (IaaS)• Per MB of storage• Per storage transaction• Per MB of transfer outbound (inboud:
free)• Pricing calculator:
http://www.windowsazure.com/en-us/pricing/calculator/
Why Azure Tables for Storage?• Cost!
• US$1.75/mo for 25GB vs US$76 with SQL Azure
• $0.01 for 100K transactions (N/A for SQL Azure)
• +80% of the data will be unstructured data (logs)!
• The option of massive-scalability• Performance
Windows Azure Storage AccountUser specified globally unique account name
Can choose geo-location to host storage account:
Azure Tables• Provides Structured Storage
• Massively Scalable Tables• Billions of entities (rows) and TBs of data• Can use thousands of servers as traffic
grows
• Highly Available & Durable
• Data is replicated several times (default)• 2x Same region (GRS)• 3x Same sub-region (LRS)
Scalability Targets for Storage Accounts• Capacity – Up to 200 TBs• Transactions – Up to 20,000
entities/messages/blobs per second• Bandwidth for a Geo
Redundant storage account• Ingress – up to 5 gigabits per second• Egress – up to 10 gigabits per second
• Bandwidth for a Locally Redundant storage account
• Ingress – up to 10 gigabits per second• Egress – up to 15 gigabits per second
Table Storage Concepts
EntityTableStorageAccount
contoso
Name =…Email = …
Name =…EMailAdd=
customers
Photo ID =…Date =…
photos
Photo ID =…Date =…
Entity Properties• Entity can have up to 255 properties
• Up to 1MB per entity• Mandatory Properties for every entity
• PartitionKey & RowKey (only indexed properties)• Uniquely identifies an entity• Defines the sort order
• RowKeys must be unique within the same partition
• Timestamp (Auto)• Optimistic Concurrency
• No fixed schema for other properties• Each property is stored as a <name, typed value> pair• No schema stored for a table
Purpose of the PartitionKey•Entity Locality
• Entities in the same partition will be stored together (Efficient querying and cache locality)
•Entity Group Transactions• Atomic multiple Insert/Update/Delete in
same partition in a single transaction•Table Scalability
• Target throughput – 500 tps/partition, several thousand tps/account
• Automatically load balance partitions• Each partition can be served by a different storage node• Scale to meet the traffic needs of your table
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Rafts 14ft Super Tourer … 1999
… … … …
Skis Fabrikam Back Trackers
… 2009
… … … …
Tents Super Palace … 2008
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe
… 2007
… … … …
Canoes Whitewater … 2009
Canoes Flatwater … 2006
Partitions and Partition Ranges
Server ATable = Products
Server BTable = Products
[Canoes - MaxKey)
Server ATable = Products
[MinKey - Canoes)
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE)
TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe
… 2007
… … … …
Canoes Whitewater … 2009
Canoes Flatwater … 2006
Rafts 14ft Super Tourer … 1999
… … … …
Skis Fabrikam Back Trackers
… 2009
… … … …
Tents Super Palace … 2008
Data Types
No Fixed Schema
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
FAV SPORT
Canoeing
Demo Time!
The Storage Client API
• OData (Atom) protocol with WCF Data Services
• Microsoft.WindowsAzure.StorageClient
• REST• LINQ• PowerShell!
InsertUpdate Merge – Partial update
Replace – Update entire entity
UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction
Create, Query, DeleteUnnormalized dataTable URL: http://<storage account>.table.core.windows.net/<table>
Not an RDBMS! Table
Entities
Single Entity Query
Server A
PartitionKey(Category)
RowKey (Title)
Action Fast & Furious
… 10000 more Action movies
Action The Bourne Ultimatum
… 100000 more Action & Animation movies
Animation Open Season 2
… 100000 more Animation movies
Animation The Ant Bully
Comedy Office Space
… 1000000 more Comedy & SciFi movies
SciFi Star Trek
… 100000 more SciFi & War movies
… 100000 more War movies
War Defiance
Client
Server B
> Where PartitionKey=‘SciFi’ and RowKey = ‘Star Trek’
> Efficient processing> No continuation tokens
28
Request
Result
Client
Server A
Server B
Table Scan Query
Request
PartitionKey(Category)
RowKey(Title)
Rating
Action Fast & Furious 5
… 999 more movies rated > 4
…
… Action and Anim. movies here with rating < 4
…
Animation A Bug’s life 2
… 100 more movies < 4 here
…
Animation The Ant Bully 3
Comedy Are we there yet? 2
… More movies here …
Comedy Office Space 5
… 800000 more movies here
…
Drama A Beautiful Mind 5
… 1200000 more movies here
…
War Defiance 4
Cont.
> Select * from Movies where Rating > 4> Returns Continuation token
> 1000 movies in result set> Partition range boundary
> Serial Processing: Wait for continuation token before proceeding
Request Cont.
Cont.
Request Cont.
Cont.
29
Returns 1000 movies
Partition range boundary hit
Return continuation
Client
Server A
Server B
Make Scans Faster
Request
PartitionKey(Category)
RowKey(Title)
Rating
Action Fast & Furious 5
… More movies here …
Comedy Office Space 5
… More movies here …
Documentary
Planet Earth 4
… More movies here
Drama Seven Pounds 4
Horror Saw 5 3
… More movies here …
Music 8 Mile 2
… More movies here …
SciFi Star Trek 5
… More movies here …
Cont.
> Split “Select * from Movies where Rating > 4” into> Where PartitionKey >= “A” and PartitionKey < “D” and Rating > 4> Where PartitionKey >= “D” and PartitionKey < “I” and Rating > 4> Etc.
> Execute in parallel> Each query handles continuation
Cont.
30
Request
Request
Cont.
Some Patterns..> Ex: Store Account Information and Rental details in
same table..> Maintain same PartitionKey to enforce transactions
> Account ID as PartitionKey> Update total count and Insert new rentals using Entity
Group Transaction> Prefix RowKey with “Kind” code: A = Account, R = Rental
> Row key for account info: [Kind Code]_[AccountId]> Row Key for rental info: [Kind Code]_[Title]
> Rental Properties not set for Account row and vice versaPartitionKey(AccountID)
RowKey(Kind_*)
Kind TotalRentals
Name Address CheckOutOn Title DueOn
… … … … … … … … …
Sally A_Sally Account
8 Sally Field
Ann Arbor, MI
Sally R_Jaws Rental 2009/11/16 Jaws 2009/11/20
Sally R_Taxi Rental 2009/11/16 Taxi 2009/11/20
… … … … … … … … …31
How to Connect to an Azure Table• Get the latest SDK (2.2)• Install the
“WindowsAzure.Storage” nuget• Get Azure Storage Account,
container and keys• Inherit class from TableEntity• Connect to table• Query and display
Demo Time!
Watch Out!
• A reduced set of supported Data types
• Initializing DateTime properties (set them as nullable)
• Retrieving error details• Issuing complex queries• Only supported operators:
• From, Where, Take, First, FirstOrDefault, Select
Demo Time!
Blobs
BlobContainer
Account
http://<account>.blob.core.windows.net/<container>/<blobname>
Pages/ Blocks
contoso
PIC01.JPG
Block/Page
Block/Page
PIC02.JPG
images
VID1.AVIvideos
Blob Storage Concepts
Demo Time!
Links
• Manage.azure.com• http://www.windowsazure.com/
en-us/pricing/calculator/• http://azurestorageexplorer.cod
eplex.com/
