Hands On Windows Azure Building A Twitter Clone

Session Code: NE.09

Windows Azure: Building a Twitter Clone

Nik Kalyani

DotNetNuke Corporation

About

Nik Kalyani• Co-founder, DotNetNuke Corporation• Technology Entrepreneur, 15 years• Microsoft MVP (2005-2009)

Online • Email: [email protected]• Twitter: @techbubble• Blog: http://www.techbubble.net

Session Objective

Provide insights on how to architect a large-scale, asynchronous application with the same capabilities as Twitter on Azure

while leveraging its non-RDMS capabilities

Agenda

• Azure Architecture

• Storage Concepts

• Micro-blogging Application – Requirements– Architecture– Processing

ServiceBus

AccessControl

Workflow

…

Database

Reporting

Analytics

…

Compute Storage Manage

Identity

Devices

Contacts

…

…

…

Your Applications

Azure Architecture

Azure Application Architecture

Cloud Storage (blob, table, queue)

Web RoleLBn

Worker Role

m

Azure Durable Storage

• Durable, scalable, available store• Simple interface

– REST– ADO.NET Data Services

• Simple abstractions

Blobs Tables

…

Queues

Azure Data Storage Concepts

Account

Container Blobs

Table Entities

Queue Messages

http://<account>.blob.core.windows.net/<container>

http://<account>.table.core.windows.net/<table>

http://<account>.queue.core.windows.net/<queue>

Blob Storage Concepts

BlockBlobContainerAccount

erica

pictures

IMG001.JPG

IMG002.JPG

movies MOV1.AVI

Block 1

Block 2

Block 3

Blobs: Features and Functions

• Store large objects

• Associate metadata with blob– Metadata is <name, value> pairs– Set/Get with or separate from blob data bits

• Standard REST Interface

• Limitations– Block Id - 64 Bytes Per Block– PutBlob - 64 MB Total– Blob MetaData - 8 KB Per Blob– PutBlock / PutBlockList - 50 GB Each– Blocks - 4 MB Each

Table Storage Concepts

EntityTableAccount

erica

users

Name =…

Email = …

Name =…

Email = …

photo index

Photo ID =…

Date =…

Photo ID =…

Date =…

Tables: Features and Functions• Massively scalable, highly available and

durable structured storage– Not RDMS -- no joins, no maintenance of foreign keys

• LINQ and REST interfaces via ADO.Net Data Services– Partition key is exposed to the application

• Limitations– Up to 255 Entity Properties (including PartitionKey,

RowKey, and Timestamp)– 1 MB Max for Combined Size of All Data in an Entity's

Properties

Queue Storage Concepts

MessageQueueAccount

erica

thumbnail jobs

128x128, http://…

256x256, http://…

photo processing

jobs

http://…

http://…

Queues: Features and Functions• An account can create many queues..

– Queue Name is scoped by the account

• A Queue contains messages..– No limit on number of messages stored in a queue– A message is stored for at most a week in a queue

• http://<Account>.queue.core.windows.net/<QueueName>

• Messages..– Message size <= 8 KB– To store larger data, store data in blob/entity storage,

and the blob/entity name in the message

• Relationships

– Any user can “follow” any other user

– A user can “unfollow” another user he/she is “following”

Micro-blogging ApplicationRequirements

• Micro-blogging

– Any user can post a message of up to 140 characters in length

– All message posts are public


• Timeline

– A public timeline of all posted messages is available to anybody for up to 24 hours from the time the message was first posted

– A user timeline of all messages posted by a user is available to anybody forever

– A “following” timeline of all messages posted by all users that a user is following in the prior 24 hours is available to each user


• Micro-blogging

– The number of users will determine volume of messages

– However, common situation is that a small number of users will post majority of the messages

Micro-blogging ApplicationArchitectural Considerations

• Timelines

– The public timeline is transient and will have a high frequency of reads and writes

– A user timeline is permanent and will have a small number of writes and a large number of reads

– A “following” timeline is transient and will have a large number of irregularly timed writes and a small number of reads

Micro-blogging ApplicationArchitectural Considerations

Micro-blogging ApplicationProfile, Messages and Relationships Storage

Table = UserPartitionKey = UsernameRowKey=string.Format("{0:D19}", DateTime.MaxValue.Ticks – DateTime.UtcNow.Ticks);

• Use Table storage sharded by user

• Entities with the same PartitionKey are stored on the same storage node

• Obtaining recent messages, followers or following list require only a single query for each

• Adding or Deleting Follower or Following requires two writes (i.e. one for each user)

Properties

EntityType = “Message”Source = UsernameMessage Text

EntityType=“Follower”Username

EntityType=“Following”Username

EntityType=“Profile”<Any profile property>

Micro-blogging ApplicationFollow

User A Partition

Follower[User C]

User B Partition

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

User A follows User B


User A Partition

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue


User A Partition

Follower[User C]

User B Partition

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

User B follows User A

Follower[User A]

Following[User B]


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Micro-blogging ApplicationMessage Posting

User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

User A posts a message


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Message added to Message Queue

User AMessage


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

After Message Worker Role processes…

Msg (A)

Msg (A)

Msg (A)

User B User C


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Msg (A)

Msg (A)

Msg (A)

After Maintenance Worker Role processes…

Micro-blogging ApplicationMessage Maintenance

User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Msg (A)

Msg (A)

Msg (A)

Msg (D)Msg (D)25 hours

Msg (X)

User A posts a message


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Msg (A)

Msg (A)

Msg (A)


Msg (X)

Message added to Message Queue

User AMessage


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Msg (A)

Msg (A)

Msg (A)


Msg (X)

After Message Worker Role processes…

User B User C

Msg (A)

Msg (A)

Msg (A)


User A Partition

Follower[User B]

Follower[User C]

Following[User B]

User B Partition

Follower[User A]

Following[User A]

Following[User D]

User C Partition

Follower[User Y]

Follower[User Z]

Following[User A]

Following[User X]

Message Queue

Maintenance Queue

Msg (A)

Msg (A)

Msg (A)

Msg (D)

Msg (X)

Msg (A)

Msg (A)

Msg (A)

After Maintenance Worker Role processes…

• Cloud app storage significantly different from RDMS…requires shift in thinking

• Queues make apps efficient … processing is done asynchronously

• Duplication of data is often required and this is OK

• Sharding is an important consideration for the scalability of your cloud application

• Absence of join, sort make application design challenging but with planning this is manageable

Micro-blogging ApplicationTakeaways

Evaluation form

Vul je evaluatieformulier in en maak kans op een van de prachtige prijzen!!

Fill out your evaluation form and win one of the great prizes!!

Session Code: NE.09