Building Scalable and Available Web Applications with Microsoft Project Code Name "Velocity"Muralidhar KrishnaprasadPrincipal Software ArchitectMicrosoft Corporation

Grocery Web Site

Typical Architecture

…Application /Web Tier

Users

Database Data tier

Application Application Application

Asp.Net app Session cart

hosted in memory

Load balancerSticky routing

Catalog sits in database

And Then…

Gas Prices Go up!!

Dow Goes Down!!

Your Web Site is now popular!!

…

Application/ Web Tier

Users

Database Data Tier

Application Application Application

Web site’s too

slow!!

Servers are crashing

Database is hot!!

Where did my

shopping cart go?

Velocity

There is hope..

Outline

The problem!Velocity – the solution!

What is it?

Why velocity?

What stage is it in?

Understanding data usageOverview and programming velocityDeployment and maintenanceVelocity futuresConclusions

Unified cache view

What is "Velocity"?An explicit, distributed, in-memory application cache for all kinds of data

(CLR objects, rows, XML, Binary data etc.)Fuse "memory" across machines into a unified cache

Velocity callers can be spread

across machines or processes

Callers access the cache as if it

was a large single cache

Cache layer distributes data

across the various cache

nodes

Where Does it Fit?

… Application/Web tier

Users

Database Cloud Data tier

Application Application Application

Velocity access layer

Velocity access layer

Velocity access layer

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Cache tier

Outline

The Problem!Velocity – The solution!

What is it?

Why Velocity?

What stage is it in?

Understanding Data UsageOverview and Programming VelocityDeployment and MaintenanceVelocity FuturesConclusions

Why Velocity?

4. High availabilityProtect from web and cache

server failure

1. Share data across applications

No more sticky routing3. Scale out by adding more boxes2. Performance (quad core box)

Operation Throughput

Latency

Read 2k 30,000 / sec

3 – 4 ms

Write 2k 18,000 / sec

3 ms

Operation Servers

Throughput

Read 2k 1 30,000 / sec

Read 2k 2 58, 600 / sec

Read 2k 3 85, 500 / sec

Velocity

And the Best Yet...

Do all this at low cost!Use inexpensive commodity hardwareV1 of velocity is currently planned to

be a Free out of band release on MSDN *

 * Requires windows server.  Final SKUs/product positioning hasn’t been figured out and some capabilities/number of nodes may require additional licenses

Outline

The Problem!Velocity – The solution!

What is it?

Why Velocity?

What stage is it in?

Understanding Data UsageOverview and Programming VelocityDeployment and MaintenanceVelocity FuturesConclusions

When Can I Get Velocity?

CTP1• Teched

CTP2• PDC

2008

CTP3• End of

Mar ‘09

RTM• Mid

2009

Sector DescriptionMedia Cache all media files and metadata; big media provider in US

Content Big content provider in Europe

Content MSDN forums is live on CTP2

Healthcare One of the large health care ISV in the world; US

Finance Big brokerage firm/banking; high performance computation

Gaming Notifications for presence;

Enterprise Largest mainframe migration to Windows

Manufacturing Manufacturing process monitoring

Insurance Session data for insurance purchase

Are Customers Using It?

Outline

The ProblemVelocity – The SolutionUnderstanding Data UsageOverview and Programming VelocityDeployment and MaintenanceVelocity FuturesConclusions

Data to be Cached...

Figure out which data to cacheExample

CatalogSession/shopping cartInventory

Cached data across sectors can be classified into

ReferenceActivityResource

Each of which has different characteristic

Caching Reference Data

Web tier

Clients

Local cache (in Proc)

Velocity

Data tier User profile

threads…

Scenario: MSDN forums

Caching Activity-Oriented Data

Data tier

Application tier

Distributed cache

Integrationhub

Clients

Web tier

Vendor services

Pricing

Vendorsources

Order, invoice, payment

Aggregated vendor

catalogs

Scenario: Insurance

Caching Resource-Oriented Data

Scenario: Flight Inventory and Pricing

Application tier

American

Distributed cache

Flight Itinerary

Flight segment

Flight price

Air

lines

Seat inventory

Booking service

Catalog

Shopping

cart

Data Classification

Inventory

Understand Data Requirements

Classify your data Reference or activity or resource data

Examine requirementsPerformance

Throughput and latency

ConsistencyHow much staleness can be tolerated

EvictionIs the data evictable?

SecurityIs there any secure data?

AvailabilityShould data survive node or cluster failures?

OutlineThe problem

Velocity – the solution

Understanding data usage

Overview and programming velocityDeployment and access API

Demo – integrate velocity into grocery site

How does velocity scale?

Demo – scale

Performance and availability

Demo – availability

Few more features

Deployment and maintenance

Velocity futures

Conclusions

Deployment

Velo

city

Serv

ice

Cache Tier

Configuration Store (Can be database, File share, etc.)Stores Global Cache PoliciesStores Current Partitioning Information

Velo

city

Serv

ice

Velo

city

Serv

ice

Install Windows Cache Service

Copy Client DLLs and add app.config

…

Users

Application/ Web Tier

Application

Velocity Client

Application

Velocity Client

Application

Velocity Client

<configuration> <configSections> <section name="dcacheClient" type=“Microsoft.Data.Caching.DCacheClientSection,CacheBaseLibrary" allowLocation="true" allowDefinition="Everywhere"/> </configSections>

<dcacheClient deployment="routing"> <localCache isEnabled="false" sync="TTLBased“ tlValue="300”/> <hosts> <host name="BL1CDB8083714" cachePort="22233" cacheHostName="DistributedCacheService"/> </hosts> </dcacheClient></configuration>

Velocity API

// Create instance of cachefactory (reads appconfig)DataCacheFactory fac = new DataCacheFactory();

// Get a named cache from the factoryDataCache catalog = fac.GetCache("catalogcache");

// Simple Get/Putcatalog.Put("toy-101", new Toy("Puzzle", .,.));

// From the same or a different clientToy toyObj = (Toy)catalog.Get("toy-101");

// Region based Get/Putcatalog.CreateRegion("toyRegion");

// Both toy and toyparts are put in the same region catalog.Put("toy-101", new Toy( .,.), “toyRegion”);Catalog.Put("toypart-100", new ToyParts(…), “toyRegion”);

Toy toyObj = (Toy)catalog.Get("toy-101“,"toyRegion");

Velocity Logical Hierarchy

HostPhysical processes hosting velocity instance

Named CachesCan span across machinesDefined in the configuration file

Cache itemKey, payload (object ), tags, TTL, timestamps, version

RegionsPhysically co-located container of cache etemsMay be implicit or explicitly created

Regions Region A

Key Payload Tags Key Payload Tags 121 xxxx “Toy” “Child”

123 yyyy “Toy” “Chair”..

Machine -> Cache Host -> Named Caches -> Regions -> Cache Items -> Objects

Velocity serviceNamed cache : product catalog

Named cache : electronics inventory

Velocity service

Velocity service

Velocity service

Access APIsTagging items

Hot or discount items in catalogUse tagging

List all items/hot items – enumerationUse enumeration

// Add TagsTag hotItem = new Tag("hotItem");Tag discItem = new Tag("discountItem");

catalog.Put("toyRegion", "toy-101", new Toy("Puzzle"), new Tag[]{hotItem});catalog.Put("toyRegion", "toy-102", new Toy("Bridge"), new Tag[]{hotItem,discItem});

// From the same or a different clientList<KeyValuePair<string, object>> toys = catalog.GetAnyMatchingTag("toyRegion", hotItem);

Usage Pattern Cache aside

// Read from CacheToy toyObj = (Toy)catalog.Get("toy-101");

Application

Velo

city

Serv

ice

// If Not present in the cacheif (toyObj == null){ // Read from backend.. toyObj = ReadFromDatabase(); // Populate Cache catalog.Put("toy-101", toyObj);

return toyObj;}

OutlineIntroduction

Do I need velocity?

Understanding data usage

Overview and programming velocityDeployment and access API

Demo – integrate velocity into grocery site

How does velocity scale?

Demo – scale

Performance and availability

Demo – availability

Few more features

Deployment and maintenance

Velocity futures

Conclusions

Grocery Web SitePlugging velocity

ASP.Net Pages Catalog.aspx product.aspx

default.aspx Business LayerCatalogService

Data Access LayerICatalogProvider SQLProvider

CacheLayerICatalogCache VelocityCacheProvid

er

OutlineThe problem

Velocity – the solution

Understanding data usage

Overview and programming velocityDeployment & access API

Demo – integrate velocity into grocery site

How does velocity scale?

Demo – scale

Performance & availability

Demo – availability

Few more features

Deployment and maintenance

Velocity futures

Conclusions

Data DistributionPartitioned cache

… Application/ Web tier

Cache tier

Velo

city

serv

ice

Velo

city

serv

ice

Users

ApplicationVelocity client

local cacheVelo

city

serv

ice

ApplicationVelocity client

local cache

ApplicationVelocity client

local cache

A C EDB G IHF

How Do We Partition Keys?

Region Key

"Cust1"

"Cust2"

"Cust33"

"ToyRegion"

"Toy101"

"ToyRegion"

"Toy102"

"BoxRegion"

"Box101"

Partition(Range of Ids)

0 – 1000

1001 - 2000

…

…

..

xxx - Maxint

Region(Name)

Default Region 1

Default Region 2

…

Default Region 256

ToyRegion

BoxRegion

Velo

city

Serv

ice

Velo

city

Serv

ice

Velo

city

Serv

ice

Region Name Hashed into Region Id

Keys Bucketized into Regions

ID Ranges mapped to Nodes

Application

Cache2Cache1

Primary forK2,v2

K2, v2

Primary for K1,V1

K1, v1

Cache3

Primary for K3,V3

K3, v3

Velocity client2

Partitioned CacheHow it works

Get(k2)

Routing table

(K2, v2)

Velocity client1

Routing table

Put

Using the routing table client routes The PUT to cache2 (primary) node

Routing table

Routing table

Routing table

K2

, v2

Operations queue for notifications, to bring up a

new secondary, etc.

Partitioning Increases Scale and Performance

Scale on data sizeMore machines => more memory to cache

Scale on cache throughput Cache throughput: Operations/sec from entire

cache

More machines => keys distributed across more

machines => better throughput

PerformanceHigh performance by scaling out data and

processing

Increased memory

Increased processing

Workload distributed across multiple cache nodes

OutlineThe problem

Velocity – the solution

Understanding data usage

Overview and programming velocityDeployment and access API

Demo – integrate velocity into grocery site

How does velocity scale?

Demo – scale

Performance and availability

Demo – availability

Few more features

Deployment and maintenance

Velocity futures

Conclusions

Demo Layout

Load test controller

…

Server 1 Server 2 Server 3

Cache client1 Cache client 2 Cache client n

Load controller spawns test

clients

Client machines run

test

Cache service cluster

Partitioned cache

Demo Output

Single ServerThroughput Increases with Increasing Load

Until Server Saturation

Load

Throughput

Latency

Server 2 AddedThroughput Increases

Latency DecreasesUntil Server Saturation

Server 3 AddedThroughput Increases

Latency Decreases

OutlineThe problem

Velocity – the solution

Understanding data usage

Overview and programming velocityDeployment and access API

Demo – integrate velocity into grocery site

How does velocity scale?

Demo – scale

Performance and availability

Demo – availability

Few more features

Deployment and maintenance

Velocity futures

Conclusions

Velocity client

Local cache

Local Cache

Local cache can help speed up access on clientsUses notification mechanism to refresh the cache on cache item changesPut(K2, V2)

Routing table

Cache2Cache1

Primary for K2,V2

K2, v2

Primary for K1,V1

K1, v1

Cache3

Primary for K3,V3

K3, v3

Velocity client

Local cacheRouting table

K2, V2

Get(K2)

Get(K2)

Application(K2, V2)

Cache2Cache1 Cache3

Primary for (K2,V2)

High Availability

Get(K2)

Primary for (K1,V1) Primary for (K3,V3)

K3, v3

Velocity client1

Routing table

K2

, v2

Using the routing table client routes The PUT to cache2 (primary) node

Put• Queues the PUT operation • puts locally • propagates operation to secondary•Returns control

Secondary for (K2,V2), (K3,V3)

K2, v2

K1, v1

K3, v3

Secondary for (K1,V1), (K3,V3)

K3, v3 K1, v1

Secondary for (K1,V1), (K2,V2)

K1, v1

Velocity client

Routing table

K2, v2

Replication agent

Cache4

Primary for (K4,V4)

K4, V4

Secondary for

K1, V1

K3, V3

Partition Manager

Global Partition Map

Cache2Cache1 Cache3

Primary for (K2,V2)

Failover

Primary for (K3,V3)

K3, V3

Routing Table

Secondary for

K2, V2

K1, V1

K3, V3

Secondary for

K3, V3

K1, V1

Secondary for

K4, V4

K2, V2

K2, V2

Detects Cache 2 failure. Notifies PM (on Cache4)

Replication Agent

Reconfiguration

Agent

Local Partition Map

Picks Cache1 as the primary for (K2,V2). Picks Cache 3 as new secondary

PM analyzes the info on secondaries of all primary partitions of Cache2 to elect the primaries.

Cache1 creates new secondary on Cache2 Cache1 initiates

reconfiguration.After reconfig, Cache1 is primary for (K1, V1) and (K2, V2)

Session Cart Integration With ASP.Net

…Session state

stored in velocity

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Load balance requestsNo more sticky routing

ApplicationVelocity

sessionstoreprovider

ApplicationVelocity

sessionstoreprovider

ApplicationVelocity

sessionstoreprovider

Velo

city

se

rvic

e

Scale your session storedynamically

Highly available

Drop in velocity sessionstoreprovider

<sessionState mode="Custom" customProvider="SessionStoreProvider"><providers> <add name="SessionStoreProvider" type="System.Data.Caching.SessionStoreProvider, ClientLibrary" cacheName="<YourNamedCache>"/></providers></sessionState>

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocity

Deployment and access APIDemo – integrate velocity into grocery siteHow does velocity scale?Demo – scalePerformance and availabilityDemo – availabilityFew more features

Deployment and maintenanceVelocity futuresConclusions

Availability Demo

Cache clustervelocity with

ASP.Net application

Northwind

Application

Asp.Net

sessionCache

VELOCITY

Velocity sessionStore provider

Northwind

Application

Asp.Net

sessionCache

VELOCITY

Velocity sessionStore provider

VELOCITY

Partitioned cache

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocity

Deployment and access APIDemo – integrate velocity into grocery siteHow does velocity scale?Demo – scalePerformance and availabilityDemo – availabilityFew more features

Deployment and maintenanceVelocity futuresConclusions

Locking

Resource contentionE.g., Inventory or session

Prevent multiple application from loading the same page

Startup Optimistic

Use when less contentionPessimistic

Optimistic Version Based Updates

GetCacheItem returns a version objectEvery update to an object internally increments it's versionSupply the version obtained along with the Put/RemovePut/Remove will succeed only if the passed in version matches

the version in the cache Version Based Update

Time Client1 Client2 (Different Thread or process)

T0 CacheItem item = catalog.GetCacheItem(“PlayerRegion”, ”Zune”);

CacheItem item = catalog.GetCacheItem(“PlayerRegion”, ”Zune”);

T1 ((ZuneObject)item.Object).inventory --;

((ZuneObject)item.Object).inventory--;

T2 catalog.Put(“PlayerRegion”, “Zune”, item.Object, item.Version);

T3 catalog.Put(“PlayerRegion”, “Zune”, item.Object, item.Version);// Version mismatch// Client must retry again

Two clients access the same item

Both update the item

Second Client gets in first; put succeeds because item version matches;

atomically increments the version

First client tries put;Fails because the versions

don’t match

K1

Pessimistic Locking

GetAndLock – Get the object and take a lockOther clients calling GetAndLock will fail (not block)Lock Expiry time expires lockRegular Get does not blockRegular Put will override the lock

PutAndUnlock – Put the new object and unlock existing lockUnLock – explicitly unlocks the object given the lock handle

Client1: GetAndLock ("k1")

Client2: GetAndLock ("k1")

Client3: Get ("k1")

Regular Get succeeds

GetAndLock gets lock handle

Other GetAndLock on same item fails

Notifications

Scalable Cache Event notification systemUses

Local Resource Synchronization – Local CachePopulate expired items

Subscribe to expiry notifications

Copy to backend databasesClients forward changes to databases asychronously

Forward changes to geo-clustersGateway client forwards to other clusters

Cache Event NotificationCalls delegate on client when changes happenNotifications at object, region or cache levelNotification misses are also sent

Cache.RegisterCacheLevelCallback( int filter, CacheChangeCallback delegate);

Cache.RegisterRegionLevelCallback( String region, int filter, CacheChangeCallback delegate);

Cache.RegisterKeyLevelCallback( String region, String key, int filter, CacheChangeCallback delegate);

public delegate void CacheChangeCallback( String CacheName, String regionName, String key, CacheItemVersion version, CacheEventType event, NotificationDescriptor nd ); 

Application

Cache2Cache1

Primary for K2,V2

K2, V2

Primary for K1,V1

K1, V1

Cache3

Primary for K3,V3

K3, V3

Scalable Notifications

Velocity client1

Routing table

Routing table Routing tableRouting table

Register notification for

key "a", "b"

Map keys to partition

ranges

Poll required

nodes

Nodes return list of changes

LSN order

Partition: P1

Last LSN: 20

Partition: P2

Last LSN: 21

Call DelegateStore Last LSN

Change Log1 Add K22 Del

K323 Del

K43

Change Log21 Add K322 Del K423 Upd K9

Change Log33 Add K134 Del

K2235 Upd

K93

Change Log1 Add K22 Del

K323 Del

K43

Change Log1 Add K22 Del

K323 Del

K43

Change Log33 Add K134 Del

K2235 Upd

K93

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocityDeployment and maintenanceVelocity futuresConclusions

Eviction

Expiry only eviction which Evicts expired items alonePeriodicPer partition

Hard-eviction (Data > Allocated Cache Size)Evicts expired items + non-expired items (in LRU order)   Per requestCan be turned off

Memory pressure based evictionA thread for detecting memory pressure (polling per second)Avoids pagingTriggers hard-eviction (mentioned above) at 85% system memory usage and asks for releasing 5% of system memory

Security

No cache securityUse firewalls, ipsec, vlans to protect cache

Domain based security optionDomain account based AuthenticationAuthorize at service level Authorize at named cache levelOnly authorized servers can join the cluster

Transport level securityUse Standard WCF security options E.g., encryption, signing

Administration and Monitoring

InstallationMSI to install on each boxSilent/remote install

Logging Provider model; default ETW, support for file logs

AdministrationPowershell Commandlets

Start and stop cluster or service, configure named caches ..Monitor Cache statistics

Perfmon integration

Size Your Servers

Number of cache servers depend on Amount of data to cacheThroughput requiredCapacity of the cache server machines

CTP3 saturation pointsLess than 6k payload CPU saturates fasterMore than 6k payload network saturates faster

Assume linear scale for throughputTry it out – you can always add machines!!

RecapBefore using Velocity

Understand type of data to cacheReference -> local cachingActivity -> HA optionResource -> HA + locking

Required characteristicsPerformance and availabilityEviction Security

Size number of servers Data sizeThroughput required

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocityDeployment and maintenanceVelocity futures

CloudAsp.Net, queries and persistanceGrid compute

Conclusions

Application Growth

You are a popular grocer!!Lots of users!

Irregular demandsPeaks during weekends or holidays

SolutionMove to the Cloud!

Move data to the CloudAzure Storage, Microsoft SQL Services, etc.

Move the application to the CloudAzure compute

Data in the Cloud

… Application/ web tier

Users

Application Application Application

Velocity access layer

Velocity access layer

Velocity access layer

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Cache tier

Storage/Microsoft SDS

App in the Cloud

… Applicationroles

Users

Application Application Application

Velocity access layer

Velocity access layer

Velocity access layer

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Velo

city

se

rvic

e

Velocity roles

Storage/Microsoft SDS

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocityDeployment and maintenanceVelocity futures

CloudAsp.Net, queries and persistanceGrid compute

Conclusions

Better Integration with ASP.Net

ASP.Net extends cache surface .Net 4.0Integrate with the application cache

System.Caching namespaceVelocity will be a provider for this namespace

Granular session updatesIPartialSessionState lists keys changed

Extensible output cache providerOutput caching not limited to single node

PersistenceCache through

Callback for read-through, write-behind Specified at named cache levelRead-through

Called when item not present in cacheCallback returns the object/serialized bytes

Write-behindWrites to cache are queued Callback called asynchronously in batchesRe-tries upon failure

Bulk access APIs

Queriable Cache

Cache2Cache1

Primary regionsPrimary regions

Toy1, 500

Cache3

Primary regions

Toy2, 350 Toy3, 400

Cache API

Local cache

Velocity client

Dispatch manager

Federated query processor

Object manager

In-memory Data Manager

Query processor

Object manager

In-memory Data Manager

Query processor

Object manager

In-memory Data Manager

Query processor

from toy in catalog<Toy>() where toy.ToyPrice > 300 select toy;

ToyRegionToy4,

100

from toy in catalog<Toy>()where toy.ToyPrice > 300 select toy;

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocityDeployment and maintenanceVelocity futures

CloudAsp.Net, queries and persistanceGrid compute

Conclusions

Computation and CachingVelocity-HPC

ScratchVelocityIntermediate

Store

RollupOperation

Final

Resu

lts

Keys

SplitMethod

Mark

et

Data

Central Market Data Store(~1 TB Tick

Data)

Final Results Store

Job Input

ScratchCalculationOperation

Velocity Data Cache

Market Data

HPC job framework

ScratchVelocityIntermediate

Store

RollupOperation

Final

Resu

lts

Final Results Store

Computation and CachingVelocity-HPC

Keys

SplitMethod

Central Market Data Store(~1 TB Tick

Data)

Job Input

Scratch

CalculationOperation

Velocity Node

Mark

et

Data

HPC Job Framework

Outline

The problemVelocity – the solutionUnderstanding data usageOverview and programming velocityDeployment and maintenanceVelocity futuresConclusions

Takeaways

Velocity is an in-memory distributed cache for .NETWhat is it? – a distributed in-memory cacheWhy would I need it? – It can help your app scale, perform, and fault tolerant

How to integrate Velocity in to a websiteUnderstand the requirements and data to be cachedWe saw how easy it is to integrate Velocity into a website

Using Velocity can help your app scale, perform, and highly-available

We saw a how Velocity can scale and make app available

Roadmap covers web, enterprise, and cloud scenarios

V1 Expected in Mid-2009Velocity will be there when you go to the grid or cloud

Velocity VisionCache for all your data

Web Scenarios

• Distributed/Global object cache• Low latency access• High scale• Availability

• Cache for reference and activity data

• Scale IIS/ASP.Net applications

Enterpris

e/ HPC Scenarios

• LINQ enabled cache

• Co-locate computation and data• Integrate with

HPC server• Persistence• Heterogeneous client

support

Software + Services

Scenarios

• Application Cache for cloud Storage• Integration

with SSDS, Windows Azure

• More data services• BI, Streaming, Reporting

• REST and SOA access

Q & A

Please Complete an Evaluation FormYour feedback is important!

Evaluation forms can be found on each chairTemp Staff at the back of the room have additional evaluation form copies

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.