10 Ways to Guarantee Your Microsoft Azure Project Will Fail!

Michael S. CollierPrincipal Cloud Architect

michaelc@aditi.com@MichaelCollierwww.MichaelSCollier.com

15

Central Ohio Cloud Computing User GroupWhat• Local user group for the sharing of ideas and

learning about all things cloud computing. The group will not focus on any one vendor. Over the year there will be meetings to discuss Windows Azure, Amazon AWS, Google App Engine, and several others.

When• Second Monday of each month• 6pm – 8pmWhere• Improving EnterprisesMore Information• www.coccug.org• @coccug

April 14th

Jared Faris

DisclaimerThis is a talk on what NOT to do.

Inspired by real projects . . . . Real people did this stuff!

Names withheld to protect the“innocent”.

Do and you will #FAILand you’re paying me a lot of money to fix your $^!+ . . . . Justice is

ServedImage courtesy of http://cheezburger.com/8106642432

http://3.bp.blogspot.com/-3d9Cp2RCaH8/TalXvZKqqDI/AAAAAAAAFUc/0j4E-2rJ6qw/s1600/angry-baby-6.jpg

More PowerPoint?!?

#10 – Lack of AutomationManual deployments . . . #1 technique

#10 – Lack of AutomationUse PowerShell to script the process

New-AzureService -ServiceName $MyServiceName -Location $MyServiceLocation

New-AzureStorageAccount -StorageAccountName $MyStorageAccountName `-Location $MyStorageLocation -Verbose

New-AzureSqlDatabaseServer –AdministratorLogin $credential.UserName `-AdministratorLoginPassword ` $credential.GetNetworkCredential().Password `-Location $Location

New-AzureDeployment -ServiceName $MyServiceName -Slot Production `-Configuration $MyConfigurationFilePath -Package $MyPackageFilePath

New-AzureQuickVM -Windows -ServiceName $svcName -Name $vmName `-ImageName $image.ImageName -Location $location -AdminUsername $user -Password $pass

#9 – Poor Subscription SegregationSingle sub for Development, QA, and

Production

“Did I do that?”

Quotas

#8 – Failure to Understand Core Features

Endpoint Protection for Windows Azure is ooooold.

You want it, you install & manage it.

FALSEMicrosoft Azure provides anti-virus protection

#8 – Failure to Understand Core FeaturesMicrosoft Azure automatically protects against

DDOSFALSEAzure will protect itself from attacks

What is “busy”?Being successful looks a lot like a DDOS

Be Ready!

#8 – Failure to Understand Core FeaturesSQL Database supports encryption (TDE)

FALSE

Understand the differences

SQL Server on an Azure VM?

!=

#7 – Everything has a Limit

Web Role Worker RoleStorage Queue SQL Database (Microsoft Azure)

100k’s devices1KB every 2 seconds

#7 – Everything has a Limit

Web Role Worker RoleStorage Queue SQL Database (Microsoft Azure)

IIS limitations

20,000 tx per account*

2,000 msgs/sec

150 GB per DB

Transient Faults / Throttling

100k’s devices1KB every 2 seconds

#7 – Everything has a Limit

Worker RoleOWIN/Katana

ASP.NET Web API

Worker Role

Storage Account

SQL Database (Microsoft Azure)

Batch messagesRetry

Table Storage

N

#6 – Infrastructure PlanningManaging infrastructure in cloud is like on-

premises

Virtual Network

CIDR /27 = 32 hosts

X10.0.0.4 10.0.0.1

210.0.0.2

010.0.0.2

810.0.0.3

6

#6 – Infrastructure PlanningManaging infrastructure in cloud is like on-

premises

Virtual Network

10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7 10.0.0.8

AppSubnet

#6 – Infrastructure PlanningManaging infrastructure in cloud is like on-

premises

Virtual Network

10.0.0.4 10.0.0.5 10.0.0.6 10.0.0.7 10.0.0.8

1. Delete logical VM.2. Create proper

network. 3. Create VMs.

AppSubnet

#5 – Failure to Embrace FailureConnecting to a Microsoft Azure SQL Database

public class OrderRepository{             public OrderRepository()          {                    }

      public void Add(Order order)       {          using (var ctx = new OrderDbContext())          {              ctx.Orders.Add(order);              ctx.SaveChanges();          }       }}

#5 – Failure to Embrace Failurepublic class OrderRepository{          private readonly RetryPolicy policy = new RetryPolicy<SqlDatabaseTransientErrorDetectionStrategy>(RetryStrategy.DefaultExponential);      public OrderRepository()          {              policy.Retrying += (sender, args) =>             {                 // Write log statement             };        }

        public void Add(Order order)         {             policy.ExecuteAction(() =>             {                 using (var ctx = new OrderDbContext())                 {                     ctx.Orders.Add(order);                     ctx.SaveChanges();                 }             });         }}

Retry FrameworksTransient Fault Handling Application Block / “TOPAZ”EF6Windows Azure Storage Client Library

#5 – Failure to Embrace FailureServices will fail

ScenarioSingle email providerUnknown SLARetry strategy – retry until success (infinite)

What could possibly go wrong?Didn’t have a max stop countDidn’t have an alternative providerDidn’t have a way to measure or enforce an SLA . . . . If the provider offered one.

#5 – Failure to Embrace FailureConstrain the RetryInvolve a humanImplement a circuit breaker pattern

More on Circuit Breaker pattern from Pattern’s & Practices - http://mcollier.net/1eCyTDu

#5 – Failure to Embrace Failure“Our site can’t go down. 100% uptime is the

requirement.”

NO CLOUD

FOR YOU!

. . . This is hard, and expensive.

Cost vs. Impact

#4 – Poor or No Scale StrategyWhat is wrong here?

web role web role

WASABi Auto-Scaler worker role

crazy.cloudapp.net

#4 – Poor or No Scale StrategyInclude scaler in a separate cloud service

web role web role

WASABi Auto-Scaler worker role

crazy.cloudapp.net

crazy-scale.cloudapp.net

#4 – Poor or No Scale StrategyInclude scaler in a separate cloud service

#4 – Poor or No Scale Strategy“Azure will auto-scale when we’re busy”

FALSE

InstrumentationLogs

#4 – Poor or No Scale Strategy“Azure will auto-scale when we’re busy”

FALSE

33% hit

25% hitX

X

#3 – No Load or Performance TestingScenario• 1 week before date of production launch. “Am I ready?”• Running of 4 instances of a Medium web role• Anticipated day one user load – 1 million users

web role instances

load balancer

#3 – No Load or Performance TestingScenario• 1 week before date of production launch. “Am I ready?”• Running of 4 instances of a Medium web role• Anticipated day one user load – 1 million users

web role instances

load balancer

#3 – No Load or Performance TestingThe TestReach 40 concurrent users before receiving errors and timeouts10 users PER SERVER!

The CauseNo cachingNo retryUnnecessary hits to storage layerInefficient codeNo tests

These are not “Azure” problems!

#2 – Throwing it Over the WallScenarioSiloed development – Ops team not involved

Ops/Security tasked with ensuring app is supportable

Didn’t follow company standards for security, DR, support process, etc.

Security Guys . . . Cloud? EW EWJimmy Fallon

#1 – Non-existent Instrumentation or TelemetryScenario1 week before date of production launch. “Am I ready?”

Well, we eventually log

any fatal errors, but that’s all.

OH . . .

Logs? Yeah . . .we really don’t have logs.

Let’s run some tests and look at your logs

I guess that’s better than

nothing.

We looked at Azure diagnostic logging but

didn’t see much value in it

#1 – Non-existent Instrumentation or Telemetry

You’re kidding? Right?

#1 – Non-existent Instrumentation or TelemetryInstrumentationGeneration of custom monitoring and debugging information

#1 – Non-existent Instrumentation or TelemetryInstrumentationGeneration of custom monitoring and debugging information

TelemetryProcess of gathering the information collected by instrumentation.

#1 – Non-existent Instrumentation or Telemetry

We Don’t Know What We Don’t Know

Preemptive vs. Reactionary

Recovery &Root Cause Analysis

What Have We Learned?3 Truths of Cloud Computing

HardwareFails

SoftwareHas Bugs

PeopleMake

Mistakes

Microsoft blog post on RCA of leap day bug in 2012

What Have We Learned?

Automation Subscription Management

Understand the Limits

Understand the Platform

Plan the Infrastructure

Scale StrategyLoad &

Performance Testing

Embrace Failure

Work Together

Instrumentation & Telemetry

Thank You!Michael S. CollierPrincipal Cloud Architect

michaelc@aditi.com@MichaelCollierwww.MichaelSCollier.com