Scaling Azure for Outgoing HTTP RequestsScaling to thousands of requests / sec

Gergely Orosz@GergelyOrosz

The Problem

Implementation

• Windows Server 2012• C#• HttpClient

GET & POST

Implementation

NetworkErrors

>~1000 req / sec

4 DCs,12 VMs each

Network Errors• System.Net.WebException:

The underlying connection was closed: An unexpected error occurred on a send.

• Sympthom of Port Exhaustion• Let’s scale up!

Number of Deployments

VMs per Deployment

Treshold for Errors (req/sec)

4 12 10004 24 10008 12 10008 24 2000

Network Errors After some more research…

Two LimitsPer deployment: ~350 req / secPer VM: ~20 req / sec

Number of Deployments

VMs per Deployment

Treshold for Errors (req/sec)

1 1 171 2 341 100 3502 100 700

What’s Going On?1. The VM Limit of ~17 / sec netstat

lots of TIME_WAIT connections TCP

TCP close sequence Windows default TCP configuration

TIME_WAIT = 4 minutesPorts to use: 1024 – 50004000 / (4 * 60) = 16.66

What’s going on?1. The VM Limit of ~17 / sec

Port exhaustionChange default Windows configurations

Decrease TIME_WAITOr increase number of ports to use

Increase number of portsnetsh int ipv4 set dynamicport tcp start=1025 num=64511New limit: 265 req / sec / VM

What’s going on?2. The Deployment Limit of ~350 req / sec

NATTIME_WAIT in Azure is 180 seconds65K ports65,000 / 180 = 360

Need to add more deployments to scale beyond this limit

Some other findings Choice of the .NET client library

Success rate under heavy loadHttpClient – 78%Shared HttpClient – 98.5%WebRequest – 99.5%

Use WebRequest when possibleOr share HttpClient instances

Some other findings Tuning of IIS

A configuration that worked better than the default onedynamic port allocation: 64KappConcurrencyLimit: 750KqueueLength: 65KminWorkerThreads: 5K, maxWorkerThreads: 10KminIoThreads: 500, maxIoThreads: 1000requestQueueLimit: 750KconnectionTimeout: 3 minute

Increased successful responses by 30%

Summary Know how TCP & NAT works

Increase number of dynamic ports on VMs to scale beyond 17 req / secIncrease number of deployments to scale beyond 350 req / sec

Use WebRequest or shared HttpClient

Tune IIS for performance

You’re now ready to scale

Thank You@GergelyOrosz

Visualtini