Addressing a Globally Distributed Network The Future of Testing at Akamai Polish Network Operators Group 2013 Thomas Higdon – Akamai Technologies [email protected]

Addressing a Globally Distributed NetworkThe Future of Testing at Akamai

Polish Network Operators Group 2013

Thomas Higdon – Akamai Technologies

[email protected]

©2013 AKAMAI | FASTER FORWARDTM

A little about me

• Software engineer at Akamai in the platform infrastructure team

• We develop the software that goes on every Akamai server out there (well over 100,000).


Some statistics

Accelerating Daily Traffic of:• 10+ Tbps • 25+ million hits per second• 1.5+ trillion deliveries/day• 50+ petabytes/day• 10+ million concurrent streams

15-30%+ of Web Traffic

Delivering 250,000+ Domains• 5,000+ Customers • 9 of the top 10 Banks

• 9 of the top 10 Social

Media Sites• All top 30 M&E Companies

• All top 60 eCommerce Sites

A Global Platform:• 132,000+ Servers • 86 Countries

• 650+ Cities

• 1,150+ Networks

• 2,200+ Locations


A little about me

• I’m addressing the problem of how to get new platform software out there faster, and with a smaller likelihood of disruption

• We’re using…



Akamai in a nutshell

• Akamai is a globally-distributed cloud of physical servers

• Machines at Akamai are grouped into functional units called networks.

• Edge networks - CDN services, object/file storage, analytics, etc.

• Infrastructure – distributed file transfer, messaging, monitoring, reporting, etc.



• Each network runs a different set of software that depends on the role of the network in the ecosystem.

• Each also runs a common set of platform software• Generally OS software and Akamai-specific

File transferMessaging Monitoring

Network-specific software

OS



Infra network File transferMonitoringMessaging

Operating system/kernel

Edge network






Edge network


File transferMonitoringMessaging


Network-specific softwareNetwork-specific software


File transferMonitoringMessaging

Operating system/kernelFile transferMonitoringMessaging

Operating system/kernelFile transferMonitoringMessaging






Edge network


Edge network


Edge network




• State of a network• Installed software and configuration• Dynamic configuration/metadata• Persistent (across installation) filesystem state• Applied manual changes

• Each network tends to have built-up state necessary for proper functionality

• not well-documented or defined


An Akamai “instance”

• A given unique set of Akamai networks that function together

• Self-contained with limited external dependencies


Akamai “instances”

Production – 105 SQA – 102

SQA – 102

SQA – 102

Dev – 102

Dev – 102

Dev – 102


Commoditizing an Akamai instance

• Instances are labor-intensive to manage

• We’d like the ability to create these as a commodity

• Automatically bootstrapping from scratch is possible, but difficult.


Commoditizing an Akamai instance

• Solution: virtualization• Represent the state of an

instance using virtual machines.

• Save the virtual machines and clone them. SQA – 102 SQA copy– 102

SQA – 102


Cloning Akamai instances

• Static IP configuration• Isolation• traditional NAT • “reverse” NAT• SOCKS proxy• Akamai “authgate”


Accessing an instance

NATSOCKSauthgate

192.168.0.0/16

192.168.0.1

NATSOCKSauthgate

192.168.0.0/16

192.168.0.1Client

172.26.238.10

172.26.238.20


Managing the Akamai side

• Private cloud infrastructure is great for managing VMs and their resources

• We needed another level of management• “Akamai” networks• Akamai instances• Expose specific Akamai machine types and services


An Akamai instance service

• Service-oriented• Language/test harness agnostic• Share common resources


An Akamai instance service

Instance service

Private Cloud Orchestrator

2. “Clone these machines”

3. Creates resources

4. “Here’s your instance!”

5. “Ok, let’s use it.”

Gateway

User1. “Give me an instance”


A well-managed instance

• Leverage expertise from around the company to create an internal “master” instance

• Each additional network gets it closer to a “real” instance.

• Forking?


Addressing existing instances

• Can also address existing instances of physical machines

• Production• Lab instances

Akamai ProductionInstance

Lab test instance

Lab test instance

Lab test instance

User

Instance service


Testing at Akamai

Unit testing

Dev QA

SQA

Checklisting

Alerting

(?)


Unify these stages

Dev QA

SQA

Checklisting

• Use the same code to write tests by using a common interface to refer to an Akamai instance.


Unify these stages

Dev QA

SQA

Checklisting

• Use the same code to write tests by using a common interface to refer to an Akamai instance.


Takeaways

• We’re virtualizing Akamai and providing this as an internal service

• We’re simplifying our testing using the power of the cloud.


Thank you!

Documents

Addressing a Globally Distributed Network The Future of Testing at Akamai Polish Network Operators Group 2013 Thomas Higdon – Akamai Technologies [email protected]