AWS re:Invent 2016| GAM401 | Riot Games: Standardizing Application Deployments Using Amazon ECS and...

Adam Rozumek

November 28, 2016

Riot Games: Standardizing

Application Deployments Using

Amazon ECS and Terraform

GAM401

RIOT GAMES | GAM 401Standardizing Application

Deployments Using Amazon ECS

and Terraform

SYSTEMS ENGINEER

ADAM ROZUMEK

INTRODUCTIONSWHO AM I?

AMAZON ECS. TERRAFORM. VIDEO GAMES.

WHAT TO EXPECT

ECS CONSOLIDATION WINS & LESSONS

Adapting existing deployments & infrastructure

TERRAFORM ENCAPSULATION STRATS

Object Oriented Development Operations

MULTI-GAME MODULAR SERVICE DESIGN

A different kind of scaling

WHAT TO EXPECT

7.5MILLION

PEAK CONCURRENT

PLAYERS

100MILLION

MONTHLY ACTIVE

PLAYERS

MORE THAN

27MILLION

DAILY ACTIVE

PLAYERS

MORE THAN MORE THAN

2016 LEAGUE OF LEGENDS STATS

DATA PRODUCTS & SERVICESOUR MISSION

Empower teams at Riot to make timely, data-informed products by maintaining a

scalable and reliable data platform

AWS re:Invent 2015 | (GAM303) Riot Games: Migrating Mountains of Big Data to AWS

Sean Maloney

PROBLEM

PROBLEMSCALING TOTAL OWNERSHIP ON AWS

Total ownership

We want to empower developers to:

• Provision their own infrastructure

• Execute their own deployments

• Monitor their own metrics

Resource attribution

• Who owns these EBS volumes?

• What applications depend on these security groups?

• Can these AMIs be deleted?

Security

• Auditing is important, but it’s reactive

• Operational time sink

Security Monkey AWS Trusted Advisor

Enforcing conventions

• Common syntax for tags

• Simplify reservations

• Standardize networks

AWS re:Invent 2014 | (GAM304) How Riot Games re:Invented Their AWS Model

Jonathan McCaffrey

Marty Chong

Enforcing conventions

• Common syntax for tags

• Simplify reservations

• Standardize networks

AWS re:Invent 2014 | (GAM304) How Riot Games re:Invented Their AWS Model

Jonathan McCaffrey

Marty Chong

Amazon EC2

Container Service Terraform

CONTAINERS

CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS

• Dockerfiles capture application dependencies

• Common use cases have great community support

• Profit from our own engineering community

Embrace the abstraction

• Plan for failure at all levels

• Avoid manual intervention whenever possible

• It’s ephemeral all the way down

Scheduling is hard

We need to:

• Quickly and “fairly” run tasks

• Prevent resource conflicts

• Provide reasonable fault tolerance

We need AWS hardware

• Enable total ownership of the AWS resources backing our containers

• Avoid the security, resource attribution, and convention degradation pitfalls

HISTORY

First iteration (2013)

• Traditional ASGs

• AMIs configured to launch a single Docker container

• Managed by Netflix’s Asgard

• Easy to adopt

• Familiar AWS concepts

• No increased resource utilization

• No elasticity improvements

Second iteration (2013)

• Third party container orchestration

• Production scale had unexpected challenges

CONTAINERS STANDARDIZED APPLICATION UNITS… ON ECS!

Third iteration: Amazon EC2 Container Service

• ECS AMI provides all necessary software

• Designed with AWS integration in mind

• Free!

AMAZON EC2 CONTAINER SERVICEKEY FEATURE TIMELINE

Nov 2014

ECS ANNOUNCEDRe:Invent 2014

Nov 2014 April 2015 Dec 2015 August 2016

ECS GENERALLY

AVAILABLE

ECR & NEW REGIONSECS becomes available in the final

missing region we need (Frankfurt)

for our globally deployed

applications

Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016

ECS ANNOUNCEDre:Invent 2014

ECS GENERALLY

AVAILABLE

applications

SERVICE

SCALINGAutomatic task count

scaling based on

CloudWatch metrics

introduced

ECS GENERALLY

AVAILABLE

applications

SERVICE

scaling based on

CloudWatch metrics

introduced

TASK SPECIFIC

IAM ROLESUnlocked a lot of cluster

sharing potential

ECS ANNOUNCEDre:Invent 2014

ECS GENERALLY

AVAILABLE

applications

SERVICE

scaling based on

CloudWatch metrics

introduced

TASK SPECIFIC

IAM ROLESUnlocked a lot of cluster

sharing potential

APPLICATION

LOAD BALANCERSSeveral key improvements

for ECS

BEYOND ECSINFRASTRUCTURE AS CODE

• At scale, orchestrating infrastructure in a consistent, reproducible way is key

BEYOND ECSINFRASTRUCTURE AS CODE

• At scale, orchestrating infrastructure in a consistent, reproducible way is key

Total ownership

TERRAFORM

TERRAFORMINFRASTRUCTURE AS OBJECT-ORIENTED CODE

• Intelligent & flexible flow control

• Configuration drift managed

• AWS resources documented

• Share common use cases

VPC NATGateway

Route 53Hosted Zone

RouteTables

VPNGateway

VPC InternetGateway

Application Subnets Tools

Instances

Availability

Zone C

Availability

Zone B

Availability

Zone A

Simplified infrastructure definitions

CASE STUDIES

ECS CLUSTERTERRAFORM BUILDING BLOCKS

VPC NATGateway

Route 53Hosted Zone

RouteTables

VPNGateway

VPC InternetGateway

Application Subnets Tools

Instances

Availability

Zone C

Availability

Zone B

Availability

Zone A

ECS Cluster

Auto Scaling Group

Security Group

Instance Instance Instance Instance Instance Instance

Launch Configuration User DataIAM Role CloudWatch

Alarms

Module 1

Module 2

Module 3

MICROSERVICESWITHOUT SERVICE ENDPOINTS

ECS Cluster

Autoscaling Group

Security Group

Instance Instance Instance Instance Instance Instance

Launch Configuration User DataIAM Role CloudWatch

ECS Service Task Definition

CloudWatch Alarms IAM Role

MICROSERVICESWITH SERVICE ENDPOINTS

ECS Cluster

ECS Service Task Definition

CloudWatch Alarms IAM Role

Application Load Balancer

Monitoring

CloudWatch Alarms SNS Topics

Security Group

Listeners

Target GroupsRoute 53

PERSISTENT DATALOSE ECS HOSTS WITHOUT LOSING DATA

ECS Cluster

ECS Service

Application Load Balancer

Monitoring

Security Group

Listeners

Target GroupsRoute 53

Attachment Group

EBS EBS EBS

Elastic Network Interface

Elastic IP

LESSONS LEARNED

LESSONSWHAT WORKED FOR US

• Socialize your templates & take advantage of the community!

• Terraform community modules

• Terraform AWS blog

• Break apart your stacks

ECS Cluster

• Break apart your stacks, but don’t overdo it

ECS Service

SNS Topics

Route 53

IAM Role

Task Definition

CloudWatch Alarms

• Break apart your stacks, but don’t overdo it

• Use modules to reduce code duplication

ECS Cluster ALB

• Use remote state files

• Be liberal with your cluster provisioning

• Don’t risk resource contention in production

• With good orchestration, additional clusters != additional operational overhead

• Tag everything all of the time

• Keep your tags organized in your Terraform templates

• Have top level variables that get applied to every resource

• Create a tag for every dimension that is useful to your business

• Centralize your logs

Amazon

CloudWatch

• Stay up to date with release blogs and application updates

• ECS updates on the AWS blog

• Terraform’s open source GitHub repository changelog

• Capture your AMI generation process

• Profile your memory requirements, monitor for scheduling issues

• Take advantage of flow control in Terraform

• Use the Terraform graph tool to visualize your dependencies

RIOT ENGINEERING BLOGTHESE PROBLEMS AND MORE

https://engineering.riotgames.com/

Riot engineering

How we use data

http://na.leagueoflegends.com/en/tag/insights

JOIN US!

ENJOY NOMS.

DRINK DRINKS.

PLAY GAMES.

Tuesday

November 29th, 2016

6:30 – 10:30PM

Paiza Lounge

The Venetian

36th Floor

Thank you!

Remember to complete

your evaluations!

AWS re:Invent 2016| GAM401 | Riot Games: Standardizing Application Deployments Using Amazon ECS and...

Technology

Terraform at Scale

Terraform Basics - Adfinis SyGroupTerraform Basics Be smart. Think open source. Write, Plan, and Create Infrastructure as Code Agenda Introduction to Terraform Setup HCL My first Terraform

Terraform innovation - Performance 3.0

Terraform: Infrastructure as Code · 2018. 9. 14. · MartinSchütte|Terraform|Kielux’18 } 16/42. terraform graph | dot -Tpdf aws_s3_bucket.importdisk provider.aws aws_sqs_queue.importqueue

Refactoring terraform

Standardizing RPAL AST’s

Jsonnet, terraform & packer

Terraform: Configuration Management for Cloud Services

Standardizing and Evaluating Risk Evaluation and ... fda/published/Standardizing-and... · REPORT: Standardizing and Evaluating Risk Evaluation and Mitigation Strategies (REMS) September

Infrastructure as Code Terraform & Ben Higginbottom · Terraform - The Downside Terraform maintains a state file (terraform.tfstate) that describes what the environment is, therefore

Terraform: What's New in Version 0.7.x? - mschuette.name · Example:UsingStateImport $ terraform import aws_instance.server i-0f83bd96e9ea45fe3 aws_instance.server: Importing from

Infrastructure as Code with Terraform · State Terraform stores state about your managed infrastructure and configuration. Used by Terraform to map real world resources to your configuration

Terraform®Wall Panels Product Introductionbmatglobal.com/wp-content/uploads/2013/12/Terraform-Curtain-Wall... · Product Introduction Terraform® wall Panel products are one of

Terraform - hwsw.hu · •Introduction to Terraform •Demo •Create instance in Oracle Public Cloud ... Terraform Chef Ansible Puppet SaltStack CloudFormation

Terraform + Mesos

Terraform Basics - docs.adfinis.com · Check > terraform Usage: terraform [-version] [-help] [args] The available commands for execution are listed below. The most

Standardizing Sustainability Context

AWS re:Invent 2017 · Are re:Invent passes required to use this space? • The meeting host must be a registered re:Invent pass holder. Meeting participants who do not have a re:Invent

Cool, but… What is the - SUSE · Terraform and Salt to the rescue... Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently Terraform can

DKUUG standardizing