Embed Size (px)
DESCRIPTION
Riot Games is a high-paced dynamic environment with many groups striving to release new content, features, and tools. Riot runs League of Legends, one of the biggest online multiplayer games, and uses AWS to host many complex sites that service millions of players everyday. In this session, Riot Games talks about the evolution of their management practice on AWS over the past two years, some lessons learned the hard way, and where they hope to be in the future. Key topics include: SSO (Single-Sign On) integration with IAM roles High-level AWS architecture (How to make it easy on your organization) VPC design, centralization, and simplification DevOps tooling and automation How and why we use Auto Scaling
Citation preview
How We
re:Invented our AWS ModelJonathan McCaffrey : Software Architect
Marty Chong : Sr. Network Engineer
GAM 304
MARTY CHONGSR. NETWORK ENGINEER
IAAS TEAM
MARTY.CHONG @RIOTGAMES.COM
JONATHAN McCAFFREYSOFTWARE ARCHITECT
MEET YOUR PRESENTERS
WHO IS RIOT?
RIOT GAMES
Developer & publisher of League of Legends
Hosts some of the largest eSports events
What is League of Legends?
2009LAUNCH
TEAMORIENTED
100+CHAMPS
MODERNFANTASY
GAMEPLAY SCREESHOT #2
Where in the World is.....
League of Legends World Championship 2014
LEAGUE OF LEGENDS STATS
7.5MILLIONPEAK CONCURRENT
PLAYERS
STATS RELEASED JANUARY 2014
67MILLIONMONTHLY ACTIVE
PLAYERS
MORE THAN
27MILLIONDAILY ACTIVE
PLAYERS
MORE THAN MORE THAN
ADAPT CONCLUSIONOUR
STORYMOVESTART
INTRODUCTION
What is LeagueOfLegends.com?
League of Legends is a Global Product
Goals for the site technology
SECURITY
SECURITY QUALITY
AGILITYSECURITY QUALITY
Lets look at how we applied these initially…
INTRODUCTION ADAPT CONCLUSIONOUR
STORYMOVE
STARTIN THE DATACENTER
STRUGGLE TO DELIVER
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
INTRODUCTION ADAPT CONCLUSIONOUR
STORYMOVE
STARTIN THE DATACENTER
STRUGGLE TO DELIVER
MONOLITHIC APPPHYSICAL SERVERSNETWORK SECURITYREVISIT GOALS
INTRODUCTION ADAPT CONCLUSIONOUR
STORYMOVE
STARTIN THE DATACENTER
STRUGGLE TO DELIVER
MONOLITHIC APPPHYSICAL SERVERSNETWORK SECURITYREVISIT GOALS
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
CMS
Frontpage
News
Media Gallery
Comments
Account Management
Refer-a-friend
CMS
Frontpage
News
Media Gallery
Comments
Account Management
Refer-a-friend
CMS
Frontpage
News
Media Gallery
Comments
Account Management
Refer-a-friend
INTRODUCTION ADAPT CONCLUSIONOUR
STORYMOVE
STARTIN THE DATACENTER
STRUGGLE TO DELIVER
MONOLITHIC APPPHYSICAL SERVERSNETWORK SECURITYREVISIT GOALS
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
?
?
IN THE CMS
Bolt-On New Apps
?
NEW APP
?
IN THE CMS
?
IN THE CMS
NEW APP
INTRODUCTION ADAPT CONCLUSIONOUR
STORYMOVE
STARTIN THE DATACENTER
STRUGGLE TO DELIVER
MONOLITHIC APPPHYSICAL SERVERSNETWORK SECURITYREVISIT GOALS
GAME SERVICES
ROUTER
DDoS MITIGATION
FIREWALL
LOAD BALANCER
EDGE
NA CMS
NA FORUMS
MySQLSERVER
FRONTEND SERVER
MySQLSERVER
FRONTEND SERVER
DDoS Mitigation
DDoS Mitigation Firewall
DDoS Mitigation Firewall Strict Monitoring
INTRODUCTION ADAPT CONCLUSIONOUR
STORYMOVE
STARTIN THE DATACENTER
STRUGGLE TO DELIVER
MONOLITHIC APPPHYSICAL SERVERSNETWORK SECURITYREVISIT GOALS
SECURITY QUALITY AGILITYREVISITGOALS
Data Center had a lot of security features No CDN
SECURITY QUALITY AGILITYREVISITGOALS
Separate stacks for CMS and Forums CMS did a lot of things Adding a new stack was very hard
SECURITY QUALITY AGILITYREVISITGOALS
Static physical server topology
INTRODUCTION ADAPT CONCLUSIONOUR
STORYSTART
MOVETO AWS
STRUGGLE TO SCALE
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICERDS
FRONTEND SERVER
RDSFRONTEND
SERVER
VPCCONNECTIONS
ELBELB
CDNELB
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
ELBELB
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICERDS
FRONTEND SERVER
RDSFRONTEND
SERVER
VPCCONNECTIONS CDNELB
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
ELBELB
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICECDNRDS
FRONTEND SERVERVPC
CONNECTIONS ELB
RDSFRONTEND
SERVER
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICECDNRDS
FRONTEND SERVERVPC
CONNECTIONS ELB
RDSFRONTEND
SERVER ELBELB
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICERDS
FRONTEND SERVERVPC
CONNECTIONS CDNELB
RDSFRONTEND
SERVER ELBELB
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICERDS
FRONTEND SERVER
RDSFRONTEND
SERVER
VPCCONNECTIONS
DEV TEAMS
ELBELB
CDNELB
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
INTRODUCTION ADAPT CONCLUSIONOUR
STORYSTART
MOVETO AWS
STRUGGLE TO SCALE
STARTING WITH VPCMICRO-SERVICESAUTOMATIONREVISIT GOALS
INTRODUCTION ADAPT CONCLUSIONOUR
STORYSTART
MOVETO AWS
STRUGGLE TO SCALE
STARTING WITH VPCMICRO-SERVICESAUTOMATIONREVISIT GOALS
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICECDNRDS
FRONTEND SERVERVPC
CONNECTIONS
ELBELB
ELB
RDSFRONTEND
SERVER
COMMUNITY CMS
TOURNAMENT CMSPLAYERS
VPC
VPC
VPC
VPC
VPC
VPC 1# of VPNs
VPC
VPC
VPC 2# of VPNs
VPC
VPC
VPC 10# of VPNs
VPC
VPC
VPC 20# of VPNs
VPC
VPC
VPC 40# of VPNs
!!# of VPNs
VPC
VPC
VPC
!!# of VPNs
VPC
VPC
VPC
INTRODUCTION ADAPT CONCLUSIONOUR
STORYSTART
MOVETO AWS
STRUGGLE TO SCALE
STARTING WITH VPCMICRO-SERVICESAUTOMATIONREVISIT GOALS
LoLGAME
SERVICE
LoLGAME
SERVICE
RIOT DC 1
LoLGAME
SERVICERDS
FRONTEND SERVERVPC
CONNECTIONS CDNELB
RDSFRONTEND
SERVER ELBELB
COMMUNITY CMS
TOURNAMENT CMS
COMMENTS SERVICE (JAVA)
NEWS ARTICLE PAGE
COMMENTS UI (JAVASCRIPT)
COMMENTS SERVICE (JAVA)
NEWS ARTICLE PAGE
COMMENTS UI (JAVASCRIPT)
COMMENTS SERVICE (JAVA)
CMS (PHP)
INTRODUCTION ADAPT CONCLUSIONOUR
STORYSTART
MOVETO AWS
STRUGGLE TO SCALE
STARTING WITH VPCMICRO-SERVICESAUTOMATIONREVISIT GOALS
EC2
EC2 INSTANCE
EC2 INSTANCECHEF
EC2 INSTANCECHEF
No AMIs
No AMIs Fragile
No AMIs Fragile Complex
INTRODUCTION ADAPT CONCLUSIONOUR
STORYSTART
MOVETO AWS
STRUGGLE TO SCALE
STARTING WITH VPCMICRO-SERVICESAUTOMATIONREVISIT GOALS
SECURITY QUALITY AGILITYREVISITGOALS
Implemented VPCs with VPNs Struggled to scale them
SECURITY QUALITY AGILITYREVISITGOALS
Take advantage of our CDN Separate services We still had a lot of stuff in the CMS
SECURITY QUALITY AGILITYREVISITGOALS
Automated deploys Automation was fragile
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
SCALING VPCAWS WORKING GROUPGOLDEN IMAGE DEPLOYSSTATIC SITE HOSTINGREVIST GOALS
WITH ALL GROWTH COMES
What brought us agility also brought us the Wild West of Cloud Computing
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
SCALING VPCAWS WORKING GROUPGOLDEN IMAGE DEPLOYSSTATIC SITE HOSTINGREVIST GOALS
PAIN POINTS
INFRASTRUCTURE
PAINORGINIZATIONAL
PAIN
PAIN POINTS
INFRASTRUCTURE
PAIN
VPC REGION 1
VPC REGION 2
AWS
VPC REGION 1
VPC REGION 2
AWSRIOT DC 1
RIOT DC 2
RIOT DC 3
VPC REGION 1
VPC REGION 2
AWS
CDN
RIOT DC 1
RIOT DC 2
RIOT DC 3
!!# of VPNs
VPC
VPC
VPC
VPCCONSOLIDATE
VPC REGION 1
VPC REGION 2
EXTEND
VPN
VPN
MULTI-POINT VPNAMI
AMI
VPC REGION 1
VPC REGION 2
MULTI-POINT VPN
DIRECT CONNECT
AWS DIRECT CONNECT
SHARED MODEL
VPC
PUBLIC
SHARED TOOLS
APPLICATIONS
MANY TEAMS – ONE VPC
VPC
PUBLIC
SHARED TOOLS
TEAM 1
TEAM 2
TEAM 3
VPC PER TEAM
VPC WEB VPC BIG DATA VPC IT
PUBLIC
SHARED TOOLS
APP 1
APP 2
APP 3
PUBLIC
SHARED TOOLS
APP 1
APP 2
APP 3
PUBLIC
SHARED TOOLS
APP 1
APP 2
APP 3
COMMODITY-BASED VPC
VPC 1 VPC 2 VPC AS NEEDED
PUBLIC
SHARED TOOLS
APP 1
APP 2
APP 3
PUBLIC
SHARED TOOLS
APP 1
APP 2
APP 3
PUBLIC
SHARED TOOLS
APP 1
APP 2
APP 3
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
SCALING VPCAWS WORKING GROUPGOLDEN IMAGE DEPLOYSSTATIC SITE HOSTINGREVIST GOALS
PAIN POINTS
ORGINIZATIONAL
PAIN
OK!
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
SCALING VPCAWS WORKING GROUPGOLDEN IMAGE DEPLOYSSTATIC SITE HOSTINGREVIST GOALS
GIT
JENKINSGIT
JENKINS PACKERGIT
JENKINS EC2PACKERGIT
UPLOAD RUN SNAPSHOT
JENKINS EC2PACKERGIT
UPLOAD RUN SNAPSHOT
JENKINS EC2PACKERGIT
UPLOAD RUN SNAPSHOT
JENKINS EC2PACKERGIT
AMI v1.0
JENKINS EC2PACKERGIT
AMI v1.0
AMI v1.1
JENKINS EC2PACKERGIT
AMI v1.0
AMI v1.1
AMI v1.2
ELB
V1 ASG EC2 EC2
ELB
V1 ASG EC2 EC2
V2 ASG EC2 EC2
ELB
V1 ASG EC2 EC2
V2 ASG EC2 EC2
ELB
V2 ASG EC2 EC2
ELB
V1 ASG EC2 EC2
V2 ASG EC2 EC2
ELB
V1 ASG EC2 EC2
ELB
V2 ASG EC2 EC2
ELB
V1 ASG EC2 EC2
V2 ASG EC2 EC2
ELB
V1 ASG EC2 EC2
PACKMULE!
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
SCALING VPCAWS WORKING GROUPGOLDEN IMAGE DEPLOYSSTATIC SITE HOSTINGREVIST GOALS
Markdown File
Markdown File Git Repo Jenkins
Spark
HTML Static Site
JSJS
JS
CSSCSS
MarkdownMarkdown
JSON
Spark
STAGE BUCKET
AMAZON S3
STAGE BUCKET
PROD BUCKET
AMAZON S3
CDN
But its not perfect…
INTRODUCTION CONCLUSIONOUR
STORYMOVESTART
ADAPTTO CHANGES
TACKLE GROWTH
SCALING VPCAWS WORKING GROUPGOLDEN IMAGE DEPLOYSSTATIC SITE HOSTINGREVIST GOALS
SECURITY QUALITY AGILITYREVISITGOALS
CDN for everything Continuing our VPC design
SECURITY QUALITY AGILITYREVISITGOALS
Static sites Full-page caching Auto-scaling Balance the AWS regions we are in
SECURITY QUALITY AGILITYREVISITGOALS
Fully automated deploys Small easy to deploy services Need to improve local and dev workflows
INTRODUCTION ADAPTOUR
STORYMOVESTART
CONCLUSION
COMMUNICATE WITHIN YOUR ORGANIZATION
DOCUMENT
<CODE>
STAY TIGHT WITH YOUR AMAZON ACCOUNT TEAM
LEARN FROM OUR MISTAKESLEARN FROM OTHERS
PLEASE COME AND VISIT OUR BOOTH IN THE EXPO HALL!
THANKS!
QUESTIONS?!MARTY CHONG
SR. NETWORK ENGINEERIAAS TEAM
MARTY.CHONG @RIOTGAMES.COM
JONATHAN McCAFFREYSOFTWARE ARCHITECT
Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals
