Dedicated Game Server Scaling on Azure Kubernetes Service

(or, some thoughts on scaling memory stateful workloads with containers)

Dimitris Gkanatsios

Technical Evangelist, Microsoft Greece/Malta/Cyprus

@dgkanatsios

open-conf – 29/3/2019

Agenda

• Problem definition

• Docker - Kubernetes 101

• Custom Kubernetes controllers

• Using Azure Kubernetes Service to scale Dedicated Game Servers

• Demo

Dedicated Game Server (DGS)DGS: a server, authoritative source of events in a multiplayer video game

It transmits enough data about its internal state to allow itsconnected clients to maintain their own accurate version of the gameworld for display to players.

DGS receives and processes each player's input.

Players must connect to the server with separate client programs in orderto see and interact with the game.

Source: Wikipedia

How do we create a stateful, isolated, multiplayer gameservice that would be as lightweight as possible,scalable on demand, able to terminate gracefully?

Docker / Containers• Think of container as a lightweight VM

• But *it is absolutely not* a VM

• A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings

• Docker Registries (e.g. Docker Hub, Azure Container Registry) host container images

• Users can create custom container images via Dockerfiles(text files with build instructions)

Images source: docker.com

Kubernetes 101 (K8s)• Orchestrator

• Control Plane

• Worker Nodes (Virtual Machines)

• Pod (1 or more tightly-coupled containers deployed together)

• ReplicaSets (manages sets of Pods – scale in/out)

• Deployments (ability to version ReplicaSets)

• Services (expose and load balances traffic to Pods)

Kubernetes solution - components

• Kubernetes cluster (demo on Azure Kubernetes Service - AKS)

• Extend Kubernetes to support DGS

• OSS solution• GitHub

• Go (Golang)

• Could be used for non-gaming memory-stateful workloads

• AKS• Managed Control Plane

• Easy • Deployment

• Scaling

• Upgrading

• Security

• Azure Container Registry integration

How to represent a DGS?• Extend the Kubernetes API - Custom Resource Definitions (CRDs) to the

rescue!

• Entities• DedicatedGameServerCollection (DGSCol)

• Acts like a “parent” to the DedicatedGameServer(s), can scale

• Servers in a collection have some common properties• e.g. same Map, same game Type

• DedicatedGameServer (DGS)• It is a “child” of the DGSCol

• It is the “parent” to the actual game server Pod

DGSCol

DGS DGS DGS

Pod Pod Pod

Dedicated Game Server CRDsapiVersion: apiextensions.k8s.io/v1beta1kind: CustomResourceDefinitionmetadata:name: dedicatedgameservers.azuregaming.com

spec:group: azuregaming.comversion: v1alpha1scope: Namespacednames:kind: DedicatedGameServerplural: dedicatedgameserverssingular: dedicatedgameservershortNames:- dgs

DedicatedGameServerCollection.go

DedicatedGameServer.go

What is a Kubernetes Controller?

Desired State Actual State

• CRDs alone are not enough. We need to manage them via a Controller• Kubernetes is a declarative system• Controllers implement the “reconciliation loop”• Kubernetes will do everything it can to make the actual state match the desired state• Check out kubernetes/sample-controller, operator framework, kubernetes-sigs/kubebuilder

Components of our solution

• Controllers• DedicatedGameServerCollection (DGSCol) controller

• Watches DedicatedGameServerCollections and DedicatedGameServers

• DedicatedGameServer (DGS) controller• Watches DedicatedGameServers and Pods

• DGSAutoScaler controller (optional)

• API Server (different than Kubernetes API Server)

• POST DGS state (string) – called by game server• POST DGS health(string) – called by game server• POST DGS MarkedForDeletion – called by game server• POST DGS ActivePlayers (int32) – called by game server• GET the list of Running Game Servers – called by by MatchMaker/Lobby service

DGS Creation Flow• User submits a request for a

DedicatedGameServerCollection

• DGSCol controller sees a new DGSCol instance, creates NumberOfReplicas DGS

• DGS controller sees the new DGS instance(s), creates respective Pods

• DGS controller also watches Pods’ creation, updates specific DGS status with Pod State• In parallel, each game server runs and

POSTs a “Healthy” game health state to the API server, which will be set to the corresponding DGS instance

• DGSCol controller watches DGS’s creation, updates DGSCol status with Pod State and DGS State

• Pod sends ActivePlayer messages to the API server which propagate to DGS

DGSColcontroller

DGS controller

Project’s API server

DGS Deletion Flow

• Or else, DGSCol shrinkage flow

• API Server is notified that a DGS (or more) is not needed any more (e.g. user-requested scale in or auto-scaling)• DGS controller is setting the DGS(s) “MarkedForDeletion” flag set to true

• DGS controller removes the DGS(s) from the DGSCol

• However, players may still be connected to the DGS(s)

• When ActivePlayers is set to zero (game is over), DGS is deleted• The “child” Pod is deleted as well (via Garbage Collection process)

Example

We will use OpenArena as a

sample gameOpen source, multiplayer real-time,

FPS, based on Quake

We have not modified source in any

way, just “Dockerized” the game

Image from openarena.wikia.com

DEMOCreate and manually scale a Dedicated Game Server Collection

Stuff we did not cover – check the repo!

• Testing• unit testing

• e2e testing with kubernetes-sigs/kind

• Auto-scaling• Pod scaling (easy problem)

• Cluster scaling (hard problem)

Summary

• AKS scaling Project

• https://github.com/dgkanatsios/AzureGameServersScalingKubernetes

• Docs

• Azure Kubernetes Service

• https://azure.microsoft.com/en-us/services/kubernetes-service/

• Microsoft Game Stack• https://developer.microsoft.com/en-us/games

• Me• @dgkanatsios

• digkanat@microsoft.com