Leveraging Multi-CDN at Riot Games

Leveraging Multi-CDN

at Riot Games

Kris BeeversFounder

NS1

Ray PanahonTech Ops

Riot Games

Intelligent DNS & Traffic Management

Leveraging Multi-CDN at Riot Games


As you scale, one CDN may not fit all your needs, even within the same use case

Distributed usersCDNs do not perform uniformly well for all geographies / networks

Fault toleranceCDNs are built for resilience, but localized outages still happen

Cost managementDiversification enables you to trade off performance and cost


Make sure you’re introducing CDNs that fit your application, not just your geo/network demands:

• CDNs often specialize by content type (small objects, large objects, video, mobile, ...)

• Using implementation-specific features? Will they translate across CDNs?

• Not just about pushing bits: invalidation, APIs, reporting, etc


DNS lookup is a good time to address multi-CDN:

• Already part of every application

• CDN typically implemented with CNAME "pointers" in DNS• cdn.example.com IN CNAME

example.somecdn.net

• Opportunity for deep control over CDN selection: cdn.example.com -> ALGORITHM(CDN1, CDN2, CDN3) -> CNAME answer


Implementing multi-CDN with DNS

Step 1: selecting CDNs

• What are you trying to optimize? Global delivery? Performance in key markets? Reliability against localized network issues? Cost?

• Ensure you select CDNs suitable for your content

• Look for CDNs with some network independence -- cover your gaps. Don’t want CDNs with same upstreams, same datacenters, etc.



Step 2: burn-in testing

• Don't go full multi-CDN from Day 0 -- big configuration risk

• Canary test: introduce new CDNs and multi-CDN selection with a small % of users

• Modern DNS traffic management platforms support granular weighting for gradual turn-up



Step 3: traffic management

• Which CDN should we use for this user right now?

• Approach depends on app & optimization needs

• Best technologies make building and evolving complex application-specific algorithms easy

• Key aspect of modern multi-CDN: getting real-time data into traffic management platform


Example: Local CDNs in Key Markets

• Basic idea: introduce CDNs with connectivity optimized for specific geographies or ISPs

• Global CDNs: great coverage in US, EU, some of APAC

• Smaller/developing markets more challenging, but can be key for a use case like Riot's -- huge active user bases in South America, Eastern Europe, etc.


Example: Local CDNs in Key Markets

Multi-CDN using a simple algorithm:

Extract GeoIP GEOFENCINGCDN Selection

ConfigurationCDN1South AmericaCDN2TurkeyCDN3AustraliaCDN4Rest of World

cdn.example.com?


Example: Selecting CDN based on response time

• Basic idea: select from among multiple CDNs globally, based on response time between CDN and ISPs/geographies/etc

• Optimizes delivery across multiple networks using real-time RUM data

• Protects against localized outages or network issues in individual CDNs


Example: Selecting CDN based on response time

Extract GeoIP + ASN, detect mobile,

etc.RUM Steering

CDN Selection

ConfigurationCDN 1 CDN 2CDN 3CDN 4

cdn.example.com?

RUM Measurement Engine

CDN 1 CDN 2 CDN 3 CDN 4

Response timemeasurements


Example: selecting CDN based on locality & application-specific performance metrics

• Basic idea: direct traffic in key markets based on a performance metric measured from within the application

• Application specific metrics: e.g., real-world throughput downloading a game patch

• Measure and direct traffic based on metrics that matter most to your application


CDN 3 CDN 4

Example: selecting CDN based on locality & application-specific performance metrics

cdn.example.com?Extract GeoIP + ASN, detect mobile, etc.

ConfigurationCDN1: S America CDN2: TurkeyCDN3: AustraliaCDN4: Global

GEOFENCING RUM Steering

RUM Measurement Engine

CDN Selection

CDN 1 CDN 2

App-specificmeasurements



Patching our games is a huge delivery problem:

• Huge number of global active users• League of Legends: patch/updates

every other Tues/Wed – 350-450Mb to every user across 13 localization regions in 36-48h

• Hot fixes / patches released on the fly• E-sports environments: competitive

play = performance sensitivity


One CDN is not enough:

• CDN network / geo diversity to deliver to global player base• Redundancy & better performance in key markets

• Niche/regional CDNs can be great in super-specific markets• Tariff free updates


One CDN is not enough:

• Other motivations for multi-CDN:• Enabling new products to be multi-CDN ready from the start• Increased reliability of front-facing products• Level playing field by driving price transparency of CDN services


Globally distributed users?24/7? (who isn’t)

Big scale?

You need multi-CDN


DNS is a great way to introduce multi-CDN

• Changing patcher code — long lead times • Minimal impact on patcher / release teams• Granular control of whom we release – targeting highly impacting

changes is huge


Today: 8+ CDNs and counting globally

Constant internal analysis & discussions:• Reviewing strengths & weaknesses of

each CDN partner• Which regions / ISPs does each CDN

serve best


So far: big success in specific tough markets

• Brazil: 60-90% increase in download throughput

• Turkey: similar• Australia: routing ISPs with download

tariffs to specific white-listed CDNs• Empowered local teams to take more

control over regional delivery


Lots of lessons

Shared origin across many CDNs is a challenge (S3, GCP, roll-your-own)


Lots of lessons

Need a normalized approach for monitoring/analyzing perf across CDNs


Lots of lessons

Flexible routing capabilities are key because different strategies work well in different markets


Lots of lessons

Operating across many CDNs puts serious workload on NOC/support staff:• Which CDN is serving bad/stale data?• Control panel / API / feature deltas


What's next?

• Performance-based routing• Local patcher software gathers

real-time data perfect for routing, e.g. real patch download throughput metrics

• Combine with NS1’s Pulsar CDN datasets & ASN/IP/geo specific rules to drive granular routing

• Leveraging same data to gain carrier-level insights for NOC/support

Thank You!

@riotgames

@nsoneinc / @beevek

Technology

Leveraging Multi-CDN at Riot Games