Preparing for CDN failure: Why and how

Preparing for CDN Failure:

Why and How

Aaron Peters TurboBytes

@aaronpeters & @ksgyoung

Kyle Young Mobify

Examples of CDN Failure

https://www.flickr.com/photos/phobia/2308371224/

Multi-continent outage for 17 minutes


100% broken in North-America and Europe.

Up and down for days, some ASNs only


Zoomed in on May 4 and May 5 …


OMG!

Japan, all networks: 5 hours of bad

Three days later, it happened again …


#1 eyeballs network in Germany

Single country, single ASN perf degradations are not uncommon in CDN land …


“System Maintenance”


Take-Aways


Expect your CDN to fail on you

They won’t tell you about it

Monitoring CDN Performance

https://www.flickr.com/photos/chasblackman/8502151556/

Synthetic monitoring


Pingdom, cron+curl, ThousandEyes, Interns, etc.



The ProsVery Specific

Known Intervals

Controlled Locations

Lots of Information



The ConsLimited Locations

Datacenters

Hyper-idealized Client Model



Still better than nothing.

RUM for Page Load Time


CDN must serve HTML + resources

Third party content adds noise

Can’t capture CDN failing

Not very useful

RUM with Test Object


Fetch small object from CDN(s) after onload

Beacon timings, or a Fail

Use Nav Timing for best insight

Challenge

get your JS on other sites too, to capture CDN failing

RUM for Page Resources


Resource Timing API

Send assets with TAO header

Onload => beacon timings

Easy, right? Not so fast …

Starting Points


Did the asset come from CDN?

Did it load fast enough?

Was it a good response (200/304)?

Fetched from network?if (total time > 20 ms) { // from network }

DOES NOT WORK !

RT API has many quirks


DNS time and Connect time always zero in IE

No data for 4xx/5xx responses, except in IE

Nothing in RT API until asset fully loaded, except in IE

FF doesn’t tell the truth

How measure CDN perf with RT API


main.css

inline JS, high in HEAD, exec only if window.chrome

setTimeOut(checkRTAPI, 5000);

at onload or when timer ended:

if ( main.css in RT API && connectTime > 3 ) { loaded fine from network } else { meh }

Too slow: Fail

transferSize attribute, FTW!


transferSize = byte size that came over the wire

if ( transferSize != 0 ) { // from network }

Status: no browser is implementing this yet

Background/discussion: https://github.com/w3c/navigation-timing/issues/3

Future

https://github.com/w3c/navigation-timing/issues/3

Take-Aways


Measure Fail Ratio too, not just Speed Use RUM for real-world performance insightAnd Synthetic monitoring for deep visibilityBeware of the many bugs in Res Timing API

Mitigating CDN Failure: Multi-CDN

https://www.flickr.com/photos/metali/294107810/

Selecting your CDNs

Feature & Behaviour Parity

Performance

Costs


Doing Multi-CDN


Perf dataDecision making

TargetingTime-to-Switch

High volume, high qualityMake good sense of the data, quicklyVery granular, very accurateAsap!

Where switch CDNs?


cs109.wac.edgecastcdn.net.

cds.z4b9c4e6.hwcdn.net

//cdn1.mydomain.com/main.css

//cdn2.mydomain.com/main.css

in DNS in HTML

OR

Traffic management in DNS


resolver authoritativeclient

I see the request comes from NL, based on resolver IP address …

… so I’ll handout the CNAME to the CDN configured for NL

CDN BCDN A

Static Geo


Always route to that CDN in that geoEasy: no need to monitor perfBut what if CDN has boo boo?

Dynamic Geo


Always route to best CDN per geoNeeds solid perf data (RUM !)Geo targeting accuracy important

Dynamic Geo + ASN


Holy grail: gives best results

Really needs RUM data, and lots of it

Targeting accuracy even more important

Geo targeting gone wrong


8.8.8.8 authoritativeclient

I see the request comes from MY, based on resolver IP address …

… so I’ll handout the CNAME to the CDN configured for MY

CDN B - best in India !CDN A

EDNS0 to the rescue !


8.8.8.8 authoritativeclient

I see the request comes from IN, based on client IP address /24 …

… so I’ll handout the CNAME to the CDN configured for IN

CDN B - best in India !CDN A

Decision Making


Look at everything, not just ‘Response Time’

Use multiple statistics, not just median

Make your ‘decider’ sensitive to Fail Ratio !

Tuning your logic takes time

Coping with low volume data


Don’t make changes

Make decisions with lower confidence

Have a dynamic targeting granularity

Experiment: do a Pat Meenan !


http://www.slideshare.net/patrickmeenan/service-workers-for-performance

Hi, I’m Pat

http://www.slideshare.net/patrickmeenan/service-workers-for-performance

Example CDN Perf Program

http://bit.ly/1KppCfo


http://bit.ly/1KppCfo




Limitations

Not Practical for Monitoring

Humans are Required

Misses Important Factors (EG SSL)

Hard to Commit to Bandwidth

War Story

Seattle Down


or How we Started Caring About MultiCDN



Yay, Monitoring!


Support Lines - A Tale of Lowered Expectations

IT Crowd - Fremantle MediaBatman - 20th Century Fox


SLA Reminder99.9% 8 hours99.99% 52 minutes99.999% 5 minutes


Best EffortSwitch over to an alternate CDN for the entire service, across the globe, as per the “Holy S#!t Handbook”.


This Isn’t PrettyCold Cache

Backend Thrashing

3 to 4 hours of Intermittent Failure


Lessons LearnedHot Standby

Geographic DNS Control, and Optimization

More Monitoring


Thank you.

@aaronpeters - [email protected]@ksgyoung - [email protected]

Questions?

mailto:[email protected]

mailto:[email protected]

Technology

Preparing for CDN failure: Why and how