Fastly - Inaugural NYC Varnish Meetup

Varnish NYC Meetup | 23 July 2014

Varnish | NYC Meetup23 July 2014


• First major user of Varnish

• Varnish contributor

• Fastly Founder and CEO

• @crucially

• [email protected]

• Please ask questions

Artur Bergman

mailto:[email protected]


• Large distributed Varnish

• Realtime

• Purging

• Logfiles

• Config

• Stats

•Since July 2011

Fastly

Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014

• Reverse proxy

• Sits between you and origin/backends

• Caches content

• Load balancer

• Advanced configuration language

Varnish


FASTLY

VCL


• Simple scripting language

• Compiles into c code

• Shared library dynamically loaded into app

VCL


sub vcl_recv { if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } }

VCL


• “The Vary field value indicates the set of request-header fields that fully determines, while the response is fresh, whether a cache is permitted to use the response to reply to a subsequent request without revalidation. For uncacheable or stale responses, the Vary field value advises the user agent about the criteria that were used to select the representation"

• Second level cache key

• You need to look up the object, get the vary signature, then compute the vary and match it

Vary


• varnishlog -i RxHeader | grep Accept-Encoding: | head -1000 | cut -c 22- | sort | uniq -c | sort -rn

• 399 Accept-Encoding: gzip, deflate• 285 Accept-Encoding: gzip• 262 Accept-Encoding: gzip,deflate,sdch• 42 Accept-Encoding: gzip,deflate• 3 Accept-Encoding: identity• 3 Accept-Encoding: deflate, gzip• 1 Accept-Encoding: identity,gzip• 1 Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3• 1 Accept-Encoding: gzip,deflate,lzma,sdch• 1 Accept-Encoding: gzip, deflate, identity• 1 Accept-Encoding: gzip, deflate, compress• 1 Accept-Encoding:gzip

Vary - Accept Encoding


Varnishlogvarnishlog -i RxHeader

| grep Accept-Encoding:

| head -1000 | cut -c 22- | sort | uniq -c | sort -rn


• So should be 1-4 different versions

• Vary is literal

• 10 different version!• Massive DOS vector

• Linear scan across all Varys



sub vcl_recv { if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } }

VCL


• Bounded number of possible combinations

• Vary: User-Agent, Referer, Accept-Encoding, Cookie • Seen in the wild!

• Be careful with what you let into you vary



sub vcl_recv { set req.http.Vary-Cookie = regsub(req.http.Cookie, “.*(userid=[^;]+).*”, “\1”);}

sub vcl_fetch {set beresp.http.Vary = “Vary-Cookie”;

}

VCL


• Ignore cookies from• ad networks• google analytics• random 3rd party tracking cookies

• Cache-Control: private for cookie content • Now you have hitpass objects for logged in users

Vary - Cookie


• return(pass) in vcl_recv• bypass all caching• bypass request collapsing• bypass hash lookup

• return(pass) in vcl_fetch• Cache that the object is uncacheable• On next request will closely follow vcl_recv return(pass)• Except it does go into vcl_hash

• Useful to avoid bad request collapsing

Hitpass


Inline CC{asm ("\t.LD2496:\n""\t.string \”xxxx.services.fastly.net\"\n""\t.LD2497:\n""\t.string \”yyy.services.fastly.net\"\n""\t.LD2498:\n""\t.string \”zzzz.services.fastly.net\"\n""\t.LD2499:\n"\tcmpb $103, %r14b # g\n""\tleaq .LS5230(%rip), %r8 # xxxxxx -> 5705\n""\tleaq .LD23896(%rip), %r9\n""\tje compare # foo\n""\tcmpb $114, %r14b # r\n""\tje L177\n""\tcmpb $48, %r14b # 0\n""\tleaq .LS1407(%rip), %r8 # zzzzzzzzzz -> 2264\n""\tleaq .LD9381(%rip), %r9\n""\tje compare # foo\n"


Inline C


FASTLY

Purging


• Cache forever• Purge on changes• Hook into CMS/Wordpress/Whatever• Fastly supports it :)

Cache invalidation


• Tag responses• Surrogate-Key: objectid userid templateid

• Purge by tag

• ban obj.http.surrogate-key ~ “objectid”

Surrogate key purging


FASTLY

Running Varnish


• -p thread_pool_stack=524288• Save ram

• -p syslog_cli_traffic=off• Don’t block if syslog hangs

• -p thread_pool_max=32000• Depends on your count of CPU

• -p thread_pools=8• -p thread_pool_min=1000• -p thread_pool_add_delay=0

• Don’t spend time waiting for threads when you need them

Start parameters


• -s file,<dir_or_file>,<size>

• Has concurrency problems• Holds lock while paging (reading) from disk!

• Solution, use many per disk (or rewrite it)• (And obviously use a SSD)

Disk storage


• vm.dirty_background_bytes = 531838208• vm.dirty_expire_centisecs = 3000• vm.dirty_ratio = 50• vm.dirty_writeback_centisecs = 500• vm.lowmem_reserve_ratio = 1 1 32• vm.max_map_count = 524288• vm.min_free_kbytes = 15000000• vm.mmap_min_addr = 65536• vm.swappiness = 0• vm.zone_reclaim_mode = 0

sysctl



Control writeback


• vm.dirty_background_bytes = 531838208• vm.dirty_expire_centisecs = 3000• vm.dirty_ratio = 50• vm.dirty_writeback_centisecs = 500• vm.lowmem_reserve_ratio = 1 1 32• vm.max_map_count = 524288• vm.min_free_kbytes = 15000000 <<< WARNING• vm.mmap_min_addr = 65536• vm.swappiness = 0• vm.zone_reclaim_mode = 0

keep memory free



Don’t fucking swap


• 2 Intel 2690 v2 (Sandy Bridge)• 10 Cores @ 3 Ghz

• 768 GB of Ram• 4x10Gb Ethernet EB82599• 24*500GB SSD

• Intel 3500• Samsung 840 Pro

Hardware


• 16 Servers

• 12 TB Ram• 192 TB of SSD• 640 Gbit/sec

Rack



FASTLY

Questions?


FASTLY

Bonus!


FASTLY

Thank you.

Technology

Fastly - Inaugural NYC Varnish Meetup