Caching with varnish

VARNISH-CACHE *Lessons Learnt from Performance Testing

* https://www.varnish-cache.orgTuesday, 25 October 11

https://www.varnish-cache.org

https://www.varnish-cache.org

WHY? FAST!

• If your throughput is currently measured in the hundreds per second, think of varnish in the thousands per second.

• Theoretical limits in hundreds of thousands per second.*

* http://kristianlyng.wordpress.com/2010/10/23/275k-req/Tuesday, 25 October 11

http://kristianlyng.wordpress.com/2010/10/23/275k-req/

http://kristianlyng.wordpress.com/2010/10/23/275k-req/

GET OUT OF JAIL. FREE.

• Crappy cache control?

• Expensive CDNs? 20¢/GB?

• Not enough hardware?

• Backend struggling?

Tuesday, 25 October 11

IT WON’T

• Do HTTPS... terminate SSL with something else, hopefully your load balancers.


INSTALL THE LATEST

• RHEL / CentOS

• Debian

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpmyum install -y varnish

apt-get install varnish


http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpm




USE 64BIT*

“It’s all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you’re somewhat on your own. However, try to fit it within 2GB. I wouldn’t recommend a cache larger than 1GB, and no more than a few hundred threads… (Why are you on 32bit again?)”

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11

http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/


SYSTEM CONFIG

• The minimum.

• Do not set “tw_reuse” to 1 (sysctl). It will cause problems for NAT based clients.

• Enable keepalives on load balancers.

echo 'fs.file-max=1000000' >> /etc/sysctl.confecho 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.confsysctl -pecho '* soft nofile 1000000' >> /etc/security/limits.confecho '* hard nofile 1000000' >> /etc/security/limits.confecho 'session required pam_limits.so' >> /etc/pam.d/loginulimit -n 1000000ulimit -n -H


THREADS

• Adjust default thread* settings in /etc/sysconfig/varnish

thread_pool_add_delay=2thread_pools = <Number of cpu cores>thread_pool_min = <800/number of cpu cores>thread_pool_max = 4000session_linger = 50sess_workspace = <16k to 5m>

* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/

“The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesn’t matter much. Our defaults are good.”




STORAGE

• Memory is fast, duh.

• Got enough RAM to store the cache?Yes! then use -s mallocUnsure? then use -s file


VCL

• Varnish Configuration Language

• Looks like C and compiled into C but ...~ “has regex tilda operator”Assignments use set keyword.Only has conditional if tests.Has a bunch of variables and some built in functions.

• Defined by backends, probes, ACLs and subroutines which hook into the varnish workflow.


VCL

• Default VCL is defined in/etc/varnish/default.vcl

• Then create your custom VCL in -f /var/custom.vclso varnish can fall through to the default.vclCan also include "/var/other.vcl";

• Keep it simple within the VCL workflow*Do most of the work in vcl_recv and vcl_fetch

* https://www.varnish-cache.org/trac/wiki/VCLExampleDefaultTuesday, 25 October 11

https://www.varnish-cache.org/trac/wiki/VCLExampleDefault

https://www.varnish-cache.org/trac/wiki/VCLExampleDefault

LOOKUP AND MISS

• request received from client

• hash request

• lookup object and miss

• fetch response from backend

• deliver to client

recv

hash

pass

pipe

misshit

fetch

deliver


LOOKUP AND HIT


• hash request

• lookup object and hit


recv

hash

pass

pipe

misshit

fetch

deliver


PASS AND DELIVER


• pass through to backend

• fetch response


recv

hash

pass

pipe

misshit

fetch

deliver


PIPE


• pipe to backend until client or backend closes

recv

hash

pass

pipe

misshit

fetch

deliver


PROBES

• Define common health checks for multiple backends.

• window is how many of the latest polls are probed for health.

• threshold is how many polls must succeed to be healthy.

probe health { .url = "/healthCheck?onFailure=true"; .expected_response = 200; .interval = 15s; .timeout = 5s; .window = 4; .threshold = 4; }


BACKENDS

• Define backend host, port and health checks inline.

• Or named.backend mysite { .host = "www.example.com"; .port = "http"; .probe = "health";}

{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }


DIRECTORS

• Group backends together using different algorithms. Random Client (based on identity) Hash (based on url) Round robin Fallback


YMMV

• I found it better to balance in front of Varnish using a hardware load balancer ...

• Random seemed a little too random using Varnish directors however was probably an artefact of the load generation.


DESIGN DECISIONS

• Share nothing. Easy to scale horizontally.Great if your site is not ‘transactional’.Backend session affinity via LB set cookie.

LB

app

varnishvarnishvarnish

appapp

Set-Cookie=BackendID


DESIGN DECISIONS

• Share everything. Varnish loves RAM, scale vertically.Varnish cache becomes the shared source of truth.

LB

app

varnish

appapp


SHARE NOTHING. FALLBACK.

LB

app

varnishvarnishvarnish

appapp

Set-Cookie=VarnishIDdirector varnishA fallback { { .backend = { .host = "a"; } } { .backend = { .host = "b"; } } { .backend = { .host = "c"; } } }

director varnishB fallback { { .backend = { .host = "b"; } } { .backend = { .host = "a"; } } { .backend = { .host = "c"; } } }

director varnishC fallback { { .backend = { .host = "c"; } } { .backend = { .host = "b"; } } { .backend = { .host = "a"; } } }


TERMINATE SSL.

LB varnish appSet-Cookie=VarnishIDPort=https

:80:443

if (req.http.Port ~ "https") { set req.backend = appa_80;} else { set req.backend = appa_443; }

• Get the LB to set a Port header. Useful when user switches from HTTP to HTTPS so varnish knows which backend port to route to.


VCL TRICKS

• Rewrite headers to/from production.com.au to testdomain.com

• vcl_recv

• vcl_deliver

set req.http.Host = regsuball(req.http.Host, "testdomain\.com", "com\.au");set req.http.Referer = regsuball(req.http.Host, "testdomain\.com", "com\.au");

set resp.http.Host = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");set resp.http.Referer = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com"); set resp.http.Location = regsuball(resp.http.Location, "\.com\.au", "\.testdomain\.com");set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");


VCL TRICKS

• Add some debug headers for hit/miss in firebug in vcl_deliverif (obj.hits > 0) { set resp.http.X-Varnish-Hit = "++"; set resp.http.X-Varnish-Hits = obj.hits; set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");} else { set resp.http.X-Varnish-Hit = "--"; set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");}


VCL TRICKS

• Normalize the Accept-Encoding headers to reduce cache usage in vcl_recv

if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { remove req.http.Accept-Encoding; }}


VCL TRICKS

• Remove Google Analytics cookies in vcl_recv

• Remember, varnish will pass any request with cookies by default. You will need to over-ride this behaviour if your web app uses cookies.

• Allow lookup of requests with basic auth (in test) or cookies in vcl_recv but beware of creating a per-user-session store.

set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

if (req.http.Authorization || req.http.Cookie) { return (lookup); }


VCL TRICKS

• Pass on the client IP to the backend servers using the X-Forwarded-For header in vcl_recv

remove req.http.X-Forwarded-For;set req.http.X-Forwarded-For = client.ip;


VCL TRICKS

• Let varnish keep objects for 2 hours beyond their TTL in vcl_fetch

• Let varnish serve stale objects up to 30 secs old for healthy servers, otherwise 2 hours in vcl_recv

• Can also use saint mode to disable backends for a period of time, otherwise if all backends unavailable, use grace.

set beresp.grace = 2h;

if (! req.backend.healthy) { set req.grace = 2h; } else { set req.grace = 30s; }


VCL TRICKS

• Force a grace on error. Restart keyword can be called any where and will return request all the way back to vcl_recvsub vcl_error { if (req.restarts == 0) { set req.http.X-Serve-Graced = "1"; return(restart); }}


TTL

• Varnish will obey cache control headers from backendCache-Control: s-maxageCache-Control: max-age

• Can also override TTL control in VCL, but best do it from the backendset obj.ttl = x


REDUCING CACHE SIZE

• Normalize / sanitize request headers

• Set Content-Length on backends

• Serve (static) digital assets from VFS cache

• Pipe large objects like videos


MONITORING VARNISH

• Handy command line toolsvarnishstatvarnishlogvarnishtopvarnishadm


VARNISHSTAT

• varnishstatHit rate % (x100)Average per secondAverage/sec since bootSum total

• Filteredvarnishstat -f \ client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail


VARNISHSTAT

• Log all statistics to disk ...

• Then graph as you see fit ...

#!/bin/bashlogdir=/var/log/perf/`hostname`/latestmkdir -p $logdir while truedo varnishstat -x >> "$logdir/varnishstat.xml" sleep 60done


VARNISHLOG

• varnishlog debugging (-d to dump shared memory) 12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80 12 TxRequest b GET 12 TxURL b / 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: 12 TxHeader b Referer: 12 TxHeader b Cookie: 12 TxHeader b X-Forwarded-For: 1.2.3.4 12 TxHeader b X-Varnish: 2092287889 12 TxHeader b Accept-Encoding: gzip 12 RxProtocol b HTTP/1.1 12 RxStatus b 401 12 RxResponse b Authorization Required 12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT 12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only" 12 RxHeader b Vary: Accept-Encoding 12 RxHeader b Content-Encoding: gzip 12 RxHeader b Content-Length: 336 12 RxHeader b Connection: close 12 RxHeader b Content-Type: text/html; charset=iso-8859-1 12 Fetch_Body b 4 0 1 12 Length b 336 12 BackendClose b xx_80[0]


VARNISHTOP

• Monitor top URLs hitting backendvarnishtop -i txurl

• Monitor top received URLsvarnishtop -i rxurl


VARNISHADM

• varnishadm for administration# varnishadmCLI connected to 127.0.0.1 6082200 -----------------------------Varnish Cache CLI 1.0-----------------------------Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit

Type 'help' for command list.Type 'quit' to close CLI session.

varnish> help200 help [command]ping [timestamp]auth responsequitbannerstatusstartstopvcl.load <configname> <filename>vcl.inline <configname> <quoted_VCLstring>vcl.use <configname>vcl.discard <configname>vcl.listvcl.show <configname>param.show [-l] [<param>]param.set <param> <value>panic.showpanic.clearstorage.listban.url <regexp>ban <field> <operator> <arg> [&& <field> <oper> <arg>]...ban.list


MORE HELP?

• Talk to the devs, read their blogs.http://kristianlyng.wordpress.comhttp://ingvar.blog.redpill-linpro.com/

• Get some online training.https://www.varnish-software.com/products-services/training/varnish-administration-course

• Consider the Varnish ‘Moral’ License.http://phk.freebsd.dk/VML/


http://kristianlyng.wordpress.com/

http://kristianlyng.wordpress.com/

http://ingvar.blog.redpill-linpro.com/

http://ingvar.blog.redpill-linpro.com/

https://www.varnish-software.com/products-services/training/varnish-administration-course




http://phk.freebsd.dk/VML/

http://phk.freebsd.dk/VML/

Technology

Caching with varnish