Do you speak VCL?

1

Do you speak VCL?

2

The Varnish Book

Dedicated writer and maintainer

Francisco Velázquez

Curriculum ManagerIRC: francisv

francisco@varnish-software.com

CDNs with Varnish

Build your own CDN with Varnish

Who am I?

Agenda

● What is VCL?

● Motivate from learning VCL

● Understanding VCL in Varnish

● VCL as finite state machine

● How to design and test your VCL code?

● VCL tips and tricks

● Typical mistakes in VCL

● Conclusions

4

What is VCL?

5

Varnish Configuration Language (VCL)

● VCL is a domain specific

language based on C

● Programming language

specialized to express solutions

for caching the Web

● Rule-based language

6

Why should you program in VCL?

● Varnish covers most common caching needs out-of-the-box, but does not intend to be a one-fit-all solution!

● VCL allows you to:○ define your own rules○ specify when to use and alternate them on run time

7

Cacher

Web Application Firewall

Hotlinking Protector

DDoS Attack Defender

Load Balancer

Authentication and Authorization Mechanism

HTTP Router

and more...

VCL makes Varnish to act as:

VCL

8

VCL Characteristics (1/2)

● Simple syntax, but pay attention to the semantics of HTTP● No loops● Functionality grouped in subroutines that

○ do not take arguments nor return values○ exchange data only through HTTP headers

● VCL can manipulate HTTP headers○ Override TTLs○ Strip cookies○ Rewrite URLs

● Extensible via Varnish modules (VMODs)

9

VCL Characteristics (2/2)

● Load multiple VCLs and switch between them instantly and seamlessly● No server restart required● Includes built-in functions that allow you to:

○ modify strings○ invalidate caches

● Supports regular expressions (regex) ● Languages evolve, so VCL does!

○ varnish3to4 script

10

Varnish Finite State Machine (FSM)

● States in Varnish FSM are per HTTP request○ HTTP is a stateless protocol

● State preservation across transactions use mechanisms such as cookies

11

Built-in VCL subroutines per state

● File location: varnish-cache/bin/varnishd/builtin.vcl12

Your VCL code *always* precedes the built-in VCL code!

13

Your VCL code

Built-in VCL code

14

Understanding how VCL works in Varnish

varnishd

VCC Process

Manager ProcessCLI...

Cacher Process

Storage...

Shared Memory Log

C-compiler Shared ObjectOne

binary program

VACvarnishadm

varnishlog...

15

How to design your VCL code?

● Design your rules first○ Create Varnish Test Cases (VTC) in varnishtest○ Attend to Arianna’s presentation after break

● Be sure you understand the HTTP caching headers and objects’ lifetime○ Refer to The Varnish Book○ Refer to RFC7232 and RFC7234

● Get ideas from VCL snippets you find in Internet, but be critical!● Test your VCL!

○ Use varnishtest

16

● Design a plan for cache invalidation

Source: https://www.posterlounge.co.uk/spiderman-pr28529.html

17

Cache Invalidation Alternatives

Bans Purge Soft Purge Hashtwo – Surrogate keys

Force Cache Misses

Targets Patterns matching Specific object Specific object Objects with common key

Specific object

Frees memory

After a request hits an object or the ban luker invalidates cached object

Immediately After grace time

Immediately No

Scalability High High High High Low

VCL Yes Yes Yes Yes YesCLI Yes No No No No

Table 18 in The Varnish Book 18

VCL tips and tricks

● Command to print VCL code compiled to C language and exit: ○ $ varnishd -C -f filename○ Useful to check whether your VCL code compiles correctly

● Data types:○ TIME + DURATION is allowed○ TIME - TIME = DURATION is allowed○ TIME + TIME is not allowed!○ Use the Varnish Standard Module (std VMOD)○ If not in std VMOD, do not assume automatic data type conversions, test

them!● Check operator precedence and more at varnish-cache.org

19

Typical mistakes

20

Typical Mistakes in VCL (1/3)

● Regular expressions● Cache with inefficient cache invalidation plan● Skipping the built-in VCL by calling return(action)

○ Caching cookies inadvertently● Order query parameters of a URL

○ example.com/test?var0=0&var1=1

○ example.com/test?var1=1&var0=0

○ Call querysort(STRING) from std VMOD● Copy/paste VCL snippets without understanding them

21

Typical Mistakes in VCL (2/3)

● Create uncontrolled variations of hash key values for the same object○ Vary: User-Agent

○ Vary: Cookie

22

Typical Mistakes in VCL (3/3)

● Understand request serialization and the hit-for-pass object● Let vcl_backend_response execute!

○ Otherwise, never set beresp.ttl = 0○ Slows down site response if skipped

● Varnish 3 has a hit_for_pass return action● Varnish 4 beresp.uncacheable = true;

23

Ask for Help and Share

● Contact Varnish Support● IRC channel #varnish on irc.linpro.net● Mailing lists: https://www.varnish-cache.org/trac/wiki/MailingLists● Consult The Varnish Book● https://www.varnish-cache.org/trac/wiki/VCLExamples ● https://www.varnish-cache.org/docs/4.0/reference/vcl.html

24

Take away points

● It is worth it to learn VCL● VCL has a simple syntax● Design Varnish Test Cases and test your VCL in varnishtest● Let the built-in VCL code execute!

25