Upload
fastly
View
393
Download
0
Embed Size (px)
Citation preview
Debugging Fastly VCL: 101
Cassandra DixonSolutions Architect | Fastly
Steven ChuobTechnical Account Manager | Fastly
Discussion
● Common Fastly VCL Debugging
● More Technical Fastly VCL
Debugging
Introduction
● Purpose of this training
● What will you learn
● Aiden’s Journey
When Contacting Fastly Support
What to provide:
● Tell us who you are
● Tell us the issue
● Be specific
● Steps to reproduce
● How did you discover it?
Aiden’s Journey through Fastly
FOR PARTICIPANTS:
ALL CODE IN THIS PRESENTATION IS BROKEN.
DO NOT USE IN PRODUCTION.
Fastly’s Shielding Service
Fastly’s Shielding Service (VCL Flow)
Origin
vcl_recv
vcl_deliver
vcl_fetch
Edge POP Shield POP
vcl_recv
vcl_deliver
vcl_fetchUser
restart
• vcl_recv - This is the first
subroutine the request goes
through
• Handles backend/shield
selections
• Modifications to hostname, url
and other req.headers
vcl_recv
• vcl_fetch - This subroutine runs
after the object has been
fetched from the origin/shield
• Does not run on a hit. Only for
miss, pass requests
• Caching logic and TTLs
vcl_fetch
● vcl_deliver - This is one of the
last subroutine that handles the
response.
● This subroutine sends the object
and its headers to a client
making a request.
● Handles modifications to the
response headers before
delivery
vcl_deliver
● vcl_log - Like vcl_deliver this is
one of the last subroutine to be
run.
● If you configure log, it is called in
this subroutine
vcl_log
• vcl_recv - This is the first subroutine the request goes
through, backend/shield selections happens in this routine.
• vcl_deliver - This is one of the last subroutine that
handles the response from the origin. This subroutine sends
the object and its headers to a client making a request.
• vcl_log - Like vcl_deliver this is one of the last subroutine
to be runned. This will send a log to a logging endpoint
after the request has been handled.
• vcl_error - This subroutine handles any error handling
when the origin responds with an error.
• vcl_miss - This subroutine runs once the object is
determined to not be in cache.
• vcl_hit - This subroutine runs once the object is
determined to be in cache.
• vcl_pass - This subroutine will either be runned after
vcl_recv or vcl_fetch.
• vcl_fetch - This subroutine runs right after the object has
been fetched from the origin and its caching TTL set (based
on what’s being sent from the origin).
Cluster Edge Cluster Node
Clustering
The more you know...
Months 1-6
● Setting up an anonymous cloud hosting origin
● Enabling Fastly’s Shielding Service
backend default { .host = “174.138.66.5”; .port = “80”; .connect_timeout = 60s; .first_byte_timeout = 60s; .between_bytes_timeout = 60s; .max_connections = 800;}
backend shield_ssl_cache_lcyxxxx_LAX { .is_shield = true; .connect_timeout = 2s; .port = "80"; .host = "23.235.47.20"; .max_connections = 1000; .share_key = "fastlyshield";} ...
Aiden’s VCL
sub vcl_recv { # custom vcl set_hostname # setting host header to match backend set req.http.host = "hostname expected by origin"; set req.backend = F_addr_xxx;
#do shield here { if (req.backend == F_addr_xxx && req.restarts == 0) { if (server.identity !~ "-LCY$" && req.http.Fastly-FF !~ "-LCY") { set req.backend = ssl_shield_lax_ca_us;} if (!req.backend.healthy) { # the shield datacenter is broken so don’t go to it set req.backend = F_addr_xxx; }}}
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } return(lookup);}
Aiden’s VCL
Aiden’s Testing
● Test URL
● Check Aiden’s VCL
● Testing Tools○ watch○ curl○ Logging
Logging
curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/shielding
< HTTP/1.1 200 OK< Server: Apache/2.4.7 (Ubuntu)< Content-Type: text/html< Fastly-Restarts: 1< Cache-Control: max-age=86400< Content-Length: 14405< Accept-Ranges: bytes< Date: Sun, 18 Jun 2017 16:22:53 GMT< Via: 1.1 varnish< Age: 0< Connection: keep-alive< X-Served-By: cache-sjc3635-SJC< X-Cache: MISS< X-Cache-Hits: 0< X-Timer: S1497802973.177423,VS0,VE171< Vary: Accept-Encoding
Test URL
● Hits to Origin multiple times for the same object
● Fastly-Restart > 0
● Served-By header shows only one POP
● Age is randomly coming back as 0.
sub vcl_recv {
# custom vcl set_hostname # setting host header to match backend set req.http.host = "hostname expected by origin"; set req.backend = F_addr_xxx;
#do shield here { if (req.backend == F_addr_xxx && req.restarts == 0) { if (server.identity !~ "-LCY$" && req.http.Fastly-FF !~ "-LCY") { set req.backend = ssl_shield_sjc_ca_us;} if (!req.backend.healthy) { # the shield datacenter is broken so don’t go to it set req.backend = F_addr_xxx; }}}
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } return(lookup);}
Reviewing: Aiden’s VCL
Fastly’s Shielding Service (VCL Flow)
Origin
vcl_recv
vcl_deliver
vcl_fetch
Edge POP Shield POP
vcl_recv
vcl_deliver
vcl_fetchUser
restart
http://altitude-stage.global.ssl.fastly.net/req.http.host == altitude-stage.global.ssl.fastly.netreq.url == / vcl_recv
req.http.host == altitude-debug-workshop.dixonkin.comreq.url == /
vcl_recv
Delivers a 500 internal domain not found to the Edge POP
vcl_deliver
vcl_fetch
A restart is triggered due to the 500 error. Cause: Edge thinks the shield is unhealthy and thus down
Origin
vcl_deliver
Easiest Solution
# OPTION 1 - Create 2 snippet # vcl_miss & vcl_pass
## Setting a hostnameif (!req.backend.is_shield) { set bereq.http.host = "hostname expected by origin";}
# OPTION 2 - Create a vcl snippet with # type “init routine” # create snippets for vcl_miss & vcl_pass
sub set_hostname { ## Setting a hostname if (!req.backend.is_shield) { set bereq.http.host = "hostname expected by origin"; }}
Better Solution
# OPTION 2 - Cont’d# Update Custom VCL file
sub vcl_miss { #FASTLY miss call set_hostname; return(fetch);}
sub vcl_pass { #FASTLY pass call set_hostname;}
sub vcl_recv { #FASTLY recv ## Setting a hostname if (!req.http.Fastly-FF) { set req.http.host = "hostname expected by origin"; }
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } return(lookup); }
What about Fastly-FF?
• Aiden creates a login page
• Set Cache-control: max-age=0
• He implemented serving stale
Month 8-10
Fastly Serve Stale
Client
Fastly Origin
Is content cached?
Yes No
Serve Stale Serve Error
Fastly Testing
● Test URL
● Testing Tools○ watch○ curl○ logging○ Fastly-Debug
curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/caching/time
< HTTP/1.1 200 OK< Server: Apache/2.4.7 (Ubuntu)< Cache-Control: max-age=0, public< Via: 1.1 varnish< Content-Length: 1533< Accept-Ranges: bytes< Date: Tue, 20 Jun 2017 21:48:21 GMT< Via: 1.1 varnish< Age: 8< Connection: keep-alive< Fastly-Debug-Digest: 6d34439e4502f525f4dee9255bb9fb1f3b4943b4a9b792c1b07f09f4d31b7c38< X-Served-By: cache-lax8635-LAX, cache-sjc3640-SJC< X-Cache: MISS, HIT< X-Cache-Hits: 0, 1< X-Timer: S1497995301.171881,VS0,VE0
Test URL
Logging
● Hits in logging
● Cache-Control: max-age=0
● Age is greater than 0.
● Includes log line with a HIT
response
● Includes curl responses;
showing Age=0 and sometimes
Age=6
Aiden’s Support Ticket
Jun 20 14:48:19 Fastly_logs
AltitudeDebugWorkshop-Staging:
2017-06-20T21:48:18+0000 SJC 8.18.217.202 "-" "-"
[20/Jun/2017:21:48:18 +0000] "GET /caching/time
HTTP/1.1" 200 1533 "" "curl/7.48.0"
recv[shield_lax_ca_us:/caching/time] hit
deliver[HIT-STALE:hits=1:age=1] HIT-STALE
sub vcl_recv { # setting host header to match backend set req.http.host = "hostname expected by origin";
# Fix encoding issues in urls set req.url = regsub(req.url, "&", "&"); set req.http.tmpLog = if (req.restarts == 0, "recv", req.http.tmpLog "; recv"); set req.backend = F_addr_xxx; #do shield here ...
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } # Strip out the service ID for readability set req.http.tmpLog = req.http.tmpLog "[" regsub(req.backend, "^[a-zA-Z0-9].*--", "") ":" req.url "]";
return(lookup); }
Aiden’s VCL - vcl_recv
sub vcl_miss { set req.http.tmpLog = req.http.tmpLog " miss";
#--FASTLY MISS START ... # Snippet set_hostname_miss : 100 if (!req.backend.is_shield) { set bereq.http.host = "altitude-debug-workshop.dixonkin.com"; }
#--FASTLY MISS END
return(fetch); }
Aiden’s VCL - vcl_miss
sub vcl_fetch { set beresp.do_stream = true; #if (beresp.status >= 500 && beresp.status < 600) {
# /* deliver stale if the object is available */ # if (stale.exists) { # return(deliver); # } ... #set stale_if_error and stale_while_revalidate (customize these values) set beresp.stale_if_error = 86400s; set beresp.stale_while_revalidate = 10s;
if (req.http.X-Debug) { set beresp.http.Backend-Name = beresp.backend.name; }
...}
Aiden’s VCL - vcl_fetch
● Cache-control: max-age=0 is not equivalent to a pass object
● TTL of zero is cacheable- serve stale applies
Aiden: No Regrets, Lessons Learned
Solutions
OPTION 1 - Update
cache-control header on your
origin
OPTION 2 - Create a cache setting
on Fastly with a condition on the
path
OPTION 3 - Custom VCL in vcl_recv after
the #FASTLY recv macros.
Break
• New Photo Gallery released on
website
• TTL needed for static pictures
• Aiden’s origin is not sending the
cache control header for this
content.
• He adds a cache header on
Fastly
Month 14 -15
sub vcl_fetch { set beresp.do_stream = true; ... #set stale_if_error and stale_while_revalidate (customize these values) set beresp.stale_if_error = 86400s; set beresp.stale_while_revalidate = 10s;
if (req.http.X-Debug) { set beresp.http.Backend-Name = beresp.backend.name; } ... #--FASTLY FETCH BEGIN ... # priority: 10 if ( req.url == "/redirect1" ) { # Header rewrite Static Page Cache : 10 set beresp.http.Cache-Control = "max-age=3600"; } ...}
Aiden’s VCL
Fastly Testing
● Test URL
● Testing Tools○ watch○ curl○ logging○ Fastly-Debug
curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/redirect1 -H 'Fastly-Debug:1'
< HTTP/1.1 200 OK< Server: Apache/2.4.7 (Ubuntu)< Cache-Control: max-age=3600< Content-Length: 1917< Accept-Ranges: bytes< Date: Wed, 21 Jun 2017 20:24:02 GMT< Via: 1.1 varnish< Age: 1< Connection: keep-alive< CDN-Process-Log: recv[shield_lax_ca_us:/redirect1] hit deliver[HIT:hits=1:age=1]< Fastly-Debug-Path: (D cache-sjc3121-SJC 1498076643) (F cache-sjc3121-SJC 1498076642)< Fastly-Debug-TTL: (H cache-sjc3121-SJC 119.237 86400.000 1)< Fastly-Debug-Digest: cfb378e8d0413b6345893969c58d8ed23b6db487be9197c0d24d8b5833df8970< X-Served-By: cache-sjc3121-SJC< X-Cache: HIT< X-Cache-Hits: 1< X-Timer: S1498076643.983655,VS0,VE0
Curl Test
Logging
Wait - What is Fastly-Debug showing?
Fastly-Debug-Path: (D cache-sjc3121-SJC 1498076643) (F cache-sjc3121-SJC
1498076642)
Fastly-Debug-TTL: (H cache-sjc3121-SJC 119.237 86400.000 1)
The first number is the TTL remaining for the object
The third number is current age of the item in cache
The second number refers to the grace
● Object is expiring after 120 seconds
● Fastly-Debug is saying the object TTL is 120 seconds
sub vcl_fetch { set beresp.do_stream = true; ... #set stale_if_error and stale_while_revalidate (customize these values) set beresp.stale_if_error = 86400s; set beresp.stale_while_revalidate = 10s; ... #FASTLY fetch ...
if (beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(s-maxage|max-age)") { # keep the ttl here } else if (beresp.cacheable) { # apply the default ttl set beresp.ttl = 3701s; }
}
Aiden’s VCL
Important VCL Logic
TTL Flow
Origin
vcl_recv
vcl_deliver
vcl_fetch
Edge POP Shield POP
vcl_recv
vcl_deliver
vcl_fetchUser
vcl_recv vcl_recv
Origin TTL = 0
TTL = 120Cache-Control: max-age=3600
vcl_fetch
vcl_deliver
vcl_fetch
vcl_deliver
TTL = 3600
● There is a default TTL of 120 that is used when the object has no TTL or Cache-Control headers from the origin.
● The starting TTL is set before vcl_fetch runs on a service
Lessons Learned
Solution
A Cache-control header created
in vcl_fetch does not reset the
TTL
To set a TTL, modify the variable
beresp.ttl in vcl_fetch
Therefore create a cache setting on Fastly
to actually change and set the beresp.ttl
• A starting point for Debugging
• Streaming Logs
• Basic understanding on VCL
subroutines
Recap
Questions?
Thank youCassandra & Steven