Sometimes servers and applications fails, sooner rather than later, that’s because part of our work is to design and continuously planning a robust and highest reliable architecture for our application to ensure his availability most of the time. Depending on the application, software used, architecture, environment we’ve different challenges and strategies, so we’re depending on multiple factors, but in that post I’ll focus in one particular piece for architectures using web servers with Varnish that can help us to improve a bit more the available with a couple of tweaks. As you know Varnish is an HTTP accelerator which is used to cache dynamic content from web servers, acting as a proxy between the client and the original web server. That’s not the objective of this post to focus on the functionality and configuration of Varnish, so you can find very well documentation on his website: https://www.varnish-cache.org/docs.
One of the features of Varnish is the support of Saint and Grace mode, both features will allow us to handle troubles with our web servers and keep the service online even if our backend servers goes down. Well this is in part true, of course we cannot guarantee the entire service will continue working just with Varnish, but unless we can keep working part of our application.
So imagine a website or API service with thousands of requests per second, some of them may be POST, DELETE or PUT requests to submit changes for the application, which that kind of requests cannot be handled in the situations of the backend servers goes down, but in the case of GET requests where the clients wants to obtain the information from the service and Varnish have that particular content on his memory cache, this can be handled perfectly and returning the content to the client even if the backends are not working. Of course this has two things to bear in mind, this requests has to be cached before from Varnish and we’ll have outdated responses to the clients, but that’s better than reply with an error page! As always this behavior it’s useful depending on the requirements and the type of application, but most of the time can can save us requests and keep part of the service working in case of failure, so in my opinion highly recommendable if you can use that.
So let’s get started with the configuration of Varnish:
Edit the VCL definition file, usually located in /etc/varnish/default.vcl and edit the following directives:
sub vcl_recv {
…
if (req.backend.healthy) {
set req.grace = 30s;
} else {
set req.grace = 6h;
}
…
}
sub vcl_fetch {
…
if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503) {
set beresp.saintmode = 10s;
if (req.request != "POST") {
return(restart);
}
}
set beresp.grace = 6h;
…
}
Let’s see with a bit more in depth what this configuration does. The “vcl_recv” is called when at a request comes from the client, and the purpose of this method is decide what to do with that request. In that configuration we’re saying if the backend servers are alive we’ll keep the content for 30 seconds beyond their TTL “set req.grace = 30s;”, in case the backends becomes unavailable we’ll keep the content for 6h to serve to the clients “req.grace = 6h;”.
The “vcl_fetch” is called when a document has been successfully retrieved from the backend. In case the backend server returns a HTTP error code of 500, 502 or 503, Varnish will not ask that backend again for this object for 10 seconds “set beresp.saintmode = 10s;”. “return(restart);” restart the HTTP request, this will automatically happen on the next available server, except for POST requests to avoid duplicate form submits, etc… The max_restarts parameter defines the maximum number of restarts that can be issued in VCL before an error is triggered, thus avoiding infinite looping.
“set beresp.grace = 6h;” Keep all objects for 6h longer in the cache than their TTL specifies, so even if HTTP objects are expired (they’ve passed their TTL), we can still use them in case all backends goes down.
References:
https://www.varnish-software.com/static/book/Saving_a_request.html