Performance Boosting with Varnish Cache

In the context of web applications, there are various types of caches. These can greatly accelerate the loading time of applications by avoiding computationally intensive operations. Since the loading time of an application is of enormous importance, especially in e-commerce, server-side caching is a proven means of improving it. Shop operators often use the web accelerator Varnish for this purpose, as it enables enormous speed improvements, especially for e-commerce applications.

What is Varnish?

Varnish is a full-page cache for Unix operating systems that is commonly referred to as a "performance booster for websites" or "web accelerator". Rightly so, because with the right settings, the page load time of a website can be improved enormously on the server side.

Since setting up Varnish is relatively complex and the application is also not suitable for every online shop, we will go into more detail below about how it works, the areas of application and the configuration.

How does the Varnish cache work?

Nowadays, web servers often generate websites dynamically. In the process, data is retrieved from a database such as MySQL and converted in a programming language such as PHP into an HTML format that can be read by browsers. The shop systems Magento and Shopware also work in this way.

The more complex the website, the higher the load on the server and the slower the loading time. In e-commerce, this can quickly result in falling conversions, lower turnover and increasing customer dissatisfaction. Depending on the hardware and software setup, a web server is limited in the number of visitors it can serve simultaneously. By using a full-page cache, the number of simultaneous visitors can be greatly increased.

This is where the Varnish cache comes into play. As a so-called HTTP cache proxy, it was designed for complex websites with dynamic content and can be placed in front of a web server such as Apache or NGINX.

When Varnish receives a request, it first checks whether this request can already be served from the cache. In this case, the underlying web server does not even have to be requested. Instead, Varnish simply delivers the cached page directly. However, if this page is not (yet) contained in the cache, Varnish sends the request to the underlying web server. If the request can be cached, Varnish sends the server's response to the browser and saves it for future visitors.

The result: the web server is relieved, content is delivered to the client within a few milliseconds and the performance of the shop remains stable even with higher access numbers.

Varnish finite state machine

When a request is processed by Varnish (whether cached or uncached), it passes through a so-called finite state machine within Varnish. Using the Varnish Configuration Language (VCL), it is possible to intervene in each individual step of the automaton and influence the behaviour of Varnish. In this way, the cache behaviour can be influenced on the one hand, but on the other hand, the request and the response of the server itself can be changed. In this way, it is determined which requests are to be answered in which way.

Varnish as a load balancer

Normally, Varnish forwards requests that were not served from the cache to the web server. For larger online shops, however, it is often necessary to address more than one web server. Here Varnish can take over the distribution of the load to several web servers.

For this purpose, the individual web servers (backends) are combined in a so-called director. By default, this director distributes all requests to the individual backends one after the other. It is also possible to direct certain requests to certain backends via the VCL language. For example, a blog can be easily separated from the shop system. Even more complex scenarios such as visitor pinning or A/B testing can be implemented in this way. In addition, selecting a backend via a header is possible using the VCL language.

Exceptions to content delivery

With dynamic websites such as online shops, the entire website is usually not cached. This is mainly for data protection reasons, but is also recommended from a usability perspective: for example, users should only see their own shopping cart and not that of another customer. To ensure that only content that should be cached is cached, there are various options.

URL exception via VCL configuration

If you want to define an exception in the cache, you can write the exception directly into the VCL file. The VCL syntax allows you to define rules that determine which pages are not cached. A common rule here is the exception based on certain URLs.

   if (req.url ~ "/customer" || req.url ~ "/checkout") 
   {
       return (pass);
   }

Thus, the code block specifies that pages beginning with /customer or /checkout should generally not be served from the cache, but can be queried from the backend. Besides the URL, other conditions are also possible.

   if (req.url ~ "/graphql" && req.http.Authorisation ~ "^Bearer") 
   {
       return (pass);
   }

Thus, this block specifies that Graphql requests cannot be cached if the "Authorization" header was sent along.

Exceptions via cache headers

In addition to fixed exceptions in the VCL file, the application can also dynamically define which pages are cached and which are not. This can be implemented via the headers Surrogate-Control, Cache-Control and Expires-HTTP. Via the headers Cache-Control and Expires it is also possible to set the TTL (Time to Live) of the cache objects. Unless the installed VCL file specifies otherwise, all responses that set a cookie are not cached by default. This is the reason why it is almost always necessary to use an application-specific VCL file.

ESI Hole punching

Often a large part of a website is identical for all visitors, but differs in certain details. An example in e-commerce is the shopping cart icon, which shows the number of products in the shopping cart. Instead of not caching the entire page because of an individual detail, only this part of the page can be excluded from caching. For this purpose, the web server sets a so-called ESI Include Tag as a placeholder. As soon as Varnish sees such a tag, the URL entered as the source is called. The result of the call is then inserted in the place of the placeholder.

Hole punching with Ajax

In addition to the method of replacing the ESI tags when the page is called up, user-specific content can also be integrated into the pages on the browser side. In this case, the private content is inserted in Javascript. Data for this is retrieved from the server via Ajax if required. This method has the disadvantage that it is more complex to implement in the application. The reason for this is that both a server logic that makes the data available and a client-side logic that compiles the content must be present. An advantage, on the other hand, is that the original page call is significantly faster and Ajax calls can run in the background. This can improve performance even further compared to the ESI method. For this reason, Magento 2 officially only supports this method in newer versions.

Check Varnish Caching

There are several ways to check whether certain pages are correctly delivered from the cache.

One reliable method is to check the Access.logs of the web server. If a request has been successfully delivered from the cache, the web server does not even receive any information about it and consequently does not show any entry here.

Furthermore, it is also possible to check the behaviour of the cache with the tool varnishlog or to send debug headers (via the VCL) to the browser.

What Varnish is technically based on

Since the Varnish cache is very complex in its functionality, we will take a look at the technical details of the proxy below.

VCL (Varnish Configuration Language)

The Varnish Configuration Language is a domain-specific language and, as its name suggests, is used specifically for configuring Varnish. Using the VCL language, it is possible to write hooks for each individual state of the finite state machine and to control the behaviour of Varnish at that point. When Varnish is started or a VCL file is loaded, it is translated into C code. In this way, extremely good performance can be achieved within Varnish. In addition to VCL, it is also possible to embed C code directly in VCL in order to be able to map more complex logics.

Cache logic

As described above, the cache logic can be adapted for the specific application via the Varnish Configuration Language. If no configuration is available, Varnish will attempt to cache content via a default behaviour.

Headers

Varnish evaluates the browser's headers. These headers are Surrogate Control, Cache Control or Expire.

If the Surrogate-Control header is set, the page is not cached if one of the flags no-cache, no-store or private is set. The TTL is not evaluated from this header by default.

If the Surrogate-Control header is not set, the Cache-Control header is evaluated instead. In addition to the flags, the TTL of the cache is also taken into account here.

If both headers are not set, the system checks whether the Expires header (which is now obsolete) is set and then evaluates it.

Cookies

If the response from the web server contains a cookie, it is also not cached. For this reason, cookies are often removed in the application-specific VCL if they are not relevant to the response.

Status code

The status code is a prerequisite for successful caching. Among other things, correctly delivered web pages (200), but also redirects (301, 302, 307) or missing pages (404, 410) are cached, but not errors of the web server (5xx).

Request types

In addition to the response of the web server, the type of request is also relevant. By default, Varnish only caches GET and HEAD requests, but not POST requests, for example.

Hashing

It is also important which scheme is used to cache a page request. This is called a hash in Varnish. By default, this consists of the URL and the host name. However, it is possible to set this up via the VCL file or the vary header. For example, there is the option of delivering a different website for mobile visitors than for desktop users. This other website can then be cached separately.

Grace time

Even if the cache TTL has expired, Varnish does not necessarily have to request the web server for every request. In the so-called Grace time, content that has actually expired is still delivered from the cache, while the page is retrieved anew from the web server in the background. This ensures that visitors always receive a web page quickly and that not too many requests are sent to the web server.

Cache Invalidation

In some situations, independent of the TTL, certain elements should be removed from the cache. Varnish offers several possibilities for this. One common method is so-called PURGE requests. The application sends an HTTP-PURGE request. In order for Varnish to evaluate this, it is necessary to write VCL logic for it.

Another method is to send a BAN command directly to Varnish. This is possible via the varnishadmn tool or via telnet, but also via the VCL logic as with PURGE. Unlike PURGE, BAN removes the objects in the background. In addition, when delivering a request, Varnish checks whether there is a pending BAN operation for this request and then removes it from the cache. BAN operations are therefore particularly suitable for deleting many objects at once.

ESI (Edge Side Includes)

With the help of so-called ESI tags, it is possible to dynamically make changes to web pages that Varnish has read and saved during a request to the web server. These tags can be used, for example, to insert user-specific content into a web page that is otherwise cached by default.

Varnish currently supports only two types of ESI tags: ESI Includes tags allow additional requests to be inserted into a specific location on a web page. ESI Remove tags or the associated comments can be used when the application does not know whether a proxy with ESI support is present or not.

With regard to user-specific context, it can also be useful to outsource shared context into ESI blocks. This is useful, for example, if certain context, such as a menu navigation, is available on several pages. The advantage is that such blocks can then be removed from the cache independently of the rest of the page content. A change to the menu therefore does not require the invalidation of all pages.

Edge Computing

Varnish is mainly used as a cache in e-commerce. With the help of the VCL, logic that changes requests and responses can optionally be generated. Furthermore, it is even possible to generate a response completely without a web server behind it.

Whether this makes sense depends very much on the setup used. Often, the logic can also be relocated to the SSL offloader at hand, provided it supports it. In addition, it must be ensured that existing fallbacks on the web server do not lead to misbehaviour.

In any case, computationally intensive operations should always remain on underlying webnodes in order to be able to ensure scalability.

VMODS

In addition to the possibility of controlling Varnish via the VCL (and included C code), the web accelerator can also be extended with additional modules, so-called VMODs. A list of available VMODs is maintained on the Varnish homepage. It is also feasible to create VMODS for Varnish yourself in order to outsource more complex functions.

Start parameters

Numerous Varnish options can be specified at start time via Parameters. Settings such as the size of the cache and timeouts, but also buffer sizes or ESI behaviour can be controlled via this.

For Magento2, the setting http_max_hdr is often particularly relevant, as Magento sometimes sends very long X-Tag headers to Varnish in order to be able to categorise the cache.

The size of the workspaces (workspace_client and workspace_backend) must also often be adapted to the application.

The most important parameters can be configured at maxcluster via the Managed Center.

Varnish Tools

There are numerous tools available for dealing with Varnish, which can be used to facilitate debugging.

varnishlog

With varnishlog it is possible to check exactly what happens inside Varnish. Therefore, it is possible to see both the incoming request and the backend request, if there is one.

If, on the other hand, you only want to look at certain requests, you can use appropriate filters for this.

varnishlog -g request -q (ReqHeader ~ 'X-Forwarded-For: 1.2.3.4' or ReqHeader ~ 'CF-Connecting-IP: 1.2.3.4') and ReqURL ~ '/fuu'.

The example above only shows requests with the IP 1.2.3.4 in the request headers X-Forwarded-For or CF-Connecting-IP and whose request goes to /fuu.

varnishadm

With the tool varnishadm, administrative tasks can be carried out via CLI. Here, for example, VCL files can be managed or elements can be removed from the cache.

varnishstat

With the help of the varnishstat tool, numerous statistics on Varnish can be read out. This is particularly helpful for carrying out long-term analyses. Among other things, it can be used to check whether the cache should be enlarged if elements are regularly displaced from the cache (MAIN.n_lru_nuked).

varnishncsa

The varnishncsa tool displays the requests processed by Varnish in a compact format. This is comparable to the Access.logs of a web server.

varnishhist

The varnishhist tool can be used to display access times in a histogram.

TLS/SSL support

As a pure HTTP cache proxy, Varnish does not include native TLS or SSL support. Consequently, Varnish cannot receive or play back encrypted HTTPS traffic. Since encrypted connections comply with modern security standards, Varnish must be combined with another proxy. This then functions as a so-called TLS or SSL offloader, which is switched in front of Varnish.

In principle, any proxy that can terminate SSL can be used here. Commonly used are NGINX, HAProxy or Hitch. At maxcluster, we have chosen NGINX because it allows us to flexibly map different requirements.

Technical configuration of Varnish

Varnish can be flexibly adapted to the corresponding requirements of a shop system through individual configuration. However, as the configuration is very complex, a lot of expertise is required to use Varnish profitably. In the following, we will show you how to configure Varnish individually depending on the shop system used.

Varnish and Magento

The Magento HTTP cache is active by default, but can be placed in the Varnish cache to achieve better performance. Instructions on how to configure this can be found in the Magento documentation. Varnish must first be activated in the Magento settings. This is possible under Stores > Settings > Configuration > Advanced > System > Full Page Cache.

A suitable VCL file can also be exported directly there. In the next step, Varnish can be activated in the maxcluster interface and the VCL file can be stored.

While the integration of Varnish in Magento 2 is very simple, in the end it must always be checked whether the respective shop works in connection with Varnish. Through additional modules and adjustments to Magento, this is not guaranteed in every case.

As soon as a single block in a page is marked as not cacheable, this means that the complete page can no longer be delivered from the cache. It is therefore very important to avoid this scenario at all costs with regard to product and category pages. If a page is to contain private content, it is instead necessary to display this via UI components independently of the actual page call. Private content via ESI includes is no longer supported by Magento 2.4.

Cache Purging

Invalid pages are invalidated automatically by Magento via PURGE requests. If all services are on the same server, this already works without having to make any changes. In the standard setup, however, all PURGE requests are routed via the SSL offloader. Varnish recognises the origin of these requests in the local host, which is authorised to clean the cache according to the ACL (Access Control List). Conversely, this means that unauthorised clients can also execute a PURGE request.

To increase security or if more than one Varnish server is in use, it is recommended to explicitly specify the Varnish instance as described in the Magento instructions. In addition, it should be specified via the ACL that only the web servers are allowed to empty the cache.

Varnish and Shopware

Varnish can be implemented in Shopware 5 as well as in Shopware 6.

Varnish integration in Shopware 5

The integration of Varnish in Shopware is possible from a Shopware version 4.3.3 and Varnish version 4.0. It is recommended to carry out the setup using the Shopware documentation.

The documentation also contains the subchapters "TLS Termination" and "Forward HTTP Headers". The configurations described there are automatically implemented at maxcluster via the Managed Center as soon as Varnish and the corresponding SSL offloader have been activated there.

You can import the suggested Varnish configuration from the chapter "Varnish configuration (vcl)" via the Managed Center, menu item Varnish SSL > Configuration. Please note that the port of the web server changes to 8000 when Varnish and SSL Offloader are active. The Varnish configuration is therefore:

vcl 4.0;

import std;

backend default {
    .host = "localhost";
    .port = "8000";
}

[...]

All further steps of the Varnish configuration can be found in the official documentation.

Varnish Integration in Shopware 6

In Shopware 6, the HTTP Cache is active by default. This can be placed as "Reverse HTTP Cache" in Varnish in favour of better performance.

In the case of Shopware 6, it is also recommended to implement it using the official documentation. Prerequisites are a Shopware 6.4 application, the active Varnish service including the activated SSL offloader as well as an empty Redis instance that is used for the tag-based invalidation.

According to the official documentation, the configuration file storefront.yaml must be created first. In this example, we have created a new Redis instance in maxcluster's Managed Center with the option "Optimised for Cache" under port 6379 and recommend setting the settings for "hosts" and "redis_url" as follows:

config/packages/storefront.yaml

storefront:
        csrf:
                enabled: true
                mode: ajax
        reverse_proxy:
                enabled: true
                ban_method: "BAN
                hosts: [ "http://127.0.0.1" ]
                max_parallel_invalidations: 3
                redis_url: "redis://127.0.0.1:6379"

In the template of the Varnish configuration, only the setting for ".host" and ".port" must be set. At maxcluster, the Varnish configuration is stored after completion via the Managed Center, menu item Varnish SSL > Configuration. As soon as Varnish and the associated SSL offloader are activated, the web server is accessible under localhost:8000. For this reason we set:

vcl 4.0;

import std;

backend default {
    .host = "localhost";
    .port = "8000";
}

[...] 

All further steps of the Varnish configuration can be found in the official documentation.

After completing the configuration steps from the official documentation, the X-Cache header can be used to check via the developer tools available in the browser whether the call in the frontend is a Cache-HIT or Cache-MISS.

Varnish hosting at maxcluster

At maxcluster, Varnish is already pre-configured via the e-commerce stack and can be integrated at no extra charge into an online shop. Regardless of the shop or CMS system used, Varnish can be activated and then configured with just one click via the Managed Center under Webserver > Varnish SSL.

All necessary settings can be found in the Managed Center under Webserver > Varnish SSL. The VCL file exported from the application or taken from the documentation can also be stored there. The port for the backend must be set to 8000 for maxcluster. The SSL offloader should also be activated directly here.

When storing, we recommend removing the health check (if present) from the VCL file, as this is not necessary due to our high availability setup and can increase the downtime after a temporary malfunction in case of doubt.

Conclusion

Varnish is a versatile web accelerator that is significantly more efficient than the HTTP caches integrated by default in Magento and Shopware in terms of web application performance. However, since the configuration of Varnish is very complex and requires some specialist knowledge, its efficient use in connection with a shop system involves a lot of effort. In most cases, this is nevertheless very worthwhile, as with the right parameters an enormous performance boost can be achieved.

Published on 11.10.2021 | JH, AS, DR