Redis part 1: Performance optimization for Shopware and Magento

25.09.2019

Online shops that have many visitors generate many shopping baskets, from which sessions are generated. The sessions can be stored in different ways. In the first part of the blog series on Redis, we summarised the advantages that the NoSQL database Redis offers in the area of sessions and application cache for the performance of Shopware and Magento.

About this Redis blog series

In its daily work with online shops, maxcluster has repeatedly found that Redis significantly improved the performance of the shops. For this reason, we have always recommended the use of our pre-configured tool to our customers. However, we were only able to prove our experience with the benchmarks conducted by Collin Mollenhour in 2012, so we decided to conduct our own benchmarks to re-evaluate the usefulness of Redis. This Redis blog series is divided into a total of 5 posts. In this first post, we give an introduction to Redis, describe our results of the benchmarks, give tips on configuring Redis and make a recommendation at the end.

The second part and the third part deal with the benchmarks. We have carried out a total of two benchmarks: for sessions and for the application cache. In the fourth part of this series we describe the integration of Redis in Shopware. In the last and fifth part of the Redis blog series, we explain the integration of Redis in Magento.

What is Redis?

Redis is an in-memory database implemented in C. It belongs to the group of NoSQL databases and offers a simple key value store.

The Redis database was created in an open source project in 2010 and was developed by Salvatore Sanfilippo. With this in-memory database, strings, sets, lists and hashes can be stored in main memory. It can be extended by virtual memory, which allows certain data sets to be outsourced.

Sessions - Does Redis bring performance benefits for online shops?

Why is it a problem for online shops to store sessions in the database or file system?

Every visitor creates a so-called session when they visit an online shop. This is where user-defined data is stored, such as a customer's shopping cart. Shopware stores these sessions in the database by default, Magento in the file system.

Usually, each session is stored in a single file. With many visitors, this very quickly leads to a large amount of files. The higher this amount is, the more the reading of these files slows down and the loading times increase. For the customer, this means that the checkout becomes very slow - mostly unnoticed!

Another problem is that the storage space required is unpredictable, as the number of visitors can fluctuate greatly. Also, the sessions cannot be efficiently saved in individual files in a backup. This is where Redis brings significant advantages. The sessions can be stored in Redis and are managed in the main memory.

Advantages for online shops with a web server

At regular intervals, the sessions are stored in a single large file on the hard disk and backed up. The sessions can be read and written with high performance due to the management in the main memory. This offers advantages both for medium-sized shops with one web server and for very large online shops with several servers.

The effects are most obvious with Shopware, as Shopware saves sessions in the MySQL database by default. This leads to a significantly worse performance compared to Redis. In addition, in our experience there are often lockings on the MySQL table. For Shopware, we can therefore unreservedly recommend Redis for all shop sizes.

Advantages for online shops with multiple web servers

Usually, a load balancer is placed in front of the web server, which distributes the requests to the web servers. While it is still easy to store sessions locally on individual web servers, this leads to problems when several web nodes are in use. In this case, the web servers would see and write different files or sessions. There are various solutions for this.

Either an online shop uses a distributed file system in which the sessions are stored - which, however, has the disadvantage of poor performance. The reason for this is that the sessions have to be synchronised at file system level, which results in an additional loss of time. Or the load balancer assigns the visitors permanently to a specific web server, which makes configuration more difficult due to the increased complexity of the setup. With Redis, the load balancer can arbitrarily distribute requests to the web servers. This is easier to set up, less error-prone and easier to expand because all web servers access the same Redis instance. This advantage also applies to the application caches.

Advantages with a high number of sessions

If the sessions are not cleaned up, a large number will accumulate within a shop over time. A high number of sessions is problematic because it disturbs the backup process and reduces performance. This is because the loading time of a file increases with the number of existing files. We were able to measure that even in a shop without traffic, it takes 48 milliseconds longer to place a product in the shopping basket if there are 5,000,000 sessions. In a shop with frequent visitors, the effects are much stronger. With Redis, on the other hand, this problem does not exist. On the one hand, the loading time remains constant with Redis due to the internal data structure, and on the other hand, Redis automatically deletes sessions as soon as they expire.

Redis compared to NVMe SSDs and MySQL

We have tested the impact of Redis on sessions. We will present the technical information on our benchmarks in another blog post: Redis part 3 - Sessions Benchmark - NVMe SSDs, Redis and MySQL.

Session benchmark on write, read and modify operations, single and multi-threaded, to compare the behaviour of Redis, NVMe SSDs and MySQL. The higher the bar, the better the result.

What does the test mean for Shopware and Magento?

Through our benchmark, we have found that it is advantageous for Shopware online shops to place sessions in Redis.

With Magento (1 & 2), the use of Redis for storing sessions is also worthwhile if several visitors access the shop in parallel. The results of our benchmark in the area of multi-threaded showed that Redis offers higher performance per second than NVMe SSDs and MySQL for write, read and change operations.

Read all technical details about the Session Benchmarks: Redis part 3 - Sessions Benchmark - NVMe SSDs, Redis and MySQL.

Application cache - Does Redis bring performance advantages for online shops?

In addition to the sessions, application caches can also be stored in Redis and distributed over several webnodes. Depending on the application, this improves performance compared to the file system. However, the advantage is not as clear as with the sessions.

In addition to caching configuration data, complete web pages can also be stored in Redis via a so-called full page cache. However, this has the disadvantage that a PHP application must still be instantiated in order to read this page cache. We therefore recommend storing page caches in Varnish instead.

We have also validated our experiences with a benchmark for application caches: : Redis part 2 - Application Cache Benchmark: Redis, NVMe SSDs and memcached.

With this benchmark, we wanted to find out which backend is better performing for the application cache: Redis, file system with NVMe SSDs (as used on our clusters) or memcached. To do this, we measured the number of delete, read and write operations per second.

The diagram below shows the write rate of the cache backends: Zend_Cache_Backend_File (Dateisystem), Cm_Cache_Backend_Redis, memcached (Zend_Cache_Backend_TwoLevels) + Zend_Cache_Backend_File und memcached (Zend_Cache_Backend_TwoLevels) + Cm_Cache_Backend_Redis.

Benchmark to the application cache with four different backends to measure the number of write operations per second. The more operations, the better the result.

We were able to establish the result that although the throughput of the read operations was higher on our NVMe SSDs, the number of write operations per second was better with Redis. In addition, the number of delete operations per second was significantly higher with Redis. This is particularly relevant for heavily visited shops, as purchases usually lead to partial cache invalidations, so we recommend using Redis for well-visited shops here as well.

What distinguishes Redis from other databases?

Due to the simple data storage, Redis is an extremely efficient key value store. It thus offers the performance, scalability and stability necessary to run large online shops. Another advantage over other databases is the native support of many web applications such as Shopware and Magento 2.

Configure Redis

To use Redis optimally, we recommend using a separate Redis instance for each connection (e.g. cache or sessions). This allows the individual instances to be configured individually. This division enables higher performance, as Redis executes most operations in a single thread.

Persistence

Redis saves the data in the working memory by default. When the Redis instance is restarted, this working memory is emptied. To prevent this, Redis offers two possible options for Persistence. On the one hand, a snapshot (RDB) of the current state can be created at certain intervals, which can be read in during a restart. In this way, very little data would be lost in the event of an unplanned restart. On the other hand, the AOF method (append-only file) is available, which permanently writes the data to a journal in order to completely avoid data loss. The disadvantage of AOF is a slightly increased latency.

For Magento and Shopware caches, we recommend completely deactivating persistence, as the caches can easily rebuild themselves. In this respect, data storage is not so efficient that online shops should take the performance disadvantage for it.

For sessions, we recommend the snapshot-based backup method RDB. This way, in the event of a failure of the instance or even the server, a large part of the sessions can be preserved. In our experience, this offers the best combination of performance and stability.

Master replica

With a master replica setup, it is possible to distribute the read accesses to several nodes or make them available to other applications. In our experience, this is usually not necessary for small and medium-sized magentoshops.

Displacement strategy

It is possible to limit the memory of Redis via the maxmemory setting This is a measure to prevent a suddenly increasing Redis instance from occupying the server's resources and endangering stability. There are various strategies available on how Redis should deal with this when the memory is fully occupied. The aim of these strategies is to create free memory again. The individual keys represent data records that contain a session or an element to be cached. The strategy can be set via the maxmemory policy:

noeviction: As soon as the memory is full, saving further data leads to an error message.

allkeys-lru: Several keys are randomly selected. Of these, the key that has not been used for the longest time is deleted.

volatile-lru: Several keys marked as volatile are randomly selected. Of these, the key that has not been used for the longest time is deleted.

allkeys-random: A random key is removed.

volatile-random: A random key marked as volatile is removed.

volatile-ttl: Several keys marked as volatile are randomly selected. Of these, the key with the lowest TTL (time to live) is deleted.

allkeys-lfu (ab Redis 4): Several keys are randomly selected. Of these, the key that has been used the least is deleted.

volatile-lfu (ab Redis 4): Several keys marked as volatile are randomly selected. Of these, the key that has been used the least is deleted.

For sessions, we recommend the volatile-lru strategy. This ensures that only those keys are removed that would be deleted at a later time anyway due to the expiry of the TTL (time to live).
For the application caches, we also recommend monitoring the occupancy of Redis to be able to react before the memory of a cache instance is fully occupied. We monitor the occupancy and intervene in an emergency or inform the customer.

For the cache instance, we also recommend the volatile-lru strategy, but this is only a measure to catch fast peaks. Through our monitoring, we ensure that we are informed before the instance reaches the maximum and can thus intervene in time.

With the volatile-lru policy, we ensure that in the event that the instance is fully occupied, only keys that may be removed are removed, but the shop remains accessible. An allkeys-lru policy, on the other hand, removes arbitrary keys and can thus potentially affect the stability and data consistency of the shop. We also strongly advise against leaving the instance permanently at the limit, as this cancels out the performance advantages of Redis.

Redis and Magento 1

Redis is not integrated in Magento 1 by default. With the plugin by Colin Mollenhour there is a simple and free possibility to store sessions in Redis. The Magento versions from Open Source 1.8.0 or Commerce 1.13.0 include this plugin by default. However, we recommend using the latest version of the plugin from Github.

For caches, the plugin Cm_Cache_Backend_Redis, also by Collin Mollenhour, can be downloaded and installed on GitHub.

Redis and Magento 2

In contrast to Magento 1, Magento 2 already provides a complete integration in all editions. This can be easily configured via env.php.

Magento has published a detailed guide here which explains the exact configuration for sessions.

The Magento cache and a full page cache can also be stored in Redis in all Magento editions. We will explain the details in the next part of this blog series.

In our experience, it makes sense to store the Magento cache in Redis, but not the full page cache. As a full page cache, we recommend Varnish for Magento. Detailed information on how to configure and use Varnish for Magento 2 can be found in the Magento DevDocs: Configure and use Varnish.

Redis und Shopware

As with Magento 2, Redis is supported by Shopware. With Shopware, the application cache, an additional model cache and the sessions can be stored in Redis.

Unlike Magento, Shopware stores the sessions in the MySQL database by default instead of in the file system. This is much more in-performant, so we strongly recommend the use of Redis for a Shopware installation.

With Shopware 5, the configuration is done conveniently via config.php in the Shopware folder or for Composer installations under app/config/config.php. Details are explained in part 3 of this blog series.

Redis and other PHP applications

A large number of applications already support Redis "out of the box".

It is still possible to redirect sessions directly to Redis at the PHP level via php.ini. To do this, the following code can be entered in php.ini. We will be happy to implement this for you on our clusters if required.

session.save_handler = redis
session.save_path = "tcp://127.0.0.1:6379?auth=password"

Own data can also be easily stored in Redis thanks to the simple Redis-API. Numerous libraries are available for PHP (and other programming languages) to control Redis.

Redis at maxcluster

Redis instances can be created and managed conveniently and without additional costs via our interface. We also offer templates for caches and sessions to optimally configure the Redis instances. When using the cache template, our team monitors the instance's memory usage and intervenes if too little memory is available.

Redis instances can be easily managed at maxcluster in the Managed Center.

If required, we can also support you in integrating Redis into your own online shop. Please contact our support team for this.

If you have any questions about Redis, we look forward to hearing from you via email at support@maxcluster.de or by phone at 05251/414130.

Conclusion

Redis is a useful tool for storing sessions. We fully recommend the NoSQL database for this purpose. In Shopware, the use of Redis relieves the database, which leads to faster loading times in the checkout and thus limits the risk of purchase cancellations due to slow loading times. Redis also reduces the load on Magento, as the sessions and the cache scale better with increasing user numbers.

With regard to caches, the efficiency of Redis depends strongly on the individual shop application and configuration. In principle, however, the use of the in-memory database is recommended when more than one web server is used. We would be happy to advise you whether the use of Redis is efficient for your individual shop and brings you performance advantages.