Elasticsearch - What are the advantages for online stores?
Today, anyone searching in an online shop or web application not only expects meaningful results immediately, but also support in the form of autocompletes or search suggestions.
Speed can become a knockout criterion when users get tired of the "long" wait, leave the site and possibly migrate to a competitor. But it is not only (milli)seconds that count to keep customers on your site. The quality of the search results is also crucial.
To ensure the highest possible speed and quality of search results, powerful search engines must be integrated into the website. One of the most important full-text search engines is Elasticsearch, which is used by companies such as Facebook, Netflix, Zalando and GitHub.
How does full-text search work with Elasticsearch? What are the advantages for online shop and website operators? And what alternatives are there to this very popular search engine? We clarify these and other questions here.
What is Elasticsearch?
Elasticsearch is both an open-source search engine and an analytics engine that has been available on the market since 2010. It is based on Apache Lucene and can be used for data of all kinds: Text data, numeric data, geodata and both structured and unstructured data. Its range of application is thus (theoretically) unlimited. For example, geodata can be used to display offers according to distance from the user. Elasticsearch thus offers online shops many opportunities for creative sales increases.
Elasticsearch offers many advantages as a search engine:
- Distributed nature: Individual nodes taken together form a cluster, which makes it possible to distribute even large amounts of data via sharding. To cushion high request loads, data can also be multiplied across multiple nodes.
- Simple operation
- Fast and stable at the same time
- Offers a wide range of functions: Data Roll-ups, Index lifecycle management
- Easy data ingestion, visualisation and reporting with Kibana
- 24 text languages
- free of charge under Apache 2 licence
- persists data, therefore it can also be used for longer-term storage
Disadvantages of Elasticsearch
With more technologies in use, the setup for an online shop or blog becomes correspondingly more complex and unfortunately Elasticsearch is not available in all environments.We have compiled a list of all shop systems compatible with Elasticsearch here.
Simple managed hosting environments in particular do not offer Elasticsearch in all cases. Even with self-hosted servers, the installation and maintenance of Elasticsearch generates additional expenses that should not be neglected.
However, those who are not put off by this additional effort will notice a significant acceleration of the search with Elasticsearch and thus potentially a recognisable increase in the conversion rate.
How does Elasticsearch work?
With Elasticsearch, raw data from various sources, such as logs, system metrics and web applications, are combined in an ingestion process and enriched with further information if necessary. The data is then stored as JSON documents, which in turn consist of a set of pairs of keys and values. In order for search queries to deliver results later, the data is indexed, for which Elasticsearch uses a data structure called inverted index.
Instead of examining all available documents for a search query, Elasticsearch checks an index in which all content has been previously stored in a prepared form. For this process, Elasticsearch creates one (or more) indices, each of which functions like a kind of database and is then later searched by the search engine.
The index created in this way is divided into several types. Documents can now be stored in these indices, each of which can be assigned to a type. In contrast to classic SQL databases, the documents do not have a fixed format, but can consist of any structure of attributes. This enables a fast full-text search and the browsing of document data in (almost) real time.
To store the data, Elasticsearch can split the index into several "shards". In order to distribute the load, these shards can be divided into several "nodes". One node is defined as the "primary node". In most cases, this node is sufficient for small and medium-sized online shops.
If you want to dive even deeper into how Elasticsearch works, you can sign up for the Elasticsearch Tutorial here.
Which shop systems are compatible with Elasticsearch?
The efficiency of the search depends strongly on the connection of the respective shop. Depending on the software used, Elasticsearch is already supported natively or by 3rd party plugins:
|Magento 1||✓||Via Extension|
|Magento 2 - Community Edition||✓||Since Magento 2.3; Important: From version 2.4 the use of Elasticsearch as catalogue search engine is enforced!|
|Magento 2 - Commerce Edition||✓||Since Magento 2.1; Important: From version 2.4 the use of Elasticsearch as catalogue search engine is enforced!|
|Shopware 5||✓||From Shopware-Version 5.5|
|Shopware 6||✓||Elasticsearch 7.1 or higher|
Elasticsearch integration has been supported by Shopware since version 5.0. However, the integration is deactivated in the default setting. To activate it, the following code must be added to "config.php" in Shopware 5:
'es' => [ 'enabled' => true, 'number_of_replicas' => null, 'number_of_shards' => null, 'version' => '5.6.5', #<-- genutzte version einfügen 'dynamic_mapping_enabled' => true, 'client' => [ 'hosts' => [ 'localhost:9200' ] ] ],
The data can be transferred using the commands
php bin/console sw:es:index:populate and
php bin/console sw:es:backend:index:populate. You can find more information about this on Shopware Developers - for version 5 or for version 6.
Since version 5.5, Elasticsearch support has been greatly improved. For example, modules can now be indexed in the backend in addition to the search, which speeds up the process considerably.
Shopware 6 also supports the use of Elasticsearch. As usual with Shopware 6, this connection is set via the environment variables in the
.env. The following variables are inserted for this purpose:
SHOPWARE_ES_HOSTS "localhost:9200" SHOPWARE_ES_ENABLED 1 SHOPWARE_ES_INDEXING_ENABLED 1
Afterwards, the commands
php bin/console es:index and
php bin/console messenger:consume are to be executed in order to fill the indices.
Elasticsearch 6 is compatible with all versions of Shopware. Note that re-indexing may be necessary, as indexing is performed differently from Shopware 5.5.0. The Shopware demo shop (here in version 5.5) shows the Elasticsearch 6 supported search function with auto-completion. Elasticsearch 7.1 or newer is required for Shopware 6.
For Magento 1, we recommend the Magento extension from Wyomind. This plugin provides an Elasticsearch-based search with numerous additional functions such as auto-completion. You can see these in the Magento 1 demo shop (see Demo 2). After entering the article you are looking for in the search field, a window opens below the search field with suggestions for auto-completion. The item descriptions, a preview image and the price are displayed. These suggestions can be selected and clicked on.
The included Magento 1 search, without the Wyomind plugin, does not include autocompletion of possible search results (see demo 3). Only a suggestion for auto-completion of a possible search term is offered.
With version Magento 2.3 there is native support for Elasticsearch in the Community Edition - support is available for Elasticsearch 6.X. In the Commerce Edition of Magento, on the other hand, this integration has already been available since Magento 2.1. From version 2.4, which will be released in August 2020, Elasticsearch will even be a fixed Magento requirement.
Elasticsearch with own applications
Self-built applications (or applications without existing integration) can also be easily connected to Elasticsearch. Using the well-documented REST API, data can be stored and retrieved in Elasticsearch.
Besides the simple string comparison, Elasticsearch search and analysis can be extended with additional analysis plugins.
- The plugin "analysis-phonetic" allows to perform a similarity search by pronunciation
- With the "analysis-icu" plugin, support for Unicode and Asian languages is improved
- In addition, there are many plug-ins for individual languages. For example, the "analysis-kuromoji-readingform" plugin allows you to deal with the different spellings of the Japanese language (kanji, hiragana, katakana and romanji).
Alternatives to Elasticsearch
Just like Elasticsearch, Solr is based on Apache Lucene and both are offered under the same Apache 2.0 licence. Solr became open source in 2006 and has been actively supported by the Apache community ever since,
Solr not only offers a very detailed functional environment, but also numerous available plug-ins.
What are the advantages:
- Fast search even with millions of articles
- Multivalue filter
- High performance through local installation and Enterprise NVMe SSD
- Easily scalable and fault-tolerant
- Easy management via our Managed Center
- Reindex without interruption and in real time
- Flexible and simple configuration
Just like Elasticsearch, Sphinx is an open source application. You can either use it to quickly and easily batch index and search data stored in an SQL database, a NoSQL store or simply in files. Alternatively, you index and search data "on the fly" and work with it in a similar way to a database server.
The properties of Sphinx:
- Large databases can be searched efficiently and quickly
- Modern algorithms - stop words, stem form reduction and real-time indexing without problems
- Simple deployment
- Easy management via our Managed Center
- Flexible and simple configuration
The MySQL software provides a very fast, multi-threaded, multi-user and robust SQL (Structured Query Language) database server. Nevertheless, Elasticsearch is much faster than MySQL's full-text search when dealing with large amounts of data, so that product searches can be carried out much faster than with conventional search functions. However, this advantage only becomes apparent with a higher number of products. Nevertheless, the speed of Elasticsearch also seems to have convinced the minds behind Magento 2, because from version 2.4, which is planned for August, Elasticsearch will be mandatory as a search engine and replace MySQL.
Set up Elasticsearch at maxcluster
At maxcluster, Elasticsearch can be easily installed in the desired edition via the Managed Center. To do this, select the Elasticsearch menu item. Then install the desired version. Finally, create a new instance that can then be used by your shop system.
If you need additional plugins for certain integrations, we can install them for you on request at email@example.com.
With Elasticsearch, thesearch function of online shops that carry a large number of products can be greatly accelerated. In addition, the installation of the plugins is often already profitable for small online shops due to the customer-friendly search thanks to auto-completion.
You have questions, requests, criticism, suggestions or just want to tell us your opinion about our blog? Here you have the opportunity to contact us directly.Send e-mail