Elasticsearch - What are the advantages for online stores?

Speed can become a knockout criterion when users get tired of the "long" wait, leave the site and possibly migrate to a competitor. But it is not only (milli)seconds that count to keep customers on your site. The quality of the search results is also crucial.

To ensure the highest possible speed and quality of search results, powerful search engines must be integrated into the website. One of the most important full-text search engines is Elasticsearch, which is used by companies such as Facebook, Netflix, Zalando and GitHub.

How does full-text search work with Elasticsearch? What are the advantages for online shop and website operators? And what alternatives are there to this very popular search engine? We clarify these and other questions here.

What is Elasticsearch?

Elasticsearch is both an open-source search engine and an analytics engine that has been available on the market since 2010. It is based on Apache Lucene and can be used for data of all kinds: Text data, numeric data, geodata and both structured and unstructured data. Its range of application is thus (theoretically) unlimited. For example, geodata can be used to display offers according to distance from the user. Elasticsearch thus offers online shops many opportunities for creative sales increases.

Advantages of

Elasticsearch offers many advantages as a search engine:

• Distributed nature: Individual nodes taken together form a cluster, which makes it possible to distribute even large amounts of data via sharding. To cushion high request loads, data can also be multiplied across multiple nodes.
• Simple operation
• Fast and stable at the same time
• Offers a wide range of functions: Data Roll-ups, Index lifecycle management
• Easy data ingestion, visualisation and reporting with Kibana
• Variety of supported programming languages such as Java, JavaScript (Node.js), PHP, Python and Ruby
• 24 text languages
• free of charge under Apache 2 licence
• persists data, therefore it can also be used for longer-term storage

Disadvantages of Elasticsearch

With more technologies in use, the setup for an online shop or blog becomes correspondingly more complex and unfortunately Elasticsearch is not available in all environments.We have compiled a list of all shop systems compatible with Elasticsearch here.

Simple managed hosting environments in particular do not offer Elasticsearch in all cases. Even with self-hosted servers, the installation and maintenance of Elasticsearch generates additional expenses that should not be neglected.

However, those who are not put off by this additional effort will notice a significant acceleration of the search with Elasticsearch and thus potentially a recognisable increase in the conversion rate.

How does Elasticsearch work?

With Elasticsearch, raw data from various sources, such as logs, system metrics and web applications, are combined in an ingestion process and enriched with further information if necessary. The data is then stored as JSON documents, which in turn consist of a set of pairs of keys and values. In order for search queries to deliver results later, the data is indexed, for which Elasticsearch uses a data structure called inverted index.

Instead of examining all available documents for a search query, Elasticsearch checks an index in which all content has been previously stored in a prepared form. For this process, Elasticsearch creates one (or more) indices, each of which functions like a kind of database and is then later searched by the search engine.

The index created in this way is divided into several types. Documents can now be stored in these indices, each of which can be assigned to a type. In contrast to classic SQL databases, the documents do not have a fixed format, but can consist of any structure of attributes. This enables a fast full-text search and the browsing of document data in (almost) real time.

To store the data, Elasticsearch can split the index into several "shards". In order to distribute the load, these shards can be divided into several "nodes". One node is defined as the "primary node". In most cases, this node is sufficient for small and medium-sized online shops.

If you want to dive even deeper into how Elasticsearch works, you can sign up for the Elasticsearch Tutorial here.

Which shop systems are compatible with Elasticsearch?

The efficiency of the search depends strongly on the connection of the respective shop. Depending on the software used, Elasticsearch is already supported natively or by 3rd party plugins:

Shopware 5

Elasticsearch integration has been supported by Shopware since version 5.0. However, the integration is deactivated in the default setting. To activate it, the following code must be added to "config.php" in Shopware 5:

    'es' => [
        'enabled' => true,
        'number_of_replicas' => null,
        'number_of_shards' => null,
        'version' => '5.6.5', #<-- genutzte version einfügen
        'dynamic_mapping_enabled' => true,
        'client' => [
            'hosts' => [
                'localhost:9200'
            ]
        ]
    ],

The data can be transferred using the commands php bin/console sw:es:index:populate and php bin/console sw:es:backend:index:populate. You can find more information about this on Shopware Developers - for version 5 or for version 6.

Since version 5.5, Elasticsearch support has been greatly improved. For example, modules can now be indexed in the backend in addition to the search, which speeds up the process considerably.

Shopware 6

Shopware 6 also supports the use of Elasticsearch. As usual with Shopware 6, this connection is set via the environment variables in the .env. The following variables are inserted for this purpose:

SHOPWARE_ES_HOSTS "localhost:9200"
SHOPWARE_ES_ENABLED 1
SHOPWARE_ES_INDEXING_ENABLED 1

Afterwards, the commands php bin/console es:index and php bin/console messenger:consume are to be executed in order to fill the indices.

Elasticsearch 6 is compatible with all versions of Shopware. Note that re-indexing may be necessary, as indexing is performed differently from Shopware 5.5.0. The Shopware demo shop (here in version 5.5) shows the Elasticsearch 6 supported search function with auto-completion. Elasticsearch 7.1 or newer is required for Shopware 6.

Magento 1

For Magento 1, we recommend the Magento extension from Wyomind. This plugin provides an Elasticsearch-based search with numerous additional functions such as auto-completion. You can see these in the Magento 1 demo shop (see Demo 2). After entering the article you are looking for in the search field, a window opens below the search field with suggestions for auto-completion. The item descriptions, a preview image and the price are displayed. These suggestions can be selected and clicked on.

The included Magento 1 search, without the Wyomind plugin, does not include autocompletion of possible search results (see demo 3). Only a suggestion for auto-completion of a possible search term is offered.

Magento 2

With version Magento 2.3 there is native support for Elasticsearch in the Community Edition - support is available for Elasticsearch 6.X. In the Commerce Edition of Magento, on the other hand, this integration has already been available since Magento 2.1. From version 2.4, which will be released in August 2020, Elasticsearch will even be a fixed Magento requirement.

Further functions as well as support for other Magento 2 versions are also offered by the free open source plugins Elasticsuite by Smile, Wyomind and Amasty.

Elasticsearch with own applications

Self-built applications (or applications without existing integration) can also be easily connected to Elasticsearch. Using the well-documented REST API, data can be stored and retrieved in Elasticsearch.

Besides the simple string comparison, Elasticsearch search and analysis can be extended with additional analysis plugins.

• The plugin "analysis-phonetic" allows to perform a similarity search by pronunciation
• With the "analysis-icu" plugin, support for Unicode and Asian languages is improved
• In addition, there are many plug-ins for individual languages. For example, the "analysis-kuromoji-readingform" plugin allows you to deal with the different spellings of the Japanese language (kanji, hiragana, katakana and romanji).

Alternatives to Elasticsearch

If you want to speed up the full-text search on your own website, you can also use Solr or Sphinx in addition to Elasticsearch.

Solr

Just like Elasticsearch, Solr is based on Apache Lucene and both are offered under the same Apache 2.0 licence. Solr became open source in 2006 and has been actively supported by the Apache community ever since,

Solr not only offers a very detailed functional environment, but also numerous available plug-ins.

What are the advantages:

• Fast search even with millions of articles
• Multivalue filter
• High performance through local installation and Enterprise NVMe SSD
• Easily scalable and fault-tolerant
• Easy management via our Managed Center
• Reindex without interruption and in real time
• Flexible and simple configuration

Sphinx

Just like Elasticsearch, Sphinx is an open source application. You can either use it to quickly and easily batch index and search data stored in an SQL database, a NoSQL store or simply in files. Alternatively, you index and search data "on the fly" and work with it in a similar way to a database server.

The properties of Sphinx:

• Large databases can be searched efficiently and quickly
• Modern algorithms - stop words, stem form reduction and real-time indexing without problems
• Quick
• Simple deployment
• Easy management via our Managed Center
• Flexible and simple configuration

MySQL

The MySQL software provides a very fast, multi-threaded, multi-user and robust SQL (Structured Query Language) database server. Nevertheless, Elasticsearch is much faster than MySQL's full-text search when dealing with large amounts of data, so that product searches can be carried out much faster than with conventional search functions. However, this advantage only becomes apparent with a higher number of products. Nevertheless, the speed of Elasticsearch also seems to have convinced the minds behind Magento 2, because from version 2.4, which is planned for August, Elasticsearch will be mandatory as a search engine and replace MySQL.

Set up Elasticsearch at maxcluster

At maxcluster, Elasticsearch can be easily installed in the desired edition via the Managed Center. To do this, select the Elasticsearch menu item. Then install the desired version. Finally, create a new instance that can then be used by your shop system.

If you need additional plugins for certain integrations, we can install them for you on request at support@maxcluster.de.

Conclusion

With Elasticsearch, the search function of online shops that carry a large number of products can be greatly accelerated. In addition, the installation of the plugins is often already profitable for small online shops due to the customer-friendly search thanks to auto-completion.

If you have any questions about the implementation or need other support, we look forward to receiving your enquiry via e-mail at support@maxcluster.de or by telephone at 05251 / 41 41 30.