Part 4: Profiler in comparison - Blackfire IO, Tideways, New Relic APM

08.01.2018

Three of the most important profilers in e-commerce are Blackfire IO, Tideways and New Relic APM. The comparison clearly shows where the strengths of the individual tools lie.

In previous posts in this blog series, we explored the questions: Why should a special e-commerce profiler be used for online stores? Which profilers are particularly efficient for online stores? What are the detailed functions and applications of these e-commerce tools? For this, we have presented Blackfire IO, Tideways and New Relic APM individually.

But which profiler fits the personal requirements? Where are the strengths and weaknesses in a direct comparison?

In this post, Blackfire IO, Tideways and New Relic APM are compared. The table below shows an overview of the main features: Performance Monitoring, Alerting, Overview, Profiling, User Management and Cost. The individual functions are then compared with each other in detail and a balance is subsequently drawn.

Profiler comparison table

Function	Blackfire	Tideways	New Relic
Performance Monitoring	Automatic, regular profiling of important pages through custom scenarios	Constant, passive profiling of all customer requests	Constant, passive profiling of all customer requests
Alerting	An alarm can only be configured in the scenarios	Alarm about the response time or error rates of customer requests	Alarm about the response time or error rates of customer requests. Also dynamic, in case of deterioration of response time.
Overview	Last results and references	Load times, requests and last measurements	Load times, score, division of database requests, division of web calls.
Profiling	Detailed profiling of time, CPU, I/O wait, memory, network calls, HTTP and SQL requests. Each in different views	Detailed profiling over time in different views. Memory profiling. Display of SQL requests and external resources.	No complete profiling possible, but performance tracings for slow requests as well as an overview of the timing of calls are possible
User management	Starting with the Premium Edition, a user administration for up to 3 users is available. Additional users can be activated for an additional charge.	Good user management	Very good user management. Possibility to belong to different organizations with one New Relic account.
Costs	0€ /month / local development server 19, 90 € / month/ production server 82,50€ / month / 1 environment, 3 collaborators	89 € / month / 5 server 159 € / month / 10 429 € / month / 20 Server Developer Editionen: 129 € / month/ 10 applications 499 € / month / Unlimited applications	75$ – 149$ / month
maxcluster support	Can be set up quickly and easily via the Managed Center. Support from maxcluster if required	Can be set up quickly and easily via the Managed Center. Support from maxcluster if required	Can be set up quickly and easily via the Managed Center. Support from maxcluster if required

With the "Performance Monitoring" function, the profilers monitor online stores as well as other web applications. Depending on the profiler, performance monitoring can apply to the entire store or to individual pages of the store.

Tideways and New Relic APM work with passive profiling. This means that the function is constantly running in the background and does not need to be activated separately. All customer requests are monitored. The advantage of constant, passive monitoring is that it can uncover problems that were not previously assessed. This includes, for example, customers behaving unpredictably differently than previously expected.

In the area of monitoring, Blackfire offers the smallest scope compared to the other profiling tools. With this tool, monitoring is possible from the Premium Edition through the "Scenarios" function. With the scenarios, the behavior of a customer can be simulated. In the scenarios, the pages of the online store must be explicitly defined for constant monitoring.

Alerting

An alert is a warning sent to the technically responsible person as soon as a metric changes negatively below a defined threshold value. It is possible to set certain reference values, e.g. in the area of response times. If these reference values are undercut or exceeded, a warning is sent via the previously defined communication channel (e.g. by e-mail).

The alarms are most extensively adjustable with New Relic APM. Here it is possible to set an alarm via the response time or via the error rates. The program (Pro Edition and higher) already detects a dynamic deterioration of the response time and issues an alert about this as well.

Tideways also has alarms on response times or error rates. In contrast to New Relic APM, however, the profiler does not detect dynamic degradation, but only alerts when the defined benchmark values are reached.

Blackfire has the least scope of the three profilers in this function. Only from the Premium Edition can tests be created in Blackfire, which are in turn combined into scenarios. An alarm can only be set via these scenarios. If such a test fails according to previously defined metrics (response time, memory consumption, etc.), a defined person can be informed and view the scenario report.

Overview

The overview clearly presents the individual functions of the respective profiler. New Relic APM has the greatest user-friendliness in this area. The overview is well structured in this tool, so that the individual functions can be called up quickly and easily. These include load times, score, breakdown of database requests, and breakdown of web page views.

Tideways concentrates on the essential functions in the overview. Two graphs provide a quick overview of the current performance. In the tables on the overview page, manual and automatic traces can be easily accessed.

With the very tidy design of Blackfire, the traces and existing reference profiles can be called up quickly and easily. Other functions can still be accessed via the menu navigation.

Profiling

The strengths of Blackfire and Tideways are clearly in profiling.

With Tideways and Blackfire, detailed call graphs and tables can be created. The main bottom line differences are in the display. Both tools have the ability to display the runtime of individual functions and locate slow database queries and external resource calls. Blackfire offers the additional option of measuring memory usage, CPU time and IO-Wait for individual PHP functions in addition to the classic profiling. Tideways, on the other hand, also has memory profiling, which offers support for memory profiling in timeline and callgraph mode and can be used to display the memory of different function calls, among other things.

New Relic APM can be used to create performance tracings for slow requests. In addition, an overview of the timing of calls can be created.

User management

New Relic APM offers the strongest management. Especially interesting for agencies is the fact that you can belong to several organizations with a single account.

Tideways also has a very clear administration. It is possible to assign individual rights to different users.

Blackfire only has user management from the Premium Edition onwards. Three users can work on a project at the same time. Beyond that, however, an additional fee is charged for each additional user.

Costs

In the area of costs, Blackfire is right in front with the free "Hack Edition". This already allows for basic profiling to detect bottlenecks. Please note that we no longer support this edition. The Profiler Edition can be implemented from €19.90 per month for production servers. The Premium Edition is available from €82.50 per month.

The cost of New Relic APM ranges from $75 to $149 per month depending on the edition. According to the current currency, this amounts to about 68 € to 136 €. New Relic can be tested free of charge for 15 days.

The German profiling and monitoring tool Tideways costs between 89 € and 499 € per month, depending on the edition and number of servers. However, it is possible to test the functions of Tideways for 14 days.

Integration at maxcluster

All three profiling tools are integrated into our e-commerce stack due to their high efficiency. After purchasing a license on the respective profiler website, the desired profiler can be activated in our Managed Center with a simple mouse click. Only the respective license key has to be entered.

Conclusion

All three of the aforementioned profilers are well suited for finding bottlenecks in the online store.

Which of these three tools is best suited for one's own online store depends very much on the requirements of the store operator. If, for example, there is a loading time problem in the online store whose origin is not easy to find, Tideways and Blackfire are particularly suitable. With the extensive profiling functions, error sources can be quickly identified. With Blackfire, even more specialized profiling can be performed, such as memory profiling.

If the focus of the requirement is on continuous monitoring of the store, New Relic APM is particularly recommended. The strength of this profiler lies in monitoring. Of all three profilers, New Relic APM has the most comprehensive monitoring capabilities.

However, if you need a mixture of monitoring and profiling, Tideways is a suitable tool. Tideways has both detailed profiling capabilities in the area of time and passive monitoring including alerts.

But in some cases, a combination of two profiling tools is also useful. For example, long-term monitoring by New Relic APM can be combined well with the detailed profiling of Tideways or Blackfire.