Very Fast Business Intelligence Tool: Arenadata QuickMarts

||Anton Kovalenko Director, Arenadata Product Marketing Department

In today's market, you can find many variants of specialized ones DBMS designed to effectively solve certain problems. One such DBMS is a product Arenadata QuickMarts (ADQM) developed on the basis of technology c. open source ClickHouse According to the results of the synthetic test and the experience of real use, ClickHouse and ADQM can perform analytical requests many times faster than many other solutions and technologies present on the international and Russian market. How else ADQM differs from similar products, where it is used and what results helps to achieve business, we will tell in this article.

About ADQM: Capabilities, Application Scenarios, Benefits

ADQM is a cluster column DBMS based on ClickHouse technology, which has become actually an industry standard for implementing operational business intelligence tasks on large amounts of data. The product is suitable for companies that need quick analytics and can be applied in various areas of business.

The product is optimized for storing information in column formats, which allows you to increase the speed of analytical reports up to 2.5 times. Such results are especially in demand in the following data applications:

• Implementation of the DataMart layer within the Enterprise Data Warehouse (QCD).
  ADQM allows you to analyze a large amount of data many times faster than traditional DBMS, while reducing the load on the main storage.
• Timeseries
  . DBMS is effective for storing and analyzing time series, for example, in tasks of analyzing data from financial markets or industrial solutions of the Internet of Things (IIoT).
• Hypothesis testing for ML tasks.
  ADQM has built-in models for using ML algorithms and can become a data source for them.
• Log analysis.
  DBMS allows efficient collection of metrics and event logs from different sources for further analysis.
• and E-commerce finance.
  The solution implements the ability to quickly save e-commerce data, such as business transactions, user behavior and, and advertizing data creates flexible BI reports based on them.

It is also worth noting that by developing ADQM, Arenadata developers are actively contributing to the ClickHouse kernel. In 2023, the company entered the top 5 main contributors to the project with source code among the global community. Specialists know the technology on the basis of which the product is built, and are able to orient it to the needs of customers.

ADQM based on the growing worldwide ClickHouse technology is considered by the company as one of the key and promising components of the Arenadata platform in terms of its further development. For this reason, over the past few years, the company has been investing significant resources in the refinement of this product, expanding the team and solving increasingly complex and ambitious tasks in terms of its adaptation for use in the corporate segment, - said Anton Kovalenko, Director of Product Marketing at Arenadata.

Among the key benefits of ADQM are:

• ADQM received a certificate of compliance with the FSTEC of the Russian Federation.
  The product is certified according to the requirements of the sixth level of trust and technical specifications and meets the highest legal requirements for domestic solutions. Based on the technical conclusions, Arenadata QuickMarts is included in the state register of the information security certification system for information security requirements dated June 8, 2023
• Original Russian-language documentation and technical support.
  For ADQM, as for all Arenadata products, original documentation in Russian and English has been created and constantly maintained, and the company's technical support service quickly responds to customer problems of any level and helps to solve them within the framework of SLA.
• Consulting services.
  The company's architects and engineers help customers with system design, technical analysis and communication of business objectives and technical implementation.
• ADQM can be used in conjunction with Arenadata DB (ADB).
  ADQM can be used independently, but the technology has a number of limitations that do not allow it to be used as a full-fledged QCD. For such tasks, we recommend that customers use ADQM and ADB products together using the new version of the Tkhemali connector.
  This connector is used to transfer data between ADB and ADQM (ClickHouse). The tool is implemented on the basis of foreign data wrapper (a tool that allows you to work with data from different sources) and foreign tables (external tables). This simplifies the syntax and provides the ability to conveniently and securely manage credentials, provide tiered connection settings, efficiently distribute loads across hosts, flexibly manage custom settings, and control resource allocation.

In addition, the technical maturity and functionality of the solution allows you to use ADQM to replace the products of many foreign vendors, including Oracle Exadata, Microsoft SQL, Teradata, SAP BW/4HANA, Vertica, Amazon Timestream and InfluxDB Enterprise.

ADQM Application: Company Case

As we noted above, ADQM is suitable for companies from different areas of business, including industry, FMCG, retail and e-commerce, banks, telecommunications and the public sector.

One of the company's clients is the federal department, which decided to build a new unified integration and analytical platform. The task arose in connection with the obsolescence of equipment, accelerated data growth, almost complete disposal of the use of database machine resources and a ban on further updates of Teradata systems.

The basis of the new platform was a data lake with a corresponding set of data layers, operating on the basis of another Arenadata product - Arenadata Hadoop. Arenadata Streaming, Arenadata DB, Arenadata QuickMarts and Picodata were also used for the creation.

As part of the project, it was necessary to transfer from Teradata systems to a new repository of 30 applications of analytical tasks, about 50 reporting forms, over 48 thousand objects - a total of approximately 95 TB of data located on two software and hardware complexes of 66.0 and 28.67 TB.

First, the Arenadata software was migrated to resource-intensive tasks, which made it possible to remove a high load from the current industrial contour, where the Teradata database machines continued to operate, and ensure optimal operation until the migration processes were completed. In the next stage of migration, blocks of complex and interrelated tasks in terms of data and processes passed to Arenadata products.

The import substitution process in this department is still ongoing. At the moment, work is underway to replace Oracle Exadata, and soon this system will cease to be a data source for the analytical platform.

The company's immediate plans include the completion of the ADQM certification procedure in FSTEC for the fourth level of trust for its use in the framework of ZOKII, as well as the implementation of the new ADQM Control module, which will significantly expand the capabilities of customers in terms of monitoring and managing ADQM clusters, - sums up Anton Kovalenko.