ADS - Arenadata Streaming

Live streaming and processing based on Apache Kafka and NiFi

• Enterprise communication bus for all applications
• Delimiting Access Rights to Data Streams

2025: Capture Data Changes

Arenadata has updated its Arenadata Streaming (ADS) streaming and data product. The company announced this on January 14, 2025. This version has received the functionality of capturing data changes (change data capture, CDC) and a monitoring service based on Prometheus. In addition, Arenadata Streaming now supports the concept of "multilayer" storage Kafka Tiered Storage and the Kafka Raft negotiation protocol.

To implement CDC, the Arenadata Streaming distribution includes two Debezium source connectors for the Kafka Connect service, designed to capture changes databases PostgreSQL in and. Microsoft SQL Server They collect data changes in, DBMS writing them to Kafka topics, and then various sink connectors can transfer these data to other databases and. storages It is possible to use Debezium connectors available out of the box, add your own and configure them from the ADS Control web interface - systems for managing streaming clusters.

This version of Arenadata Streaming implements the concept of Tiered Storage - "multilayer" storage of data in Kafka topics. The incoming hot data is located in the local file system, and then, according to the specified policies, it is transferred to an external system, which is better adapted for the long-term storage of cold data. Arenadata Streaming supports HDFS1 and S3 as an external system.

Tiered Storage reduces costs and provides additional flexibility when historical data storage is required. Due to the limited set of local segments, the need for premature scaling is reduced, the time for expanding and rebalancing the cluster, as well as for its recovery from failures and maintenance, "said Alexander Anisimov, Technical Director of Arenadata Streaming.

The monitoring service is included in Arenadata Streaming. It deploys its Prometheus server to ADS, which collects and stores cluster monitoring metrics (system as well as service metrics Kafka, ZooKeeper, Kafka REST Proxy, Schema-registry, ksqlDB, NiFi, MiNiFi, Kafka Connect). You can view and analyze the collected data in the Prometheus web interface or in the Grafana web application.

The updated version of Arenadata Streaming supports the Kafka Raft Negotiation Protocol (KRaft), which allows you to manage metadata inside Kafka, eliminating dependence on the external ZooKeeper service. With KRaft, Kafka is easier to use, System Recovery Time Target (RTO) is reduced, and the available number of batches supported by a single cluster is multiplied. In the current version of the product, the functionality is provided in preview mode and is not intended for use in the production environment.

2024

Java 17 Support

Arenadata has updated the Arenadata Streaming (ADS) streaming and data processing product and its complementary Arenadata Streaming Control (ADSC) system, designed to manage and monitor streaming clusters. The company announced this on October 24, 2024. Releases include a number of improvements in the context of security, stability and ease of administration.

This version of Arenadata Streaming supports Java 17, which contains improvements to improve performance and security. Java 17 is a long-term support (LTS) version that will receive updates for a long time.

Arenadata Streaming includes the kafka-rest-security plugin, designed for authorization in the Kafka REST Proxy service. By default, the user can authenticate to Kafka with only one predefined principal created when Kerberos is enabled. The plugin developed by the Arenadata team removes this restriction and allows authentication with different principals, which makes it easier to control user access to resources.

The release included an update to versions of all Arenadata Streaming components and services, as well as a number of improvements and bug fixes. In particular, the Kafka service configuration page has been updated and work with user parameters has been optimized.

When creating a roadmap for product development, we take into account the requirements of the security services of corporate customers. As part of these requirements, the new release included the ability to distinguish between Kafka REST Proxy users' access to Kafka topics, "said Alexander Anisimov, technical director of Arenadata Streaming.

In addition, the Arenadata Streaming Control service has been updated, which allows administrators to manage all aspects of ADS Kafka clusters. The management of Kafka Connect connectors has also been improved, expanding the broker's ability to access external systems.

The web interface of the updated version of the service has a Consumer groups page containing information about connected consumer groups in the cluster. In addition, ADSC implements tabs for the Topics page, designed to display information about topics and manage them in a cluster: the Messages tab allows you to read and write messages, and Consumer groups shows groups of users connected to the topic.

MWS Cloud Availability

MTS, a digital ecosystem, on July 26, 2024 announced the conclusion of a strategic partnership between MTS Web Services (MWS), part of the MTS group, and Arenadata. As part of the agreement, MWS launched five services on software products from Arenadata, including Arenadata Streaming (ADS). Read more here.

Availability from the CROC Cloud

Croc Cloud Services and Arenadata entered into a partnership agreement in which Arenadata products became part of the services provided to customers on the basis. CROC clouds In particular, provider Arenadata Streaming (ADS) is now available to cloud customers. CROC announced this on January 17, 2024. More. here

2023

Red OS Support

Arenadata on November 1, 2023 released an updated release of the Arenadata Streaming (ADS) data streaming product, which included support for the Russian operating system RED OS.

Thus, the list of operating systems supported by ADS was expanded, which now includes:

• RED OS 7.3;
• CentOS 7.6.1810;
• RHEL 7.6;
• AltLinux 8.4 SP;
• Astra Linux 1.7 "Eagle" SE with Axiom JDK.

Another important change in Arenadata Streaming for users was the implementation of encryption of NiFi configuration parameters, designed to increase cluster security.

In addition, the updated version of ADS included the NiFi update to version 1.20, which added additional processors and fixed a number of information security issues.

For Arenadata customers, the ability to influence the development map of our products is important, and this Arenadata Streaming release confirms this. So, in response to the wishes of one of our customers, an additional flow provider GitFlow Provider was added to NiFi, - commented Alexander Ermakov, technical director of Arenadata.

Cloud Availability Cloud.ru

and Provider cloudy AI-technology Cloud.ru has become a strategic partner Russian of the vendor ON for storages and processing. data Arenadata Arenadata products can now cloud Cloud.ru be used in. Arenadata announced on August 29, 2023.

According to TAdviser representatives of Arenadata, the following products of the company are available in the Cloud.ru cloud: ADB - Arenadata DB, ADH (Arenadata Hadoop), ADQM (Arenadata QuickMarts) and ADS (Arenadata Streaming). Read more here.

Support for Astra Linux Special Edition "Eagle"

Arenadata has released an updated product release for streaming Arenadata Streaming (ADS), which includes support for the Astra Linux Special Edition operating system (Eagle release). The functionality is already available to users. Arenadata announced this on August 10, 2023.

Compatibility with the Russian operating system Astra Linux means that the product is suitable for a large range of corporate IT infrastructures, including those in need of increased security.

For Arenadata customers, a high level of information security is important. And this release of Arenadata Streaming, which received support for Astra Linux, will allow them to use a certified operating system. To do this, we did a lot of work in terms of changing the configuration mechanisms of the service and component management system for the operating system, - commented on the update of the ADS functionality Alexander Ermakov, technical director of Arenadata.

This Arenadata Streaming release has included other changes. The user management functionality in the interface of the universal orchestrator of the hybrid landscape Arenadata Cluster Manager (ADCM) and individual service settings for one or a group of hosts were redesigned. And also fixed a number of known problems.

Arenadata Streaming is a scalable fault-tolerant system for real-time streaming, adapted for enterprise use and built on the basis of Open source technologies Apache Kafka and Apache Nifi. The product includes a monitoring and management system, advanced security support, as well as a number of additional tools for implementing replication, data quality verification and other functions necessary for product operation.

Arenadata Streaming allows you to:

• Build real-time streaming pipelines that reliably transfer data between systems or applications.
• develop real-time streaming applications that convert or respond to data streams;
• Store record streams in a fail-safe, durable manner
• Differentiate between access rights to data streams.

Arenadata Streaming has received a certificate of state registration of the computer program. In addition, the product is included in the unified register of Russian programs for electronic computers and databases.

Availability in beeline cloud

The product ecosystem Arenadata is now available to customers. cloudy provider beeline cloud Arenadata announced this on August 3, 2023. As TAdviser representatives of Arenadata specified, among the products available through beeline cloud:,, Arenadata DB Arenadata Hadoop Arenadata Streaming,, and. Arenadata Postgres Arenadata QuickMarts Arenadata Cluster Manager Read more here.

Rubbles MLOps Suite Compatibility

IT companies Rubbles and Arenadata have made their software products compatible. Now the Rubbles MLOps Suite platform and Arenadata data storage and processing solutions can work seamlessly in a single software package. Arenadata announced this on July 19, 2023.

A single software package combines Rubbles MLOps solutions and Arenadata products, including Arenadata Streaming. Read more here.

Release of Apache Kafka Connector Monitoring and Management Tool as part of Arenadata Streaming

Russian developer of the collection storages and processing platform - data the company Arenadata announced on January 11, 2023 the release of market a connector monitoring and management tool Apache Kafka as part of the Arenadata Streaming (ADS) product Arenadata Streaming Command Center. It was named (ADSCC). ADSCC supports the management of multiple clusters with Kafka services and Kafka Connect installed, and provides the ability to create, edit, delete and manage Kafka Connect connectors.

Arenadata Streaming Command Center complements ADS and allows you to configure real-time data transfer time between source and receiver using - and source sink - connectors, get a quick overview of the cluster state, and observe and control Apache Kafka connectors to sources and receivers. The most popular scenarios for using this tool are setting up real-time data transfer between different systems, replication and Disaster Recovery of Kafka clusters.

By releasing the Command Center, the company allows the customer to quickly configure the Disaster Recovery Kafka cluster and connect external connectors (for example, Debezium to configure the CDC). At the same time, we lay the foundation for the development of functionality for customer requests with vendor support, as well as the inclusion of other versions of Kafka with KRaft support in the ADS product, explained the director of the Arenadata product management department Alexander Bolshakov.

Previously, when working with Open Source Kafka Connect, users needed to manually create configuration files and figure out which parameters are important and which are not. This took up a lot of resources from the system administrator, there was a high risk of errors, in addition, the administrator needed to know the Kafka Connect API and do custom monitoring of connectors.

Arenadata Streaming Command Center allows you to perform all these manipulations directly from the user interface. Almost instantly, you can organize the transfer of data from one point in the architectural landscape to another, and now it is also possible:

• Add and manage Kafka connectors
• Monitor the status of Kafka Connect clusters
• Work with multiple clusters at the same time
• Flexible configuration of connector parameters (select connector type, group parameters, JSON mode for deep configuration)
• Work with custom connectors.

Arenadata Streaming Command Center is available as a separate bundle as part of the enterprise version of Arenadata Streaming starting with version 1.7.1. You can install the bundle through the Arenadata Cluster Manager.

In the immediate development plans of Arenadata Streaming Command Center, the implementation of the following functionality:

• LDAP adding/LDAPS -;authentications
• monitoring (number of messages, throughput, etc.) and topic management;
• reassigning partitions to other nodes;
• adding dashboards and graphs;
• select leader replica.

2022: Kerberos-authorization to prevent unauthorized access

On February 24, 2022, Arenadata announced that it had provided users with three of its products - Arenadata Hadoop (ADH), Arenadata Streaming (ADS) and Arenadata Platform Security (ADPS) - the ability to prevent unauthorized access to services and cluster data. Now it is enough to enable Kerberos authorization for all components, which will allow you to store authorization data in Active Directory. Read more here.