Yandex Database (YDB)

Yandex Database (YDB) is a distributed fault-tolerant NewSQL DBMS developed by Yandex. According to the developers, it can be an alternative to existing solutions in the following cases:

• In systems that require support for ACID transactions between tables with an uneven time load;
• In systems that require processing and storage of a large amount of data;
• When developing a new product, for which, already at the design stage, a large load is expected with a high probability (thousands of requests per second to the database) and the requirement of strict consistency;
• As a backend for databases with a poorly predictable load.

2024

Support for analytical data processing

Yandex December 20, 2024 announced the launch of the YDB DWH component to support analytical data processing in Yandex DBMS (YDB). It supports complex analytical queries (OLAP), is capable of storing hundreds of terabytes of data, scales horizontally due to automatic volume and load partitioning (sharding), and provides fast query execution thanks to a massively parallel architecture (MRP).

The analytical functionality of the YDB DWH component includes column tables with consistent data storage and mixed load management. The Yandex DBMS platform also includes an engine for executing queries with spilling data to disk, a cost optimizer, federated SQL queries to external data sources (S3, PostgreSQL, Greenplum, Oracle, MS SQL), which allows you to implement a scenario for creating an enterprise data warehouse. There are plans to implement SQL support for the PostgreSQL dialect for seamless integration in both OLAP and OLTP scripts.

The platform is distributed under a commercial license for installation in the customer's loop, as a managed service in Yandex Cloud.

Thanks to the launch of the YDB DWH component, Yandex DBMS has become a universal solution for high-performance transaction processing and analytical requests. We are ready to share this solution with a market that obviously lacks universal DBMS that can work in business-critical scenarios. The competence centers created by Yandex and the team that has been developing DBMS for more than 10 years will allow customers to quickly achieve results during migration. This is especially true for solutions in the financial, telecommunications and retail sectors, "said Aleksei Kurochka, commercial director of YDB.

According to Yandex, the market for data processing systems in Russia in 2025 will amount to 55 billion rubles. The share of analytical DBMS will be 32% of the total data processing market.

In addition to YDB DWH, Yandex's line of licensed DBMS add-ons also includes the YDB Topics data bus and the YDB Search machine learning task add-on. The main component of the platform is the YDB Enterprise component for high-performance transaction processing.

Based on PAC for business-critical OpenYard systems

Server hardware manufacturer OpenYard will launch a hardware and software suite for business-critical systems based on YDB technology. This industrial solution is designed for highly loaded corporate infrastructures with increased requirements for reliability and fault tolerance - for example, large banks, industrial, telecommunications companies. The hardware and software complex will be available for pre-order as early as 2024. OpenYard announced this on October 11, 2024. Read more here.

Ex-director of Oracle Systems in Russia appointed commercial director of Yandex DBMS 

In June 2024 , Aleksei Kurochka headed the commercial department of Yandex's YDB database management system. Read more here.

2022: DBMS Source Disclosures

On April 20, 2022, Yandex announced the release of the YDB database management system (Yandex Database) to the public. Code, as well as documentation, SDK tools and all database solutions are provided under an open license Apache 2.0. The documentation is published in the GitHub repository.

{{quote 'Opening a project creates an interesting win-win situation for everyone. The community, on the one hand, has the opportunity to use unique developments in which Yandex has invested hundreds of man-years, get acquainted with the code, freely launch and develop solutions based on YDB, "Yandex said in a statement on Habra. }}

Yandex revealed the source code of its DBMS

Another reason for the discovery of DBMS sources in the company was called a strong increase in user variability. Yandex will be able to receive feedback from the world community and improve the product. The company added that it is necessary to eliminate the barrier for users who are interested in the technology, but stop for fear of being closed or unable to use it on their equipment or in their clouds.^[1]

We are confident that the rapid development of technology that we have seen in recent decades would not be possible without the use of open source code. The availability of technologies promotes their dissemination and stimulates interest in them. For example, you can no longer imagine the Internet without databases such as MySQL, PostgreSQL or ClickHouse. By publishing the YDB code, we continue to follow the principles of openness and share with all the developments that we have spent almost ten years on. Now the technology that allows Yandex to quickly scale and move forward will be available to everyone under the free Apache 2.0 license, "said Alexey Bashkeev, CEO of Yandex Cloud.

It is noted that by April 2022, Yandex has been using the YDB system for more than five years, projects in it are hosted by the teams Alice, Taxi, Market, Metrics and other services. There are nearly 500 projects in YDB. You can deploy on your own or third-party servers - including Yandex Cloud or other cloud services. The company noted that YDB runs on standard hardware, it can be deployed everywhere using the Kubernetes operator or manually, while the code of most systems of leading cloud providers is closed, and some of these products are tied to specialized equipment. According to the developers, the YDB DBMS can handle millions of requests per second.^[2]

Yandex systematically uploads open source to its key technologies. So, in the public domain there is a library that machine learning CatBoost is used Search in other Yandex services. In 2016, the company published the source code - ClickHouse high-performance analytical. database management systems

2020: General availability in Yandex.Cloud

On September 23, the new database management system Yandex Database (YDB) became publicly available Yandex. began offering it as a service in both regular and serverless variants on its cloud platform. Yandex.Cloud

The Data Base belongs to the class of NewSQL solutions, which was conceived as a DBMS for cases when there is a lack of performance or functionality of classic DBMS, says Alexey Bashkeev, head of the Yandex.Cloud platform. Yandex itself faced this many years ago. Therefore, the company developed and began to use this DBMS, he explained. Now this system is used in Yandex.Search, Yandex.Cloud, voice assistant Alice, Yandex.Collections, Yandex.Messenger, on the Авто.ру website

For external users, since the announcement in 2019, YBD has been available in Yandex.Cloud in a non-commercial on-demand preview mode. Alexey Bashkeev told TAdviser that it was then made available in the cloud to test the demand for these solutions. Several hundred companies have shown interest, he said.

One of the main types of demand that we felt was large implementations, where the very capacities or approaches of standard DBMS are not enough and you want something else. At the same time, we saw that the market for serverless solutions is also growing. Therefore, we added such an opportunity to YDB, "he explained to TAdviser.

Yandex Database left the preview stage "(image - studio of Yulia Bedrosova)"

The software interface of the serverless version of YDB is compatible with the API of the popular serverless solution Amazon DinamoDB. This will significantly expand the boundaries of YDB applicability, the company expects. Oleg Koverznev, Operations Director of Yandex.Cloud, says that the company is not in vain ensuring the compatibility of its DBMS with the global database, which is already in large-scale use. Now the company is doing the most to lower the threshold for entering this technology, he added.

Rather, we are counting not on 1-2 large implementations, but on hundreds, thousands of small projects so far, which will grow and develop on the basis of this technology, "says Koverznev.

According to Alexei Bashkeev, YDB is a long-term investment. The company will adjust further plans for the development of YDB depending on how much demand for it will see, added the head of the Yandex.Cloud platform.

The company recalled that Yandex, like Mail.ru, also has an open source solution - the ClickHouse DBMS, which has long been presented in the cloud.

YDB has become publicly available as part of a major update to Yandex.Cloud services. In addition to it, the company has introduced a number of new services for storing and managing data.

Notes