Bank "St. Petersburg" and GlowByte implemented a fault-tolerant solution for the data management platform on the Hadoop stack

2024: Building a Data Management Platform Solution

Bank Saint Petersburg together with IT the partner GlowByte , he implemented a project to create a fault-tolerant solution for the platform data management on the stack Hadoop using the tool. This was announced Data Ocean Flex Loader on May 14, 2024 by representatives of GlowByte.

Flex Loader is a tool within the Data Ocean platform of the Russian developer Data Sapience. The solution is designed to automate the loading of data into such popular solutions of the Big Data class as Hadoop, GreenPlum, S3 object storage, as well as to provide inter-cluster replication in near-real-time mode.

The data management platform is a source for critical reports of the Bank, including regulatory ones. The platform is distinguished by increased fault tolerance and belongs to the business critical class, which makes it special in the Russian market. Systems of this class require the creation of a backup loop, which in itself is a difficult task in a Hadoop environment out of the box, said Yan Guzov, CDO PJSC "Bank St. Petersburg."

It was important for us to have a state of backup circuit close to real time. After examining all available tools on the market and evaluating the possibility of using open source solutions, we opted for the Data Ocean FlexLoader tool. As part of the project work, Flex Loader has been integrated with our solutions for the formation of analytical calculations and reporting, it is, in fact, part of a continuous regulatory process. Thanks to the use of Flex Loader, we not only achieved fault tolerance targets, but also implemented data change isolation, which is not available in the basic Hadoop distribution. noted Gleb Smirnov, owner of the Data Management Platform, deputy CDO of PJSC Bank St. Petersburg.

Our company is always happy to offer products for data management platforms, including Flex Loader, a tool within the Data Ocean family designed to automate the upload of data to target systems based on the GreenPlum DBMS, Hadoop ecosystem and S3 object storage. The key features of Data Ocean Flex Loader are metadata synchronization and quality control of downloaded data, work in incremental mode, flexible priority management, multithreading and competition. In addition, the tool allows you to replicate data between two GreenPlum or Hadoop clusters in Disaster Recovery mode, noted Alexey Rundasov, commercial director of Data Sapience.