For Monopoly.Online moved "data lake" to Yandex Cloud using BI.Qube

2023: Moving a "data lake" to Yandex Cloud with BI.Qube

On July 10, 2023, it became known that the digital platform Monopoly.Online, which offers transport services and solutions in the field of organizing and managing the logistics business in the FTL transport format, completed the migration of data from 10 sources to the Yandex Cloud cloud platform. The project was implemented using BI.Qube. As a result of the project, Monopoly.Online organized a single corporate data warehouse and modern technologies for analyzing business processes, including for segmenting the client base using machine learning technologies.

As reported, Monopoly. Online has created several services that help transport companies search for and place cargo on the platform, automate the operation of gas stations, organize cargo transportation and optimize the work of truckers. All these services generate heterogeneous data that is processed and stored using the Yandex Cloud data platform services.

The key goal of the project was to localize the company's "data lake." In addition, the company sought to optimize the quality of analytics and reporting, as well as the efficiency and security of processing large amounts of transactional data related to logistics activities.

Monopoly. Online has chosen the Yandex Cloud data platform, which provides an ecosystem of services from collection and storage to data analysis and visualization. Services operate on the basis of Open source and easily integrate with each other.

The Yandex Cloud platform, to which Monopoly.Online migrated, contains Open source components provided as managed services. This approach makes it possible to quickly deploy and flexibly configure the infrastructure, manage performance and cost of use in accordance with the PAYG accounting model (payment as used).

I liked the BI.Qube toolkit and the expertise of the development team, the approach to choosing and implementing an architectural solution. The Yandex Cloud data platform allows you to quickly test the updated concept and easily scale the resulting solutions. We got the opportunity to develop using Yandex Cloud machine learning services. commented Dmitry Vlaskin, Head of Data Analytics at Monopoly.Online

The data migration project was completed in four months using BI.Qube. BI.Qube includes industry models, tools for building a data model in the no-code/low-code concept and algorithms on the Yandex Cloud platform. The tools provide automatic migration to open source platforms and technologies with support for high-speed extraction of large amounts of data from distributed sources of any type: SQL Server, PostgreSQL, MySQL, REST API, Excel, CSV, Google BigQuery, Kafka.

We are pleased that our BI.Qube platform products have helped Monopoly.Online quickly organize work with data on. Russian software This was an interesting and technical sight project that required flexible approaches to development and non-trivial solutions. We dealt with a large number of heterogeneous sources, complex data processing logic and high quality control requirements. I will not hide that this was our first project of this scale, and we coped thanks to a team of specialists with inter-platform expertise and experience in building analytical repositories in accordance with modern Data Vault methodologies, and. DevOps Agile told Dmitry Polikovsky, Development Director of BI.Qube

The total volume of data transferred to the cloud was 4 TB, the number of heterogeneous sources was more than 10, the number of objects with data was more than 300. The project was implemented in accordance with the Agile methodology and DevOps principles, which ensured optimal development speed and solution quality. The entire main load and functionality of the solution has been transferred to the updated technology stack, and a continuous stream has also been organized for the prompt delivery of current data to numerous storefronts.

Transport and logistics companies process in real time a large amount of data from disparate services - for example, for finding and placing goods, automating the operation of gas stations and roadside services, tracking the movement of transport. All these systems generate data, when working with which it is important to take into account time stamps. In order to effectively work with this information and build analytics, cloud technologies are well suited, one of the advantages of which is the ability to quickly scale. Managed services allow you to focus on data management rather than infrastructure maintenance. The experience of projects like Monopoly.Online can be useful for others in the industry who are exploring opportunities to efficiently organize a data lake. reported by Alexander Chernikov, Commercial Director of Yandex Cloud