Gazprom Neft (ADH - Arenadata Hadoop)

2019: Arenadata Hadoop implementation

The directorate of regional sales of PJSC Gazprom Neft in cooperation with Arenadata implemented the project on creation of the modern "lake of data", the major components of the corporate platform of data management.

The directorate of regional sales of PJSC Gazprom Neft at the end of 2017 initiated the Smart Lake of Data project on implementation of the complex platform of processing and data storage with the integrated Data Governance components. The need for secure storage location of the "crude" and initially integrated data acted as one of premises of the project. All information arriving from internal and external sources contained on layers of data of the centralized analytical infrastructure in the closed format that interfered with effective work with it: for example, it was possible to transfer data only in the form of file packets or specially developed show-windows outside. A significant amount of the initiatives and projects connected with processing of unstructured data and data bulks started on a wave of digitalization became other premises of creation of "the smart lake of data".

After approbation of different solutions for primary integration and storage of crude data (data lake), the choice fell on Arenadata Hadoop — a domestic distribution kit.

In 2019 the Directorate of regional sales of PJSC Gazprom Neft integrated the lake of data on the Arenadata Hadoop platform into structure of the complex platform of data management.

The first problems of "the smart lake of data" included transaction processing of Gazprom Neft gas Station network, calculation of segments for client analytics, the analysis of a feedback from clients.

Besides, data of considerable number of external sources, in particular, of the St. Petersburg commodity and raw exchange, the websites of the Central Banks of Russia and the CIS, geographical and meteorological resources, metrics and withdrawals of Google, App Store, "Yandex", open data of social networks, different data of partners and the information about competitors, data of mobile applications were integrated into the platform.

"The smart lake of data" is unrolled in the Data processing center of Gazprom Neft in St. Petersburg. Its users are analysts of divisions of Directorate of regional sales and subsidiaries. Besides, the created solution is a supplier of data for the different systems in a circuit of the company and in a target type — for external partners.

Feature of the project of steel unique solutions for Big Data of a landscape in the field of security. At the level of all a component, landscapes and a role model security requirements of information on standards of the Gazprom group were provided, the solution is successfully certified for work with a trade secret and personal data.

In particular, separate groups of access, for example, for developers, analysts, administrators were created. Between their rights and powers the thin edge is recorded, and role models are constructed so that users saw only required data. Also integration between solution components and the adjacent systems is executed with observance of corporate information security policy.