VTB implemented Hadoop and PostgreSQL for work with Big Data

As at the beginning of July, 2017 CNews knew, in VTB bank "pilot" on implementation of tools of Big Data about use of the free software came to the end. In the organization the system of formation of the analytical and management reporting on the open Hadoop platform using technologies of data processing Apache Spark and Apache Zeppelin was unrolled. As relational DBMS free PostgreSQL was involved. The managing director of department of transaction business of VTB bank Andrey Novakov told about it. As he explained, PostgreSQL a direct part of a system is not and can be in case of need replaced with other database. The amount of investments into a system does not reveal.^[1]

According to VTB, the Luxoft company acted as vendor of the project. At the same time the rights to a system after start into commercial operation under the agreement will pass to VTB bank, license fees will not be required.

The project started in November, 2016, and its key stage was completed in May, 2017. Then the decision on development of functionality by September, 2017 was made.

A system received the name GAUSS — Global Transaction Business Analytic Unified Source & System, a uniform analytical system source for transaction business. The GAUSS is used for creation of the reporting, but also its application for assessment of different risks (credit, client, partner), detection of fraudulent schemes, modeling of target commercial offers and so forth is considered. In plans of VTB — integration into the analytical Microsoft Business Intelligence system which already works in bank and will be adapted for GAUSS.

The GAUSS works at the clusters consisting of a set of nodes where duplication of a system on a case of failure of one of them is provided and support of several work copies of data is performed.

"Hadoop was selected for creation of a system as its work is based on the principle of parallel processing of data — explained in bank. — It allows to increase the speed of reporting and creation of forecasts. A system differs in fault tolerance and a possibility of parallel operation of both users, and programmers at the same time".

A system GAUSS for the first time in VTB Group was implemented according to the agile development method Agile scrum. As consider in bank, when using traditional approaches the project could stretch for a year, having borrowed, thus, is twice more than time.

During operating time over GAUSS analytical work with databases of bank was carried out, in a system information arrays for 2014-2016 are already created. As a result conditions for a request of materials on an unlimited combination of parameters and options are created.

"A system soon will begin to obtain data from alternative sources, and necessary analytical forms for the purposes of modeling and monitoring of sales of products of transaction business will be developed" — reported in bank.

The data model created within the project for one of lines of business can become afterwards a basis for ontology and a data model of all bank, emphasized in VTB.

For bank it became the first experience in area of Big Data though in general in VTB Group (in particular, in VTB 24) proprietary solutions of Teradata, SAS and Oracle are already implemented. According to Novakov, the stack of technologies of open products used in VTB bank is economically more effective.

Notes