Rostelecom (ADB - Arenadata BD)

2019: Creation of a single technology platform

Project scale

• 115000 man-hours
• 3000 automated jobs
• 500 subscribers

Objectives

• Consolidation of federal HD by the organization of information-analytical space based on a single failsafe scalable platform (TsHD), as uniform source of the reliable, interpreted, qualitative, up-to-date information at optimal cost
• Execution of the directive of the Government on import substitution by balance shift towards domestic software, own developments and open source
• Reduction of TCO/TCC of reporting systems and analytics
• Implementation within TsHD of control function by data, techniques of the consolidated data preparation, KKD, the business glossary, interpretation in business terms of initial data, integration with MDM/RDM systems, providing data lineage, a possibility of impact of the analysis in borders of complete lifecycle of attributes
• Cultural development of a business intelligence, use of tools of a research of data (Predictive Analytics, ML, Text Mining), increase in monetization of data, democratization.
• Reduction of T2M by practicians of prototyping, self-service BI/ETL

Store and Compute used by software, DBMS:

• Hadoop (HDP),
• Hive,
• PostgreSQL,
• Oracle DB,
• Arenadata DB

During the project RDM and MDM systems are implemented, integration with TsHD is provided, the uniform Glossary is created, the register of reports, the medium map is made, DG tools are developed, democratization of data is provided. The target stack of technologies is defined, purchase of software, hardware upgrade is carried out.

The implemented equipment allowed to increase multiply possibilities of the platform (24 GP nodes, 65 HDP, 7.5 PB a space disk). The platform is prepared for growth of amount of data, number of users of BI, show-windows, sandboxes. Several circuits (DEV/TEST) are unrolled in addition that with implementation of release process, the practician of CI/CD allows to increase quality of end products.

Transition to open source, own development reduced the required purchasing amount of licenses of proprietary products several times that reduced the project budget by millions of dollars, allowed to level corporate and sanctions risks.

The self-service BI/ETL service as the certain platform is provided to commands of other divisions for independent development and demonstration of prototypes (a "grey" circuit). TsHD provides with infrastructure and support of the application software, third-party commands are concentrated on processes of the analysis and development. The prototypes approved by the customer, undergoing refactoring, are transferred further to a target circuit that significantly reduces T2M.

In 2019 TsHD it is officially put into commercial operation.