Neoflex developed date platform for work with Big Data for Mediascope

2020: Creation of Mediascope Data Platform

On March 12, 2020 the Neoflex company announced project implementation on creation date platform for Mediascope research company. The basis of the platform was formed by solutions of Neoflex for work with Big Data based on technologies of the Hadoop family. The project is brought into commercial operation.

Mediascope Data Platform allows to collect and process in the unified type big arrays of diverse data on contact of the person with media and advertizing, his consumer behavior. Thanks to it the platform becomes a technology basis for cross-media of analytics in the company. Except own data of Mediascope, to the platform data of partners can be loaded and processed: Internet platforms, telecom operators, third-party data on purchases and consumer behavior of the person.

"Crude" data on consumption media content are included in the platform in streaming mode through the manager of queues Kafka and are loaded into primary layer on HDFS through Apache NiFi. Further there is a formation of an analytical layer where data are consolidated, cleaned and calculations are made. It is performed with the help Apache Spark under control of Apache Airflow. Access to already ready analytics is organized using a management system databases Apache Hive which allows to execute requests, to aggregate and analyze the data which are stored in Hadoop, using traditional SQL- the interface.

The important success factor of the project was use of our accelerator of development of Datagram which allows to project data streams in the visual editor and to generate the performed Scala – the code automatically. It considerably accelerated and simplified development process and also gave the chance to attract ETL and SQL developer to design of flows of data processing of c with use of Apache Spark library, commented Ivan Okopny, the head Big Data Solutions, Neoflex

We managed to find balance of approaches of classical marketing research and data science at the same time to remain the reliable supplier of analytics and to answer requests of Big Data. The platform will allow to provide data processing about audience of all leading players of the media and advertizing market – TV channels, Internet platforms, radio stations, publishing houses. This amount of data is measured by tens a terrabyte. Using the Mediascope platform will be able to provide to clients data access at the deep level with high degree of efficiency and quicker to start analytical products, noted Vasily Kuzmin, the director of work with data of Mediascope