Big Data in Russia

End-to-end analytics

Main Article: End-to-End Analytics

2023

The Ministry of Digital Development of the Russian Federation creates a big data factory

November 8, 2023 in Ministry of Digital Development of the Russian Federation told about the creation of a big data factory. We are talking about a state project using artificial intelligence technologies.

A state big data factory will be created, where the accumulation and formation of data sets will take place, primarily on the basis of data that the state already has, "said the head of the Ministry of Digital Development Maksut Shadayev on November 8, 2023.

He noted that developers of AI technologies will be able to use data from this factory to train various neural networks. The head of the Ministry of Digital Development named education, health care, security and services as priority areas where AI will be used.

Second. Based on the data that the state will take from business on certain conditions. In this sense, it will also be a big regulatory amendment. And in this state Data Lake, Data Set will be formed and will be provided to technology developers so that they can train their neural networks to further provide services on a competitive basis, Shadayev said.

According to him, the state focused on the transition from strategy to practical implementation and emphasis on data, which, as the minister added, should "translate into concrete proposals."

On November 8, 2023, Shadayev also said that in the near future it is planned to discuss the national project "Data Economics" at the strategic session. It is planned to discuss the main activities, funding, as well as specific regulatory mechanisms.

The national project should provide for the collection of data, including using highly sensitive sensors based on quantum sensors, data transmission and the development of new generation communication systems, the creation of an infrastructure for calculating and storing data using domestic equipment, technologies and software, ensuring data security, including using quantum encryption technology.^[1]

Putin urged to use big data

On March 16, 2023, speaking at the congress of the Russian Union of Industrialists and Entrepreneurs (RSPP), Russian President Vladimir Putin called for the widespread development of digital solutions and technologies Big Data. Read more here.

2021

Russia approved the first national standard in the field of big data

Russia has approved the first national standard in the field of big data. This became known in mid-July 2021.

We are talking about GOST "Information technologies. Big data. The review and dictionary "is identical to the international standard Information technology - Big data - Overview and vocabulary, write" Sheets"with reference to the developers of the document from the National Center for the Digital Economy MSU and. Information Society Development Institute

The standard is designed to provide in the subject area of ​ ​ "big data" mutual understanding between authorities, commercial companies and the scientific and educational community. GOST translates English-language terms related to big data into Russian and decrypts their meanings. The key characteristics of big data (volume, processing speed, diversity and variability) are also given with an explanation of their value.

The adoption of the first national standard sets the vector of joint actions of the authorities, business, and the scientific and educational community for the development of the data economy in Russia, - said the chairman of the board of directors of the Institute for the Development of the Yury Khokhlov Information Society. "Our goal is to reduce the backlog of big data standardization.

In the future, it is planned to adopt eight more national standards in the field of big data regarding their reference architecture, security, analysis methods, use cases, etc., Khokhlov said. He added that some of them have been developed by mid-July 2021, and the other part will be developed together with international standards.

According to Nikita Utkin, chairman of the Cyber ​ ​ Physical Systems technical committee at Rosstandart, the terminology standard is really necessary, since it should allow big data market participants to communicate in the same language.^[2]

The Ministry of Digital Development will create a big data state operator

On May 18, 2021, it became known about Ministry of Digital Development of the Russian Federation 's plans to create a big data state operator. Its services are expected to be paid for companies.

The proposal of the Ministry of Digital Development to provide big data accumulated by ministries and departments, commercial developers of artificial intelligence is stated in the concept of access to state data sets presented by the ministry at a meeting of the Digital Economy ANO working group.

As they write Sheets"" with reference to the relevant document, big data a specialized state organization will provide access to state ones. It will form data sets based on requests from AI developers, anonymize and depersonalize them, and also ensure the creation and operation of an infrastructure for accessing state data sets. Also, the state operator will decide what data can be provided to a particular customer. For example, big data with one degree or another of secrecy will be shared with organizations accredited by relevant departments (for example,). FSB

The ability to create a single data factory makes the state a unique operator that has at its disposal an almost unlimited number of datacets and their combinations. Such bases are of interest to businesses that build marketing communications and development plans based on data analytics, "explained Deputy Prime Minister Dmitry Chernyshenko.

The introduction of artificial intelligence will boost enterprise productivity and accelerate digital transformation, he said.

Deputy Head of the Ministry of Digital Development Oleg Kachanov noted that working with personal data requires strict order and urged operators to ensure the impossibility of personalization when forming sets, first of all.^[3]

A national standard on the structure of the reference architecture of big data has been developed in the Russian Federation

On January 12, 2021, it became known about the development in Russia of a national standard on the structure and process of applying the reference architecture of big data. It was created by the National Center for Digital Economics of Moscow State University named after M.V. Lomonosov and the Institute for the Development of the Information Society.

National Standard "Information Technologies. Big Data Reference Architecture. Part 1: Structure and Application Process "is part of a series of five standards on the reference architecture of big data and is a Russian-language adaptation of the international technical report ISO/IEC TR 20547-1: 2020 Information technology - Big data reference architecture - Part 1: Framework and application process.

The document describes the structure of the reference architecture of the system for working with big data, and also provides a solution to the problem of displaying possible uses of big data in the reference architecture. The provisions of the national standard can be applied by organizations to describe the architecture of specific systems for working with big data and implementing these systems, taking into account the technologies used, as well as roles/performers and their needs.

The document provides a conceptual representation of the reference architecture of big data, as well as basic concepts: reference architecture, interest, interested party, scope, architecture structure, user and functional representation, application process, identification of stakeholders, etc.

The presented draft national standard, along with other parts of the 20547-Kh series of standards, will contribute to the effective use of end-to-end digital technology "big data" to solve economic and social problems in the implementation of the national program "Digital Economy," the authors of the initiative believe^[1]

2020

Big Data in Russia: how to speed up the use of such technologies and who benefits from them

From pilot projects to informed implementation

Big data technologies are already used in almost all sectors of the economy. If a few years ago pilots and approvals were hiding behind the announcements about the introduction of Big Data, now organizations are seriously engaged in the development of corporate strategies for working with data, taking into account business tasks and the formation of technological infrastructure.

A few years ago, these technologies were used mainly by large network and online retail, insurance companies, banks, marketing divisions of large production corporations. Today, Big Data technologies on one scale or another are used everywhere: this trend is most obvious in the public sector, social sphere, extractive industries, construction, "says Alexey Baryshkin, an expert on digital interaction between government and business at Netrika.

According to Alexei Artemenko, regional director of Qlik Russia and the CIS, the term Big Data itself recently began to be used more consciously. The hype about the term has passed, and those companies who really need to analyze large amounts of data are deploying appropriate solutions.

There is a smooth growth in demand, confirms Anton Chekhonin, CEO of Norbit (part of the Lanit group). According to him, those who use this technology have become more aware that its application requires a more detailed study of business processes. Data normalization tasks are also identified and solved to increase the efficiency of using this technology.

Alexey Vyskrebentsev, head of the Forsyth Solutions Expertise Center, gives an example that a couple of years ago the Forsyth product. The analytical platform "was implemented mainly for" classic "business analytics, while now there are quite a few requests for systems with" advanced "tools for analyzing data arrays and the ability to receive business insights.

Government agencies have become the real driver of the use and promotion of Big Data technologies throughout the country in recent years, believes Alexey Mamonov, General Director of the Center for Analytical Systems.

Now it is already impossible to imagine a single system-forming state corporation that would not lead large projects in the field of Big Data. Accordingly, in the coming years we will see that both everyday and strategic management practices will be based on "big data." Moreover, the state is now one of the main providers of basic data for use in corporate systems. Obviously, these processes will only develop, and the business, in turn, will bring to the market more and more solutions for information processing and embedding analytical tools in corporate IP, "says Alexey Mamonov.

The hype around Big Data is subsiding, but the tasks of working with large amounts of information are preserved, and, accordingly, there is a need for technologies that can process growing - often exponentially - volumes of data, says Artem Grishkovsky, commercial director of Triaflai (Trusted Environment, part of the Group of company Systematica).

According to him, working with such arrays without the use of new solutions in the field of Big Data is impossible and will not give the expected effect. Currently, there is a process of gaining experience, mastering opportunities and understanding the prospects of new technologies both on the part of customers and on the part of performers, and in the coming years the use of BI technologies will be expanded.

Andrey Krasnopolsky, CEO of the ATK Consulting Group, believes that more and more companies not only understand the need to use Big Data technologies, but also begin to experiment by introducing various solutions and methodologies that, among other things, increase the efficiency of the development and analysis team. The business has learned to work with data in different sections, from data discovery and working with facts, to predictive analytics, and already machine learning, he said.

Aleksandr Elin, CEO of Alan IT, notes that the number of companies that are ready to pay for forecasts and subsequent results is growing, but most organizations still do not understand the business value of data analytics.

Need for ready-made practices

Dmitry Savvin, architect of Qlev Solutions BI solutions, believes that customers rather need ready-made value extraction practices than research projects. They would love to take advantage of the result that someone has already received. But figuring out how to extract something valuable from disparate data is a completely different business and rather the task of vendors.

A simple example. The benefits of video analytics and recognition of visitors in retail are undoubted. But the customer is preferable not to invest in technology, but to buy a ready-made solution, the benefit of which will be immediately clear, - explains Savvin.

Of course, customers want to use Big Data and AI. But the basis of such a desire today is often not a pragmatic calculation, but rather a fear of losing competition in the future. For example, due to the fact that some trend or "big thing" was missed.

Opportunities to Accelerate the Big Data Market

According to Yuri Efarov, CEO of Sapiens solutions, the emergence of a culture of "trust" in cloud solutions and technologies in companies could accelerate the introduction of Big Data technologies in Russia.

At the moment, there is a conservative desire of companies to store their own data in their own infrastructure, while BI solution providers are increasingly offering cloud solutions, explaining their advantages.

Another important factor is the stability of the economy and investment. In calm times, companies are more actively and confidently investing in innovative solutions.

According to Alexander Yelin, CEO of Alan IT, financial investments are needed to develop and market a high-quality product that meets the needs of users.

Another aspect is the personnel. The situation is such that even in specialized specialties, students do not receive the necessary practice and are not informed about the possibility of working in the field of Big Data.

Шаблон:Quote 'Cooperation with universities and internships by companies, financing from the state and investors, could speed up the introduction of Big Data technologies in Russia, - said the general director of Alan IT.

Also, the greater openness of those companies that were the first in their industry to work with big data could accelerate the introduction of Big Data technologies.

Such companies are "locomotives": it is always more difficult for them, but they can also help others move in the right direction. Therefore, I would like companies that have successful cases to share them with other representatives of their industry, "says Svetlana Belous, Business development manager at Navicon.

Ivan Vakhmyanin, CEO and co-founder of Visiology, believes that the leaders' determination is lacking for the accelerated development of the Big Data market, since both technology and implementation experience often already exist.

When implementing Big Data and analytics, there is an element of risk, because ROI from more effective control and management is not always easy to calculate, but the risk is more than justified in modern conditions. I think that it is possible to accelerate the introduction of Big Data technologies, and indeed any digital technologies, only by developing the competencies of managers in the field of principles and opportunities for digitalization, "says Ivan Vakhmyanin.

According to Alexei Mamonov, a regular dialogue between business and the state (where the former create professional tools and the latter act as a data provider), designed to create a real mechanism for introducing these technologies throughout the country, could give a new impetus to the development and implementation of Big Data in Russia. But the strategic task is to put "on an industrial basis" the training of specialists, to ensure a massive influx of qualified managers, engineers and analysts; teach them how to work properly with data.

The ability to correctly use data (Data Literacy) for highly effective management should be "in their blood." This can only be achieved by joint efforts of the state, business and educational institutions. Perhaps I would even insist here on creating new departments and specializations in universities of various profiles. But there is no need to wait, - already now you can use the affordable, simple and powerful tools presented on the market that allow you to work with big data "here and now," says the general director of the Center for Analytical Systems.

Who benefits from the introduction of Big Data technologies

The benefits of Big Data analysts are received by those companies in which a high culture of working with data is built, experts say. In such organizations, data has owners interested in their quality, completeness and relevance, and data lifecycle management processes are established.

They, regardless of the industry, believe in the data and are ready to make informed decisions based on knowledge obtained from the data, says Rustem Ibragimov, Deputy General Director of BARS Group.

Such processes, he said, historically started earlier in the banking sector, retail, pharmaceuticals and a number of other areas. They have become the basis of competitive advantage, but these trends have long gone beyond industries and are becoming the de facto standard of modern management tools in both the corporate and public sectors.

Artem Grishkovsky, commercial director of Triaflai (Trusted Environment, part of the Group of company Systematica), believes that the public sector, retail, the financial sector, and such technological areas as robotics and the Internet of Things can receive the greatest benefits from the introduction of Big Data analytics.

The arrays of data accumulated in the databases of federal and regional authorities (for example, data on citizens, appeals, public services...) store a huge potential that has not yet been extracted. Large data arrays are based on solutions to the problems of predictive analytics, the creation of imitation and econometric models, the construction of digital twins for retail, financial and corporate sectors. A huge amount of data for obtaining new useful knowledge is provided by information flows from various sensors, he comments.

At the state level, everyone wins - from federal ministries to regions and municipalities, says Alexey Mamonov, General Director of the Center for Analytical Systems.

Most territorial entities are already ready for this, because this leads to a quick result and allows the implementation of long-term development strategies, he notes.

In short, any company that has big data, introducing Big Data analytics, reduces many costs, says Andrey Krasnopolsky, General Director of the ATK Consulting Group.

For example, these are large retail chains that need to purchase, store, move and deliver goods, as well as production companies of any industry, which imply multi-iterative processes, he says.

Sergey Shestakov, CEO of the Luxms Group of Companies, adds that the benefits of implementing Big Data analytics can be obtained wherever there is a need for operational management in terms of indicators, with the ability to quickly and clearly detail up to primary data. First of all, these are telecom, retail and e-commerce, the financial sector, the transport industry and, of course, public administration.

Data analytics have an understandable impact on sales and production, said Alexander Yelin, CEO of Alan IT. For example, forecasting the volume of sales, the volume of goods necessary for production and sales, the optimal place for opening branches - the questions that are most often asked by the owners of stores, wholesale and production enterprises and the introduction of analytics gives an answer to them.

However, in fact, in any area where there are factors that can be digitized and evaluated, the introduction of analytics, according to him, provides a measurable effect: science, metallurgy, agriculture. Systems collect data on metrics and factors that have changed and influenced metrics, and then build a forecast, allowing the company to accurately plan activities.

For example, in agriculture, milk yield is affected by weather, quality and amount of feed, other indicators, tracking which can achieve a steady increase in milk yield by more than 15%. Depending on the size of the farm, the value of 15% in monetary terms can be very, very impressive. The more factors affect the result and the more difficult it is to collect data together and establish a connection, the more profitable the introduction of Big Data analytics can be, - believes Alexander Yelin.

Svetlana Belous, Business development manager at Navicon, believes that companies from the CPG (Consumer Packaged Goods) industry can receive great benefits from the introduction of Big Data analytics. They actively accumulate data and invest in their analysis and processing, as they understand: it is data that allows you to build personalized relationships with customers and contribute to sales growth.

Thanks to Big Data processing tools, CPG companies study the preferences of their customers and strive to offer each of them a high-quality and most suitable product. As an IT specialist, I like this kind of careful work with data. But I can also rate it as a consumer of CPG goods: in my opinion, personalized mailings, discounts or recommendations bribe and highlight the brand against the background of others, she notes.

Alexey Baryshkin, an expert on digital interaction between power and business at Netrika, adds that Big Data analyst will receive a powerful impetus in medicine, biotechnology and personal security. According to him, the serious effect of high-quality Big Data analytics in the light of the fight against the pandemic, overcoming its consequences, preventing further spread in many cases is already visible.

The Ministry of Telecom and Mass Communications withdrew the bill on the regulation of the Big Data market

On June 15, 2020, the Ministry of Telecom and Mass Communications announced the withdrawal of the bill on the regulation of the Big Data market. We are talking about amendments to the Law on Information, Information Technology and Information Protection, introducing new rules for the treatment of big data. 

Taking into account the discussions, including in the Government of the Russian Federation, a decision was made to withdraw the said bill from the Government of the Russian Federation, - said in a letter from State Secretary - Deputy Minister Lyudmila Bokova sent to the President of the Russian Union of Industrialists and Entrepreneurs (RSPP)  Alexander Shokhin.

Earlier in 2019, Alexander Shokhin wrote a letter to the Minister of Digital Development, Communications and Mass Media  Maksut Shadayev with comments on the bill. The letter said that the RSPP Committee on Intellectual Property and Creative Industries and the RSPP Commission on Communications and Information and Communication Technologies concluded that the document does not meet the needs of doing business - it does not create guarantees for entrepreneurial activity based on the creation and processing of large amounts of data, as well as the protection of intellectual rights.

According to the relevant commissions and the committee, the bill does not meet not only the needs of business, but also society, since it does not spell out the restriction of the turnover of data on citizens or the possibility of applying antimonopoly measures to aggregators of such data. The problem of the availability of data that government agencies collect as part of the execution of public functions also remains unresolved.

Among the shortcomings of the bill: inclusion in the category of big data of almost any information regardless of the source and method of receipt, high corruption capacity, excessive regulation of "big data operators" (under this definition, according to the bill, any person who processes something, including manually, falls under this definition).

Russia has developed a fundamental national standard for Big Data

On May 8, 2020, it became known about the development in Russia of a fundamental national standard for big data. The corresponding project was presented by the National Center for Digital Economy of Moscow State University named after M.V. Lomonosov and the Institute for the Development of the Information Society.

Standard "Information Technologies. Big data. Overview and Dictionary "establishes terms and definitions of basic concepts in the field of big data technologies. The use of such technologies is relevant in the telecommunications sector, banking, power, health care and other industries. 

The standard is designed to ensure in the subject area of ​ ​ "big data" mutual understanding between stakeholders - authorities, commercial companies and the scientific and educational community. The unification of the conceptual apparatus will contribute to the unity of perception of information, increase the speed of its dissemination, and also create prerequisites for mutual penetration of domestic and world research in the field of big data technologies.

The national standard is part of a series of national standards harmonizing international documents in the field of big data, and is identical to the provisions of the current international standard ISO/IEC 20546:2019 Information technology - Big data - Overview and vocabulary.

Big data technologies have reached a high level of maturity, their use brings tangible effects in various sectors of the economy and areas of the social sphere, "says Yury Khokhlov, chairman of the board of directors of the Institute for the Development of the Information Society, head of the big data working group of the Technical Committee 164" Artificial Intelligence. " - Standardization of the processes of development and use of big data storage and analysis technologies allows to exchange best practices, use approaches and solutions that have confirmed their effectiveness both in Russia and around the world.[2]

How Moscow uses Big Data when providing public services

Speaking at the TAdviser Big Data and BI Day conference on March 4, Alexander Filatov, Head of the Analytics and Monitoring Department of the Public Services Development Department of DIT Moscow, spoke about the experience of using predictive analytics tools in the city. Over the years, the department has gone from providing simple statistical reporting data to using advanced mathematical methods in its day-to-day work, he noted.

For government agencies, unlike commercial organizations, it is not the achievement of any economic indicators that comes to the fore, but the achievement of state program indicators and a clear adherence to service regulations, including deadlines. The first two tasks of the division are associated with this.

The third task is related to the collection, storage and processing of data that arise when citizens interact with authorities. In total, about 30 data sources are used, each of which has its own history and problems. To control this, mathematical models of predictive analysis are also introduced.

Another task in which the analytics and monitoring department of the development of public services indirectly takes part is to increase the attractiveness of services for the user.

Among the data that are used, for example, those provided by the user to provide services to him, as well as transaction data that arise when the user interacts with authorities.

Data from sources is collected in storage, regulatory reference information is added to them and all this is input to mathematical models implemented by a separate layer on a microservice architecture, and the final result is output.

Predictive analytics is used, for example, to search for time series in order to predict the values ​ ​ of indicators and identify some process anomalies, says Alexander Filatov.

At the input to the algorithm, we submit a time series of transactions, and at the output we get a forecast value and an interval within which this value may fluctuate, - explained the representative of the Moscow DIT. - This can be used to predict the indicators of state programs.

Another example is calculating the load on the infrastructure and predicting the need to allocate an additional pool of resources for some bursts. The value of the performance indicators of the processes of rendering public services is also monitored.

For example, if we see that there is an abnormally large number of refusals to provide any service, this is a signal to our bodies that are engaged in control and supervisory activities and methodological support of the process in order to go "into the field" and figure out on the spot what is happening, - says Alexander Filatov.

Another direction is related to the management of data sources: it is predicted how many records with data each source must transmit in the event that some abnormal value is observed.

A large layer of work is associated with the study of user behavior. User data about transactions is taken, digitized and translated into a vector space. Based on classification and clustering methods, you can look at groups of users, which groups of services interest them, or you can look at a selection of services - which categories of users are interested in these services.

If we add time stamps to data sets, we can use algorithms of associative rules, and then you can look not only at groups, but also see the sequence of services that were ordered by the user and predict the chain of his subsequent actions, - explained Alexander Filatov. - Thus, we are going to form "super services" based on an analysis of user preferences - the most acceptable packages of services for him.

The Ministry of Telecom and Mass Communications proposed to regulate Big Data

The Ministry of Digital Development, Communications and Mass Media in February, 2020 drafted the bill directed to regulation of the market of big data (big data). In the document the ministry enters definitions of concepts: big data, operator of big data and processing of big data. Kon­tro­liro­vat obo­rot big data bu­det Ros­komnad­zor. For this purpose department will create the register of operators of big data. Ig­ro­ki ryn­ka na­zyva­yut^[3]Mass Communications proposed to regulate big data.

According to the bill, big data is defined as follows: "Big data is a collection of non-personalized data, classifying by group characteristics, including informational and statistical messages, information on the location of movable and immovable objects, quantitative and qualitative characteristics of activities, behavioral aspects of movable and immovable objects obtained from different data owners or from different structured or unstructured data sources, by means of collection using technologies, data processing methods, technical means, providing said collection of data to be combined, reused, systematically updated, the form of presentation of which does not imply their assignment to a specific individual. "

The draft amendments to the Federal Law "On Information, Information Technologies and Information Protection" say that big data means all data that can be obtained from owners of structured and unstructured sources using any technology and means.

Big data operators can be government agencies, municipal bodies, legal entities or individuals, self-regulatory organizations or public associations (NGOs and foreign agents too), which organize or process big data themselves. Big data processing goals, their composition and algorithm of actions with them are determined.

Big data processing refers to an action or set of actions that big data operators perform with or without automation. We are talking about the collection, recording, systematization, accumulation, storage, update, change, as well as the extraction, use, transfer, deletion, destruction and analysis of such data.

According to the text of the bill, the Government of the Russian Federation will determine the principles, rights and obligations of big data operators, the procedure, control and conditions for their circulation.

2019: Creating a Code for Big Data Market Self-Regulation

At the end of August 2019, it became known that the Institute for Internet Development (IRI) and the Big Data Association (ABD) have developed a code of self-regulation, which is expected to avoid additional legislative restrictions.

The initiative, among other things, is designed to resolve the issue of the possibility of freely using public data - for example, posted on social networks, transmits. Kommersant

Consent to the processing of personal data can be obtained in any form, including remotely. But at the same time, the use of data for targeted marketing will be recognized as ethical. However, only if the proposals allow the potential purchaser to "ensure the optimal choice of goods"  and will not be "unreasonably intrusive."

IRI General Director Sergei Petrov noted that in this case "it is necessary to take into account not only market laws, but also  consumer rights."

The association notes that within five years the Russian market will grow 10 times, to 300 billion rubles by 2024, which requires its "professional regulation."

The development of a single law at this stage may not meet expectations. Each type of data has its own specifics, which is difficult to prescribe in the language of jurisprudence, - says Sergei Petrov.

Regulation of big data is discussed in the government and the State Duma, their concepts were proposed by the competence center of the ANO "Digital Economy" and the Internet Initiatives Development Fund (IIDF).

According to Leonid Levin, Chairman of the State Duma Committee on Information Policy, self-regulation of the market will not be enough - legislative initiatives are also necessary.

It is necessary to determine how much information about users a business can use and process and how this data should be stored, he believes.[4]

2018

Boston Consulting Group for the Big Data Association

In which areas Big Data technologies are most in demand

It is difficult to name an industry where big data analysis technologies will not be in demand in the near future. At the same time, Big Data is developing more actively in companies that have accumulated large layers of structured and unstructured information: financial sector, telecommunications, Internet commerce, retail.

Telecom operators work with a large amount of data about their users. They apply Big Data technologies to a number of areas: predicting subscriber outflows, predicting complaints, planning measures to retain customers, preventing fraudulent financial transactions, etc.

In retail, using big data analytics, you can, for example, aggregate information about the interests of store visitors and, based on this very accurate audience slice, predict the effects of various marketing campaigns and promotions.

In the near future there will be more implementations using big data technology in the public sector. The vast amounts of data accumulated by federal government agencies are a huge resource that can be used to develop a digital society and increase the efficiency of public administration processes, "said Yulia Kudryavtseva, director of strategic development at Forsyth.

The demand from the state customer and state corporations is largely caused by import substitution and the development of the Digital Economy. The state customer understands that the accumulated data is a valuable asset in the field of public administration, because federal information systems store a huge industry Big Data.

For example, in healthcare, Uniform State Health Information System allows you to collect statistical information about the industry and monitor each settlement for the provision of the necessary equipment and medical services. In the construction industry - an information and analytical system created to implement the reform of pricing in construction. The system calculates the estimated price of construction resources and services in the context of each region of the country, as a result of which catalogs are published with a total volume of about 50,000 items of estimated prices. In the financial sector - GIIS "Electronic Budget," in which there is a full process of preparing the draft law of the Federal Budget of Russia, - says Timur Akhmerov, General Director of BARS Group.

According to experts, the list of areas where Big Data technologies are in demand will soon be replenished with the transport industry, power, oil and gas and food industries.

Transport companies, for example, Big Data technologies allow you to optimize logistics planning and its tariff regulation by tracking the state of the transport fleet, fuel consumption, monitoring customer applications, "explains Timur Akhmerov.

In addition, the Internet of Things is most closely connected with big data. All kinds of sensors, measuring devices for water consumption, equipment in robotic factories, smart transport - all of them generate a huge amount of information that is transmitted from the machine to the machine, and then subjected to research and processing by people.

Finally, Big Data tools are mastering the pool of companies that require instant decision-making depending on the change in the situation in the market and in business - now almost any Russian business falls under this definition, - adds Roman Konovalov, General Director of ID Management Technology.

Agriculture, construction and some other industries do not always boast a high penetration of Big Data technology. This is mainly due to the large life cycle of the product within each industry: it takes a long time to build a building and to develop a new variety of plant, which affects the collection of a suitable sample of data for subsequent analysis.

However, universal digitalization and automation of many processes will contribute to the fact that deep data analysis will be used in these areas in the future, "says Denis Afanasyev, CEO of CleverDATA (Lanit Group of Companies).

If we talk about specific areas of application of Big Data technologies, then such solutions as analysis of video, images and other unstructured types of data cannot be ignored. New technologies have made it possible to analyze them not only efficiently, but also quickly.

Applied solutions have already been created for use in law enforcement agencies and law enforcement agencies. Thus, the facial recognition system using video cameras allows you to detain those who are wanted, and intelligent video surveillance systems in stores detect suspicious behavior of customers, "says Alexey Davletyarov, head of the development group of the integrated design department of information systems, company" Force - Development Center. "

In general, experts talk about two approaches in working with big data technologies. The first is when they purchase not the technology, but the finished product, where Big Data technologies are built inside, and the client as a whole does not care how it works inside. These are often cloud solutions. The second is to create a solution based on these technologies internally with the involvement of external experts or independently. The second approach is actively used by telecommunications, manufacturing companies, retail, banking and insurance sectors.

These are all industries in which a large amount of data has already accumulated, and where business units have realized their value and are striving to gain a competitive advantage and monetize data through Big Data technology, "says Yegor Osipov, an expert on big data at CROC.

Cognitive data processing and Big Data analysis tools will improve those business processes where large amounts of unstructured information need to be processed at short notice.

Firstly, this is the field of marketing. To calculate the effectiveness of a marketing campaign, predict the outflow of buyers, or the next possible purchase, the functionality of traditional BI systems is often insufficient - to improve the accuracy of forecasts, tools are needed that can simultaneously take into account many parameters, factors and all available information about customers.

In turn, solutions based on cognitive analytics will allow, for example, to form patterns and patterns of consumer behavior, based on a comprehensive history of all interactions with each client - and on this basis make personal offers to him or predict care, - believes Artem Kaptsov, head of the department of integration services and complex solutions Navicon.

Another area that will benefit from the introduction of smart data tools is finance. Machine learning and artificial intelligence will help here predict accurate budgets for the period and quickly reschedule them.

Finally, big data processing tools will greatly simplify the life of back offices, doing an excellent job of reading data and documenting, including contracts, agreements and all types of reporting, according to predetermined templates and with minimal human resources.

For example, thanks to artificial intelligence recognition tools, smart BI systems can freely read information from scans of documents in different formats and use it for analysis, along with other data, the Navicon expert notes.

What is holding back the development of the Big Data market in Russia

High solution costs and no quick results

Although interest in BI and Big Data solutions is growing in all areas, the main deterrent, especially in medium-sized businesses, remains a survival strategy in the absence of a development strategy and a breakthrough, and, as a result, savings on the IT budget. Customers don't just need IT, they need competitive business ideas and economic impact in the near future.

In other words, many customers of medium-sized businesses are not ready to work for the future, they live one day without looking beyond the horizons of the exactly known and necessary right now and saving on investments, "explains Denis Seroshtanov, head of information and analytical systems at Interprocom.

Big data tools require a lot of computing power, and therefore are expensive to purchase, install, and use.

Business users in such circumstances want to see a return on investment in equipment in the very near future. However, this does not happen in reality - like any analytical tools, Big Data systems are aimed at optimizing the business and do not bring "fast" revenues, - said Roman Konovalov, CEO of ID Management Technology.

Artem Kaptsov from Navicon, adds that while the developers cannot make Big Data solutions so simple that they are available to everyone. But as soon as the Big Data market moves into a more "massive" phase of development, we will see a sharp simplification of user interfaces and a rapid drop in solution prices, he is sure.

Yulia Kudryavtseva, Director of Strategic Development at Forsyth, also considers budgets and customers' desire to assess the effectiveness of investments in advance as market restrictions. According to her, innovative projects or complex optimization problems are associated with research processes and numerous iterations of verification of methodological models. However, not everyone is ready to go to a project that does not promise a guaranteed result.

Shortage of specialists

There is still a shortage of specialists on the market who know how to implement projects in the field of Big Data. In Russia, competence centers have not yet been formed that would deal with their mass training. Therefore, successful cases are more likely the stories of individual companies and developers.

Many companies are trying to find specialists in outsiders, but due to a shortage of qualified personnel, many projects simply do not "shoot," says Yegor Osipov, an expert on big data at CROC.

In addition, Russia does not yet have a professional community that would take on a big task - informing the market from the inside.

The request is from both developers and customers, and individual vendors and specialists have competencies. I think it is worth using as many different formats as possible to create an expert community. We are all participants in a single IT space, and the exchange of experience will strengthen the potential of the market for domestic IT development of Big Data technologies, "said Timur Akhmerov, CEO of BARS Group.

According to CleverData CEO Denis Afanasyev, the use of Big Data in practice strongly depends on the competencies and skills of specialists, so it is important for companies to develop their own expertise. To benefit from the data, analysts combining skills and mathematics, both developer and business analyst are required. The synergy of these competencies allows you to understand simultaneously in the field of data analysis, statistics, take into account the possibilities of technical implementation of projects and the practical use of Big Data.

Andrey Baibutov, Director of Business Development of the Department of BI Corus Consulting GC "," says that it is often quite difficult to motivate and attract competent people to the project in Russia, since most of the highly qualified specialists with experience in building a highly loaded Big Data architecture work on projects abroad.

The Russian market for Data specialists is currently at an early stage, but it is actively developing. And if in the Western market many companies already have the necessary experts on the staff to build their own digital products and monetize data, then in the Russian market only large players have begun work in this direction. The presence of highly qualified data scientists allows businesses to increase their revenue structure thanks to the implementation of digital projects in addition to the company's main activities, says Andrey Baibutov.

Poor data

For Russian customers, the problem of poor quality data is still relevant - on the basis of disparate or inaccurate data, it is impossible to effectively solve analytical problems.

But it is important that the direction is indicated and, in general, the movement forward is traced, and there are Russian BI tools on the market that provide integration with various data sources, which is vital for the implementation of Big Data projects, and advanced analytics tools at the same time. For example, "Forsyth. Analytical platform. " It provides integration with commercial platforms, including Teradata, Oracle Exadata, SAP Hana, HP Vertika, IBM Netezza, etc.), as well as open source products (for example, PostgreSQL, Hadoop, etc.), - says Yulia Kudryavtseva from Forsyth.

Limited choice of solutions

There are not many solutions on the market that can actually work effectively with large amounts of unstructured data. At the same time, only the largest market players can use them, the volume of data of which is calculated by petabytes: telecom, retail, finance.

And even among them, not everyone is satisfied with the real results from the introduction of existing solutions - they still need to be seriously improved, made more practical-oriented. Big data analytics should not be introduced for the sake of analytics itself, otherwise the business will not receive financial results in the foreseeable future, - notes Artem Kaptsov, head of the department of integration services and complex solutions at Navicon.

Copyright/IP Policy

One of the main restraining factors for the development and improvement of analytics tools in Russia is customer concerns in the field of data privacy.

Despite the fact that new generation cyber protection tools are being actively introduced into business practice, users of Big Data systems are still wary of leaks of confidential information about companies, as well as personal data of customers, - says Roman Konovalov, General Director of ID Management Technology.

More active market development is hindered by consumer distrust of technology, as well as certain issues of market regulation.

To work with data of Internet users and their application, it is necessary to ensure confidentiality and special conditions for storing personal data, - adds Denis Afanasyev from CleverData.

Other factors

Big Data technologies are often perceived negatively, because there has been an extremely lot of noise around them in recent years, but many companies have not seen obvious use cases for themselves. As a result, some organizations make an erroneous conclusion that this is more fashionable than useful technology.

It is important to understand that market leaders are ready for projects with Big Data technologies today. For the rest of the companies, Big Data is not a key driver of development. Some companies today still do not even have an enterprise data warehouse or Data Governance strategy, so they do not think about applying these technologies.

We advise such companies to build the right architecture of their analytical system at once. Modern data analytics solutions are quite complex, consisting of more components. It is not at all necessary to immediately install all the components, including Hadoop or some other heavy solutions. You can only use components that meet your current needs. But it is important to choose an architecture that, over time, when the company realizes that it is ready to work with large amounts of data, can be easily expanded, "explains Yegor Osipov, a big data expert at CROC.

Finally, the lack of real practical experience of most manufacturers and integrators restrains the serious growth of the market.

The customer wants to see the real effect that the implementation of the system has brought to their competitors or other industry companies. When an IT company cannot show project experience, trust in it and the solution being implemented is sharply reduced, says Roman Konovalov.

It should also be noted that the practice of obtaining data from external sources is insufficient.

This means not social networks, sites, open data - everything that can be obtained from there is used as efficiently as possible. But solving a number of problems sometimes requires data belonging to other organizations and not available due to legislative restrictions, - explains Alexey Davletyarov, head of the development group of the integrated design department of information systems, Force - Development Center.

Using Big Data to Spot Illegal Rental Housing

On July 20, 2018, it became known about the development in Moscow of a Big Data analysis system to identify illegal apartment rentals. Finding defaulters is technically difficult, but the Moscow Department of Information Technologies (DIT) has solved the problem, department head Artem Ermolaev told RBC on the sidelines of the Moscow Urban Forum.

He explained that the mechanism that will allow determining those who do not pay taxes has already been tested and, "at a certain point," these data will be used. However, to fully launch the technology, it will be necessary to amend the legislation.

This is a complex scheme, because it is organizational and regulatory. Here is the intersection of the zones of responsibility, - said Ermolaev.

Back in 2017, Artem Ermolaev spoke at an urban forum that work is underway to identify landlords who are hiding from paying taxes. Then he said that it is supposed to analyze the largest Internet resources on which apartments are offered for rent, and compare these data with statistics on tax payments. After such an analysis, data on those apartments that may be illegally rented out were planned to be transferred to the tax service for verification.

The data director of Weborama Russia Dmitry Egorov said that from a technical point of view, the mechanism that the DIT is talking about is solvable. However, in practice, the authors of ads do not always indicate the exact address of the delivery object. In addition, realtors often rent apartments, not owners. And the commercial director of AmberData Viktor Mityunin believes that DIT will be able to get the data of landlords, since advertisements with phone numbers are in the public domain.

According to the Department of Economic Policy and Development of Moscow, in the first nine months of 2017, about 27 thousand apartments were illegally leased in the city. At the same time, 200-300 thousand apartments are offered for rent in the capital annually.^[5]

2017: Big Data Market Trends and Prospects

Roman Baranov, head of business analytics and data storage at CROC, spoke about trends in the Russian Big Data market in August. According to him, the Big Data concept, which has been among the "hot" top analysis technologies in recent years, is gradually falling out of fashion. IT specialists no longer expect revolutionary changes in this area, but only clarifications of approaches and opportunities for implementing certain tasks, but the set of products necessary for work has already been determined.^[6]

Is Big Data losing relevance?

Of course not, says expert CROC. This is still one of the key trends in the analytics market. At the first stage of the "hype cycle" Big Data was perceived as exotic. The new concept based on open source technologies was new to Russian business, which is used to boxed solutions by well-known foreign developers. The words "Big Data" helped to speed up the approval of the project by the business almost twice, even if the decision itself referred to large data very conditionally. New products were not easy to integrate and operate; today these problems are solved using all kinds of connectors, visual tools designed for operation and work with data arrays. The emergence of Russian Hadoop distributions and products for working with them makes the market clearer, procurement, support and training processes become more transparent. All this, ultimately, levels the previous gap between Big Data technologies and Russian IT reality, says Roman Baranov.

Big Data as a way to save money

Big Data is an excellent option for situations where traditional solutions are too expensive or difficult to operate, the expert noted.

From recent examples, I can recall recent events in the collection market, which changed a lot on January 1, 2017. A law came into force that severely limited opportunities for communication with the debtor. One bank asked us to help ensure control in this area and combine the interests of the customer with the requirements of the law. The use of classical technologies turned out to be quite expensive, since then it would be necessary to keep inside one process huge amounts of information collected from all branches throughout the country. And Big Data made it possible to significantly reduce the cost of the decision and implement the project in a few months, - emphasized Roman Baranov.

Big Data in Trade

According to the expert's forecasts, in the near future, in addition to the banking segment, interest in big data will be in the field of e-commerce. Here these technologies are going through a rebirth. Business understands how to earn money and what solutions will help him in this. For example, sales of logistics services that cannot completely go online, and with Big Data become a source of additional income for many companies and especially service aggregators.

In retail, Big Data is actively used in the field of Wi-Fi analytics, which allows, using signals from mobile devices of visitors, to draw up a representative analytical section: the duration of a visit to the store, the frequency of visits, travel routes, distribution of visitors across the territory, crossing the audience with other objects of the shopping center, the share of store visitors from among all passing by, etc.

Video Analytics and Facial Recognition

Another solution within the framework of the Big Data concept, the interest in which is only flaring up - video analytics and facial recognition. This technology automatically detects and extracts customer-relevant information from a huge video stream, allowing you to count, for example, the number of visitors, analyze the loading of the trading floor, monitor queues, predict the interests of buyers, monitor staff activity and cash transactions, instantly detect suspicious actions, send personalized ads, etc.

Safety

Big Data also allows you to resolve a number of security issues. Enterprises now do not need to rely on the attentiveness of the security service, which monitors ten monitors at the same time. A system using the capabilities of Big Data technology can automatically recognize suspicious actions on the territory of an enterprise or, for example, a shopping center or airport. There may be a lot of scenarios here, it all depends on the peculiarities and tasks of the business, said the representative of CROC.

Customers often want to understand the pace of return on investment. Big data technologies have reached a level of development where businesses clearly understand their application value and can appreciate the positive effect in money. We can expect that the Internet of Things and blockchain will go further along this path, which will bring the now fashionable digital transformation closer to the people, - concluded Roman Baranov.

2016

The state of the Russian Big Data market

The Russian Big Data market is at an early stage of development and this term is often understood as traditional BI approaches. The main consumers of big data technologies, as well as the main carriers of large amounts of data, are companies in the banking sector, telecom and trade. For them, analyzing large amounts of data related to the analysis of customer solvency, consumer behavior and market conditions is the most important tool for maintaining a competitive advantage.

In recent years, in all companies from the big three mobile operators there have been divisions specializing in working with big data, and they are not just information divisions for the development of client profiles, they are business units that are designed to generate additional profit.

It was in telecom that they began to transfer the large circuit of systems to. Hadoop Large arrays of data from billing systems CRM and other sources are added to Hadoop, aggregated and already built on this information, BI which allows you to understand where the subscriber is at the moment and what his needs are in order to offer the best service, to make the most attractive offer for each specific client, "says Andrey Baybutov, Business Development Director of the BI GC Department."Corus Consulting

Retail is also among the pioneers of the Big Data market. More and more companies from this segment are creating separate data departments in order to dive as deeply as possible into the lines of checks for 2 + years and find new hidden relationships, adds Baibutov.

Ivan Vakhmyanin, CEO of Visiology, believes that the first hype has already passed, and now there is an accumulation of real experience, both among customers and performers.

We expect interesting cases in the near future, not only in the "traditional" industries for Big Data, such as finance, telecom and retail, but also in industry, logistics, construction and healthcare, adds Vakhmyanin.

Pavel Adylin, executive director of the company Artezio (Artesio) (group of companies), LANIT believes that public sector companies should also act as a potential customer for Big Data projects in Russia in the near future, since they have huge accumulated amounts of data suitable for analysis.

Konstantin Chernousov, Deputy General Director of Vesolv, cites an example of a implemented project in the public sector: "For example, the Federal Tax Service completed the first project using Big Data to track the chain of VAT payers and suppress VAT withdrawal fraud."

As for the solutions offered by developers, these are either international commercial products from Oracle, SAP and the like, or solutions based on open source technologies. There is practically no domestic software for processing a large amount of data, adds Chernousov.

Andrey Nugmanov, partner at AT Consulting, director of the BI block, believes that in the sector of "new BI" - big data analysis, event processing and real-time decision-making - the SPO stack is actively crowding the products of traditional vendors. It is developing in light of the updated vision of functional requirements for BI and has technologically largely caught up with the proprietary stack.

Open code, transparency of development, legal purity and accessibility, support guaranteed and not closed to one vendor, tolerance to equipment, the highest popularity of software, primarily among young and promising specialists, all this becomes the reasons for the active displacement and washing out of the "old" proprietary stack from traditional niches, - Nugmanov is sure.

Vendors are trying, if not to ride the wave, then at least not to be buried by it. Someone opens the code and switches to an open source business model, trying to revive interest from the public, and therefore from opinion leaders among buyers, in their traditional products. Others are actively integrating with major Hadoop stack support service providers, trying to reduce the cost of ownership of their traditional products by leveraging the open capabilities of Big Data and achieve the synergistic effects of a hybrid solution.

The client is not always ready to immediately pay for licenses to the vendor and tries to independently test the technology, understand the degree of its applicability and gain the necessary expertise for further operation. The selection of SARs allows you to quickly implement the functionality of interest without royalties and - due to the lack of procurement procedures - in the shortest possible time. We do not see any serious obstacles in the development of these technologies among customers. And the examination is present on the market, and at least there is a clear business case that ensures a reduction in operational costs for storing significant amounts of information, "says Nugmanov.

From the point of view of technology, AT Consulting observes that solutions using In-Memory Data Grid (IMDG) come to the fore.

Hadoop allows you to collect heterogeneous information and store. Now it's time for the next step - to conduct complex analytical calculations online. Classic MPP platforms can no longer provide a quick reaction due to the presence of read and write operations to disks and the specifics of the operating environment. The issue of the cost of such technologies is also important, - says the partner of AT Consulting. - We see that in-memory solutions are increasingly used for serious analytical tasks. They enable high-performance parallel query execution on heavily loaded analytical systems to serve thousands of users in high-availability mode.

Roman Baranov, head of business analytics and data storage at CROC, notes the importance of understanding that the term Big Data itself is becoming more and more blurred every year. The list of technologies that can be attributed to this concept is becoming more and more. They are already the everyday reality of most modern companies. In addition, when pronouncing "big data," many have long meant not only data collection and storage, but also analytics, the Internet of Things, and much more.

Trends in the Russian and world Big Data market

The main trend of the Russian Big Data market is the penetration of big data technologies in areas that were previously difficult to imagine.

If earlier a huge number of segments, for example, production, did not so actively pay attention to big data technologies, now the opportunity to collect information from all sensors and other plant equipment gives gigantic opportunities.

This will significantly optimize work at the production site itself, as well as increase the efficiency of planning and convert the information received into money that is lost when deviating from the plan or is not processed in terms of lost profit, "says Andrey Baibutov, Business Development Director of the BI Department of Korus Consulting Group.

According to Konstantin Chernousov, Deputy General Director of Vesolv, the general trend is that everyone wants to use Big Data, since big data analysis increases the efficiency and competitiveness of the company. And one of the driving facts is, oddly enough, the emerging fears that a competitor has begun to capitalize using the new technology.

If we talk about global trends, then first of all we can talk about the trend of transferring the Big Data infrastructure to the cloud, says Ivan Vakhmyanin, CEO of Visiology.

This makes sense for many companies, as server capacity for Big Data is very expensive, and is not always needed on an ongoing basis. For example, at Visiology, we conduct most of our Big Data experiments in the Amazon cloud. In addition, cloud Big Data products often greatly facilitate the work of engineers - the entry threshold of most Big Data software products in terms of deployment is very high, and in the cloud you can immediately get an already configured cluster, "says Vakhmyanin.

The second trend, he said, is streaming analytics, which allows you to analyze incoming data in real time. This capability is especially important for applications built on top of data collected from sensors (IoT, IIoT).

Pavel Adylin from Artezio adds that the global market is characterized by the division of the Big Data direction, which we still understand in general, into many independent directions that solve narrower specific problems.

For example, according to his data, it is possible to distinguish: software and hardware for storing big data, means of parallel data processing, means of filtering data and building models, means of visualizing data and their relationships, means of working with images, machine learning, intelligent interfaces, automation of mental labor.

This division is also associated with the emergence of ready-made industry solutions for small and medium-sized businesses, running both standalone applications and models of SaaS or BDaS (Big Data as Service).

Big Data Russian market barriers

Shortage of specialists

One of the main problems of the Big Data market in Russia is the difficulties in finding qualified specialists.

According to Ivan Vakhmyanin from Visiology, the shortage of such personnel is observed not only because they must have a rather complex set of skills and competencies, but also because today few people understand how to prepare, evaluate and organize their work correctly.

Konstantin Chernousov, Deputy General Director of Vesolv, says that now such a profession as Data Scientist is gradually coming into use. It is quite rare, but the demand for it is already colossal: one resume of such a specialist accounts for about 50 requests for work.

In Russia, there are few such specialists who will tell management about the possibilities of analysis using Big Data, calculate the budget and implement the project, and it will not be possible to quickly increase their number, since there are not just courses, but even materials in Russian, Chernousov notes.

Andrei Tiunov, General Director of BI Partner (I-Teco Group of Companies), clarifies that Data Scientist is experts from the customer company who understand the trends of their market, know the business well, find opportunities for its growth and are able to use the potential of the data owned to solve certain problems. It is they who have key competencies in Big Data solutions, says Tiunov.

Andrei Baibutov, Business Development Director of the BI Department of Korus Consulting Group, also believes that there are critically few good specialists in the Big Data field on the market.

If you now enter the resource market in search of a good specialist with experience with Big Data, machine learning, IoT, etc., you can hardly immediately find a person with experience from two to five years of work, and even with the necessary product portfolio. Therefore, many companies are trying to cultivate their own specialists for these tasks, - the expert explains.

Liubov Vedeshina, head of business analytics practice at Interprocom, sees the problem as the fact that an expert community of analysts in the field of big data has not yet been formed in Russia, a competent customer and a competent contractor have not appeared.

On the side of potential customers, we see a lack of specialists who would be equally well versed in both industry specifics and approaches, tools and methods of processing big data. Experts in the field of world-class big data have already appeared on the side of the performer, some are even in the world top. But there are only a few of them, - emphasizes Vedeshina.

The big data analytics profession has yet to become mainstream. Universities do not have appropriate training programs, again because there are few competent teachers so far. Corporations are partly compensating for the shortage of specialists by offering their own training programs. For example, SHAD (School of Data Analysis) from Yandex and paid courses at Bilaine.

However, these courses are not enough, it must take some time for the number of qualified big data analysts to change the quality of supply and demand in the Big Data market, Liubov Vedeshina believes.

Pavel Adylin, executive director of Artezio, adds that due to the lack of specialists in the field of Big Data in Russia, the professions of Data Scientist, Data Analyst and Data Engineer are most often not divided. If the first of them is the creator of new technologies for extracting information from data, machine learning algorithms, artificial intelligence, then the latter is the developer of software or hardware complexes for solving specific big data problems. To prepare these various specialists, it is already required to introduce various methodological approaches, Adylin is sure.

Lack of implementation experience

A number of experts call the main deterrent in the development of the Big Data market in Russia a small number of Russian cases on which both customers and integrators could rely. As a result, Big Data projects are risky.

Often you hear from customers - "guarantee us that the introduction of Big Data analytics will bring us savings of N rubles," but it is impossible to give such guarantees, at least before conducting an operational analysis of the accumulated data and building and verifying the first models, which in itself requires investment resources, "- notes Ivan Vakhmyanin, CEO of Visiology.

The same is said by Liubov Vedeshina, head of business analytics practice at Interprocom. In her opinion, potential consumers do not understand the benefits to their company and industry from big data technologies . Customers doubt that their investments in big data processing and analysis technologies will pay off.

Konstantin Chernousov from Vesolv has a similar opinion. According to him, the lack of knowledge about the possible benefits of using Big Data restrains the development of the Russian market.

The representative of Korus Consulting Andrei Baibutov also refers to the experience of implementations as a market barrier:

I only know about units, maximum - a couple of dozen implementations. Most of the big data projects are often done on open source products, which Russian specialists also have little experience with. As a result, there is a methodological unavailability, which prevents you from understanding how to do projects, and technological - due to the lack of necessary software competencies.

There are few Western cases impressing anyone, because the Russian realities are quite different. Therefore, at this stage, the Big Data market is driven by companies that are not afraid to experiment, invest in research projects, counting on the benefits and competitive advantages that Big Data can bring, adds Ivan Vakhmyanin.

Data Quality Issues

An important problem for the use of Big Data technologies in Russia is the lack of practice of accumulating big data and the poor quality of this data.

According to Liubov Vedeshina from Interprocom, even if a potential customer has formed an understanding of his benefits from Big Data analysis and found industry expert analysts in the field of big data, he is faced with the problem of the quality and amount of data that he has accumulated. As a rule, data spontaneously accumulated by customers is in a state that is not suitable for analysis and benefit for the company, she notes.

Pavel Adylin from Artezio sees the same problem. According to him, the quality of the data leaves much to be desired due to the presence of distortions (emissions) and insufficient depth. Thus, it is necessary to significantly expand the data sets for analysis, but this is not possible, because due to the protection of personal data in our country there is practically no market for the purchase/sale of information in the form of data exchanges (Data Exchange).

Perhaps the accumulation of data could be helped by a program of state support for open sources of digitized data, for example, access to primary data from Rosstat, etc., the expert believes.

The head of Roskomnadzor believes that the state operator Big Data is needed in Russia

The head of Roskomnadzor Alexander Zharov believes that in Russia it is necessary to create a state operator of large user data. The official justified this by the fact that, in his opinion, such information is a national treasure, and not the property of companies processing data.

"I

believe that the state operator of big user data should be. I support the position that the experts voiced. The fact that this is a national treasure, not the property of companies that process data. This is obviously the property of a citizen. But every person cannot understand to what depth information about personality should be structured. This should be a national treasure, "RNS quotes Zharov's comment.

The head of the ILV clarified that the concept of "big user data" includes geolocation, biometrics, user behavior on various sites, etc.

"All this leaves traces on the Internet, is the subject of analysis of transnational Internet companies and, obviously, also requires regulation, as the 152nd Law -" On the Protection of Personal Data "is currently working," Zharov added. According to him, we are talking about the creation of a new law, and this is already being done by a working group led by Assistant to the President of the Russian Federation Igor Shchegolev. At the moment, the issue of regulation is being worked out with experts; by the end of 2016, specific proposals should be formulated.

2014

TAdviser - 100 Big Data Customer Strategies

Analysts at the TAdviser center conducted a study of the Big data market in Russia. During the study, experts interviewed key IT customers in order to determine the existing demand for such technologies, as well as indicate its potential.

Read more: 100 profiles of top Russian companies on strategy in Big data (report)

CNews Analytics: Market maturity level rises

According to the results of the study СNews Analytics and, the Oracle level of maturity of the Russian market in Big Data 2014 increased.

Customers surveyed demonstrated a higher degree of awareness of these technologies, as well as an understanding of the potential of such solutions for their business. More than a third of the respondents have already started using Big Data technologies in Russia. It can be noted that a single conceptual field of this segment is already developing on the Russian market.

Yandex Data Factory Shapes Big Data Offerings

According to Interfax, Yandex's Yandex Data Factory (YDF) division intends to develop proposals for retaining subscribers. In early 2015, it was reported that the YDF analyzed more than 100 parameters describing the behavior of 100,000 World of Tanks players. The resulting model for predicting the outflow of players turned out to be 20-30% more accurate than the analysis tools standard for the gaming industry.

Leonid Delitsyn, investment holding analyst: Finam

It is unlikely that Yandex will really take on the task of developing proposals for retaining other people's subscribers. Only the operator himself can keep his subscribers. I couldn't find the blog post or corporate section referenced by the news outlet, but I think it was only about predicting subscriber churn. To do this, it is necessary to evaluate the so-called "survival function" of the subscriber, or, alternatively, the risk function of outflow of the user. These tasks are well known in statistics, as opposed to successful cases of their application in practice.

Yandex, of course, needs a stream of positive news, and big data is a grateful topic in terms of their generation. We are entering an era when machines collect information about us every second, send it somewhere, archive and store it somewhere, and sometimes even recycle it. Cars already know much more about each of us than we do ourselves. It is unlikely that the players in World of Tanks themselves are able to measure their behavior using 100 parameters. And it is unlikely that they all know today how long they will remain in the game. Of course, not all of these 100 parameters are equally important, but it is not known in advance which ones are important and which ones are not very important. To highlight important parameters, you need to study the behavior of 100 thousand players - but since Yandex prompts the answers to a hundred million visitors, he can do this.

Generally speaking, there are quite a few applications of machine learning methods to big data, many of them are quite bright and can generate popular news. That's good for the company, both on its own and to address the challenge of encouraging stock market analysts. It is this task that is the most difficult for Yandex today. And the task of predicting the outflow of customers is quite old and well-studied, approaches to solving it have been published. The only question is the practical implementation of those approaches that are efficient when client bases number millions of users. In this field, "grinding" solutions to make it suitable for practical problems can take more years and means than developing a theory. According to news outlets, thanks to the use of proprietary algorithms, Yandex manages to build 20-30% more accurate solutions than competitors can. This difference does not look so significant. I think that the main competitive advantage of Yandex is the presence of a staff of experienced specialists, software and hardware.

In addition to activating Yandex in the b2b sector, the following is also important. Apparently, the story with SaaS repeats itself (or continues), when instead of the emergence of new, independent companies, we see the emergence of a new sales channel among large manufacturers and distributors. software Big data is the second most popular investment area among venture capital funds, according to the results of the Venture Barometer Russia 2014 study, they are second only to financial technologies, and even then, the lag cannot be called significant. Investors understand the promise of machine learning during a growing wave of M2M technologies - after all, devices must not only exchange data, but also change their behavior depending on the information received. In short, investors want to put money into the 'big data' direction. But they hardly can.

The fact is that in the case of big data, firstly, a powerful hardware base is required, and secondly, indeed, high technologies. You can't build much on just superficial knowledge of machine learning methods, since methods must be fast and stable to work with big data. Almost useful methods have been polished over years of practice - and Yandex has this practice, but new players most likely do not. The volume of initial investments, apparently, also exceeds the amount by which venture investors would be willing to take a chance to try a new direction. Simple tasks that can be solved with the help of not very big data and ordinary specialists for a relatively small amount are unlikely to bring high profits - simply because many can solve them, which means that fierce competition is inevitable. Yandex in this situation turned out to be protected by a unique barrier, naturally built over the years on the search engine, Yandex market and contextual advertising management system.

Therefore, in the field of "big data," as in the SaaS industry, the main market players will be not independent startups, but already well-known large companies. By the way, the service for predicting the outflow of subscribers to Web sites could be provided not only by Yandex, but also, for example, by Russian Internet counters, LiveInternet, OpenStat and others. If Yandex promotes this service, then it will become fashionable, and for other players there will be a field of activity - the search for the lowest cost implementation of solutions to the most popular problems.

2013

Several dozen pilots across the country

The explosive growth of data, as well as the desire to gain more knowledge about their customers, pushes large companies to search for technologies that will help preserve huge amounts of data, receive and analyze information from previously inaccessible or inaccessible sources (for example, streaming video, sentence analysis when accessing a call center). In addition, in traditional systems, the cost of storing 1TB data is high enough, the use of solutions based on Big Data technologies can significantly reduce costs by using cheaper equipment.

For 2013-2014, the Big Data trend has not yet fully reached Russia. The spread of this concept in our country is still limited to pilot implementations and approbation. For example, CROC had three pilots in 2013, said Maxim Andreev, head of business applications at CROC. One of them is for a large telecommunications company to track social ties between subscribers and identify levels of influence. The goal of the project is to reduce the outflow of customers.

At the same time, all the developments available in the world in the field of Big Data are available in Russia at this time - from open source to solutions of large vendors. As for cost, it varies depending on the task and the solution suitable for it. The use of this technology for companies is an opportunity to break away from competitors by improving the products or services offered, or by significantly optimizing production and business processes. Therefore, Big Data is in demand by all companies living in conditions of high competition, in particular, banks, retail, telecom operators and others.

"Interest in Big Data is still visible, but there are only a few projects. Until now, most specialists do not understand what Big Data is, and to be honest, there are not many companies in Russia that have really a lot of data and are not structured, "commented Georgy Naneishvili, Director of Partner Relations at Qlik

. According to him, telecom and search engines use these technologies, but so far to solve a very limited class of problems.

Thus, technologies have been around for more than five years, however, it is in recent years that real projects have appeared in companies, including Russian ones, whose business is significantly less than, for example, Facebook. Many Russian companies of various levels operating in the field of telecom and Internet services, public administration, financial sector and other areas can afford Hadoop technology itself, he said.

Yuri Kolbasin, director of the BI competence center of AT Consulting (ATI Consulting Group) (AT Consulting), told TAdviser that his company also completed a pilot project in one of the Russian mobile operators. Based on the open source solution, a cluster was created into which all CDRs are loaded and analytics about the subscriber's geolocation are built at the time of the transaction. This allows you to get information about the distribution of subscribers on the map, and also helps to work with subscribers more targeted. The entire data stream was processed tens of times faster than in traditional systems, while the cost of a cluster is orders of magnitude cheaper. As a result of the pilot project, areas of application of big data technology in the telecom industry were formed, their effectiveness was shown both in terms of cost reduction and in terms of productivity improvement.

All leading vendors are represented in the Big Data eco-system. In addition, there is open source Apache Hadoop software, which forms the basis for commercial releases. In the Russian market, from the point of view of analytical systems based on BigData, the most "convenient" are solutions from IBM, Oracle and Teradata, system integrators consider.

In Russia, Big Data technologies can be in demand in any companies where managers are ready for serious innovations with a return horizon of 5 years. In itself, the volume of these roles practically does not play, they always exist, you just need to have at least some idea of ​ ​ how to extract value from them for business. Industrial companies, for example, can turn to the PCS level, there is enough data there, but so far there are not a lot of ideas on how they can help at the management level, at the level of business processes.

In general, experts interviewed by TAdviser expect that after 2103 the big data technology market in Russia will move from the testing stage and customer interest in real commercial deployments.

In Russia, the Big Data market is still small. These are several dozen projects - pilot, or at the initial stage of implementation. IDC believes that this market sector is about $340 million. About $100 million is accounted for by SAP business analytics solutions, approximately $250 million is made up of similar solutions from Oracle, IBM, SAS, Microsoft.

EMC Data

In October 2013, Dell EMC published a survey in which 678 IT managers of Russian enterprises shared their views on what tasks and opportunities, including new competencies, associate them with big data and IT transformation.

Russian experts note that the use of big data leads to a significant improvement in decision-making processes, has a positive effect on the competitiveness of companies and simplifies risk management.

• 70% of respondents in Russia believe that analyzing their company's data will help make more informed decisions, and 35% of respondents confirm that the top management of their companies relies on the results of big data analytics to make principled business decisions.
• 31% of respondents reported that their companies gained a competitive advantage as a result of the introduction of big data technologies, and 51% of respondents believe that industries that use such tools will show the highest growth.
• More than half (51%) of respondents agree that big data analytics technologies will play a crucial role in identifying and preventing cyber attacks; this may be a decisive factor, since only 67% of respondents in Russia are confident that they will be able to fully recover all their data if necessary.

At the same time, the survey revealed a number of reasons that affect the decision-making on the implementation of big data analytics in Russian companies:

• 25% of the companies in the survey do not currently plan to implement big data technologies.
• among respondents who do not plan to implement big data, 37% cited irrelevance to business as the main reason that prevents their implementation.

Since companies in Russia still see IT-Innovations as the basis of competitive advantage in the domestic and foreign markets:

• The most common business priorities that drive IT transformation include business processes efficiency/operations (68%), customer service improvement, and customer engagement (37%).
• 76% of respondents note that investing in technology is a strategically important factor in achieving
• business objectives of their enterprise;
• 71% of respondents predict that in the next three years it will be an important task to maintain the skills of specialists at a level consistent with the pace of development of IT technologies.

See also Big Data

Big Data

Big Data - Catalog of systems and projects

Big Data (Big Data) Global Market

BI and Big Data Overview

Business Intelligence (Russian market)

The main trends of the BI market in Russia

Big Data Specialist

Big Data

Business Intelligence (Global Market)

Geography of BI projects

Russian BI: Industry Specific

BI Implementations in Russia - Typical Errors

Big data in civil aviation

Notes