Big Data in Russia
The main consumers of Big Data technologies are banks, telecom operators and large retailers. The main problems of development of the direction of Big Data are the shortage of qualified personnel, lack of sufficient experience of the Russian implementations and also high cost of solutions.
BI and Big Data overview
As Moscow uses Big Data when providing state services
Speaking at the TAdviser Big Data and BI Day conference on March 4, the head of the analytical department and monitoring of management of development of state services DIT of Moscow Alexander Filatov told about experience of use of tools of predictive analytics in the city. For several years the department passed a way from providing simple data on statistical reporting before use of advanced mathematical methods in the daily work, he noted.
For state structures, unlike the commercial organizations, to the forefront there is not an achievement of some economic indicators, but achievement of indicators of state programs and accurate following to regulations of services, including completion dates. The first two tasks of division are connected with it.
The third task is connected with collecting, storage and data processing which arise in interaction of citizens with authorities. In total about 30 data sources are used, each of which has history and the problems. To manage, are implemented including mathematical models of the predictive analysis.
One more task in which the department of analytics and monitoring of management of development of state services indirectly takes part is an increase in attractiveness of services for the user.
Among data which are used, for example, those which are provided by the user for rendering services to him and also data of transactions which arise in interaction of the user with authorities.
Data from sources are collected in storage, the normative reference information is added to them and all this moves on an input to the mathematical models implemented by a separate layer on microservice architecture and at the exit the end result is issued.
The predictive analytics is used, for example, for search of time series to predict measure values and to reveal some anomalies of processes, Alexander Filatov says.
| ||On an input we give to an algorithm a time series of transactions, and at the exit we receive forecast value and an interval within which this value can fluctuate, - the representative of DIT of Moscow explained. – It can be used for forecasting of indicators of state programs.|| |
One more example – calculation of load of infrastructure and forecasting of need of selection of an additional pool of resources under some splashes. Also monitoring of a measure value of work of processes of rendering state services is performed.
| ||For example, if we see that there is abnormally large number of failures in providing any service, it is a signal for our bodies which are engaged in control and supervising activity and methodical ensuring process to leave "in the field" and to understand on the ground that occurs, - Alexander Filatov says.|| |
Other direction is connected with management of data sources: it is predicted how many records with data each source should transfer if some abnormal value is observed.
Big layer of work is connected with studying of behavior of users. User data about transactions undertake, are digitized and transferred to a vector space. On the basis of methods of classification and a clustering it is possible to look on user groups what groups of services interest them, and it is possible to watch selection on services – to what categories of users these services are interesting.
| ||If we add time marks to data sets, we can use algorithms of associative rules, and then it is possible to look not only groups, but also to see the sequence of services which were ordered by the user and to predict a chain of his subsequent actions, - Alexander Filatov explained. – Thus, we are going to create "super-services" on the basis of the analysis of preferences of the user – the service packages which are most acceptable for it.|| |
The Ministry of Telecom and Mass Communications suggested to regulate Big Data
The Ministry of Telecom and Mass Communications in February, 2020 drafted the bill directed to regulation of the market of Big Data (big data). In the document the ministry enters determinations of concepts: Big Data, operator of Big Data and processing of Big Data. Roskomnadzor will control big data turnover. For this purpose department will create the register of operators of Big Data. Players of the market call the bill "crude" and unreasoned.
According to the bill, Big Data are defined so: "Big Data - the set of not personified data classifying by group signs including information and statistical messages, data on location of movable and immovable objects, quantity and quality characteristics of types of activity, behavioural aspects of the movable and immovable objects obtained from different owners of data or from different struktuirovanny or nestruktuirovanny data sources by means of collecting using technologies, methods of data processing, technical means providing consolidation of the specified data set its repeated use, systematic updating which form of representation does not assume their reference to the specific individual".
In the draft of amendments to the Federal Law "About Information, Information Technologies and on Data Protection" it is said that Big Data are meant as all data which can be received from owners of the structured and unstructured sources, using any technologies and means.
State agencies, municipal bodies, legal persons or natural persons, self-regulatory organizations or public associations (NPO and foreign agents too) which will organize can be operators of Big Data or process big data. Define the purposes of processing of Big Data, their structure and an algorithm of actions with them.
Processing of Big Data is meant as action or set of actions which is made by operators of Big Data using the automation equipment or without their use. It is about collecting, record, systematization, accumulation, storage, updating, change and also about extraction, use, transfer, removal, destruction and the analysis of such data.
According to the text of the bill to define the principles, the rights and obligations of big data operators, an order, control and conditions of their turnover will be the Government of the Russian Federation.
2019: Creation of the code of self-regulation of the market of Big Data
At the end of August, 2019 it became known that the Institute of Development of the Internet (IDI) and the Association of Big Data (ABD) drafted the code of self-regulation which, as expected, will allow to avoid additional legislative restrictions.
The initiative is among other things designed to resolve an issue of an opportunity to freely use public data — for example, posted in social networks, Kommersant tells.
Consent to personal data processing can be received in any form, including remotely. But at the same time use of data for targeted marketing will be recognized ethic. However only if offers will allow the potential acquirer "provide the optimal choice of goods" and will not be "unreasonably persuasive".
The CEO of IRI Sergey Petrov noted that in this case "it is necessary to consider not only laws of the market, but also the right of consumers".
The association notes that within five years the Russian market will grow by 10 times, to 300 billion rubles by 2024 that requires it "a professional regulation".
| ||Development of the uniform act at this stage can not meet expectations. At each type of data the specifics which are difficult for registering law language — Sergey Petrov considers.|| |
| ||It is necessary to define what information volume about users can use and process business and how these data should be stored — he believes.|| |
Boston Consulting Group for Association of Big Data
In what spheres Big Data technologies are most demanded
It is difficult to call the industry where technologies of the analysis of Big Data will not be demanded in the short term. At the same time the Big Data direction develops more actively in the companies which saved up big layers of the structured and unstructured information: financial sphere, telecommunications, Internet commerce, retail.
Telecom operators work with the large volume of data on the users. They apply Big Data technologies to a number of the directions: forecasting of outflow of subscribers, forecasting of complaints, planning of actions for customer retention, prevention of fraudulent financial transactions, etc.
In retail by means of analytics of Big Data it is possible to aggregate, for example, information on the interests of visitors of shops and on the basis of this very exact cut of audience to predict effects of different marketing campaigns and actions.
| ||In the near future there will be more implementations using technology of Big Data in a public sector. The extensive data arrays which are saved up by federal state agencies are a huge resource which can be used for development of digital society and increase in process performance of public administration, - Yulia Kudryavtseva, the director of strategic development of Foresight company notes.|| |
Demand from outside a goszakazchka and state corporations, is in many respects caused by import substitution and development of Digital economy. The state customer understands that the saved-up data – a valuable asset in the field of public administration, federal information systems store huge industry Big Data.
| ||For example, in health care of Uniform State Health Information System allows to collect statistical information on the industry and to monitor each locality regarding security with the necessary equipment and medical services. In the construction industry – the information and analytical system created for implementation of reform of pricing in construction. In a system the estimate price of construction resources and services by each region of the country therefore directories with a total amount about 50,000 positions of the estimate prices are published is calculated. In the financial sector – GIIS "Electronic Budget" in which there is a complete process of preparation of the bill of the Federal budget of Russia, - Timur Akhmerov, the CEO of BARS GROUP tells.|| |
According to experts, the list of spheres where Big Data technologies are demanded, in fast time will be replenished with also transport industry, power, the oil and food industry.
| ||To transport companies, for example, Big Data technologies allow to optimize planning of logistics and its tariff regulation due to tracking of a status of the transport park, an expense on fuel, monitoring of requests of clients, - Timur Akhmerov explains.|| |
Besides, very closely Internet of Things is connected with Big Data. Various sensors, measuring devices for a water consumption, the equipment at the robotic plants, smart transport – all of them generate a huge number of information which is transferred from the machine to the machine, and then is exposed to a research and processing by people.
| ||At last, the Big Data tools are mastered by a pool of the companies in which instant decision making depending on change of a situation in the market and in business is required – now under this determination almost any Russian business gets, - Roman Konovalov, the CEO of ID-Management Technologies adds.|| |
Agriculture, construction and some other industries can not always brag of high penetration of Big Data technologies. It is connected, mainly, with big product lifecycle in each industry: construction of the building and for removal of a new sort of a plant requires a lot of time that influences collecting of suitable data sampling for the subsequent analysis.
| ||However general digitalization and automation of many processes will promote that deep data analysis will be used in the future and in these spheres, - Denis Afanasyev, the CEO of CleverDATA (Lanit group) is sure.|| |
If to speak about the specific directions of use of Big Data technologies, then it is impossible to ignore such solutions as the analysis of video, images and other unstructured types of data. New technologies allowed to analyze them not only effectively, but also quickly.
| ||Applied versions of solutions for application in security agencies and law enforcement agencies are already created. So, the system of face recognition using video cameras allows to delay those who are wanted, and intelligent systems of video surveillance in shops reveal suspicious behavior of buyers, - Alexey Davletyarov, the head of group of development of department of complex design of information systems, Swagger — Development Center company tells.|| |
In general experts speak about two approaches in work with technologies of Big Data. The first – when purchase not technology, but already ready-made product where inside Big Data technologies are built-in, and to the client in general all the same as it works inside. Often it is cloud solutions. The second – creation of the solution based on these technologies in the company with involvement of external experts or independently. The second approach is actively applied by telecommunication, production companies, retail, bank and insurance sectors.
| ||All this the industries in which a large number of data and where business divisions realized their value already collected and aim to receive competitive advantage for the account of Big Data technology and to monetize data, - Egor Osipov, the expert in Big Data Croc tells.|| |
Cognitive data processing and tools of the analysis Big Data will improve those business processes where it is necessary to process big arrays of unstructured information in short terms.
First, it is the sphere of marketing. For calculation of efficiency of a marketing campaign, prediction of outflow of buyers or the following possible purchase the functionality of the traditional BI systems does not suffice often – to increase forecast accuracy, tools which can consider at the same time a set of parameters, factors and all available customer information are necessary.
| ||In turn, solutions based on cognitive analytics will allow to create, for example, patterns and templates of consumer behavior, being based on the complex history of all interactions with each client – and on this basis to do it personal offers or to predict leaving, - Artem Kaptsov, the Head of Department of integration services and complete solutions of Navicon believes.|| |
One more sphere which will benefit from implementation of smart instruments of data processing – finance. Machine learning and artificial intelligence will help to predict exact budgets for the period here and to quickly re-plan them.
At last, instruments of processing of Big Data will significantly simplify life of back offices, perfectly coping with data reading and execution of documentation, including agreements, agreements and all types of the reporting, on the preset templates and with the minimum attraction of human resources.
| ||For example, thanks to instruments of recognition based on artificial intelligence the smart BI systems can freely read out information from scans of documents in different formats and use it for the analysis, on an equal basis with other data, - the expert of Navicon notes.|| |
What constrains market development of Big Data in Russia
High cost of solutions and lack of fast results
Though interest in the solutions BI and Big Data grows in all spheres, the pacing restraining factor, especially in the companies of medium business, there is a strategy of survival in the absence of the development strategy and breakthrough, and, as a result, economy on the IT budget. Customers need not just IT technologies, they need the competitive business ideas and economic effect in the near future.
| ||In other words, many customers of medium business are not ready to work for perspective, they live in one afternoon, without looking for the horizons precisely known and necessary right now and saving on investments, - Denis Seroshtanov, the head of information and analytical systems Interprocom explains.|| |
Instruments of processing of Big Data require big computing powers and consequently they, are expensive in purchase, installation and use.
| ||Business users under such circumstances want to see return of investments into the equipment in the near-term outlook. However in practice it does not occur – as well as any analytical tools, the Big Data systems are aimed at business optimization and do not bring "fast" income, - Roman Konovalov, the CEO of ID-Management Technologies notes.|| |
Artem Kaptsov from Navicon, adds that so far developers cannot make the solution Big Data so simple that they were available to everyone. But as soon as the Big Data market will pass into more "mass" phase of development, we will see sharp simplification of user interfaces and rapid falling of the prices of solutions, he is sure.
Yulia Kudryavtseva, the director of strategic development Foresight, also ranks budgets and aspiration of customers in advance to estimate efficiency of investments as market restrictions. According to her, the innovation projects or difficult tasks of optimization are connected with processes of a research and numerous iterations of verification of methodological models. However not all are ready to go to the project which does not promise the guaranteed result.
Deficit of specialists
In the market deficit of specialists who are able to implement projects in the field of Big Data is still observed. In Russia competence centers which would be engaged in their mass preparation were not formed yet. Therefore successful cases are rather story of separate companies and developers.
| ||Many companies try to find specialists on an outsource, but because of deficit of qualified personnel many projects just do not "shoot", - Egor Osipov, the expert in Big Data Croc notices.|| |
Besides, in Russia there is so far no professional community which would undertake a big task – informing the market from within.
| ||The request is from outside both developers, and customers, and separate vendors and specialists have competences. I think that it is worth using as much as possible different formats for creation of expert community. All of us are participants of uniform IT space, and exchange of experience will allow to strengthen the market potential of domestic IT development of Big Data technologies, - Timur Akhmerov, the CEO of BARS GROUP notes.|| |
According to the CEO of CleverData Denis Afanasyev, application of Big Data in practice strongly depends on competences and skills of specialists therefore it is important to companies to develop own examination. For extraction of advantage from data the analysts combining skills and the mathematician, both the developer, and the business analyst are required. The synergy of these competences allows to understand at the same time in the field of data analysis, statistics, to consider engineering feasibilities of projects and practical application of Big Data.
Andrey Baybutov, the director of business development of department of BI of Corus Consulting Group, tells that it is often quite difficult to motivate and attract in a command for the project in Russia competent people as the most part of the highly qualified specialists having experience of creation by the high-loaded Big Data of architecture works at projects abroad.
| ||The Russian market of Data-specialists at the moment is on initial stage, but it actively develops. And if in the western market many companies already have in staff of necessary experts for creation of own digital products and monetization of data, then in the Russian market only large players began work in this direction. Existence of highly skilled data scientists allows business to increase structure of revenue thanks to implementation of digital projects in addition to primary activity of the company, - Andrey Baybutov says.|| |
The problem of low-quality data is still relevant for the Russian customers – on the basis of separate or false data effectively it is impossible to solve analytical problems.
| ||But it is important that the direction is designated and in general advance is traced, and at the market there are Russian BI tools which provide integration with different data sources that is vital for implementation of the Big Data projects, and tools of advanced analytics at the same time. For example, "Foresight. Analytical platform". In it is provided integration into commercial platforms among which – Teradata Oracle Exadata SAP Hana HP Vertika IBM Netezza , etc.) and also open source products (for example PostgreSQL Hadoop , etc.), - Yulia Kudryavtseva from Foresight company says.|| |
Limited choice of solutions
In the market there are not a lot of solutions which are really capable to work with large volumes of unstructured data effectively. At the same time only the largest players of the market whose amount of data is calculated by petabytes can use them: telecom, retail, finance.
| ||And even among them not everyone is happy with real results from implementation of existing solutions – they need to be finished, done still seriously more praktiko-focused. The analytics of Big Data should not be implemented for the sake of the analytics, otherwise business will not receive financial results in the foreseeable future, - Artem Kaptsov, the Head of Department of integration services and complete solutions of Navicon notes.|| |
One of the pacing restraining factors for development and improvement of tools of analytics in Russia are concerns of customers in the field of confidentiality of data.
| ||In spite of the fact that instruments of cyber defense of new generation, actively are implemented into business practice, users of the Big Data systems still are careful of drainings of confidential information on the companies and also personal data of clients, - Roman Konovalov, the CEO of ID-Management Technologies tells.|| |
More active market development is interfered by mistrust of consumers to technologies and also single questions of regulation of the market.
| ||For work with data of Internet users and their application it is necessary to provide confidentiality and special storage conditions of personal data, - Denis Afanasyev from CleverData adds.|| |
Big Data technologies are often perceived negatively since noise around them in recent years there was very much, but many companies for themselves did not see obvious scenarios of application. As a result – some organizations draw a wrong conclusion that it is rather fashionable, than useful technology.
It is important to understand that market leaders are ready for projects with Big Data technologies today. For other companies Big Data – not the key driver of development. A part of the companies still has today even no corporate data warehouse or the strategy of Data Governance therefore they do not think of applying these technologies.
| ||We advise such companies to build at once the correct architecture of their analytical system. Modern solutions for analytics of data rather complex, consisting of bigger quantity of components. It is not obligatory to put at once to itself all components, including Hadoop or any other difficult decisions at all. It is possible to use only those components which meet current demands. But it is important to select such architecture which over time when the company understands that it is ready to work with large volumes of data, it will be possible to expand easily, - the expert in Big Data Croc Egor Osipov explains.|| |
At last, the lack of real practical experience at most of producers and integrators contains the serious growth of the market.
| ||The customer wants to see real effect which system implementation brought them to competitors or other industry companies. When the IT company cannot show project experience, trust to it and to the implemented solution sharply decreases, - Roman Konovalov says.|| |
Also it should be noted insufficiency of practice of data acquisition from external sources.
| ||Not social networks, the websites, open data – everything that can be received from there mean, it is used most effectively. But for the solution of a number of tasks the data belonging to other organizations and not available owing to legislative restrictions occasionally are required, - Alexey Davletyarov, the head of group of development of department of complex design of information systems, Swagger — Development Center company explains.|| |
Use of Big Data for identification of illegal lease of housing
On July 20, 2018 it became known of development in Moscow of an analysis system of Big Data for identification of illegal lease of apartments. Find non-payers technically not easy, but the Department of Information Technologies (DIT) of Moscow solved a problem, the head of department Artem Yermolaev on the sidelines of the Moscow urbanistic forum told RBC.
He explained that the mechanism which will allow to define those who do not pay taxes is already tested and, "at some point" these data will be used. However for full start of technology it will be required to make changes to the legislation.
| ||It is the complex circuit because it is organizational and normative and legal. Here intersection of areas of responsibility — Yermolaev noted.|| |
Artem Yermolaev in 2017 at an urbanistic forum said that work on identification of lessors who disappear from payment of taxes is conducted. Then he told that it is supposed to analyze the largest Internet resources on which apartments in lease are offered and to compare these data with statistics on payment of taxes. After such analysis data on those apartments which, perhaps, are illegally leased, it was going to transfer to tax administration for check.
Date director of Weborama Russia Dmitry Egorov said that from the technical point of view the mechanism about which speak in DIT — solvable. However in practice authors of declarations not always specify the exact address of a subject to delivery. Besides, often apartments are leased by realtors, but not owners. And the commercial director of AmberData Victor Mityunin believes that he at DITHAT will turn out lessors as the declarations with phone numbers are in open access given to get.
According to the Moscow Department of Economic Policy and Development, in nine months 2017 in the city illegally leased about 27 thousand apartments. At the same time annually in the capital in lease 200-300 thousand apartments are offered.
2017: Trends and perspectives in the Big Data market
Roman Baranov is the head of a business intelligence and data warehouses of Croc company — in August told about trends in the Russian Big Data market. According to him, the concept of Big Data which entered a "hot" top of technologies of the analysis in recent years gradually gets out of fashion. IT specialists do not wait any more for revolutionary changes in this area but only amendments of approaches and sales opportunities of these or those tasks, but a set of products, necessary for work, was already defined.
Whether the relevance loses Big Data?
Of course, no, the expert claims Croc. It is still one of key trends in the market of analytics. At the first stage "hype cycle" Big Data it was perceived as exotic. The new concept based on open source-технологий was a new experience for the Russian business which got used to boxed solutions of the famous foreign developers. The words "Big Data" helped to accelerate approval of the project business practically twice even if the solution belonged to Big Data very conditionally. New products are difficult to be integrated and operated; today these problems are solved using various connectors, visual tools intended for operation and work with data arrays. Emergence of the Russian Hadoop distribution kits and products for work with them does the market more clear, processes of purchases, support and training become more transparent. All this, eventually, levels a former gap between Big Data technologies and the Russian IT reality, Roman Baranov considers.
To save Big Data as a method
Big Data represents a great option for situations when traditional solutions are too expensive or difficult in operation, the expert noted.
| ||From the last examples I can remember recent events in the collection market which strongly changed on January 1, 2017. Became effective the law which strongly limited opportunities for communication with the debtor. One bank addressed us that we helped to provide control in this sphere and to combine the interests of the customer with requirements of the legislation. Use of classical technologies was rather expensive as then it should hold the huge arrays of information collected from all branches through the whole country in one process. And Big Data allowed to reduce the price considerably of the solution and to execute the project for several months — Roman Baranov emphasized.|| |
Big Data in trade
According to forecasts of the expert, in the near future, in addition to a bank segment, interest in Big Data will be in the field of e-commerce. Here these technologies experience a rebirth. Business understands how to earn and what solutions will help it with it. For example, sales of services of logistics which cannot go completely to online and with Big Data become a source of an additional income at many companies and especially service aggregators.
In retail trade of Big Data it is actively applied in the field of Wi - Fi - analytics which allows, having involved signals from mobile devices of visitors, to make a representative analytical cut: duration of a visit to shop, frequency of visits, movement routes, distribution of visitors by the territory, intersection of audience with other objects of shopping center, a share of visitors of shop from among all passing by, etc.
Video analytics and face recognition
One more solution within the concept of Big Data, interest in which only inflames — video analytics and face recognition. This technology automatically detects and selects information, significant for the customer, from a huge video flow, allowing to count, for example, the number of visitors, to analyze loading of a trading floor, to monitor queues, to foresee the interests of buyers, to control activity of personnel and cash transactions, to instantly detect suspicious actions, to send the personified advertizing, etc.
Big Data also allows to solve a number of questions, concerning security. The enterprises do not have now need to hope for attentiveness of the security service monitoring ten monitors at the same time. The system using possibilities of Big Data technology can automatically distinguish suspicious actions in the territory of the enterprise or, for example, shopping center or the airport. Scenarios can be quite a lot here, everything depends on features and tasks of business, the representative noted Croc.
| ||Customers want to understand rates of return of investments more often. Technologies of Big Data reached that level of development when business accurately understands their applied relevance and can estimate positive effect in money. It is possible to wait that further on this way there will pass Internet of Things and a blockchain that will bring closer digital transformation fashionable now to the people — Roman Baranov concluded.|| |
Status of the Russian Big Data market
The Russian Big Data market is on initial stage of development and this term often is understood as traditional BI approaches. The main consumers of technologies of Big Data, as well as the main carriers of large volumes of data, the companies in the banking sector, a telecom and trade are. For them the analysis of large volumes of the data connected with solvency analysis of clients consumer behavior and market conditions is the major tool for maintenance of competitive advantage.
In recent years in all companies from the big three of mobile operators there were divisions specializing in work with Big Data, and they are not just information divisions for development of client profiles, they are business units which are designed to generate an additional profit.
| ||In a telecom began to transfer a big circuit of systems to Hadoop. Data bulks from billing systems, CRM and other sources develop in Hadoop, are aggregated and already over this information build on BI allowing to understand where at present there is a subscriber and what his requirements to offer the best service, to make the offer for each specific client of the most attractive, - Andrey Baybutov, the director of business development of department of BI of Corus Consulting Group tells.|| |
Retail is in number of pioneers of the Big Data market too. More and more companies from this segment create separate divisions on work with data that as it is possible to plunge more deeply into lines of checks for 2+ years and to find the new hidden interrelations, Baybutov adds.
| ||We expect emergence of interesting cases in the nearest future, and not only in the industries, "traditional" for Big Data, such as finance, telecom and retail, but also in the industry, logistics, construction and health care, - Vakhmyanin adds.|| |
Pavel Adylin, the chief executive of Artezio company (LANIT group), considers that they should act as the potential customer of Big Data of projects in Russia in the nearest future as well the companies of a public sector since they have the huge saved-up amounts of data suitable for the analysis.
Konstantin Chernousov, the deputy CEO of Vesolv, gives an example of the implemented project in a public sector: "For example, the Federal Tax Service completed the first project with use of Big Data on tracking of a chain of payers of the VAT and suppression of frauds on withdrawal of the VAT".
As for the solutions proposed by developers, it or the international commercial products from Oracle, SAP and similar, or the solution based on open source of technologies. There is practically no domestic software for processing of large volume of data, Chernousov adds.
Andrey Nugmanov, the partner of AT Consulting, the director of the BI block, considers that in the sector of "new BI" – the analysis of Big Data, processing of events and decision making in real time – the stack of Open Source actively restricts products of traditional vendors. It develops in the light of the updated vision of functional requirements to BI and technology in many respects caught up with a proprietary stack.
| ||The open code, the transparency of development, legal purity and availability guaranteed and the support which are not closed on one vendor, tolerance to the equipment, the highest popularity of Open Source, first of all among young and perspective specialists, – all this becomes the reasons of active replacement and washing away of a "old" proprietary stack from traditional niches, - Nugmanov is sure.|| |
Vendors try if not to ride out a wave then not to be buried by it. Someone opens the code and passes to Open Source model of business, trying to revive interest at public, so and at leaders of opinions among buyers, to the traditional products. Others are actively integrated with large suppliers of services for support of a stack of Hadoop, trying to reduce the cost of ownership of the traditional products due to use of open opportunities of Big Data and to reach synergy effects of the hybrid solution.
| ||The client is not always ready to pay for licenses to vendor at once and tries to test independently technology, to understand degree of its applicability and to gather necessary examination for further operation. The choice of Open Source allows to provide rapid implementation of the interesting functionality without license fees and – thanks to lack of procurement procedures – in the minimum terms. We do not see any serious obstacles in development of these technologies in clients. And examination is present at the market, and at least there is a distinct business case providing reduction of operating costs of storage of considerable information volumes, - Nugmanov says.|| |
In terms of technologies, in AT Consulting observe that to the forefront there are solutions using In-Memory Data Grid (IMDG).
| ||Hadoop allows to collect diverse information and to store. Now time of the next step – to carry out difficult analytical calculations in the online mode came. Classical MPP platforms cannot provide fast reaction because of existence of read operations and record to disks and specifics of operating environment any more. Also the question of cost of such technologies is also important, - the partner of AT Consulting tells. - We see that solution in-memory are even more often applied to serious analytical problems. They provide a possibility of high-performance side-by-side execution of requests on strongly loaded analytical systems for service of thousands of users in the mode of high availability.|| |
Roman Baranov, the head of a business intelligence and data warehouses of Croc company, notes importance of understanding that the term Big Data becomes more and more indistinct every year. The list of technologies which can be carried to this concept becomes more and more. They are already ordinary reality of most the modern companies. Besides, saying "Big Data" many mean not only collecting and data storage, but also analytics, and Internet of Things for a long time, and many other things.
Trends of the Russian and world market of Big Data
Top trend of the Russian Big Data market — penetration of technologies of Big Data into those areas to whom before them it was difficult to provide.
If earlier huge number of segments, for example, production, not so actively paid attention to technologies of work with Big Data, then now an opportunity to collect information from all sensors and other plant equipment gives huge opportunities.
| ||It will allow to optimize significantly work on the production and also to increase efficiency of planning and to convert the acquired information into money which is lost at a deviation from the plan or profits do not dozarabatyvatsya from the point of view of lost, - Andrey Baybutov, the director of business development of department of BI of Corus Consulting Group says.|| |
According to Konstantin Chernousov, the deputy CEO of Vesolv, the general trend is that all want to use Big Data as the analysis of Big Data increases efficiency and company competitiveness. And one of the movable facts is, strangely enough, the appearing concerns from the fact that the competitor began to benefit, using new technology.
If to speak about world trends, then first of all it is possible to speak about a trend of transfer of infrastructure of Big Data in a cloud, Ivan Vakhmyanin, the CEO of Visiology company considers.
| ||It makes sense for many companies as server capacities for Big Data cost very much, and are necessary not always on a permanent basis. For example, we carry out most the Big Data of experiments to Visiology in a cloud of Amazon. Besides, cloud Big Data products often strongly facilitate work of engineers – a threshold of an input of most Big Data of software products in respect of deployment very high, and in a cloud it is possible to receive already configured cluster at once, - Vakhmyanin tells.|| |
The second trend, according to him, is a stream (streaming) analytics which allows to analyze the arriving data in real time. This opportunity is especially important for the applications created over the data collected from sensors (IoT, IIoT).
Pavel Adylin from Artezio, adds that separation of the Big Data direction which at us is understood in a general view, on a set of the independent directions solving narrower specific problems so far is characteristic of the world market.
For example, according to it, it is possible to select: software and hardware tools of ensuring storage of Big Data, means of parallel processing of data, means of data filtering and creation of models, visualization tools of data and their interrelations, means of work with images, machine learning, intelligent interfaces, automation of mental work.
Emergence of the ready-made industry solutions for small and medium business working as autonomous applications, and according to the SaaS or BDaS models (Big Data as Service) is also connected with such separation.
Barriers of the Russian Big Data market
Shortage of specialists
One of the main problems of the Big Data market in Russia - difficulties with search of qualified specialists.
According to Ivan Vakhmyanin from Visiology, deficit of such personnel is observed not only because they should have quite difficult enrollment of skills and competences, but also because today very few people understand how to train them, to estimate and to correctly organize their work.
Konstantin Chernousov, the deputy CEO of Vesolv, tells that now such profession as Data Scientist gradually becomes current. It is quite rare, but demand for it already enormous: about 50 requests for work are the share of one curriculum vitae of such specialist.
| ||In Russia such specialists who will tell the management about analysis opportunities using Big Data, will count the budget and implement the project, little, and to increase their quantity quickly it will not turn out as there are no rates not just, and even materials in Russian, - Chernousov notes.|| |
Andrey Tiunov, the CEO of the company BI Partner"I-Teco(Group), specifies that Data Scientist are experts from customer company who understand trends of the market, perfectly know business, find opportunities for its growth and are able to use the potential of data which own, for the solution of these or those tasks. They have core competencies according to the solutions Big Data, Tiunov says.
Andrey Baybutov, the director of business development of department of BI of Corus Consulting Group, also considers that good specialists in the field of Big Data in the market critically are not enough.
| ||If you enter the market of resources in search of the good specialist with experience with Big Data, machine learning, IoT, etc. now, hardly to a descent will be able to find the person with experience from two to five years of work moreover and with a necessary product portfolio. Therefore many companies try to cultivate own specialists under these tasks, - the expert explains.|| |
Lyubov Vedeshina, the head of practice of a business intelligence of Interprocom company, sees a problem that in Russia the expert community of analysts in the field of Big Data was not created yet, the competent customer and the competent contractor did not appear.
| ||On the party of potential customers we see the shortage of specialists who equally well would understand both industry specifics, and approaches, tools and methods of processing of Big Data. On the party more accurately experts in the field of world-class Big Data already appeared, some even are in a world top. But their units, - Vedeshina emphasizes.|| |
The analyst's profession in the field of Big Data for the present did not become mass. In universities there are no appropriate programs of preparation, besides because so far poorly competent teachers. Corporations partly compensate the shortage of specialists, offering own training programs. For example, SDA (School of Data Analysis) from "Yandex" and paid rates in Beeline.
| ||However these rates are not enough, should pass still some time that the number of the qualified analysts of Big Data would change quality of demand and supply in the Big Data market, - Lyubov Vedeshina believes.|| |
Pavel Adylin, the chief executive of Artezio, adds that owing to a shortcoming of specialists in the field of Big Data in Russia professions of Data Scientist, Data Analyst and Data Engineer most often are not separated. If the first of them is a creator of new technologies of information extraction from data, algorithms of machine learning, artificial intelligence, then the last is a developer of complexes of program or hardware-software providing for the solution of specific objectives of Big Data. For training of these different specialists it is required to implement different methodological approaches already now, Adylin is sure.
Shortage of experience of implementations
The pacing restraining factor in market development of Big Data in Russia a number of experts call a small amount of the Russian cases on which both customers, and integrators could rely. Therefore, projects are Big Data risky.
| ||Often it is necessary to hear from clients – "guarantee to us that implementation of Big Data of analytics will bring us economy N rubles", but such guarantees cannot be given, at least, before carrying out the eksplorativny analysis of the saved-up data and creation and verification of the first models that in itself requires an investment of resources", - Ivan Vakhmyanin, the CEO of Visiology company notes.|| |
About same also Lyubov Vedeshina, the head of practice of a business intelligence of Interprocom company speaks. According to her, potential consumers do not understand what benefit for their company and the industry is born by technologies of application of Big Data. Customers doubt that their investments into technologies of processing and the analysis of Big Data will pay off.
Also Konstantin Chernousov from Vesolv holds the similar opinion. According to him, the lack of knowledge of possible benefits from Big Data use constrains development of the Russian market.
The representative of Corus Consulting Andrey Baybutov also refers to experience of implementations, as to a market barrier:
| ||I know only about units, at most — couple of tens of implementations. The most part of projects with Big Data is done often on products open source with which the Russian specialists have some experience too. As a result there is a methodological unavailability that prevents to understand how to do projects, and technology — due to the lack of necessary program competences.|| |
| ||The western cases little on whom make an impression because the Russian realities quite strongly differ. Therefore at this stage the Big Data market is moved by the companies which are not afraid to experiment, invest in research projects, counting on those benefits and competitive advantages which Big Data can bring, - Ivan Vakhmyanin adds.|| |
Problems of quality of data
According to Lyubov Vedeshina from Interprocom even if the potential customer created understanding of the benefits from the analysis of Big Data and found industry experts-analysts in the field of Big Data, it faces a problem of quality and number of data which at it are saved up. As a rule, the data which are spontaneously saved up by customers are in the status not suitable for the analysis and obtaining benefit for the company, she notes.
The same problem is seen also by Pavel Adylin from Artezio. According to him the quality of data leaves much to be desired because of existence of distortions (emissions) and insufficient depth. Thus, it is required to expand considerably data sets for the analysis, but for this purpose there is no opportunity since in connection with personal data protection in our country there is practically no market of sale and purchase of information in the form of the exchanges of data (Data Exchange).
| ||Perhaps, data storage could be helped by the program of the state support of open sources of the digitized data, for example, access to primary data of Rosstat, etc, - the expert considers.|| |
The head of Roskomnadzor considers that in Russia the Big Data state operator is necessary
The head of Roskomnadzor Alexander Zharov considers that in Russia it is necessary to create the state operator of Big user data. The official proved it by the fact that, according to him, such information is national property, but not the property of the companies processing data.
"I consider that the state operator of Big user data should be. I support a position which was sounded by experts. That this national property, but not the property of the companies which process data. It obviously is the property of the citizen. But to understand up to what depth information on the personality should be structured, each person cannot. It should be national property" — RNS quotes Zharov's comment.
The head of RKN specified that the geolocation, biometrics, the user behavior on the different websites, etc. enter a concept of "Big user data".
"All this leaves marks in the Internet, is a subject of the analysis of the transnational Internet companies and, obviously, requires also regulation as now the 152nd law — "About Personal Data Protection" works, - added Heats. According to him, it is about creation of the new law, and the working group under the leadership of the Assistant to the President of the Russian Federation Igor Shchegolev is already engaged in it. At the moment the issue of regulation is handled with experts, until the end of 2016 specific proposals should be formulated.
TAdviser - 100 strategy of customers in the field of Big Data
Analysts of the TAdviser center conducted a research of the Big data market in Russia. During the research experts polled key IT customers to define the existing market conditions on similar technologies and also to designate its potential.
CNews Analytics: Level of a maturity of the market increased
The polled customers showed higher degree of awareness on these technologies and also understanding of potential of similar solutions for the business. More than a third of respondents already started use of Big Data technologies in Russia. It is possible to note that in the Russian market there is already a uniform conceptual field of this segment.
Yandex Data Factory creates offers in the field of Big data
According to the message Interfax, division of Yandex Data Factory (YDF) of Yandex company she intends to be engaged in a problem of development of offers for deduction of subscribers. At the beginning of 2015 it was reported that YDF analyzed more than 100 parameters describing behavior of 100 thousand players of World of Tanks. The forecast model of outflow of players which turned out as a result, it appeared for 20-30% more precisely than analysis tools, standard for the game industry.
It is unlikely Yandex will really undertake a problem of development of offers for deduction of foreign subscribers. Only the operator can hold the subscribers. I did not manage to find the blog entry or in the corporate section to which refer the news agency, but I think that it was talked only of forecasting of outflow of subscribers. For this purpose it is required to estimate so-called 'function of survival' of the subscriber, or, alternatively, risk function of outflow of the user. These tasks are well-known in statistics, unlike successful cases of their practical application.
Yandex, certainly, needs a flow of positive news, and Big Data – a grateful subject in terms of their generation. We enter an era when machines every second collect about us information, somewhere it is sent, somewhere archived and stored, and sometimes even overworked. Machines already know about each of us much more, than ourselves. It is unlikely players are capable to measure the behavior by means of 100 parameters in World of Tanks. And hardly all of them know today how all of them long will remain in a game. Of course, not all these 100 parameters are equally important, it is only in advance unknown what – are important and what – not really. To select important parameters, it is required to study behavior of 100 thousand players - but as Yandex prompts answers to one hundred millions of visitors, this task it can do.
Generally speaking, there is a lot of applications of methods of machine learning to Big Data, many of them - quite bright can also generate popular news. It is good for the company as in itself, and for the solution of a problem of encouragement of analysts of the stock market. This task – the most difficult today for Yandex. And a problem of forecasting of outflow of clients – quite old and well studied, approaches to its solution are published. A question only in implementation of those approaches which are efficient when customer bases total millions of users. In this area on 'grinding' of the solution with the purpose to make it suitable for the solution of practical tasks more years and means, than on development of the theory can leave. According to messages of the news agencies thanks to proprietary algorithms Yandex manages to build more exact solutions for 20-30%, than competitors manage. This difference does not look such essential. I think that the main competitive advantage of Yandex – available staff of experienced specialists, software and 'iron'.
In addition to activation of Yandex in b2b the sector, it is important still following. Apparently, repeats (or a story with SaaS proceeds) when instead of emergence of the new, independent companies we observe emergence of the new sales channel at large producers and software distributors. Big Data are the second of the directions of investment, most popular with venture funds, according to results of the research Venture Barometer Russia 2014, they concede only to financial technologies and that, it is impossible to call lag significant. Investors understand prospects of machine learning in the period of the growing wave of M2M-technologies – devices should not just exchange data, but also change the behavior depending on the acquired information. To put it briefly, investors want to invest money in the 'Big Data' direction. But hardly can.
The matter is that in case of Big Data, powerful hardware base, and secondly, really, high technologies are required, first. On only one superficial knowledge of methods of machine learning you will not construct much as for work with Big Data methods should be fast and steady. Practically useful methods are ground for years of practice – and Yandex has such practice, and at new players is it, most likely not. The amount of initial investments, apparently, too exceeds the amount for which venture investors would be ready to risk to try the new direction. Simple tasks which can be solved by means of not really Big Data and specialists of normal qualification for rather small amount will hardly make high profit, - just because can solve them many, so, fierce competition is inevitable. Yandex in this situation was protected by the unique barrier which is naturally constructed for years of work on the search engine, Yandex market and a management system for contextual advertizing.
Therefore in the field of 'Big Data', as well as in SaaS industry, not independent startups, but already well-known large companies will be the main players of the market. By the way, not only Yandex, but also, for example, the Russian Internet counters, LiveInternet, OpenStat and others could provide to the Websites service in forecasting of outflow of subscribers. If Yandex untwists this service, then it will become fashionable, and for other players there will be field of activity – search of minimum expensive implementation of solutions of the most running tasks.
Several dozens of pilots all over the country
Explosive growth of data and also aspiration to gain more knowledge of the clients pushes the large companies on search of technologies which will help to save huge amounts of data, to receive and analyze information from sources unavailable or hardly accessible earlier (for example, stream video, the sentence-analysis at the appeal to call center). Besides, in the traditional systems the cost of storage of 1TB of data is sufficient is high, award enforcement based on Big Data technologies allows to reduce considerably costs due to use of cheaper the equipment.
For 2013-2014 Big Data trend did not reach Russia fully yet. Distribution of this concept in our country is limited to pilot implementations and approbation so far. For example, the Croc company in 2013 had three "pilots",Maxim Andreyev, the head of business applications of CROC company told. One of them - for large telecommunication company on tracking of social communications between subscribers and to identification of levels of influence. A project objective - reduction of outflow of clients.
At the same time in Russia all practices which are available in the world in the field of Big Data are available at this time – beginning from open source, finishing with solutions of large vendors. As for cost, it varies depending on a task and the solution suitable under it. Use of this technology for the companies is an opportunity to break away from competitors due to improvement of the offered products or services, or significant optimization production and business processes. Therefore Big Data are demanded by all companies living in conditions of high competition, in particular, banks, retail, telecom operators and other.
"Interest in Big Data is still visible, and here projects of unit. Still most of specialists do not understand that such Big Data and, frankly speaking, so far is not a lot of companies in Russia at which this really there is a lot of and they are not structured", - Georgy Naneishvili, the partnership director of Qlik commented.According to him, the telecom and search systems use these technologies, but so far for the solution of very limited class of tasks.
Thus, technologies more than five years, however, exactly in recent years appeared real projects in the companies, including Russian whose business is significantly less, than, for example, than Facebook. Many Russian companies of different level working in the field of telecom and Internet services, public administration, the financial sphere and other directions are able to afford Hadoop technology, he considers.
Yury Kolbasin, the director of competence center of the BI block of AT Consulting company (AT Group) (AT Consulting) told TAdviser that his company also completes a pilot project in one of the Russian cellular operators. Based on open source of the solution the cluster into which CDR are loaded all was created and the analytics about the subscriber's geolocation at the time of transaction commission is formed. It allows to obtain information on distribution of subscribers on the card and also helps to work with subscribers is more targetted. All data stream was processed tens times quicker, than in the traditional systems, at the same time the cost of a cluster is cheaper on orders. As a result of a pilot project scopes of technology of Big Data in the telecom industry were created, their efficiency both in terms of cost reduction, and in terms of performance improvement is shown.
All leading vendors are provided in Big Data ecosystem. Besides, there is an open source software of Apache Hadoop which forms a basis in commercial releases. In the Russian market, in terms of the analytical systems based on BigData, solutions of IBM, Oracle and Teradata are the most "convenient", consider system integrators.
In Russia Big Data technologies can be demanded in any companies where heads are ready to serious innovations with the return horizon in 5 years. The amount of data of a role in itself practically does not play, they always are, it is only necessary to have though some idea how to take from them value for business. The industrial companies, for example, can address the PCS level, there are enough data there, but so far not really there are a lot of ideas as they can help at the managerial level, at the level of business processes.
In whole respondents of TAdviser experts expect that after 2103 the market of big data technologies in Russia will pass from a test stage and interest at customers to real commercial deployments.
In Russia the Big Data market is still small. These are several tens of projects – pilot, or on initial stage of implementation. The IDC company believes that this sector of the market makes about $340 million. About $100 million are the share of solutions on a business intelligence of SAP, about $250 million make similar solutions of Oracle, IBM, SAS, Microsoft.
Data of EMC
In October, 2013 Dell EMC published survey results within which 678 IT heads of the Russian enterprises shared the views of what tasks and opportunities, and, including, new competences, they connect with Big Data and IT transformation.
The Russian specialists note that use of Big Data leads to significant improvement of decision making processes, has a positive impact on competitiveness of the companies and simplifies risk management.
- 70% of respondents in Russia consider that data analysis of their company will help to make more weighed decisions, and 35% of respondents confirm that the top management of their companies relies upon results of analytics of Big Data at acceptance of basic business solutions.
- 31% of respondents reported that their companies got competitive advantage as a result of implementation of technologies of Big Data, and 51% of respondents consider that the industries in which such tools are used will show the highest growth.
- More than a half (51%) of respondents agree that technologies of the analysis of Big Data will play a crucial role in identification and prevention of cyber attacks; it can be a decisive factor as only 67% of respondents are sure of Russia that they will be able to recover all the data in case of need completely.
At the same time poll revealed a variety of reasons, the Big Data influencing decision making about implementation of analytics in the Russian companies:
- 25% of the companies participating in poll at the moment are not going to implement technologies of Big Data.
- among the respondents who are not planning implementation of Big Data, 37% called the basic reason interfering their implementation, prevarication for business.
As the companies in Russia still see in IT-Innovations a basis of competitive advantage in domestic and foreign market:
- the number of the most widespread priorities for business stimulating transformation of IT included efficiency of businesses processes / operating activities (68%), improvement of service of customers and interaction with them (37%);
- 76% of respondents note that investment into technologies is strategically important factor of achievement
- business objectives of their enterprise;
- 71% of respondents predict that in the next three years maintenance of skills of specialists at the level corresponding to rates of development of IT technologies will become an important task.