Storage systems in analytics

World market

According to IDC Big Data Drives Big Demand for Storage^[1] of April, 2013, demand for technologies and Big Data services directly affects the market of the storage systems which are key components of solutions for work with "Big Data". For this reason during the period from 2011 to 2016 the market of storage systems for analytical solutions will show unprecedented annual average growth rate – 53%.

As the number of the data generated, processed and placed in storages will continue to grow promptly, storage systems will remain to some of the most expensive components of infrastructure of the solutions Big Data and analytical solutions, the author of a research Ashish Nadkarni believes.

Proceeds from sales of DWH it is specific in the Big Data market and analysts (BD&A) will grow from $379.9 million in 2011 to $6 billion in 2016. Growth of expenses on DWH will stimulate in analytics as well that circumstance that more and more companies address processing of new data sources, in general changing approach in work with data from search to data discovery so expenses on infrastructure of data and the data organization platform grow.

During the survey of IDC conducted in the first quarter 2013, 68.6% of the companies called performance a key factor when choosing system architecture of storage. On the second place on the importance users placed the cost (59.5%). 31% of respondents noted that do not keep projects on creation of infrastructure for the systems of the business analysis yet, but are going to begin like those in the next half of year.

Russian realities

Creation of the data warehouse makes one of main articles of an expense of the BI project. Up to 70-90% of the project budget of creation of a system of the business analysis it can be spent for the organization of the corporate data warehouse (CDW). At the same time costs for analytical applications in itself in comparison with these costs are small also regarding the cost of licenses, and regarding services in a technical part of implementation.

The role of data warehouses in BI projects is big not only taking into account financial reasons: KHD is a fundamental element for the system of the business analysis without which the last it will not be simple to work. As respondents of TAdviser experts recognize, approximately in 50% projects of implementations of BI systems there is a need to alter or to cardinally rebuild the data warehouse. Basic reasons: insufficient completeness available in an information warehouse in relation to requirements of the new BI solution and also a performance concern for which solution optimization, and sometimes and reorganization of the data warehouse is necessary.

The main requirements to data warehouses – high quality of information which is contained in them, reliability and coordination of data, their completeness and independence of applications, -Olga Gorchinskaya, the head of research projects and the Big Data direction of FORS company told. Lately in process of growth of the sizes of storages and increase in number of users of BI tools, the high performance, scalability, availability of information is added to these standard requirements.

In architecture of storage and the hardware-software platform the possibility of its optimization should be put. Besides, it is important to use the best practices, recommendations and modern methodology of creation of data warehouses. To a system it is better to involve specialists of vendor to assessment of technical requirements, especially when developing a configuration of the hardware, - Olga Gorchinskaya considers.

Andrey Rozanov, the head BI of RDTECH company, says that if the BI project means creation of the analytics based on given from a large number of different sources, then in this case it is difficult to do without creation of the data warehouse.

The main requirements to such storage are: correctly designed architecture; industry logical and physical data models; processes of periodic data-refresh and check of their quality. "The accumulated experience of creation of the similar systems says that it is very important to pay from the very beginning of the project special attention to quality loaded and afterwards the analyzed data. For this purpose it is necessary to provide use in loading process and data translations accomplishment of different procedures of data validations with a possibility of the subsequent analysis of results of their work", - Andrey Rozanov explained.

Besides, the systems of creation of data warehouses should have surely means of cleaning to provide the user with complete and qualitative information.

All advanced technological solutions for creation of corporate data warehouses are sufficiently presented at the Russian market, however, problems in such projects are frequent. They are connected first of all with the incorrect organization of architecture of the storage and the basic principles of its work so one and all system integrators are advised to select for such projects of the experienced partner.

According to world statistics, to a half of projects of data warehouses as a result are considered not successful. Errors in such unsuccessful implementations are similar: the storage is created not under requests of business users, and under available types of data.

What needs to be made for correct work of KHD and the system working over it? If the storage is already used, appears a good opportunity to check, the declared principle that at implementation of the data warehouse business requirements should be considered was how successfully executed practically always.

Let's say that all of them are considered and approved with requirements imposed to the BI system implemented over storage. At such optimistic scenario only optimization at the level of storage, small completions, refining of algorithms of the indicators calculated at the level of the data warehouse is required. Quite often happens and so that these storages are not used, are not checked by business users, and it (storage) becomes actually useless, when it comes to data application in the BI system seen by all.

Whenever possible it is necessary to integrate tasks of implementation of the data warehouse and a BI system in one project. The general analysis stage will allow to consider real business requirements at design of the solution. In turn, captious testing of data in storage using BI tools will increase the probability that users will operate further with the correct, demanded by business data.

At the same time it is not necessary to try to make so that the storage covered all functional areas at once, to try to place in it all available data. It is better to use the iterative approach at which after each iteration the limited, but most verified by business users set of qualitative data will be stored in storage.

It is also necessary to find out how given will be used and to create the concept in the field of reports and analytics, having separated data on priorities, on types of users, on the speed of response to a request. After processes of use of storage are thought over and described, there is a logical and technical work which has much more chances of success.

Olga Gorchinskaya, reminded that it is always necessary to remember quality of data and in advance to think over and design processes of consolidation, approval and check of correctness of data, not to forget about solutions of the class Master Data Management and to use specialized means of cleaning and standardization of data. At last, it is necessary to pay attention to system characteristics of storage, a scaling option, high performance at data loading and during the work with information of a large number of users.

See Also

Software of DWH (world market)

DWH (world market)

Business Intelligence, BI (world market)

Big Data world market

Business Intelligence (Russian market)

Geography of BI projects

The main trends of the market of BI in Russia

Russian BI: industry specifics

Implementations of BI in Russia: typical errors

Notes