Data management

Data management - a process related to the accumulation, organization, storage, updating, storage of data and information retrieval.

Data management includes

• Data analysis
• Data modeling
• Database Management
• Working with Data Stores
• Data Extraction, Conversion, and Loading
• Data extraction
• Data Quality Assurance
• Data protection
• Data encryption
• Managing Metadata (Data Repositories)
• Data architecture

A separate TAdviser article deals with the very concept of "Data."

Split between Data Governance and Data Management

Just as the auditor controls financial processes but does not actually manage financially, data management ensures proper data management without directly performing data management.

Data management is an integral division of responsibilities between supervision and performance

2024

Data Management Culture - A Meaningful Approach to Doing Business

Over the year, more discoveries have been made in the field of digitalization and artificial intelligence than in the previous 20 years. Moreover, most of the new technologies and tools turned out to be related to the field of data management. Distributed data centers, the CDO (chief data officers) organizational structure as a factor in the cost of companies, the Data Fabric concept - "fresh" ideas seem to be enough.

Unfortunately, entrepreneurs still cannot simply offer staff advanced analytical capabilities. It is important to teach people to understand, data and to analyze and use them to inform opinions when making decisions. Creating a culture (data management Data Culture) is the task of each company that wants to follow the path of digital transformation and remain effective, shared TAdviser to article in a data management expert prepared for. Kirill Kuznetsov

Over 5 years, the share of Russian companies implementing data management initiatives has grown 5 times

Over 5 years, the share of Russian companies implementing data management initiatives has grown 5 times. Such data are presented on June 3, 2024 in the Data Management Maturity Assessment study prepared by DIS Group.

The study was attended by more than 200 leading Russian companies, including 24% of representatives of the financial sector, 20% - the production sector, 11% - telecom companies, 9% - the oil and gas sector and retail, 7% - IT companies, educational institutions, the energy sector, 4% - real estate and 2% - the public sector.

The document says that between 2018 and 2023, the share of companies that are in the process of implementing data management initiatives increased from 17% to 89%. In 2018, more than 60% of companies did not use special data management tools, preferring manual methods or their own developments. By 2023, the share of companies using proprietary data management software reached 82%, of which 48% are hybrid solutions using both proprietary software and open source.

Significant development of data management tools is expected as a consequence of the context and challenges facing the business. A significant jump will occur in areas such as master data management (23% growth), data quality (22% growth), business glossary (20% growth) and data catalog (20% growth). According to the study, by 2028, the share of companies implementing data management initiatives will approach 98%. At the same time, the share of companies using a hybrid combination of proprietary and open source software will reach 90%.

In the next 5 years, significant growth and improvement of data management tools using elements of AI and machine learning is expected. The implementation of such tools leads to a reduction in data management costs by up to 30%, an increase in employee productivity by up to 25% and an increase in data quality by up to 90%, said Pavel Likhnitsky, CEO of DIS Group.

We really noted that in data management 5 years ago there were mainly large companies and the B2C market. Over time, it came to key business processes and cross-industrial cases in other industries. So the growth from 17% to 89% seems organic. These indicators are generally correlated with the trends in the development of the big data market, which grew taking into account various CAGRs by segments and annual inflation by 12.4% in 2022, 6% in 2023 and, according to our data, will grow by 4% in 2024.

When replacing departed products, companies choose more modern and progressive tools that contain AI functions or are combined with its tools, instead of inherited ones. These are intelligent control and means of improving the quality of data in the company, data marking tools, data mining, etc., where the share of manual work of specialists has historically been high, "said Alexey Neiman, executive director of the Big Data Association.

In an age of digital transformation, data is a critical asset of organizations and plays a key role in decision-making, product innovation, and competitive advantage. However, the value of data is significantly dependent on the effectiveness of the data management methods used, "said Pavel Likhnitsky.

If companies use under-performing tools and invest little in data management, then business value in companies implementing relevant initiatives may remain at 2023 levels.

In addition, economic instability can inhibit the development of infrastructure and the use of tools for data management, and the lack of qualified specialists will prevent the development of Data Governance and the growth of competencies in the field of data management.

Among the possible calls in the study are the following situations:

1. Problems with tools: The lack of sufficiently developed domestic data management tools can undermine business confidence in this process.
2. Economic instability: Adverse economic conditions may limit investment in Data Governance development, which will negatively affect maturity growth.
3. Lack of qualified personnel: A shortage of specialists with the necessary skills in data management can hinder the development of Data Governance.

The DIS Group study identified a number of key global trends in data management.

Compliance: Implementing strict data protection rules leads to increased focus on security and data management.

Data democratization: Companies seek to provide data access to more employees, requiring the development of specific data management strategies.

Integration of advanced technologies: artificial intelligence, machine learning and blockchain are actively introduced into data management processes, automating tasks and improving the quality of data.

Data Quality and Integrity: Recognizing the importance of high-quality and reliable data, companies are actively introducing metrics and tools to monitor them.

With the increasing volume and complexity of data, there is a growing demand for specialized tools and platforms for efficient data management.

According to the Big Data Association, according to the 2024 Big Data Market Development Strategy, in 2023, market growth trends and the effect of the introduction of big data technologies are higher than the trends of 2019 and 2020. In comparison with the forecast of the 2022 Strategy, the projected market volumes for 2024 from 176 billion rubles. increased to ~ 319 billion rubles., and the effect for the same year - 1.5 times - up to 1.6 trillion rubles.

Master data management, data quality, business glossary and data catalog are certainly the first priorities in companies to automate and implement specialized tools, so growth is more likely to occur due to "catching up" industries and companies.

Experts note that now Russian organizations continue to actively work with big data. They invest significant resources in developing digital tools, improving data management processes, and training professionals. This process will not stop anyway, because big data is the key to improving efficiency, creating new products and strengthening competitive positions in the modern world.

The effectiveness of investments in digital instruments is confirmed by actual results. Companies are increasingly using data to develop innovative products and improve decision-making processes, resulting in higher profits. Analytical tools play a key role in this, allowing you to process large amounts of data, identify hidden patterns and predict the development of events. Thus, EVRAZ, one of the largest metallurgical and mining companies in the world, estimated the annual effect of the introduction of digital projects at $100 million.

The need to process continuously growing volumes, information digitalization, industries economies stimulation by the state determines the growth of the popularity of the corresponding software and is the driver of the rapid development of the market DBMS and data processing tools. According to our estimates, at the end of 2023, the market almost completely leveled the fall of 2022, and its total volume reached 67 billion rubles. For comparison, in 2021 the market was estimated at 72 billion rubles, in 2022 - 56 billion rubles. At the same time, most of the market is occupied by domestic players, the share of foreign vendors is rapidly falling. In 2023, it was less than 20%, and by 2027 it will decrease to 1% at all. The average annual growth rate of the market volume until 2030 is expected to be higher than the global one and, according to estimates, will amount to 20% from 2023. By 2030, its volume will reach more than 234 billion rubles, and this growth is almost 3.5 times, - said the Deputy General Director of the CSR Ekaterina Kvasha.

Between 2018 and 2023, data management practices were substantially strengthened and CDO offices were established, as separate functional areas supporting business development. This caused significant progress in the development of data management tools and accelerated the transition from an initial stage, where its own development and focus on basic tools dominated, to a much more mature and integrated approach. In particular, the companies paid special attention to the development of master data management tools, modeling, Data Lineage and improving data quality. This highlights the growing awareness of the value of data to business and contributes to increased demand for data analytics, "summed up Pavel Likhnitsky.

2020: Aruba: There are not enough tools for organizations in Europe, the Middle East and Africa to work with data

On May 21, 2020, it became known that according to a report by Aruba, Hewlett Packard Enterprise, organizations face the problem of benefiting from data, as systems are rebooted due to a growing number of requests.

The report, based on a survey of more than 170 IT thought leaders and network engineers in the EMEA region's Aruba ecosystem, revealed that the majority of respondents were frustrated by the shortcomings in data management. According to the survey, 61% of respondents "do not fully control their company's data," 51% "cannot view the data holistically," and 52% "cannot extract useful information from the data obtained."

The Data to Decisions: A Rulebook to Unlock the Power of your Data report outlines six steps to help organizations better manage their data. Key topics of the report:

• Respondents confirm that the potential of data allows them to be used for more efficient and customer-oriented work. They attribute the greatest benefits of data management to the ability to provide customers with better services (60%), increased efficiency (38%) and implementation of production innovations (35%).

However, respondents also pointed out the shortcomings of existing systems and infrastructure that may limit these capabilities. When asked how they would like to use data that cannot be used for May 2020, the respondents replied that their priority is to increase the level of security (63%), implement more real-time analytics (55%), make more informed decisions based on data (55%), as well as more efficient use of data to fulfill the main tasks of the business (45%).

• For May 2020, another major cause for concern is the lack of appropriate skills and knowledge among company employees. More than a third (36%) of those surveyed said "a lack of skills to manage skyrocketing data" was a major concern for them. At the same time, the biggest fear among respondents (28%) is that "employees will disagree with data management policies."

In order to improve the skills and knowledge base of data, organizations should focus on improving the skills of employees with deep industry knowledge, hiring a director of data processing and analysis (Chief Data Officer), as well as create data management teams that will include specialists who make decisions on all key business tasks, which will ensure that their needs are reflected in the strategy for work and data management.

Companies already have most of the most valuable knowledge. Training employees to communicate with IT experts and use APIs is often much more valuable than asking technical candidates to quickly understand the industry. narrated by Tom Chatfield, report author and technical analyst

• In addition to being at the top of IT's priority list for May 2020, data security also causes them the biggest fears. Two of the three biggest issues related to security or compliance with company requirements and policies: 21% of respondents fear hacker attacks, and 12% that their organization will be fined in accordance with the European Regulations for the Protection of Personal Data (GDPR).

The report makes a number of recommendations to improve the level of security, such as: classify data according to different levels of risk, develop an action plan in the event of a data breach, and train employees based on practical scenarios to increase awareness of threats.

Employees are the last line of safety. You can use the best firewalls and VPN solutions. But if someone saves a file with passwords on their Gmail mailbox and it is hacked, then everything is out of control. noted Andrew Grill, report author, futurist

Each organization understands what opportunities the data provides for innovation, product development, and customer experience improvement. As data and solutions move to the edge of the network, companies need to be confident that their systems and processes will handle it. As stated in this study, organizations need to improve the skills and training of their employees, create effective management structures, and pay special attention to data security. In a pandemic, these recommendations are even more relevant, as working at home becomes the norm, and home Wi-Fi does not meet key security requirements, while the number of targeted cyber attacks is growing. concluded Morten Illum, Vice President of Aruba EMEA

2019: Key questions and errors in data management

The relevance of the Data Governance topic is growing every year. Indeed, the need to organize processes to improve the efficiency of data collection, processing, storage and use as a valuable asset is already evident to almost all companies. A lot has been said about the benefits that companies bring to properly structured data management processes, and many organizations have already begun implementing this initiative. At the same time, organizations often make similar mistakes that negatively affect the pace of implementation and the effectiveness of the created data management processes. About what mistakes they are, how to avoid them and what questions the organization should find answers to during the Data Governance implementation process, [http://www.tadviser.ru/a/498875 in the material prepared for TAdviser, says Svetlana Bova, Chief Data Officer of VTB Bank].

2016: Efficient Data Management Increases Sales by 3X

Experian Companies are increasingly aware of the economic potential of the information, with 79% of those surveyed saying customer data will drive most of the selling decisions by the end of the decade, according to research involving 1,400 organisations from eight countries around the world. However, at the same time, almost all respondents - 90% - still believe that they lack a constructive approach to effective data management.

On average, almost 25% of customer data contains inaccuracies. Among the most common errors are:

• incomplete or missing data
• out-of-date information
• data duplication
• conflicting information

According to research by Experian, companies believe they could increase sales by almost 29% with accurate customer data.

According to estimates, in 2012, an impressive amount of data was created every day -2.5 quintillon of bytes - and this is at a time when mobile communications, according to estimates, covered 61% of the world's population. Assuming that this figure rises to 70% by 2017, it becomes obvious that the volume of data will grow exponentially in the coming years.

Such "explosive" data growth is a challenge for companies. As 84% of respondents noted, their organization currently considers data an integral component of the business strategy. In this regard, 82% of companies in the next five years plan to hire data specialists. As established by previous research by Experian, today the importance of the Director of Data (CDO) is growing, which develops data work in the framework of the company's strategic activities, ensuring improved financial results.

Data management is also becoming more complex due to increasing amounts of information and changes in the use of communication channels. According to 39% of respondents (versus 10% last year), their organization has 50 or more contact bases. In addition, social media use has risen 81% in the past 12 months.

75% of those surveyed this year in an Experian study cited email as the communication channel most widely used for marketing purposes. At the same time, 73% of this group can give an example of the negative impact of inaccurate data on the business. The poor quality of electronic address data entails not only a decrease in revenue, but also business problems such as poor customer service, inability to communicate with subscribers and useless costs.

97% of organizations strive to see the client as a whole, so the effective use of data that allows each client to be considered as a separate person, with its own individual characteristics, and develop commercial proposals taking into account the special needs of each such client.

The advantages associated with such a holistic understanding of the client were cited as an increase in loyalty, an increase in sales and the possibility of strategic decision-making. Therefore, commercial companies recognize the importance of accurate knowledge about customers and their impact on the financial results of the organization.

2013: Specifics of financial institutions

In December 2013, Qlik (QlikTech) released the results of a study that showed that despite the fact that data management has acquired special importance since the global financial crisis, more than two-thirds (63%) of financial company executives do not consider it necessary to consider this issue at the level of senior management.

According to a study conducted for QlikTech by the analytics company Lepus, a third (31%) of financial institutions did not allocate roles and responsibilities in the field of data management, despite the fact that many factors in business indicate the need for data management in the financial sector.

In addition to the unprecedented increase in the volume of data processed daily, many regulations related to data management have been put into effect - such as the Sarbanes-Oxley Act, Rule 17a-4 of the US Securities Commission, Basel III and Solvency II standards, as well as the principles of the Basel Banking Supervision Committee BCBS 239 (aggregation of risk data). In response, financial institutions had to form common approaches to architecture and data modeling; solve this problem and provide data management.

The study aimed to explore data management programs in,, and banks Europe North America Asia-Pacific countries. It showed the need to consider these issues and take responsibility for them at the senior management level in order to ensure that these management infrastructures are consistent with the goals of the business.

This is all the more important given that the study identified many business factors that contribute to the implementation of data management programs, as well as a number of critical functions in the management structure. 94% of bank executives recognized the existence of such a business factor as the need to ensure regulatory compliance and reduce legal risks. The next most important were to ensure consistent data across the institution (88%) and increase the openness and transparency of financial data and information (63%).

However, the basic function of the data management infrastructure for different banks participating in the study varies greatly. 44% of respondents named risk mitigation as one of the two main functions, and the other as the realization of business benefits, but only 13% consider compliance to be the primary function of the data management infrastructure.

Paul Van Siclen, Director of Financial Services Development at QlikTech, noted that data management has become even more important since the global financial crisis, as financial institutions have been forced to conduct a comprehensive analysis of their data strategies. "But in the absence of agreed goals and without considering these issues at the level of senior management, banks are unlikely to achieve the desired result," he said.

How to use analytics to develop quality IT services

White Paper: Using Analytics to Develop IT Services

Loss of organizations from downtime and IT failures

White Paper: Enterprise Losses from IT Downtime and Failures

Data management

• Master Data Management (Management of the main master data) Catalogue of systems and projects
• Regulatory Reference Information Management Systems in Russia.
  Top players and top trends

• Improving Data Management Models
• Chief Data Officer (CDO)
• Data Science
• Data scientist
• Chief Digital Officer, CDO
• Chief Digital Officer (CDO) in Russia
• CIO (Chief Information Officer)
• Chief information security officer (CISO)
• CFO - Chief Financial Officer
• System Administrator
• DataOps (data operations, data pipeline)

• Big Data
• Big Data in Russia
• Big Data (Big Data) Global Market
• Big Data Specialist
• Data Mining