Dark data

The introduction to the concept of Data Fabric for TAdviser was prepared by Svetlana Vronskaya, author of the telegram channel Analytics Now.

The concept of dark data was introduced by Gartner analysts, defining them as information assets that the company collects, processes and stores on an ongoing basis, but usually does not use (for example, to analyze, identify the relationships of factors and direct monetization). As with dark matter in physics, dark data is often the majority of data stored in the company. Many store dark data only to follow the requirements of the regulator - personal data, for example. The reasons for storing unused data may be the need to maintain archives. And some organizations believe that hidden data can be useful to them in the future, when more advanced analytics and business intelligence technologies appear.

At the same time, the cost of storing and ensuring the security of this data is greater than its value. And the data of such a sea. According to Mellon Rahul Telang, one of the professors at Carnegie University, the total volume of dark data can be 90%.

In an industrial context, dark data may include information collected by sensors and telematic devices. IBM cites data from sensors and other devices, and says that 90% of this data is never used.