Diasoft: Digital Q.DataFlows

The main articles are:

• Data management
• Data Quality

2023

Add functionality for setting up data quality control rules and low-code data processing

Diasoft has added functionality to the Digital Q.DataFlows platform to configure data quality control rules and low-code data processing. The developer announced this on November 15, 2023.

This component simplifies data management by automatically generating an ETL data quality process from a template based on the objects described in the catalog. Data quality can be tracked in the metric and detailed validation protocol.

The platform now includes three components:

• Data Quality Rules Designer allows you to create data quality checks based on templates.
• Data Streamer provides information collection and processing (ETL) and allows you to integrate various information systems with each other. Supports both streaming and batch scenarios.
• Data Catalog is responsible for collecting and maintaining metadata from all information sources.

Digital Q.DataFlows implements the ability to group ETL processes and include them in the created applications. Grouping is necessary for combining ETL processes and transferring data to an application solution project and further sending for automatic deployment at the stands.

The Data Streamer component has the ability to process data in low-code mode based on the meta-description of objects from the data catalog. The ETL process is now automatically generated. You can also configure the ETL process to start automatically on a scheduled basis.

The Data Catalog component has expanded the functionality of generating scripts and capturing metadata. Sets of changes are divided into atomic parts, which reduces the chance of their failure in the database. Starting metadata capture from databases automatically generates an ETL process managed from the Data Catalog, and generates a detailed protocol in which you can track the entire metadata capture process.

The Digital Q.DataFlows platform has been transferred to the Digital Q.Palette technology platform. This made the interface design more comfortable for users.

Data Archiving Product Development

Diasoft has developed a product for archiving product data as part of Digital Q development platforms. The company announced this on July 4, 2023.

The volume of data is constantly growing, often this leads to a slowdown in the work IT of solutions, the execution of routine operations, updating the structure databases and other processes. Along with the growth of data, the amount of resources required for them also increases. storages

Data Archiving is designed to archive data, improve solution performance, and clear the database of irrelevant data. The product is implemented in a microservice architecture on the Digital Q.DataFlows technology platform, which ensures its operation in 24/7 mode and high performance of data archiving processes.

"Data archiving" allows you to reduce the size of the working database, free up disk space, reduce the time for routine operations, and eliminate the need to back up the database. Reducing the size of the database reduces the cost of ownership of the IT solution running on it.

Archiving algorithms are flexible: they can be run in a product-by-product, periodically on-demand, or on-schedule. The data archiving protocol is maintained automatically. The Data Archiving product stores information related to the archived data and has the function of quickly searching and viewing the archived data.

2022

Integration with PostgreDB

On April 17, 2023, Diasoft announced the start of the release of versions of products included in the development platforms Digital Q.FinancialMarkets, Digital Q.Consumer, Digital Q.Corporate and Digital Q.Payments (formerly Diasoft FA# products), with support for the work on the import-dependent database Digital Q.DataBase products, which is an automatic feature of the assembly of software products from the Basis. This allows you to get two versions of products with a single source code at the same time: in MS SQL syntax and PostgreSQL syntax, which pass functional tests equally successfully.

To minimize the number of errors that may arise in connection with the features of PostgreSQL, Diasoft specialists have modified and built into the pipeline product assemblies a code inspector that detects syntactic constructs and their sequences that are incorrect from the point of view of conversion.

The conversion of code from MS SQL syntax to PostgreSQL syntax is provided by a converter developed by Diasoft specialists. It does not change the application business logic of the products and the composition of the data they operate with, which minimizes the number of possible errors when using software products. Also, the code converter can be used to adapt the bank's own developments to support PostgreSQL.

Now the products for integrated bank automation included in the development platforms Digital Q.FinancialMarkets, Digital Q.Consumer, Digital Q.Corporate and Digital Q.Payments (previously - products of the Diasoft FA# line), integrated with the PostgreDB component of the import-independent DBMS - the Digital Q.DataBase infrastructure platform. The component is based on the open source PostgreSQL solution, which Diasoft took under responsible ownership and significantly improved to ensure the correct and efficient operation of products for comprehensive bank automation in PostgreSQL syntax. The Digital Q.DataBase platform is part of the Digital Q digital transformation ecosystem.

As of April 2023, load testing of products for integrated bank automation is being carried out at Digital Q.DataBase.

Digital Q.DataQuality for data quality control

The success of any organization depends on its ability to work with large amounts of data. The quality of incoming data requires special control: the use of incomplete or inaccurate data for decision-making can lead to damage to the business. Digital Q.DataQuality, a solution of Diasoft as part of the Digital Q.DataFlows technology platform, will help to check the data quality. It provides data quality analysis, monitoring quality metrics and eliminating errors in data coming from various sources. This was reported on January 20, 2022 by Diasoft.

The functionality of Digital Q.DataQuality allows you to configure data verification processes taking into account individual requirements for data quality parameters and control frequency, which is integrated into the business processes of the organization. Based on the results of inspections, detailed reports are created with the ability to form analytical certificates, visualize protocols on the data quality dashboard.

"Errors are unavoidable when loading data from external sources, changing the format of its storage, and other actions with large amounts of data. Digital Q.DataQuality helps to organize a full-fledged cyclical process of data quality management, centrally monitor and eliminate all types of errors that occur. Specialists from various organizations will no longer need to spend time and effort on data reconciliation, and most importantly, the risk of the influence of incomplete or inaccurate data on business indicators will decrease, "said Olga Shiryaeva, head of products at the Data Economics department of Diasoft.

Digital Q.DataQuality is built on a microservice architecture and is easily embedded in the IT landscape of any organization. It can work both in the Diasoft architecture and in combination with the products of other vendors. As of January, Digital Q.DataQuality has been tested and is already used in internal business processes of Diasoft.

2021: Creation of the Digital Q.DataFlows platform

On July 12, 2021, the company Diasoft"" announced that it was working on the creation of the Digital Q.DataFlows technology platform, based on microservice architecture Open Source and technologies. The platform is designed to provide to banks a set of tools data management and easy access to these tools. This allows you to easily create, deploy and fill data various analytical systems. The platform will combine different approaches to data processing, which will make it possible to carry out updates of analytical systems in real time.

Our goal is to provide customers with a set of tools that can be easily embedded in the IT landscape of a data processing organization and allow them to create analytical applications in different architectures. We invest a lot in creating a technological platform, use our developments and ready-made, proven Open Source technologies, said Igor Shabanin, Deputy Director of the Data Economics Department of Diasoft.

For July 2021, Diasoft is working on the creation of three components independent of each other - this will allow separately to use platform tools, integrating them into the bank's architecture.

The Q.DataCatalog component of the Digital Q.DataFlows platform is responsible for automating metadata management from any information system in a single place, providing users with the full context for data analytics and reporting. With this tool, you can visualize the Data Line at a detailed level from sources to end storefronts and reports.

Q.DataStreamer, the second component of the platform that allows you to quickly create data extraction and conversion processes in a simple interface, the user of the system will be able to draw ETL processes without coding. The component also makes it possible to automate complex processes from many data streams for working with Data Lake and Data Warehouse.

Quality Control, or the Q.DataQuality component, in turn, allows you to embed checks into data streams to control the quality of data within the framework of general processes of the organization - with it you can develop, track, maintain statistics on control rules and work with incidents. The component makes it possible to form analytics and visualize the results of checks and protocols on the data quality dashboard. This component can work both in the Diasoft architecture and with third-party systems.

Already in 2021, Diasoft specialists plan to significantly advance the development of all components of the Digital Q.DataFlows technology platform and make them available for implementation.