MIPT: A way to increase the performance of network artificial intelligence systems

The main articles are:

• Machine Learning
• Artificial intelligence (AI, Artificial intelligence, AI)

2023: Introducing a solution to accelerate network AI

Researchers from the Moscow Institute of Physics and Technology, in collaboration with colleagues from the UAE, have proposed a solution that can significantly improve the performance of network artificial intelligence systems and reduce the cost of their operation. The university announced this on December 4, 2023.

source = MIPT

Network artificial intelligence is a software complex that has the ability to process large amounts of data and identify patterns in them. Based on these skills, they form recommendations for decision-making and offer answers to various tasks. One of the main qualities of such complexes is their ability to self-learn and improve their skills over time. This allows them to constantly improve and become more effective.

As of December 2023, network artificial intelligence is used in many industries industries areas of public life. For example, they are used in analysis, in social networks recommendation systems, algorithms in, in speech recognition automatic translators and many other applications.

To develop such complexes, specialists use algorithmic machine learning. At the same time, the effectiveness of the process largely depends on how optimally communication is built between end-user devices and servers on which data is processed. Inefficient communication slows down model analysis and update.

With increasing data and modeley̆ size, more parallel and distributed computing is required to solve real-world machine learning problems. Meanwhile, distributed approaches have a significant bottleneck - this is the cost of communications, - commented Alexander Beznosikov, co-author of the study, head of the laboratory of fundamental research at MIPT - Yandex.

He added that at the heart of machine learning are variational inequalities. This is a mathematical tool that incorporates different classes of optimization problems. Such tasks are familiar to many from school. For example, they find the minimum value of some objective function. In machine learning, it is necessary to solve the same, but much more complex problems.

{{quote 'Most real machine learning problems can be represented as variation inequalities. At the same time, the main methods that specialists use to reduce the number of communication rounds and the cost of each round when solving variational inequalities in a distributed way are methods with parcel compression, methods using the similarity of local data and methods of local steps, - said Alexander Beznosikov. }}

He explained that the first of these methods involves sending not a complete packet of information, but only a part of it (for example, sending a random part of a parcel or rounding numbers). The second is based on the assumption that if the data on computing devices are similar, then for successful communication it is possible to transfer only the differences between them. The third method speeds up the machine learning process by updating the data on each node before exchanging with other nodes.

In a sense, the specialist noted, the second and third methods are the opposite of the first. During compression, traffic is reduced by reducing "parcels." And in approaches based on data similarity and local steps, the cost of data exchange decreases, because communications occur less often.

According to Alexander Beznosikov, each of the listed methods has its own advantages and disadvantages. However, in the study, the scientists combined three methods into one and obtained a powerful synergistic effect.

The essence of our method is that on one of the devices - conditionally, the main, some kind of server - the data should be in some sense similar to those that are available on the entire network. At the same time, on all other devices, the data can be very heterogeneous, "the scientist explained.

According to him, the implementation of this method allows you to speed up network communications ten times compared to basic algorithms and about twice in relation to the most advanced of them. In addition, the algorithm is good because most computing operations take place on the server. At the same time, user devices (phones, tablets and computers) remain unloaded and, therefore, can safely perform their direct tasks.

This method correlates with one of the most promising machine learning technologies - Federated learning. This technique implies that data remains on user devices, and the model is updated on the server by aggregating trained models from different devices.

Alexander Beznosikov emphasized that during the study this method was tested on simple experimental problems. In the future, scientists intend to test it on more complex software complexes. Including on language models - artificial intelligence systems that are used to predict the following words and phrases based on the previous ones.