Sberbank Kandinsky Video Neural network for generating full-fledged video

The main articles are:

• Neural networks (neural networks)
• Big Data

2023: Presentation of the first generative model in Russia for creating videos by text

Sber presented the Kandinsky Video neural network - the first generative model in Russia for creating full-fledged videos based on text description. This was announced on November 22, 2023 to TAdviser by representatives of Sberbank. According to Alexander Vedyakhin, First Deputy Chairman of the Management Board of Sberbank, the model generates a video sequence lasting up to eight seconds at a frequency of 30 frames per second.

The Kandinsky Video architecture consists of two key blocks: the first is responsible for creating key personnel that make up the plot structure of the video, and the second is responsible for generating interpolation personnel that allow you to achieve smoothness of movement in the final video. The two blocks are based on an updated image synthesis model based on text descriptions Kandinsky 3.0.

The format of the generated video is a continuous scene with the movement of both the object and the background. This is what distinguishes the videos synthesized by the Kandinsky Video model from animated videos in which the dynamics are achieved by modeling the camera span of a relatively static scene. The neural network creates videos with a resolution of 512 x 512 pixels and a different aspect ratio. The model is trained on a datacet of more than 300 thousand text-video pairs. Video generation takes up to three minutes.

"We recently trained Kandinsky to create animated videos by text description, and today we are introducing a completely different level model - the first model in Russia to generate full-fledged videos by text. This is an important contribution to the development of Russian generative neural networks. Users will have even more opportunities for creativity and the implementation of their creative ideas of any orientation, "said Alexander Vedyakhin, First Deputy Chairman of the Management Board of Sberbank.

As he added, people will be able to create unique videos absolutely free of charge. And the model itself will be available in open source.

Previously, active users of Kandinsky 2.2 in test mode have the ability to create animated videos. On one request, you can create a video four seconds long with the selected animation effect, at 24 frames per second and a resolution of 640 x 640 pixels. Users of the Kandinsky 3.0 neural network can also create videos by text description in animation mode. Telegramboat^[1].

The neural network was developed and trained by Sber AI researchers with the partner support of scientists from the AIRI Institute of Artificial Intelligence on the combined Sber AI datacet and SberDevices.

Notes