D. lab: Multimedia Content Manufacturing Solution

The main articles are:

• Neural networks (neural networks)
• Speech Recognition (Technology, Market)
• Speech technology: On the path from recognition to understanding
• Speech synthesis
• Computer vision (machine vision)

2023: Neuroproduction testing begins in GPM Radio, Rutybe, Premier and Yappy projects

Neuroproduction studio D. lab, which Gazprom-Media Holding launched in September 2023, presented the first demo samples of AIGC - multimedia content of various formats and genres, produced by artificial intelligence with minimal human participation. Samples are created using your own D.lab solution. Now the studio is moving on to testing its technologies on the projects of Children's Radio, Rutube, PREMIER and Yappy. Gazprom-Media Holding announced this on November 15, 2023.

The D.lab solution allows you to create new content formats. For example, animated retellings of literary works or brief retellings of full-length films. The solution can synthesize musical design and voice-over, voice visualize audio content, generate digital characters, and stylize videos. The solution is hybrid - it is based on more than 20 open source and commercial models of artificial intelligence, additionally trained and combined by the D.lab team, for the production of multimedia content. The solution also uses technologies, computer vision synthesis and. speech recognition

The key advantages of the D.lab solution are saving professional time, increasing the speed of routine tasks and reducing production costs. For example, instead of manually rendering various backgrounds, characters or details, you can choose from the options proposed by neural networks, created on the basis of special industrial queries. Human participation is required only when setting a task, making stylistic changes and monitoring results.

Шаблон:Quote 'author = said Edouard Maas, head of D.lab.

RUTUBE channel D.lab^[1] presents the first examples of AIGC works: animated retellings of "Words about Igor's Regiment" and Isaac Asimov's story "Liar!" from the famous cycle "I, Robot," video retellings of film classics - "Metropolis" Fritz Lang and "Battleship Potemkin" Sergei Eisenstein, as well as a sample of animated stylization of the video.

How the D.lab solution works:

In the cycle of creating animated retellings of literary works from D.lab, an average of 7 stages:

• analysis of source text by LLM models
• Script writing by LLM models
• forming a style concept by Text-to-Image models
• draft storyboard of a video with a text description of LLM models
• generation of scenes and characters by Text-to-Image models
• adding animation (with human input if necessary)
• voicing by Text-to-Speech models

In "Liar!," the quality of neurosynthetic voicing did not suit the D.lab team, and it was decided to use the classic version with a real human voice. It now takes about two weeks to prepare such videos from books.

In the video stories of full-length films, D.lab made 6 types of works:

• film analysis by a neural network ensemble
• processing of the obtained results by LLM models
• writing a video script by LLM models
• allocation of key mounting points by the neural network ensemble
• video editing
• voicing by Text-to-Speech models

D.lab's stylization solution allows you to quickly "change" any video, for example, turn a film into a cartoon. Two-step solution:

• text description of style by Text-to-Image models
• superimposing stylistics on the original video by Image-to-Image models.

How the tests will take place on Children's Radio, in RUTUBE, PREMIER and Yappy:

The subholding of GPM Radio was interested in animated retellings, and it was decided to test the D.lab neuroproduction in visualizing the content of the only station in Russia for young listeners - Children's Radio. A visualization of the popular audio podcast is already being prepared.

The RUTUBE team is testing the capabilities of the D.lab solution in creating short videos from their original shows. In parallel, the platform analyzes the capabilities of AIGC in several directions at once - integration into new releases of current projects, the development of premiere shows based on neurocontent, the use of such material in broadcasts of sports and cultural events, visualization of audio content, remounting content in different formats.

Online cinema PREMIER has chosen video stylization for testing. The trailer for one of the top TV series of the service will be presented in an unusual form.

Yappy also chose stylization as the most suitable and operational tool for processing current content. The platform team wants to use the D.lab solution to improve the quality of the original video (light, stabilization, focusing, etc.).

Notes