Zeliboba

2021: Start neural network to generate text

In early June 2021, Yandex introduced the neural network for generating YaLM texts, as well as the Zeliboba service, based on these language algorithms. A resource based on artificial intelligence is able to "predict" the construction of a sentence. You can specify parameters and the desired text style. The output will be aphorism, toast, anecdote, advertising slogan, news, publication for a social network, etc.

Such a model has exactly one task - to generate each subsequent word in the sentence. To make the text coherent and grammatically correct, during training, the model evaluates each predicted word: for example, whether the word "run" or the word "frame" can go after "Mother of soap...," the service description says.

YaLM trained on Yandex-indexed pages, including Wikipedia, news articles and books, as well as user records on social networks and forums. In total, several terabytes of Russian-language texts were used to teach the language model.

As part of the project, a number of models were created that differ in the number of parameters - from 1 to 13 billion (Zelibob uses a version with 3 billion parameters). All of them studied on a variety of samples of Russian-language texts. Developers used both their own and publicly available datasets, including Taiga and RDT.

By June 7, 2021, access to Zelibobe was closed, although the service worked three days earlier. Yandex explained that the technology is still under internal testing. The publication "N + 1" managed to try the platform and shared the corresponding results:

Yandex claims that the algorithm is very quickly trained to work in a new style - for this it needs from five to several dozen examples in the target style.^[1]

Notes