Fugatto (Foundational Generative Audio Transformer Opus 1)

2024: Product Announcement

On November 25, 2024, the company Nvidia introduced a model (generative artificial intelligence Genia) called Fugatto, designed to create various sounds by text description. The developers describe the new neural network as a "Swiss army knife for sound."

The Fugatto model (Foundational Generative Audio Transformer Opus 1) can generate new or modify existing music, voice and sound files based on text prompts. As input instructions, a phrase like "downpour with increasing thunder rolls" or "dog barking in the pipe" can be introduced. The neural network is able to form musical fragments based on a simple description, remove or add musical instruments to an existing composition, change the accent or emotion in the voice, etc.

𝓲

Nvidia

We wanted to create an AI model that understands and generates sounds in the same way that people do, "says Rafael Valle, an application audio research manager at Nvidia and one of the project participants.

Fugatto is a generative transformer model that builds on the results of previous work in areas such as speech modeling, audio coding, and audio material analysis. The novelty uses 2.5 billion parameters. The training used Nvidia DGX high-performance systems based on H100 Tensor Core GPUs.

Fugatto is expected to find application in various areas. For example, musicians and performers could use an AI model to quickly prototype or edit a song by trying different styles, voices, and instruments. Video game developers will be able to use Fugatto to generate variations of sounds, etc. The model can be used when creating personalized tools for learning the language.^[1][2]

Notes