Artificial Intelligence and Music Creation

Main article: Artificial Intelligence

2024

Free AI audio generator released that creates sounds according to description

On November 25, 2024, Nvidia introduced a generative artificial intelligence (GENI) model called Fugatto, designed to create various sounds by text description. The developers describe the new neural network as a "Swiss army knife for sound." Read more here.

An open neural network was launched in Russia to create music

In July 2024, an open neural network for creating music was launched in Russia, developed by engineers of the Perm company Double Yu Expo. The new technology based on artificial intelligence is able to generate musical works in modern genres, create unique lyrics and customize compositions to suit the tastes of the target audience. Read more here.

Launched a neural network capable of generating full-fledged three-minute songs

On April 3, 2024, Stability AI introduced the Stable Audio 2.0 artificial intelligence model, which is said to set new standards for the generation of audio materials. In particular, this neural network is capable of creating full-fledged tracks lasting up to three minutes. Read more here.

Adobe introduced Photoshop for editing music and creating it based on text descriptions

On February 28, 2024, Adobe announced the Project Music GenAI Control tool based on generative artificial intelligence (Genia). This is a kind of "photoshop" for editing music and creating it based on text descriptions. Read more here.

2023

A neural network has been created in Russia that allows you to "conduct" the reproduction of classical musical works

On November 9, 2023, the Skolkovo Institute of Science and Technology (Skoltech) announced the development of a neural network that allows anyone who wants to "customize" musical works to their preferences. Moreover, you can "conduct" music playback using voice, gestures and even facial expressions.

The system uses an artificial intelligence model that is trained on a public corpus of 1,067 musical performances of 236 works of piano music. Notes are used as input: the model learns to play on them, predicting performance characteristics - local tempo, position, duration and volume. The result is a performance of the work in a particular style.

Tags in the passage of the score to Beethoven's Sonata No. 17. Blue indicates guidance on tempo, red and orange indicates guidance on volume, green indicates accents for notes.

Through a special mobile application, the user can control the AI model: for this, a microphone and camera of a smartphone or tablet are used. To influence the performance of a work, just press a button and record video or audio. With voice commands or expressions of emotion on the face, you can ask the AI model to play music differently - for example, slow down the tempo or make a lullaby out of a classical melody.

To control the model, we use execution instructions that have already been written in notes. The scores have labels that indicate to the performer how to play this or that part of the work: faster, slower, louder, quieter, and so on. We take all the data that is, and based on it we convert the user's voice instructions into these instructions, the developers say.

A neural network allows a person who does not have the skills to play musical instruments to influence how a particular work of classical music is played. Further plans by the creators include improving the interface of the system and expanding the base of musical works. The user-AI interaction process is expected to be fully interactive.^[1]

AI for the first time in the world soloed at a concert in Vladivostok

On September 10, 2023, as part of the Eastern Economic Forum, a concert was held in Vladivostok, where artificial intelligence from Sberbank became a soloist and improviser. Such an experiment was carried out for the first time in world history. Read more here.

2019: Warner Music lands first-ever contract with Endel algorithm artist

On March 27, 2019, information appeared that Warner Music had signed the first ever contract with an artist algorithm Endel creating musical compositions to the user's mood at the moment. Under the terms of the contract, within a year neuronet Endel will release 20 unique albums. As of March 2017, five albums are already available for download in, iTunes while all albums are created, as the developers say, "by pressing a button."

Endel was developed by the startup of the same name, the co-founder and executive director of which is Russian gaming journalist Oleg Stavitsky. According to the developers of the algorithm, Endel adapts to the user's mood and helps him depending on the tasks set - the music of the neural network helps to play sports, work, fall asleep or meditate. At the same time, AI itself determines what a person needs at the moment, analyzing many parameters: time of day, geolocation, weather outside the window, pulse and frequency of beating a person's heart.

We are on the verge of flooding the market with automatic and procedural-generated music systems, the product of which will satisfy most people in most situations, says market analyst Holly Herndon.

According to the expert, the further transformation of the music and show business market will occur at an accelerated pace. As you know, in China and Korea, concerts of fully virtual performers are already commonplace, and the number of fans of "digital" idols is in the millions. Now this trend, the analyst believes, will spread in the more conservative Western market.

2018

At the intersection of two growing industries

^[2] 2018 is expected^[3]globally companies' income from the use of artificial intelligence will amount to $1.2 trillion at the end of 2018, which is 70 percent more compared to 2017, and in 2022, these benefits are projected to reach $3.9 trillion. And such rapid growth is far from a secret for anyone: artificial intelligence can be called the defining technology of the 21st century. Artificial intelligence differs from traditional data analysis programs in its ability to learn to recognize patterns using machine learning algorithms, and then make autonomous decisions based on these patterns, being clearly not programmed to do so.

At the same time, the global music production software market is^[4]grow to $6.26 billion by the end of 2022, according to^[5]. A key factor that will drive the growth of the industry is the growing demand for digital audio content worldwide. Audio content, in turn, is gaining popularity due to the recently begun growth^[6] of streaming services. This leads to an increase in the number of artists and music producers creating music content, which ultimately increases the demand for composing software. AI algorithms have been used for several years to determine musical taste and customize personal recommendations on streaming services, and amid a surge in research and investment in artificial intelligence technology in general, there has been a flurry^[7] activity] around the subject of composing music using this technology. And according to^[8], AI will greatly affect the process of creating musical content.

Use Cases: Large Companies

NSynth Super (Google)

NSynth Super^[9] is part of an ongoing experiment under the name Magenta^[10]research project at Google that "studies how machine learning technology can help artists create in new ways" and look at the creative process from a different perspective.

Various sound effects^[11]changes the frequency and amplitude of sound, electric musical instruments^[12] are all examples of other sounds created using technology. Machine learning can now be included in the list of such technologies, because progress in this area opens up unconventional opportunities for generating unusual sound.

Based on past research in this area, Magenta created NSynth (Neural Synthesizer), a machine learning algorithm that uses a deep neural network to study various characteristics of sound, and then creates a completely different sound based on them. According to the developers^[13], instead of combining or mixing sounds, NSynth synthesizes sound using the acoustic qualities of the original sounds. Thanks to this, you can get a sound that is both the sound of the flute and the sound of the sitar - all at once.

Prototype NSynth Super - an experimental tool capable of generating new sounds

Since the release of the NSynth algorithm, Magenta has continued to experiment with various musical interfaces and instruments to infer the NSynth algorithm more understandable to the layman and reproducible. As part of this research, they created NSynth Super in collaboration with Google Creative Lab. It is an experimental open source tool that gives musicians the ability to create music using new sounds generated by the NSynth algorithm from 4 different basic source sounds. The prototype experience (pictured above) was then shared with a small community of musicians to better understand how they can use it in their creative process. For example, you can watch^[14] as London-based music producer Hector Plimmer explores the sounds generated by NSynth Super.

Sounds generated by NSynth Super

According to the developers on their website, "using NSynth Super, musicians have the opportunity to explore more than 100,000 new sounds generated using the NSynth algorithm."

Flow Machines ( Sony )

Flow Machines^[15], (literal translation - Streaming Machines) is a research project aimed at expanding the boundaries of human creativity in music.

The central part of this project is Flow Machines Composer. So far, this is not robot with a consciousness that seeks to express its experiences through music, but it is a set of complex algorithms that have gained their knowledge and "sense" of music through the study of a huge (databases 15,000 songs).

To write a song using Flow Machines Composer, you first need to set the style of the melody, and then^[16] takes place inside the system]: the algorithm obtains a selection of songs with a similar style, runs an analytical model known as a Markov chain that identifies patterns in those samples, and then mimics and modifies them to create its own original composition.

Next, the computer calculates the probability of certain chord progressions^[17], melodic connections and rhythmic patterns^[18], and uses these probabilities to create plausible and sound successful (in terms of music) variations.

But at the next stage of the composition, the system requires human intervention. The user can save the desired part of the resulting melody, and abandon the rest, and then run the program again in anticipation of the next successful generated variations. And this can be done until the melody and chord sequence appear that you will be happy with.

In September 2016, Flow Machines unveiled their single "Daddy's Car" - a rather lively pop song based on a selection of Beatles^[19].

Single "Daddy's Car"

Use Cases: Startups

AIVA

AIVA^[20] (Artificial Intelligence Virtual Artist) is a startup based Luxembourg in and funded by the $768,000^[21] to^[22]which, as the developers write on their^[23] website^[24]is "to enable people to create personalized soundtracks using artificial intelligence." According to the developers, AIVA is capable of composing emotional soundtracks for films, video games, commercials and any other type of entertainment content.

AIVA studied the art of composing music by "reading" a large collection of music scores written by composers (Mozart, Beethoven, Bach, etc.), and "created a mathematical model for representing what music is^[25]It is this model that is used to compose modern music. It is worth mentioning the fact that AIVA recently became a virtual composer whose works were registered with the copyright society (SACEM^[26] One of these works can be heard below^[27]

AIVA - "Genesis" Symphonic Fantasy in A minor, Op. 21

Popgun

Launched in January 2017, Australian startup Popgun, according^[28], uses deep learning to create musical artificial intelligence that can "complement" artists in real time.

The Popgun project was Alice's artificial intelligence, which can predict that a musician will play on, play along with him, and even improvise a little on the musical theme of a human musician's score.

Alice learned all this not immediately, but gradually: it will best display^[29] video^[30]which shows the evolution of the technology from January 2017 to July 2018. The video begins with Alice's artificial intelligence demonstrating its forecasting skills: the musician plays short melodies on the piano, and Alice responds by saying that, according to her assumption, the musician will play on.

Alice artificial intelligence can "complement" performers in real time

By August 2017, Alice was capable of improvisation: that is, she first listened to a sequence of notes played by a person, and then changed it, but at the same time retained the main musical theme. By the end of the year, Alice could create original piano compositions in various styles without human input.

For January 2019, Popgun has artificial intelligence (or, better to say, an artificial intelligence group) that can play along with a person on the piano, on the bass or on the drums (or all at once). If the melody is equipped with human vocals, then the developed algorithm can compose a suitable instrumental part, as well as reduce several musical passages into one and perform final sound processing.

The future of artificial intelligence for composing music

All this, of course, is surprising, but the available technologies are far from the fact that artificial intelligence has the ability to create something fundamentally new: a machine can pass a huge amount of data through itself, and "understand" how it can be done and how it has already been done, but it cannot catch fire with a sudden attack of inspiration and embody its creative idea. Melodies composed by artificial intelligence still need a person to have a hand in them regarding music theory, musical production and orchestration. Otherwise, they will sound a little unusual and rather chaotic for the human ear.

A big problem for artificial intelligence is understanding creative and artistic solutions (however, sometimes this is a problem for music experts). In addition, machines still do not have the elusive creative principle that is vital for the creation of art. In general, AI experts agree that artificial intelligence will never be able to replace a person in the field of creating music, but, as mentioned above, it will be able to significantly change the entire process.

2017: Release of Terin Southern I AM AI album co-written by AI

American singer Terin Sauzern released the single Break Free in the summer of 2017, which opens her new album I AM AI - "I Am Artificial Intelligence." In the official announcement, it was indicated that the entire album, including this track, is being co-created with the unknown musician Amper. However, the seemingly non-suspicious creative duet turned out to be not at all what could be expected^[31].

The Amper project is artificial intelligence, the result of the joint work of technical specialists and professional musicians. He is capable of writing, performing and producing music. Ampere is the first artificial intelligence in history to release its own music album.

Ampere is unique in its kind. AIs generating music existed before it, however, early models worked according to a certain algorithm and the final product required serious human alteration, up to changing chords and entire parts of the melody, before it could be considered a full-fledged musical work.

Ampere does not need help when he creates his own tracks - he independently selects the necessary sounds and builds chord structures. The person processing the resulting melody can only correct the rhythm and style - Ampere does everything else himself in just a few seconds.

2016: Google launches Magenta to create music and video with neural networks

In May 2016, Google even launched a special Magenta project, the main task of which is to study the creative capabilities of a neural network. Scientists plan to master the wisdom of the creative process gradually: first, an algorithm will be developed to create musical works. Then it will be the turn of video and visual arts. Data on the results of the work is planned to be posted in the public domain on GitHub.

several reasons why I wanted to form Magenta, and one of them was the desire to see full, honest and amazing improvements in the art of in-depth learning," says project manager Douglas Eck^[32].

Just a couple of weeks after Magneta opened, a program was launched to import MIDI format music files into the TensorFlow system in order to train artificial intelligence to create music. However, so far the results of the program are not so successful (although in fairness it must be said that not much time has passed).

Notes