е MinD-Video

2023: Creating Technology

On May 19, 2023, researchers from the National University of Singapore and the Chinese University of Hong Kong announced the development of technology that allows the generation of video materials based on information about brain activity. The project was named MinD-Video.

The development is based on Stable Diffusion software with artificial intelligence algorithms. This software is designed to form images based on text descriptions. Scientists have combined the capabilities of Stable Diffusion with functional magnetic resonance imaging (fMRI). As a result, it was possible to create a new system for "reading thoughts."

Researchers report on development of technology that allows generation of video footage based on brain activity information

The authors of the project showed volunteers videos with various life situations, for example, with animals or a car moving in a wooded area. Then, using the MinD-Video model, a synthesized video sequence based on the brain signals of the subjects was formed. It turned out that the differences between the original and reconstructed rollers are insignificant. In particular, the AI system correctly reproduced the plot, color palette and other parameters. Overall, accuracy was at 85%.

In the course of the study, experts found that the fMRI encoder works according to a hierarchical principle, moving from structural information to more abstract and visual elements. Moreover, the system is able to develop over time, capturing more detailed information. This makes it possible to generate videos, the quality of which increases as you learn. In the future, the MinD-Video model can find application in different areas. On its basis, in particular, advanced brain-computer interfaces or specialized applications can be created to help people with disabilities establish communication with the outside world.^[1]

Notes