DIRFA (DIverse yet Realistic Facial Animations)

2023: Product Announcement

On November 16, 2023, Singaporean researchers from the School of Computer Science and Engineering as part of Nanyang Technological University announced the development of an artificial intelligence-based program that allows the generation of video materials based on a single photo and audio recording. A system called DIRFA is capable of reproducing facial expressions and head movements of a talking person.

The DIRFA platform, or DIverse yet Realistic Facial Animations, uses special AI algorithms to create 3D video with realistic and consistent facial animation synchronized with audio recording. The new solution, it is claimed, circumvents the shortcomings of similar programs, which can face problems when varying poses and reproducing emotions. Over 1 million audiovisual clips from more than 6,000 people from The VoxCeleb2 Dataset open source database were used to train the generative AI model. As a result, the program learned to predict speech signals and associate them with facial expressions and head movements.

Singapore researchers report on the development of a program that allows the generation of video materials based on a single photo and audio recording

Creating realistic facial expressions based on audio recordings presents a challenge, the researchers say. People pronounce the same words differently in different contexts. Therefore, multiple facial expressions may be appropriate for identical phrases. The authors of the project emphasize that speech usually has strong associations with lip movements, but weaker associations with facial expression and head position. Therefore, the team focused on creating a program that reproduces exactly the movements of the lips as accurately as possible.

DIRFA could lead to new applications in a variety of areas, including healthcare, according to the developers. For example, more realistic avatars can be created that will help people with speech disorders or paralyzed patients more accurately convey their thoughts and emotions.^[1]

Notes