NeuMan

An open neural network has been launched to create 3D models of a person using video from a smartphone

At the end of August 2022, it became known about the launch of an open neural network to create 3D models of a person based on video from a smartphone. This is the development of Cornell University, its sources are posted on GitHub.

A system called NeuMan, using machine learning, creates a three-dimensional human model using an iPhone camera. The researchers tackled the problem of photorealistic rendering, which takes a long time and requires high performance from the machine. They created a neural network, which is enough to provide a video shot on a smartphone camera (the team used an iPhone). Ready-made models can be used to create scenes in videos or augmented reality, giving them different poses and appearance.

An open neural network has been launched to create 3D models of a person using video from a smartphone

According to the researchers, they have trained two AI models, one focused on creating the model and the other on creating the scene. To teach NeuMan, methods for estimating coarse geometry were used - an approximate estimate allows you to create a deforming field from observation space to canonical space, and the rendering result does not depend on the original posture of the prototype.

Based on the models obtained, the authors of the development made small videos with tricks in which you can evaluate the work of the neural network. They left the original background, but forced 3D models to perform various actions that the actors did not perform, such as jumping over obstacles, making a wheel, tumbling, dancing. In some cases, 3D models are somewhat blurred, and the shadow of a real person is noticeable on the environment, but in general the result looks good.

According to the developers of the technology, until recently, modern neural networks are capable of providing high-quality rendering for creating 3D human models, but they require a large amount of input data, expensive model training and numerous angles for forming a picture. The NeuMan system differs in that it needs it, the authors of the project say.^[1]

Notes