Researchers Introduce gDNA: a Novel AI Method that Learns to Generate Detailed 3D Shapes of People in a Variety of Garments with Corresponding Skinning Weights

This article summary is based on the research paper: 'gDNA: Towards Generative Detailed Neural Avatars'

All credits for this research goes to the authors of this paper. 👏 👏 👏 👏 

Please don't forget to join our ML Subreddit

Need help in creating ML Research content for your lab/startup? Talk to us at [email protected]

The capacity to quickly produce a variety of high-quality virtual humans with complete control over their position has numerous applications in filmmaking, gaming, virtual reality/augmented reality, architecture, and computer vision. Modern computer graphics techniques achieve photorealism, but they often require a great deal of expertise.

Continuous and resolution-independent neural 3D representations have powered recent advances in generative modeling of 3D rigid objects. Due to the complicated interaction of clothes, their topology, and pose-driven deformations, modeling clothed humans is a challenging task.

Recent research has used implicit neural surfaces to learn high-quality articulated avatars for a single subject, but these methods aren’t generative, meaning they can’t create new human identities or shapes. 

By learning a generative model of individuals, researchers at the Max Planck Institute for Intelligent Systems hope to make 3D human avatars broadly available. They offer the first approach for generating diverse 3D virtual individuals with unique identities and shapes, appearing in various clothing styles and positions, with realistic details such as wrinkles in garments.

The team proposes gDNA, a method that synthesizes 3D surfaces of novel human shapes, with control over clothing design and poses, producing realistic details of the garments, as the first step toward completely generative modeling of detailed neural avatars.

The team creates a multi-subject implicit generative representation to take advantage of raw 3D scans. SNARF, a recent method for learning single-subject articulation-dependent effects that have been demonstrated to generalize effectively to unknown poses, is the foundation for the work. The team adopts a multi-subject technique that is taught from a small number of posed scans of many different subjects. This is accomplished by including a latent space for the conditional production of clothed human shape and skinning weights. 

The team demonstrates the first approach that can produce a wide variety of 3D clothed human shapes with detailed wrinkles under pose control, using only posed scans. The learned skinning weights can be used to reposition the generated samples. Researchers test gDNA extensively and find that it outperforms baselines significantly. They also show that gDNA can be used to fit and reanimate 3D images, outperforming the current state-of-the-art (SOTA) models.

Conclusion 

The researchers at the Max Planck Institute for Intelligent Systems developed gDNA, a 3D clothed human generative model that can generate a wide range of clothed people with precise wrinkles and explicit posture control. Learning from only a few posed scans per participant is possible using implicit multi-subject forward skinning. The team shows that gDNA outperforms state-of-the-art approaches in a variety of applications, including animation and 3D fitting.

Paper: https://ait.ethz.ch/projects/2022/gdna/downloads/main.pdf

Project: https://xuchen-ethz.github.io/gdna/

Github: https://github.com/xuchen-ethz/gdna