Latest Artificial Intelligence (AI) Research Proposes ECON, A Method To Reconstruct Detailed Clothed 3D Humans From A Color Image

Future video games, movies, mixed reality, telepresence, and the “metaverse” will rely heavily on human avatars. We need to accurately reconstruct detailed 3D persons from color images taken in the field to create realistic and customized avatars at scale. Due to the difficulties involved, this issue still needs to be solved. People dress differently, accessorize differently, and posture their bodies in various, frequently innovative ways. A decent reconstruction technique should capture them precisely while standing up to creative attire and positions. These techniques need a more specific understanding of the anatomy of the human body and thus tend to overfit the positions observed in the training data.

As a result, people frequently create deformed forms or disembodied limbs for pictures of unknown stances; see the second row of Figure 1. The third and fourth rows of Figure 1 show how follow-up work regularises the IF using a shape prior supplied by an explicit body model to account for such artifacts, however, this can restrict applicability to novel apparel while attenuating form details. In other words, robustness, generality, and detail may all be traded off. The robustness of explicit anthropomorphic body models and the adaptability of IF to capture various topologies are what we want, though.

In light of this, we note two important facts: (1) Inferring a 3D geometry with comparable precise features is still difficult, even if it is very simple to infer detailed 2D normal maps from color photographs. Using networks, we can imply precisely “geometry-aware” 2D maps that we can lift into 3D. (2) It’s possible to think of a body model as a low-frequency “canvas” that “guides” the sewing of finely detailed surface portions. We create ECON, a revolutionary technique for “Explicit Clothed People Obtained from Normals,” with these considerations in mind. An RGB picture and an inferred SMPL-X body are the inputs for ECON. Then, it produces a 3D person wearing free-form clothes with an advanced degree of detail and robustness (SOTA).

Step 1: Normal rebuilding of the front and back. Using a conventional image-to-image translation network, we forecast front- and back-side clothed-human normal maps from the input RGB picture, conditional on the body estimation. 

Step 2: Reconstruction of the front and back surfaces. To create accurate and cohesive front-back-side 3D surfaces, MF, MB, we use the previously predicted normal maps and the matching depth maps produced from the SMPL-X mesh. To accomplish this, we extend the recently published BiNI method and create a new optimization strategy to achieve three objectives for the surfaces that result:

1. Their high-frequency components agree with dressed-human normals.
2. Their low-frequency components and discontinuities agree with SMPL-X ones.
3. The depth values on their silhouettes are coherent with one another and consistent with the SMPL-X-based depth maps.

The occluded and “profile” sections of the two output surfaces, MF, and MB, lack geometry, making them detailed but incomplete.

Step 3: Complete the 3D form. The SMPL-X mesh and the two d-BiNI surfaces, MF and MB, are the two inputs for this module. The aim is to “paint” the geometry that is lacking. Existing solutions have trouble solving this issue. On the one hand, Poisson reconstruction naively “fills” gaps without taking advantage of a shape distribution prior, resulting in “blobby” forms.

However, data-driven methods need help with (self-)occlusion-related missing pieces and lose information available in supplied high-quality surfaces, leading to degenerate geometries. We overcome the restrictions above in two steps: (1) For SMPL-X to regularise form “infilling,” we expand and retrain IF-Nets to be conditioned on the SMPL-X body. Triangles near MF and MB are discarded, while the remaining triangles are kept as “infilling patches.” (2) Using Poisson reconstruction, we join the front- and back-side surfaces as well as the “infilling patches”; take notice that the gaps between them are small enough for a universal technique.

ECON combines the best features of explicit and implicit surfaces to produce strong and detailed 3D reconstructions of clothed people. As seen at the bottom of Figure 1, the outcome is a complete 3D form of a dressed person. We assess ECON using real-world photos and well-known benchmarks (CAPE, Renderpeople). According to a quantitative study, ECON performs better than SOTA. Qualitative findings show that ECON generalizes more effectively than SOTA to a wide range of positions and attire, even when the topology is extremely loose or complicated. This is supported by perceptual research, demonstrating that ECON is highly favored over rivals in difficult positions and loose apparel when competing with PIFuHD in fashion photographs. Code and models are accessible on GitHub.

Check out the Paper, Code, and Project. All Credit For This Research Goes To Researchers on This Project. Also, don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.