Adobe Researchers Introduce An AI Framework That Enables Training A Neural Network To Learn A Data-Driven Prior For Distortion-Aware Mesh Segmentation

Extraction of a sizable surface patch surrounding a point that can be accurately mapped to the 2D plane is necessary for many interactive workflows like decaling, texturing, or painting on a 3D model. Because they are intrinsically user-interactive, may achieve lower distortion than their global equivalents, and are computationally more effective, local parameterizations are desirable in some modeling contexts. But until now, methods for finding surface patches that can be parameterized locally have mostly depended on algorithms that strike a balance between compactness, patch size, and developability priors. This study focuses on segmenting a small sub-region around a point of interest on a mesh for parameterization instead of global parameterization techniques that map the whole mesh to 2D while introducing as few cuts as feasible.

Instead, distortion-aware local segmentations that are best for local parameterization are learned in this study using a data-driven methodology. Their suggested system predicts a patch surrounding a point and its accompanying UV map using a unique differentiable parameterization layer. This enables self-supervised training, which allows us to avoid the shortage of parameterization-labeled datasets by encouraging their network to forecast area-maximizing and distortion-minimizing patches through a sequence of properly crafted priors. Their approach, which they call the Distortion-Aware Wand (DA Wand), produces soft segmentation probabilities from an input mesh and an initial triangle selection. By creating a weighted variant of the traditional parameterization technique LSCM, which they refer to as wLSCM, they include these probabilities in their parameterization layer.

A probability-guided parameterization results from this adaptation, over which distortion energy may be calculated to allow for self-supervised training. They show the direct relationship between probabilities and binary segmentation in the parameterization context by proving the theory that the wLSCM UV map converges to the LSCM UV map as the soft probabilities converge to a binary segmentation mask. As UV distortion increases monotonically with patch size, reducing UV map distortion and increasing the segmentation area are competing goals. They achieve these goals in harmony by creating a unique thresholded distortion loss that penalizes triangles with distortion above a user-specified threshold. The simple addition of these goals results in subpar optimization with unwanted local minima.

They create a brand-new segmentation dataset that is nearly developable, along with an automated creation technique that can be used immediately and pre-train on it to establish the weights of their segmentation network. The network is then trained end-to-end using their parameterization layer with distortion and compactness priors on a dataset of unlabeled natural forms. They use a MeshCNN backbone to learn directly from the triangulation of input data, which allows for sensitivity to sharp features and a big receptive field that provides for patch expansion. Additionally, their approach maintains rigid-transformation invariance by using intrinsic mesh properties as input. Additionally, they promote compactness by using a smoothness loss modeled after the graphcuts technique.

A user may interactively choose a triangle on the mesh using DA Wand to get a sizable, significant region around the selection that can be UV parameterized with little distortion. In contrast to current heuristic approaches, which stop at the limits of high curvature zones, they demonstrate that the neural network can prolong the segmentation with the least amount of distortion gain. Their approach outperforms competing approaches by producing user-conditioned segmentation at interactive speeds. In Figure 1 above, they show an engaging, interactive application of the DA Wand where various areas on the sorting hat mesh are successively picked and decaled. The framework code is freely available on GitHub.

Check out the Project. All Credit For This Research Goes To Researchers on This Project. Also, don’t forget to join our Reddit page and discord channel, where we share the latest AI research news, cool AI projects, and more.