How Single-View 3D Reconstruction Works?

Traditionally, models for single-view object reconstruction built on convolutional neural networks have shown remarkable performance in reconstruction tasks. In recent years, single-view 3D reconstruction has emerged as a popular research topic in the AI community. Irrespective of the specific methodology employed, all single-view 3D reconstruction models share the common approach of incorporating an encoder-decoder network within their framework. This network performs complex reasoning about the 3D structure in the output space.

In this article, we will explore how single-view 3D reconstruction operates in real-time and the current challenges these frameworks face in reconstruction tasks. We will discuss various key components and methods utilized by single-view 3D reconstruction models and explore strategies that could enhance the performance of these frameworks. Additionally, we will analyze the results produced by state-of-the-art frameworks that employ encoder-decoder methods. Let’s dive in.

Single-View 3D Object Reconstruction

Single-view 3D object reconstruction involves generating a 3D model of an object from a single viewpoint, or in simpler terms, from a single image. For instance, inferring the 3D structure of an object, such as a motorcycle from an image, is a complex process. It combines knowledge of the structural arrangement of parts, low-level image cues, and high-level semantic information. This spectrum encompasses two main aspects: reconstruction and recognition. The reconstruction process discerns the 3D structure of the input image using cues like shading, texture, and visual effects. In contrast, the recognition process classifies the input image and retrieves a suitable 3D model from a database.

Current single-view 3D object reconstruction models may vary in architecture, but they are unified by the inclusion of an encoder-decoder structure in their framework. In this structure, the encoder maps the input image to a latent representation, while the decoder makes complex inferences about the 3D structure of the output space. To successfully execute this task, the network must integrate both high-level and low-level information. Additionally, many state-of-the-art encoder-decoder methods rely on recognition for single-view 3D reconstruction tasks, which limits their reconstruction capabilities. Moreover, the performance of modern convolutional neural networks in single-view 3D object reconstruction can be surpassed without explicitly inferring the 3D object structure. However, the dominance of recognition in convolutional networks in single-view object reconstruction tasks is influenced by various experimental procedures, including evaluation protocols and dataset composition. Such factors enable the framework to find a shortcut solution, in this case, image recognition.

Traditionally, Single-view 3D object reconstruction frameworks approach the reconstruction tasks using the shape from shading approach, with texture and defocus serving as exotic views for the reconstruction tasks. Since these techniques use a single depth cue, they are capable of providing reasoning for the visible parts of a surface. Furthermore, a lot of single-view 3D reconstruction frameworks use multiple cues along with structural knowledge for estimating depth from a single monocular image, a combination that allows these frameworks to predict the depth of the visible surfaces. More recent depth estimation frameworks deploy convolutional neural network structures to extract depth in a monocular image. 

However, for effective single-view 3D reconstruction, models not only have to reason about the 3D structure of the visible objects in the image, but they also need to hallucinate the invisible parts in the image using certain priors learned from the data. To achieve this, a majority of models currently deploy trained convolutional neural network structures to map 2D images into 3D shapes using direct 3D supervision, whereas a lot of other frameworks deployed a voxel-based representations of 3D shape, and used a latent representation to to generate 3D up-convolutions. Certain frameworks also partition the output space hierarchically to enhance computational and memory efficiency that enables the model to predict higher-resolution 3D shapes. Recent research is focusing on using weaker forms of supervision for single-view 3D shape predictions using convolutional neural networks, either comparing predicted shapes and their ground-truth predictions to train shape regressors or using multiple learning signals to train mean shapes that helps the model predict deformations. Another reason behind the limited advancements in single-view 3D reconstruction is the limited amount of training data available for the task. 

Moving along, single view 3D reconstruction is a complex task as it not only interprets visual data geometrically, but also semantically. Although they are not completely different, they do span different spectrums from geometric reconstruction to semantic recognition. Reconstruction tasks per-pixel reasoning of the 3D structure of the object in the image. Reconstruction tasks do not require semantic understanding of the content of the image, and it can be achieved using low-level image cues including texture, color, shading, shadows, perspective, and focus. Recognition on the other hand is an extreme case of using image semantics because recognition tasks use whole objects and amounts to classify the object in the input, and retrieve the corresponding shape from the database. Although recognition tasks can provide robust reasoning about the parts of the object not visible in the images, the semantic solution is feasible only if it can be explained by an object present in the database. 

Although recognition and reconstruction tasks might differ from one another significantly, they both tend to ignore valuable information contained in the input image. It is advisable to use both these tasks in unison with one another to obtain the best possible results, and accurate 3D shapes for object reconstruction i.e. for optimal single-view 3D reconstruction tasks, the model should employ structural knowledge, low-level image cues, and high-level understanding of the object. 

Single-View 3D Reconstruction : Conventional Setup

To explain the conventional setup and analyze the setup of a single-view 3D reconstruction framework, we will deploy a standard setup for estimating the 3D shape using a single view or image of the object. The dataset used for training purposes is the ShapeNet dataset, and evaluates the performance across 13 classes that allows the model to understand how the number of classes in a dataset determines the shape estimation performance of the model.

A majority of modern convolutional neural networks use a single image to predict high-resolution 3D models, and these frameworks can be categorized on the basis of the representation of their output: depth maps, point clouds, and voxel grids. The model uses OGN or Octree Generating Networks as its representative method that historically has outperformed the voxel grid approach, and/or can cover the dominant output representations. In contrast with existing methods that utilize output representations, the OGN approach allows the model to predict high-resolution shapes, and uses octrees to efficiently represent the occupied space. 

Baselines

To evaluate the results, the model deploys two baselines that consider the problem purely as a recognition task. The first baseline is based on clustering whereas the second baseline performs database retrieval. 

Clustering

The the clustering baseline, the model uses the K-Means algorithm to cluster or bunch the training shapes in K sub-categories, and runs the algorithm on 32*32*32 voxelizations flattened into a vector. After determining the cluster assignments, the model switches back to working with models with higher resolution. The model then calculates the mean shape within each cluster, and thresholds the mean shapes where the optimal value is calculated by maximizing the average IoU or Intersection over Union over the models. Since the model knows the relation between the 3D shapes and the images within the training data, the model can readily match the image with its corresponding cluster. 

Retrieval

The retrieval baseline learns to embed shapes and images in a joint space. The model considers the pairwise similarity of 3D matrix shapes in the training set to construct the embedding space. The model achieves this by using the Multi-Dimensional Scaling with Sammon mapping approach to compress each row in the matrix to a low-dimensional descriptor. Furthermore, to calculate the similarity between two arbitrary shapes, the model employs the light field descriptor. Additionally, the model trains a convolutional neural network to map images to a descriptor to embed the images in the space. 

Analysis

Single-view 3D reconstruction models follow different strategies as a result of which they outperform other models in some areas whereas they fall short in others. To compare different frameworks, and evaluate their performance, we have different metrics, one of them being the mean IoU score. 

As it can be seen in the above image, despite having different architectures, current state of the art 3D reconstruction models deliver almost similar performance. However, it is interesting to note that despite being a pure recognition method, the retrieval framework outperforms other models in terms of mean and median IoU scores. The Clustering framework delivers solid results outperforming the AtlasNet, the OGN and the Matryoshka frameworks. However, the most unexpected outcome of this analysis remains Oracle NN outperforming all other methods despite employing a perfect retrieval architecture. Although calculating the mean IoU score does help in the comparison, it does not provide a full picture since the variance in results is high irrespective of the model. 

Common Evaluation Metrics

Single-View 3D Reconstruction models often employ different evaluation metrics to analyze their performance on a wide range of tasks. Following are some of the commonly used evaluation metrics. 

Intersection Over Union

The Mean of Intersection Over Union is a metric commonly used as a quantitative measure to serve as a benchmark for single-view 3D reconstruction models. Although IoU does provide some insight into the model’s performance, it is not considered as the sole metric to evaluate a method since it indicates the quality of the shape predicted by the model only if the values are sufficiently high with a significant discrepancy being observed between the low and mid-range scores for two given shapes. 

Chamfer Distance

Chamfer Distance is defined on point clouds, and it has been designed in a way that it can be applied to different 3D representations satisfactorily. However, the Chamfer Distance evaluation metric is highly sensitive to outliers that makes it a problematic measure to evaluate the model’s performance, with the distance of the outlier from the reference shape significantly determining the generation quality. 

F-Score

The F-Score is a common evaluation metric actively used by a majority of multi-view 3D reconstruction models. The F-Score metric is defined as the harmonic mean between recall & precision, and it evaluates the distance between the surfaces of the objects explicitly. Precision counts the percentage of reconstructed points lying within a predefined distance to the ground truth, to measure the accuracy of the reconstruction. Recall on the other hand counts the percentage of points on the ground truth lying within a predefined distance to the reconstruction to measure the completeness of the reconstruction. Furthermore, by varying the distance threshold, developers can control the strictness of the F-Score metric. 

Per-Class Analysis

The similarity in performance delivered by the above frameworks cannot be a result of methods running on different subset of classes, and the following figure demonstrates the consistent relative performance across different classes with the Oracle NN retrieval baseline achieving the best result of them all, and all methods observing high variance for all classes.  

Furthermore, the number of training samples available for a class might lead one to assume it influences the per-class performance. However, as demonstrated in the following figure, the number of training samples available for a class does not influence the per-class performance, and the number of samples in a class and its mean IoU score are not correlated. 

Qualitative Analysis

The quantitative results discussed in the section above are backed by qualitative results as shown in the following image. 

For a majority of classes, there is no significant difference between the clustering baseline and the predictions made by decoder-based methods. The Clustering approach fails to deliver results when the distance between the sample and the mean cluster shape is high, or in situations when the mean shape itself cannot describe the cluster well enough. On the other hand, frameworks employing decoder-based methods and retrieval architecture deliver the most accurate and appealing results since they are able to include fine details in the generated 3D model. 

Single View 3D Reconstruction : Final Thoughts

In this article, we have talked about Single View 3D Object Reconstruction, and talked about how it works, and talked about two baselines: Retrieval and Classification, with the retrieval baseline approach outperforming current state of the art models. Finally, although Single View 3D Object Reconstruction is one of the hottest topics and most researched topics in the AI community, and despite making significant advances in the past few years, Single View 3D Object Reconstruction is far from being perfect with significant roadblocks to overcome in the upcoming years.