Oxford Researchers Introduce Splatter Image: An Ultra-Fast AI Approach Based on Gaussian Splatting for Monocular 3D Object Reconstruction

Single-view 3D reconstruction stands at the forefront of computer vision, presenting a captivating challenge and immense potential for various applications. It involves inferring an object or scene’s three-dimensional structure and appearance from a single 2D image. This capability is significant in robotics, augmented reality, medical imaging, and cultural heritage preservation. Overcoming this challenge has been a focal point in the realm of computer vision research, leading to innovative methodologies and advancements.

Despite notable progress, challenges persist. Accurate depth estimation, handling occlusions, capturing fine details, and achieving robustness to varying lighting conditions and object textures remain ongoing hurdles. Additionally, generalizing the learned representations across diverse object categories and scenes poses a challenge in achieving consistent and accurate reconstructions.

Researchers at the University of Oxford have introduced the splatter image technique to tackle the inherent difficulty in computer vision of reconstructing 3D shapes from a single view. Their approach leverages Gaussian Splatting as the foundational 3D representation, capitalizing on its rapid rendering capabilities and high-quality outputs. This method forecasts a 3D Gaussian entity for every pixel within the input image, facilitated by an image-to-image neural network.

It is important to acknowledge that despite the network’s exposure to only a singular side of the object, Splatter Image can generate a complete 360-degree reconstruction by utilizing prior knowledge obtained during the training phase. 

That comprehensive information representing the full 360-degree view is encoded within the 2D image by assigning distinct Gaussians in a specific 2D vicinity to various sections of the 3D object. Additionally, the researcher’s findings reveal that numerous Gaussians are inactive in practical scenarios by adjusting their opacity to zero. Consequently, these inactive Gaussians can be removed through post-processing methods.

Remarkably, their model’s efficiency allows for training on a single GPU using standard benchmarks for 3D objects, whereas other approaches often necessitate distributed training across multiple GPUs. Furthermore, they expand the capabilities of Splatter Image to accommodate multiple views as input. This extension involves consolidating the Gaussian mixtures forecasted from individual views, aligning them to a shared reference, and combining them to form a unified representation.

Differing from these approaches, their technique anticipates a 3D Gaussian blend in a direct, forward-moving process. Consequently, their method excels in rapid inference, attaining real-time rendering capabilities while delivering top-tier image quality across various metrics in the widely recognized single-view reconstruction benchmark.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..