Researchers from MIT and Meta Introduce PlatoNeRF: A Groundbreaking AI Approach to Single-View 3D Reconstruction Using Lidar and Neural Radiance Fields

Researchers from the Massachusetts Institute of Technology(MIT), Meta, and Codec Avatars Lab have addressed the challenging task of single-view 3D reconstruction from a neural radiance field (NeRF) perspective and introduced a novel approach, PlatoNeRF. The method proposes a solution using time-of-flight data captured by a single-photon avalanche diode, overcoming limitations associated with data priors and shadows observed by RGB cameras. 

It leverages two-bounce light measured by lidar, employing lidar transient data for supervision in modeling optical paths within NeRF. This approach distinguishes PlatoNeRF from existing methods, as it enables the reconstruction of both visible and occluded geometry without relying on data priors or controlled ambient lighting. The researchers also demonstrate improved generalization under practical constraints on sensor spatial and temporal resolution.

The significance of PlatoNeRF in the context of emerging single-photon lidars becoming prevalent in consumer devices such as phones, tablets, and headsets. Notably, PlatoNeRF showcases accurate single-view 3D reconstruction without hallucinating details and demonstrates robustness to ambient light, scene albedo, and spatial-temporal resolution constraints. The method’s implicit representation allows for improved generalization to lower resolutions than existing lidar methods.

The comparison was made with PlatoNeRF with two methods, one that uses two-bounce lidar for single-view 3D reconstruction without learning and one that uses shadows measured by an RGB camera to train NeRF. Through the experiments, it was observed that the proposed model performed better than both  BF Lidar and S³ -NeRF across L1 depth and PNSR metrics on the reconstructed depth images.  The model was able to reconstruct the visible and occluded parts of the scene, providing accurate scale and absolute depth, achieving much smoother results than BF lidar. The method’s efficiency was further demonstrated in real-world scenarios, showcasing competitive performance compared to Bounce-Flash Lidar.

In conclusion, PlatoNeRF offers a promising direction in the field of 3D reconstruction by combining the strengths of NeRF and lidar, particularly as single-photon lidars become increasingly prevalent in consumer devices. The method’s ability to reconstruct visible and occluded geometry from a single view without data priors or strict lighting conditions marks a significant advancement in the realm of 3D scene understanding.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..