Neural Radiance Fields Transformed: This AI Approach Can Extract Accurate 3D Meshes from NeRFs

For decades, we have imagined a digital world where we can experience the physical world in all its three-dimensional glory, but until recently, achieving this has been a significant challenge. While we’ve been able to communicate with others through calls, videos, and photos, these experiences have been limited to a 2D representation of reality. We’ve always wanted more – the ability to see people, objects, and places in 3D, to immerse ourselves in the world around us. However, accurately reconstructing 3D scenes and objects has been a complex and challenging task, requiring significant advances in technology and computational methods.

Accurate 3D scene and object reconstruction is a crucial problem in various fields such as robotics, photogrammetry, AR/VR, etc. Recently, neural radiance fields (NeRFs) have been the de-facto solution for 3D scene reconstruction. They can synthesize novel views quite accurately using a 3D representation where each location in space can emit radiance. The impressive results of NeRF have attracted attention in the literature, and there have been numerous attempts to improve its performance. 

Most works have focused on improving NeRF in terms of image quality, robustness, training speed, and rendering speed. Though, there is a problem with these works; almost all of them focus on optimizing NeRF for the novel view synthesis (NVS) task. So, we cannot use them to obtain accurate 3D meshes from radiance fields, and that’s why we cannot directly integrate NeRF with most computer graphics pipelines. 

What if we want to extract geometrically accurate meshes from NeRFs so that we can actually integrate them into computer graphics pipelines? How can we extract accurate 3D meshes from NeRFs? Time to meet NeRFMeshing.

NeRFMeshing is designed to extract geometrically accurate meshes from trained NeRF-based networks efficiently. It can produce 3D meshes with accurate geometry that can be rendered in real-time on commodity hardware. 

NeRFMeshing is built on top of trained NeRF networks by introducing a new structure called a signed surface approximation network (SSAN). SSAN acts as a post-processing pipeline that determines the surface and appearance of a NeRF render. It generates a precise 3D triangle mesh of the scene and employs a small appearance network to generate view-dependent colors. NeRFMeshing is compatible with any NeRF and allows for easy integration of new developments, such as better handling of unbounded scenes or reflective objects.

SSAN calculates both a Truncated Signed Distance Field (TSDF) and a feature appearance field. By utilizing the NeRF estimated geometry and training views, the trained NeRF is distilled into the SSAN model. The 3D mesh is then extracted from the SSAN and can be rendered on embedded devices using rasterization and the appearance network at a high frame rate. This method is highly flexible, allowing for fast 3D mesh generation that is not limited to object-centric scenes and can even model complex surfaces.

NeRFMeshing is a novel method for capturing accurate 3D meshes from NeRFs. It can be integrated into any existing NeRF network, enabling advances in NeRF to be used with it. With this breakthrough, we can now extract accurate 3D meshes from NeRFs, which can be used in various fields such as AR/VR, robotics, and photogrammetry.

Check out the Paper. Don’t forget to join our 19k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club