Attention Gaming Industry! No More Weird Mirrors With Mirror-NeRF

NeRFs or Neural Radiance Fields use a combination of RNN and CNN to capture the physical characteristics of an object, such as the shape, material, and texture. They can generate realistic images of objects in different lighting conditions. They have proved most useful in medicine, robotics, and entertainment due to their ability to create high-resolution images.

3D reconstruction and rendering of scenes with mirrors, which exist ubiquitously in the real world, have been a long-standing challenge in computer vision. Dealing with the inconsistencies in reconstruction with mirrors with NeRF, researchers at Zhejiang University are introducing Mirror-NeRF that correctly renders the reflection in the mirror in a unified radiance field by submitting the reflection probability and tracing the rays following the light transport model of Whitted Ray Tracing. 

NeRF, RefNeRF, and NeRFReN all three methods generated mirror reflection from new viewpoints by interpolating the previously learned reflections. However, they have limitations regarding reliably inferring reflections not seen during training and synthesizing reflections for newly introduced items or mirrors in the scene. The freshly introduced technique Mirror-NeRF can accurately draw the reflection in the mirror and serve various scene modification applications by integrating the physical ray tracing into the neural rendering process. 

Five synthetic and four real datasets were created, and quantitative comparisons of novel view synthesis on the metrics Peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) were made. Due to the mirror’s bumpy surface greatly affecting the reflection quality, several regularisation terms were also introduced in the optimization process. When all regularisation terms were enabled, we successfully obtained the precise reflection in the mirror with the highest image quality. 

The findings showed NeRF, Ref-NeRF, and NeRFReN struggled to produce the reflection of the objects whose reflection has high-frequency variations in color, such as the distorted hanging picture in the mirror of the meeting room, a blurry curtain in the mirror of the office and the lounge, and a “fogged” clothes in the mirror of the clothing store. On the other hand, the new method rendered detailed reflections of objects by tracing the reflected rays. Although there is immense advancement in the work with mirrors, researchers are yet to incorporate refraction in the framework. 

In conclusion, this breakthrough promises new avenues in the gaming and film industries. Artists may desire to create complex visual effects and utilize mirror manipulation, for example, substituting the reflections in the mirror with a different scene. We can synthesize the photo-realistic view of the new scene in the mirror with multi-view consistency. 

Check out the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.