Is Real-Time 3D Rendering on Mobile Devices Now Possible? Researchers from China Introduced VideoRF: An AI Approach to Enable Real-Time Streaming and Rendering of Dynamic Radiance Fields on Mobile Platforms

Neural Radiance Fields (NeRF) represent an innovative 3D scene depiction technique in computer graphics and computer vision. Leveraging neural networks, this method is a sophisticated means to render scenes and objects within a three-dimensional space. NeRF functions by employing a neural network to model the volumetric scene as a continuous function, where inputting a 3D coordinate allows the prediction of scene color and density at that specific point. This approach enables the generation of highly detailed and lifelike images from various viewpoints, eliminating the need to store 3D models or point clouds explicitly.

However, their dependence on canonical space limits effectiveness in sequences with large motions or topological changes. These techniques fall short for dynamic scenes as their per-frame representation becomes too bulky for real-time loading. Despite the existence of NeRF solutions for dynamic scenes and mobile optimization, it is a cohesive approach that effectively addresses both, which still needs to be improved.

Researchers at Shanghai Tech University propose VideoRF. It is a novel neural modeling approach that enables real-time streaming and rendering of human-centric dynamic radiance fields on common mobile devices. Their idea is to view the 4D feature volumes reconstructed from a dynamic scene as a 2D feature image stream, which is friendly to video codec. Their method involves rendering pipelines for each representation.

The Rendering pipelines in VideoRF use a set of mapping tables to connect 3D spaces with the 2D feature images. To reduce the computational complexity of rendering, we adopt the deferred shading model with a global tiny MLP to decode the integrated feature vectors into pixel colors. They use a sequential training scheme to generate the 2D feature image stream effectively.

VideoRF can adaptively group frames by analyzing motion from the sequential data to ensure temporal stability for mapping table generation. VideoRF contains spatial and temporal continuity regularizers that can be applied to the mapped 2D feature images using the mapping table. They deploy 3D and 2D Morton sorting techniques to improve spatial consistency in the mapping. Researchers show that off-the-shelf video codecs can efficiently compress this feature image stream with spatiotemporal sparsity and reach high compression rates.

Researchers also built a cross-platform player based on VideoRF that can play FVVs in real-time, supported by video hardware accelerators. Users can interactively drag, rotate, pause, play, fast forward, rewind, and jump to specific frames, providing a viewing experience as seamless as watching online videos on various devices, including smartphones, tablets, laptops, and desktops, which were unseen before.

Their method currently lacks support for temporal interpolation, and their task will be to resolve it in the future. In conclusion, VideoRF innovatively processes feature volumes as 2D feature streams and adopts deferred rendering to leverage classical video codecs and rasterization pipelines effectively. Their method, with the unique ability to render dynamic radiance fields on mobile devices in real time, marks a significant step forward in neural scene modeling and immersive VR/AR applications.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..