This AI Research from China Introduces ‘City-on-Web’: An AI System that Enables Real-Time Neural Rendering of Large-Scale Scenes over Web Using Laptop GPUs

The conventional NeRF and its variations demand considerable computational resources, often surpassing the typical availability in constrained settings. Additionally, client devices’ limited video memory capacity imposes significant constraints on processing and rendering extensive assets concurrently in real-time. The considerable demand for resources poses a crucial challenge in rendering expansive scenes in real-time, requiring rapid loading and processing of extensive datasets.

To tackle the challenges encountered in the real-time rendering of extensive scenes, researchers at the University of Science and Technology of China proposed a method called Cityon-Web. Taking inspiration from traditional graphics methods used for handling large-scale scenes, they partition the scene into manageable blocks and incorporate varying Levels-of-Detail (LOD) to represent it. 

Radiance field baking techniques are employed to precompute and store rendering primitives into 3D atlas textures organized within a sparse grid in each block, facilitating real-time rendering. However, loading all atlas textures into a single shader is unfeasible due to inherent limitations in shader resources. Consequently, the scene is represented as a hierarchy of segmented blocks, each rendered by a dedicated shader during the rendering process.

Employing a “divide and conquer” strategy, they guarantee that each block has ample representation capability to reconstruct intricate details within the scene faithfully. Moreover, to maintain high fidelity in the rendered output during the training phase, they simulate blending multiple shaders aligned with the rendering pipeline. 

These representations based on blocks and levels-of-detail (LOD) enable dynamic resource management, simplifying the real-time loading and unloading process according to the viewer’s position and field of view. This adaptable loading approach significantly reduces the bandwidth and memory requirements of rendering extensive scenes, leading to smoother user experiences, especially on less powerful devices. 

The experiments conducted illustrate that City-on-Web achieves the rendering of photorealistic large-scale scenes at 32 frames per second (FPS) with a resolution of 1080p, utilizing an RTX 3060 GPU. It uses only 18% of the VRAM and 16% of the payload size compared to existing mesh-based methods.

The combination of block partitioning and Levels-of-Detail (LOD) integration has notably decreased the payload on the web platform while enhancing resource management efficiency. This approach guarantees high-fidelity rendering quality by upholding consistency between the training process and the rendering phase.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, LinkedIn Group, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..