RecolorNeRF: A Novel User-Friendly Color Editing Approach For The Neural Radiance Field

Neural radiance fields (NeRF) are a potent representation of 3D scenes, making it possible that they may one day replace photos and movies as a new type of media. Supporting the editing of such a new representation is essential to achieving this goal. Recent publications on the subject have explored editing NeRF in terms of geometry deformation, appearance editing, and style transfer, among other things. Recoloring is appearance editing that often entails adjusting certain color tones in a scene for improvement or correction. This process is crucial to the process of making movies. In the example of Fig. 1, the red automobile may be changed into a blue one in a photorealistic manner using a recoloring edition.

Palette-based color editing (PCE), one of the methods for recoloring a picture now in use, offers the most naturally flowing means of engagement. PCE entails these three essential steps: 1) Extraction of a palette. The first step is to choose a group of representative colors and create a palette based on the landscape. 2) Decomposition of layers. They specify a matching picture layer with a consistent color value for each item in the palette. The major aim of this stage is to choose the best method for blending these layers to recreate the original image. Editing in color. By changing the color of each layer, the scene may be naturally recolored based on the previous two processes.

Tan and colleagues presented a convex-hull simplification technique for palette extraction as one of the SOTA approaches of PCE for pictures. Layer decomposition is then expressed as an optimization problem using the acquired palette. The assumption of the blending weights’ sparsity makes the issue manageable. In this study, they put forth a brand-new technique called RecolorNeRF, which, to their knowledge, is the first attempt to use a completely learnable palette for layer decomposition in photorealistic PCE for NeRF representation. Although PosterNeRF has experimented with NeRF recoloring based on a palette, the results could be more realistic since color tweaking can only be enabled after posterization. As is well known, multiview pictures are frequently used to rebuild the NeRF of a scene.

Thus, another possible way to perform PCE of NeRF is extracting palettes from the pixels in all input images, following the method of, and then conducting layer decomposition and color editing for each rendered view of the pre-trained NeRF. Though it is easy to implement, this strategy suffers from three major issues: First, the recoloring in this way becomes post-processing of NeRF rendering, which causes expensive computational costs. Second, as each view is independently processed, the results need more view consistency. Third, the palette extraction is achieved by a heuristic method, which may make the palette color less representative and the layer decomposition not clean enough, interfering with the color manipulation. Their main suggestion is to improve the palette, the layer blending weights, and the volumetric radiance fields in a single framework to address the problems mentioned earlier. They then employ “over” composition to deal with complex scenarios as the final picture formulation. Specifically, alpha blending of a collection of ordered layers, each corresponding to an alpha weight, is used to represent each pixel. Then, for each layer, they construct a volumetric alpha field, which, like the radiance field, may also be represented by an MLP. Keep in mind that different levels employ various MLPs. Therefore, they need to jointly optimize the MLPs for the density field and the MLPs for the blending weights.

As is well known, each of the earlier PCE systems carried out palette extraction individually. The first attempt at optimizing the palette is what they have offered. Two novel designs are put up to help with the joint optimization problem: 1) A innovative convex-hull regularisation is proposed to allow a limited palette of colors to depict the entire scene faithfully. 2) Per normal, sparsities on the blending weights are employed to make the palette color more realistic. The sparsity constraint is given a unique order-aware weighting mechanism to improve the capacity to simulate complicated situations. RecolorNeRF can create photorealistic pictures with adjustable color schemes and robustly deconstruct the implicit representation, according to experiments. To their knowledge, they are the first to consider jointly optimizing the palette and the alpha blending weights, for which a novel convex-hull regulation is designed to make it solvable. The whole RecolorNeRF framework is carefully designed, allowing the color editing of NeRF representation to be done in a photorealistic manner using a fully learnable palette for layer decomposition. The code is yet to be released but a video demo can be found on the project website

Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our Reddit Page, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.