Oxford University Researchers Introduce A Diffusion Model Called RealFusion That Can Generate 360-degree Reconstructions Of Objects From An Image

With the introduction of Large Language Models and their increasing popularity, a number of tasks are conveniently getting executed. Models like DALL-E, developed by OpenAI, are already being used by more than a million users. It is a text-to-image generation model that generates high-quality images based on the entered textual description. Diffusion models behind these generative LLMs enable the user to easily produce an image from a text by iteratively modifying and updating variables representing the image. Apart from this functionality, some models are also being used to generate an image from an image. These models edit an image to produce the required target image by maintaining a lot of fine detailing.

Generating an image from an image has become possible, but reconstructing a two-dimensional image into a three-dimensional one is still difficult. This is because it is difficult to retrieve enough information from a single image that would be required to produce a 3D image. A research team from Oxford University has introduced a new diffusion model capable of generating 360-degree reconstructions of different objects from a single image. Called RealFusion, this model overcomes the challenge of 360-degree photographic representation, as the traditional approaches believe that without access to multiple views, reconstruction is not possible.

The team has used a neural radiance field to extract 3D information from a currently existing 2D model by expressing the 3D geometry and the image’s appearance. They have optimized the radiance field by keeping in mind two primary objectives –

1. Reconstruction objective – This has been used to make sure that the radiance field imitates the fed input image. This objective is from the viewpoint of the field.
2. Score Distillation Sampling (SDS) – This is an SDS-based prior objective that has been used to ensure that the object samples produced by the diffusion model and their novel viewpoints imitate the radiance field.

The researchers have utilized the idea of creating 3D images and constituting different views using prior understandings of the pretrained diffusion models like Stable Diffusion. 

Some of the primary contributions by the team are as follows –

1. RealFusion can extract a 360-degree photographic 3D reconstruction from a single image without considering any assumptions like the 3D supervision or the kind of object that has been imaged.
2. RealFusion works by leveraging a 2D diffusion image generator via a new single-image variant of textual inversion.
3. The team has also introduced some new regularizers with their effective implementation using InstantNGP.
4. RealFusion outperforms the traditional methods by showing the state-of-the-art reconstruction outcomes on several images from existing datasets and wild images.

RealFusion is a breakthrough in image generation as it caters to the domain of dimensions. Comparing RealFusion with currently existing approaches, it showed a better quality of the produced images along with better shape, appearance, and extrapolation features. It is undoubtedly a great addition to the category of diffusion models.

Check out the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 14k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.