Meet ProlificDreamer: An AI Approach That Delivers High-Fidelity and Realistic 3D Content Using Variational Score Distillation (VSD)

Text-to-image diffusion models are getting significantly popular recently for their ability to generate high-quality, diverse images. With the power of capturing complex data distributions using Generative Artificial Intelligence, several industries, including animation, gaming, virtual reality (VR), and augmented reality (AR), are making use of these models. These domains have undergone radical change due to the development of 3D content and technologies by improvisation in perceiving, interacting with, and visualizing complicated settings and things that closely mirror real-world situations. 

Text-to-3D models have emerged as a promising approach to streamline the 3D content creation process. By automating the creation of 3D material from textual descriptions, these innovative models help in doing away with the need for manual design and modeling, all thanks to diffusion models. To train a diffusion model to recognize the connection between the text and the related 3D scene representations, a huge dataset of paired text-to-3D image examples is used. The model gains the ability to accurately represent the statistical relationships between the text and the 3D scene elements.

A technique that has been showing a good amount of potential in the production of text-to-3D models by using pre-trained large-scale text-to-image diffusion models is Score Distillation Sampling (SDS). Considering its limitations, including oversaturation, over-smoothing, and low diversity issues, a team of researchers has come up with a new approach called variational score distillation (VSD).

This principled particle-based variational framework overcomes the issues in the text-to-3D image generation with the main idea of modeling the 3D parameter as a random variable rather than a constant, unlike SDS, which thereby helps in optimizing the generation of 3D scenes. SDS is a specific instance of VSD where the variational distribution is a single-point Dirac distribution, which explains the limited variety and accuracy of the 3D scenes produced by SDS. The researchers have mentioned how VSD can learn a parametric scoring model with just one particle, which may have better generalization than SDS.

The team has also proposed ProlificDreamer, a holistic solution that includes VSD and design space enhancements made for text-to-3D generation. Improvements have been made to the distillation time schedule and density initialization which are the two unexplored areas but are orthogonal to the distillation algorithm.

With these improvements contributing towards enhancement of the overall performance of the text-to-3D generation process and the capabilities of VSD, ProlificDreamer produces Neural Radiance Fields (NeRF) with high fidelity and high rendering resolution, notably 512×512, rich structure, and sophisticated effects like smoke and drops. It can even successfully construct complex scenes with multiple objects in 360-degree views based on textual prompts. The team has even optimized the created meshes using VSD after initializing using NeRF, producing meticulously detailed and photo-realistic 3D textured meshes.

Examples of generated textured meshes, such as a Michelangelo-style statue of a dog reading news on a cell phone, a delicious croissant, an elephant skull, etc., have been shared in the released research paper. Apart from that, examples of generated NeRFs have also been shared, like a DSLR photo of a hamburger inside a restaurant and of an ice-cream sundae inside a shopping mall.

Check out the Paper and Project Link. Don’t forget to join our 22k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club