Meet Paella: A New AI Model Similar To Diffusion That Can Generate High-Quality Images Much Faster Than By Using Stable Diffusion

Over the past 2-3 years, there has been a phenomenal increase in the quality and quantity of research done in generating images from text using artificial intelligence (AI). One of the most groundbreaking and revolutionary works in this domain refers to state-of-the-art generative models called diffusion models. These models have completely transformed how textual descriptions can be used to generate high-quality images by harnessing the power of deep learning algorithms. Moreover, In addition to diffusion, a range of other powerful techniques exists, providing an exciting pathway to generate near-photorealistic visual content from textual inputs. However, the exceptional results achieved by these cutting-edge technologies come with certain limitations. A number of emerging generative AI technologies rely on diffusion models, which demand intricate architectures and substantial computational resources for training and image generation. These advanced methodologies also reduce inference speed, rendering them impractical for real-time implementation. Furthermore, the complexity of these techniques is directly linked to the advancements they enable, posing a challenge for the general public to grasp the inner workings of these models and resulting in a situation where they are perceived as black-box models.

Intending to address the concerns mentioned earlier, a team of researchers at Technische Hochschule Ingolstadt and Wand Technologies, Germany, have proposed a novel technique for text-conditional image generation. This innovative technique is similar to diffusion but produces high-quality images much faster. The image sampling phase of this convolution-based model can be accomplished with as few as 12 steps while still yielding exceptional image quality. This approach stands out for its remarkable simplicity and reduced image generation speed, thus, allowing users to condition the model and enjoy the advantages lacking in existing state-of-the-art techniques. The proposed technique’s inherent simplicity has significantly enhanced its accessibility, enabling individuals from diverse backgrounds to grasp and implement this text-to-image technology readily. To validate their methodology through experimental evaluations, the researchers additionally trained a text-conditional model named “Paella” with a staggering one billion parameters. The team has also open-sourced their code and model weights under the MIT license to encourage research around their work.

A diffusion model undergoes a learning process where it progressively eliminates varying levels of noise from each training instance. During inference, when presented with pure noise, the model generates an image by iteratively subtracting noise over several hundred steps. The technique devised by the German researchers draws heavily from these principles of diffusion models. Like diffusion models, Paella removes varying degrees of noise from tokens representing an image and employs them to generate a new image. The model was trained on 900 million image-text pairs from LAION-5B aesthetic dataset. Paella utilizes a pre-trained encoder-decoder architecture based on a convolutional neural network, with the capacity to represent a 256×256 image using 256 tokens selected from a set of 8,192 tokens learned during pretraining. In order to add noise to their example during the training phase, the researchers included some randomly chosen tokens in this list as well.

To generate text embeddings based on the image’s textual description, the researchers utilized the CLIP (Contrastive Language-Image Pretraining) model, which establishes connections between images and textual descriptions. The U-Net CNN architecture was then employed to train the model in generating the complete set of original tokens, utilizing the text embeddings and tokens generated in previous iterations. This iterative process was repeated 12 times, gradually replacing a smaller portion of the previously generated tokens with each repetition. With the guidance of the remaining generated tokens, the U-Net progressively reduced the noise at each step. During inference, CLIP produced an embedding based on a given text prompt, and the U-Net reconstructed all the tokens over 12 steps for a randomly selected set of 256 tokens. Finally, the decoder employed the generated tokens to generate an image.

In order to assess the effectiveness of their method, the researchers employed the Fréchet inception distance (FID) metric to compare the outcomes obtained from the Paella model and the Stable Diffusion model. Although the results slightly favored Stable Diffusion, Paella exhibited a significant advantage in terms of speed. This study stands out from previous endeavors, as it focused on completely reconfiguring the architecture, which was not considered previously. In conclusion, Paella can generate high-quality images with a smaller model size and fewer sampling steps as compared to existing models and still achieve appreciable outcomes. The research team emphasizes the accessibility of their approach, which offers a simple setup that can be readily adopted by individuals from diverse backgrounds, including non-technical domains, as the field of generative AI continues to garner more interest with time.

Check Out The Paper and Reference Article. Don’t forget to join our 24k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we missed anything, feel free to email us at Asif@marktechpost.com

Featured Tools From AI Tools Club

🚀 Check Out 100’s AI Tools in AI Tools Club