Alibaba AI Research Proposes Composer: A Large (5 Billion Parameters) Controllable Diffusion Model Trained on Billions of (Text, Image) Pairs

Nowadays, text-based generative picture models are capable of creating a wide range of photorealistic images. Many recent efforts have expanded the text-to-image models to further accomplish customized generation by adding conditions such as segmentation maps, scene graphs, drawings, depth maps, and inpainting masks or finetuning the pretrained models on a small amount of subject-specific data. When it comes to applying these models for real-world applications, however, designers still need more control over them. For instance, it is typical in real-world design projects for generative models to need help to reliably produce pictures with simultaneous demands for semantics, form, style, and color.

Researchers from Alibaba China introduce Composer. It is a large (5 billion parameters) controllable diffusion model trained on billions of (text, image) pairs. They contend that compositionality—rather than just conditioning—is the secret to controlling image formation. The latter introduces many possible combinations, which can dramatically enlarge the control space. Similar ideas are investigated in the disciplines of language and scene comprehension. In these fields, compositionality is called compositional generalization, the ability to recognize or create a finite number of unique combinations from a limited number of available components. Based on the previously mentioned concept, they provide Composer in this work with an implementation of compositional generative models. They refer to generative models that can smoothly reassemble visual elements to create new pictures as compositional generative models. They use a multi-conditional diffusion model with a UNet backbone to implement Composer. Each Composer training iteration has two phases: the decomposition phase, where computer vision algorithms or pretrained models are used to break down batches of images into individual representations, and the composition phase, where Composer is optimized to reconstruct the images from the representation subsets.

Composers can decode unique pictures from unseen combinations of representations that may come from multiple sources and may be incompatible with one another while merely having been trained with a reconstruction purpose. Composer is surprisingly effective despite its conceptual simplicity and ease of use, enabling encouraging performance on both conventional and previously unexplored image generation and manipulation tasks, such as but not limited to text-to-image generation, multi-modal conditional image generation, style transfer, pose transfer, image translation, virtual try-on, interpolation and image variation from various directions, image reconfiguration by modifying sketches, dependant image translation, and image translation.

Additionally, Composer can limit the editable region to a user-specified area for all of the operations above, which is more flexible than the conventional inpainting operation, while preventing pixel modification outside of this region by introducing an orthogonal representation of masking. Despite having undergone multitask training, Composer obtains a zero-shot FID of 9.2 in text-to-image synthesis on the COCO dataset while utilizing the caption as the criterion, demonstrating its capacity to deliver excellent outcomes. Their decomposition-composition paradigm indicates that the control space of generative models may be greatly increased when conditions are composable rather than employed individuals. Consequently, a wide range of conventional generative tasks may be recast using their Composer architecture, and hitherto unrecognized generative capabilities are revealed, inspiring more study into various decomposition techniques that might attain higher controllability. Also, based on classifier-free and bidirectional guidance, they demonstrate many approaches to employing Composer for different picture production and alteration tasks, providing helpful references for subsequent studies. Before making the work publicly available, they plan to carefully examine how Composer can reduce the danger of abuse and maybe provide a filtered version.

Check out the Paper, Project, and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 15k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.