KAIST Researchers Propose SyncDiffusion: A Plug-and-Play Module that Synchronizes Multiple Diffusions through Gradient Descent from a Perceptual Similarity Loss

In a recent research paper, a team of researchers from KAIST introduced SYNCDIFFUSION, a groundbreaking module that aims to enhance the generation of panoramic images using pretrained diffusion models. The researchers identified a significant problem in panoramic image creation, primarily involving the presence of visible seams when stitching together multiple fixed-size images. To address this issue, they proposed SYNCDIFFUSION as a solution.

Creating panoramic images, those with wide, immersive views, poses challenges for image generation models, as they are typically trained to produce fixed-size images. When attempting to generate panoramas, the naive approach of stitching multiple images together often results in visible seams and incoherent compositions. This issue has driven the need for innovative methods to seamlessly blend images and maintain overall coherence.

Two prevalent methods for generating panoramic images are sequential image extrapolation and joint diffusion. The former involves generating a final panorama by extending a given image sequentially, fixing the overlapped region in each step. However, this method often struggles to produce realistic panoramas and tends to introduce repetitive patterns, leading to less-than-ideal results.

On the other hand, joint diffusion operates the reverse generative process simultaneously across multiple views and averages intermediate noisy images in overlapping regions. While this approach effectively generates seamless montages, it falls short in terms of maintaining content and style consistency across the views. As a result, it frequently combines images with different content and styles within a single panorama, resulting in incoherent outputs.

The researchers introduced SYNCDIFFUSION as a module that synchronizes multiple diffusions by employing gradient descent based on a perceptual similarity loss. The critical innovation lies in the use of the predicted denoised images at each denoising step to calculate the gradient of the perceptual loss. This approach offers meaningful guidance for creating coherent montages, as it ensures that the images blend seamlessly while maintaining content consistency.

In a series of experiments using SYNCDIFFUSION with the Stable Diffusion 2.0 model, the researchers found that their method significantly outperformed previous techniques. The user study conducted showed a substantial preference for SYNCDIFFUSION, with a 66.35% preference rate, as opposed to the previous method’s 33.65%. This marked improvement demonstrates the practical benefits of SYNCDIFFUSION in generating coherent panoramic images.

SYNCDIFFUSION is a notable addition to the field of image generation. It effectively tackles the challenge of generating seamless and coherent panoramic images, which has been a persistent issue in the field. By synchronizing multiple diffusions and applying gradient descent from perceptual similarity loss, SYNCDIFFUSION enhances the quality and coherence of generated panoramas. As a result, it offers a valuable tool for a wide range of applications that involve creating panoramic images, and it showcases the potential of using gradient descent in improving image generation processes.

Check out the Paper and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 31k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on WhatsApp. Join our AI Channel on Whatsapp..