[Paper Review] SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions

Yuseung Lee, Kunho Kim+2 more|arXiv (Cornell University)|Jun 8, 2023

Advanced Vision and Imaging4 citations

TL;DR

SyncDiffusion proposes a plug-and-play module that enhances global coherence in panoramic image generation by synchronizing multiple diffusion processes through gradient descent using a perceptual similarity loss computed on denoised predictions. It achieves significantly higher coherence (66.35% in user study) than prior methods while maintaining image quality and prompt alignment, outperforming joint diffusion approaches in both seamlessness and semantic consistency.

ABSTRACT

The remarkable capabilities of pretrained image diffusion models have been utilized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted to address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus on seamless montage generation, often yield incoherent outputs by blending different scenes within a single image. To overcome this limitation, we propose SyncDiffusion, a plug-and-play module that synchronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, we compute the gradient of the perceptual loss using the predicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs compared to previous methods (66.35% vs. 33.65% in our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt (as measured by CLIP score). We further demonstrate the versatility of our method across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation. Our project page is at https://syncdiffusion.github.io.

Motivation & Objective

• To address the lack of global coherence in panoramic image generation using joint diffusion models, which often produce inconsistent scenes across stitched regions.
• To enable seamless yet semantically consistent montages by synchronizing multiple diffusion processes across overlapping regions.
• To provide a zero-shot, plug-and-play solution compatible with existing diffusion models and prompting frameworks.
• To maintain high image fidelity and prompt compatibility while enhancing structural and semantic consistency in generated panoramas.
• To demonstrate versatility across diverse applications such as layout-guided generation, conditional image synthesis, and 360-degree panorama creation.

Proposed method

• Synchronizes multiple diffusion processes by computing a perceptual similarity loss between denoised predictions from different image windows at each denoising step.
• Uses gradient descent to update noisy latent features based on the perceptual loss, guiding the reverse diffusion process toward coherent outputs.
• Computes the perceptual loss on predicted denoised images rather than noisy ones, enabling effective guidance through backpropagation.
• Integrates with existing joint diffusion frameworks that average latent features in overlapping regions to preserve local seamlessness.
• Leverages pre-trained diffusion models (e.g., Stable Diffusion 2.0) without retraining or fine-tuning, enabling zero-shot deployment.
• Supports various perceptual losses (e.g., LPIPS, Style Loss) and can be applied to diverse generation tasks via plug-and-play integration.

Experimental results

Research questions

• RQ1 Can perceptual similarity loss computed on denoised predictions effectively guide joint diffusion to improve global coherence in panoramic generation?
• RQ2 How does gradient-based synchronization of multiple diffusion processes compare to feature averaging alone in maintaining semantic consistency?
• RQ3 To what extent does SyncDiffusion preserve image quality and prompt alignment while enhancing coherence?
• RQ4 Can SyncDiffusion be effectively applied to diverse generation tasks such as layout-guided, conditional, and 360-degree image generation?
• RQ5 What is the computational overhead introduced by the synchronization module compared to baseline joint diffusion?

Key findings

• SyncDiffusion achieved 66.35% coherence in a user study, more than double the 33.65% of MultiDiffusion, demonstrating significant improvement in global consistency.
• The method maintained high image quality, with a GIQA score of 57.19%, indicating strong perceptual fidelity.
• Prompt compatibility was improved to 59.50% (CLIP score), showing better alignment with input text prompts compared to prior methods.
• The approach effectively enhanced coherence in layout-guided image generation, producing natural backgrounds instead of incoherent or mismatched scenes.
• In conditional image generation with ControlNet, SyncDiffusion enabled coherent multi-view outputs that preserved the input condition (e.g., Canny edge maps).
• For 360-degree panorama generation, SyncDiffusion produced views that appeared to originate from the same scene, unlike vanilla MVDiffusion, which generated disjoint or conflicting perspectives.