[Paper Review] PanoDiffusion: 360-degree Panorama Outpainting via Diffusion

Tianhao Wu, Chuanxia Zheng+1 more|arXiv (Cornell University)|Jul 6, 2023

Remote Sensing and LiDAR Applications4 citations

TL;DR

PanoDiffusion introduces a novel bi-modal latent diffusion model that leverages both RGB and depth panoramas during training to generate high-quality, diverse, and wraparound-consistent 360° indoor RGB-D panoramas from partially visible RGB inputs. It achieves state-of-the-art performance by incorporating progressive camera rotations during denoising and a two-end alignment mechanism, significantly improving structural coherence and robustness to depth input variations.

ABSTRACT

Generating complete 360-degree panoramas from narrow field of view images is ongoing research as omnidirectional RGB data is not readily available. Existing GAN-based approaches face some barriers to achieving higher quality output, and have poor generalization performance over different mask types. In this paper, we present our 360-degree indoor RGB-D panorama outpainting model using latent diffusion models (LDM), called PanoDiffusion. We introduce a new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data during training, which works surprisingly well to outpaint depth-free RGB images during inference. We further propose a novel technique of introducing progressive camera rotations during each diffusion denoising step, which leads to substantial improvement in achieving panorama wraparound consistency. Results show that our PanoDiffusion not only significantly outperforms state-of-the-art methods on RGB-D panorama outpainting by producing diverse well-structured results for different types of masks, but can also synthesize high-quality depth panoramas to provide realistic 3D indoor models.

Motivation & Objective

• To address the challenge of generating realistic, diverse, and structurally consistent 360° indoor panoramas from narrow field of view (NFoV) RGB images with missing regions.
• To overcome limitations of GAN-based methods, such as mode collapse and unstable training, by adopting a stable latent diffusion framework.
• To improve panorama outpainting quality by integrating depth information during training, even when depth is not available at inference.
• To ensure wraparound consistency in generated panoramas, where the left and right ends must align seamlessly due to the cylindrical nature of 360° images.
• To enable simultaneous high-fidelity synthesis of both RGB and depth maps, facilitating realistic 3D indoor scene reconstruction.

Proposed method

• A bi-modal latent diffusion model is proposed that jointly conditions on RGB and depth panoramic features during training, enabling the network to learn spatial layouts and object structures.
• Progressive camera rotation is introduced during each denoising step, where the model is trained to generate consistent content across rotated views, enhancing global coherence.
• A two-end alignment mechanism is applied at every denoising step to explicitly minimize the difference between the leftmost and rightmost regions of the output panorama.
• The model is trained on the Structured3D dataset using masked RGB inputs and corresponding depth maps, with no depth required during inference.
• During inference, only partially visible RGB images are used as input, while the model generates both complete RGB and depth panoramas in a single forward pass.
• The architecture leverages latent diffusion with cross-attention mechanisms to attend to both RGB and depth features, enabling joint modeling of appearance and geometry.

Experimental results

Research questions

• RQ1 Can a latent diffusion model effectively generate diverse and realistic 360° RGB-D panoramas from incomplete RGB inputs without requiring depth at inference?
• RQ2 How does incorporating depth supervision during training improve the quality and structural consistency of outpainted panoramas?
• RQ3 To what extent does progressive camera rotation during denoising enhance wraparound consistency in generated panoramas?
• RQ4 How robust is the model to varying degrees of depth input visibility during inference compared to existing methods?
• RQ5 Can the model simultaneously synthesize high-quality depth maps, indicating effective learning of 3D scene geometry from RGB-only inputs?

Key findings

• PanoDiffusion achieves state-of-the-art performance on the Structured3D dataset, outperforming SOTA methods like BIPS and OmniDreamer in both RGB and depth synthesis across all mask types.
• Even with only 50% sparse depth input during training, PanoDiffusion produces better RGB outpainting results than standard LDMs trained with full depth.
• The model is robust to depth input sparsity at inference: performance remains stable and superior to BIPS, which degrades significantly with reduced depth visibility.
• PanoDiffusion’s two-end alignment mechanism reduces the Left-Right Consistency Error (LRCE) to 85.04 on random box masks, significantly lower than BIPS (148.78) and OmniDreamer (136.68).
• The model generates accurate depth maps, achieving the best performance on most metrics in Table 2(c), demonstrating effective 3D layout understanding.
• Surprisingly, fully visible depth at test time slightly degrades performance, indicating that the model has learned to generalize across modality imbalances and is not over-reliant on depth.