[Paper Review] DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models

Gyeongnyeon Kim, Woo-Seok Jang+4 more|arXiv (Cornell University)|Dec 17, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes Depth-Aware Guidance (DAG), a novel method that enhances diffusion models by integrating depth-aware generation through pseudo-labeling and depth prior guidance using internal representations of pretrained diffusion models. By leveraging label-efficient depth estimation and two new guidance strategies, DAG significantly improves geometric plausibility in generated images, achieving state-of-the-art performance on depth-aware image generation with a 17.31 dFID score on LSUN-Church.

ABSTRACT

Generative models have recently undergone significant advancement due to the diffusion models. The success of these models can be often attributed to their use of guidance techniques, such as classifier or classifier-free guidance, which provide effective mechanisms to trade-off between fidelity and diversity. However, these methods are not capable of guiding a generated image to be aware of its geometric configuration, e.g., depth, which hinders their application to areas that require a certain level of depth awareness. To address this limitation, we propose a novel guidance method for diffusion models that uses estimated depth information derived from the rich intermediate representations of diffusion models. We first present label-efficient depth estimation framework using internal representations of diffusion models. Subsequently, we propose the incorporation of two guidance techniques based on pseudo-labeling and depth-domain diffusion prior during the sampling phase to self-condition the generated image using the estimated depth map. Experiments and comprehensive ablation studies demonstrate the effectiveness of our method in guiding the diffusion models towards the generation of geometrically plausible images.

Motivation & Objective

• To address the lack of geometric awareness in diffusion models, which often produce images with implausible depth and cluttered layouts.
• To develop a label-efficient depth estimation framework using internal representations of pretrained diffusion models.
• To propose two novel guidance strategies—depth consistency and depth prior guidance—that inject depth awareness during the sampling phase.
• To introduce a new evaluation metric based on depth estimation networks to better capture geometric feasibility in generated images.
• To demonstrate the utility of DAG-generated images as effective data for training monocular depth estimation models.

Proposed method

• Leverages rich intermediate features from a pretrained diffusion model to train a label-efficient depth predictor without requiring ground-truth depth annotations.
• Applies pseudo-labeling via consistency regularization: the model is guided to improve poor depth predictions by treating better predictions as pseudo-labels.
• Introduces depth prior guidance by using a separate pretrained diffusion U-Net as a prior network to inject depth information during sampling.
• Uses a U-Net-based backbone for depth estimation, fine-tuned on synthetic data generated via DAG to improve monocular depth prediction performance.
• Employs a multi-scale depth map interpolation strategy to align generated depth maps with the resolution of the prior network during depth prior guidance.
• Proposes a new evaluation metric, dFID, computed using a pretrained depth estimation network to measure depth feasibility of generated images.

Experimental results

Research questions

• RQ1 Can internal representations of pretrained diffusion models be effectively leveraged for label-efficient depth estimation?
• RQ2 Can consistency regularization and depth prior guidance jointly improve geometric plausibility in diffusion-based image generation?
• RQ3 Does the proposed DAG framework outperform baseline diffusion models in generating depth-aware images, especially in complex 3D scenes?
• RQ4 Can DAG-generated images serve as high-quality synthetic data to improve monocular depth estimation models?
• RQ5 How does the choice of guidance scale and prior network resolution affect the quality of depth-aware image generation?

Key findings

• The proposed DAG framework achieves a dFID score of 17.31 on the LSUN-Church dataset, outperforming the baseline (17.69) and ablation variants.
• The depth prior guidance (DPG) with a 128×128 prior network achieves the best dFID (25.07), outperforming smaller resolutions.
• The guidance scale of 40 yields optimal performance for both depth consistency guidance (DCG) and depth prior guidance (DPG).
• Training a depth estimator on DAG-generated images improves its performance, achieving 77.54% accuracy under δ<1.25, compared to 72.55% on unguided samples.
• Qualitative results show that DAG-generated images have clearer scene layouts, more consistent depth maps, and better surface normal predictions than baseline models.
• The ablation study confirms that both DCG and DPG contribute independently to improved geometric plausibility, with their combination yielding the best results.