[Paper Review] DDP: Diffusion Model for Dense Visual Prediction

Yuanfeng Ji, Zhe Chen+7 more|arXiv (Cornell University)|Mar 30, 2023

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

DDP proposes a simple, efficient diffusion framework for dense visual prediction tasks such as semantic segmentation, depth estimation, and BEV map segmentation. By applying a conditional denoising diffusion process that iteratively refines predictions from noise using a lightweight decoder guided by image features, DDP achieves state-of-the-art or competitive performance across six benchmarks, including 83.9 mIoU on Cityscapes and 0.05 REL on KITTI, while enabling dynamic inference and uncertainty awareness.

ABSTRACT

We propose a simple, efficient, yet powerful framework for dense visual predictions based on the conditional diffusion pipeline. Our approach follows a "noise-to-map" generative paradigm for prediction by progressively removing noise from a random Gaussian distribution, guided by the image. The method, called DDP, efficiently extends the denoising diffusion process into the modern perception pipeline. Without task-specific design and architecture customization, DDP is easy to generalize to most dense prediction tasks, e.g., semantic segmentation and depth estimation. In addition, DDP shows attractive properties such as dynamic inference and uncertainty awareness, in contrast to previous single-step discriminative methods. We show top results on three representative tasks with six diverse benchmarks, without tricks, DDP achieves state-of-the-art or competitive performance on each task compared to the specialist counterparts. For example, semantic segmentation (83.9 mIoU on Cityscapes), BEV map segmentation (70.6 mIoU on nuScenes), and depth estimation (0.05 REL on KITTI). We hope that our approach will serve as a solid baseline and facilitate future research

Motivation & Objective

• To develop a general-purpose framework for dense visual prediction that avoids task-specific architectural design.
• To extend denoising diffusion models into modern perception pipelines with high efficiency and strong generalization.
• To enable dynamic inference and uncertainty awareness—properties absent in standard single-step discriminative models.
• To achieve state-of-the-art performance on diverse benchmarks without architectural or training trickery.

Proposed method

• DDP employs a conditional diffusion pipeline where noise is progressively removed from a Gaussian-distributed latent map under guidance from image features.
• During training, ground-truth maps are encoded and corrupted with noise via a noise schedule, then fused with image features from a backbone like Swin Transformer.
• A lightweight map decoder learns to denoise the noisy map using the image-conditioned features, producing clean predictions.
• At inference, the model generates predictions by iteratively refining a random noise sample through the reverse diffusion process, guided by the input image.
• The framework decouples the image encoder (run once) from the diffusion decoder (run iteratively), enabling efficient multi-step inference.
• Label encoding uses class embedding with a learnable scaling factor (optimal at 0.01), and the noise schedule is optimized as cosine for better denoising capacity.

Experimental results

Research questions

• RQ1 Can a simple, unified diffusion framework outperform task-specific discriminative models in dense visual prediction without architectural customization?
• RQ2 How does the choice of noise schedule and label encoding strategy affect performance in dense prediction tasks?
• RQ3 Can diffusion models naturally support dynamic inference and uncertainty estimation, unlike standard single-step models?
• RQ4 What is the trade-off between inference speed and accuracy in a diffusion-based dense prediction framework?

Key findings

• DDP achieves 83.9 mIoU on Cityscapes semantic segmentation using a ConvNeXt-L backbone, surpassing specialist models.
• On nuScenes BEV map segmentation, DDP reaches 70.6 mIoU, significantly outperforming the BEVFusion baseline (62.7 mIoU).
• For depth estimation, DDP achieves a leading 0.05 REL on KITTI with a Swin-L backbone, demonstrating strong performance in regression tasks.
• With only one sampling step, DDP achieves 46.1 mIoU on ADE20K, outperforming UperNet and K-Net, and improves to 47.0 mIoU with three steps.
• The model exhibits dynamic inference: increasing sampling steps improves accuracy with only moderate FLOPs and FPS cost, maintaining efficiency.
• Ablation studies show that cosine noise schedule and class embedding with scaling factor 0.01 yield optimal results, and a 6-block decoder balances performance and parameter efficiency.