[Paper Review] DiffusionEdge: Diffusion Probabilistic Model for Crisp Edge Detection

Yunfan Ye, Kai Xu+3 more|arXiv (Cornell University)|Jan 4, 2024

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes DiffusionEdge, the first diffusion probabilistic model for general edge detection, which generates crisp and accurate edge maps directly without post-processing. By training in the latent space with uncertainty-aware distillation and adaptive Fourier filtering, it achieves state-of-the-art performance on multiple benchmarks, improving ODS, OIS, and AC by up to 30.2%, 28.1%, and 65.1% respectively on NYUDv2.

ABSTRACT

Limited by the encoder-decoder architecture, learning-based edge detectors usually have difficulty predicting edge maps that satisfy both correctness and crispness. With the recent success of the diffusion probabilistic model (DPM), we found it is especially suitable for accurate and crisp edge detection since the denoising process is directly applied to the original image size. Therefore, we propose the first diffusion model for the task of general edge detection, which we call DiffusionEdge. To avoid expensive computational resources while retaining the final performance, we apply DPM in the latent space and enable the classic cross-entropy loss which is uncertainty-aware in pixel level to directly optimize the parameters in latent space in a distillation manner. We also adopt a decoupled architecture to speed up the denoising process and propose a corresponding adaptive Fourier filter to adjust the latent features of specific frequencies. With all the technical designs, DiffusionEdge can be stably trained with limited resources, predicting crisp and accurate edge maps with much fewer augmentation strategies. Extensive experiments on four edge detection benchmarks demonstrate the superiority of DiffusionEdge both in correctness and crispness. On the NYUDv2 dataset, compared to the second best, we increase the ODS, OIS (without post-processing) and AC by 30.2%, 28.1% and 65.1%, respectively. Code: https://github.com/GuHuangAI/DiffusionEdge.

Motivation & Objective

• To address the long-standing challenge of simultaneously achieving high correctness and crispness in edge detection without relying on post-processing.
• To explore whether diffusion probabilistic models (DPMs) can overcome the architectural limitations of encoder-decoder networks in edge detection.
• To enable effective training of a DPM in latent space while preserving uncertainty-aware supervision from pixel-level cross-entropy loss.
• To improve inference efficiency and feature adaptability through a decoupled architecture and adaptive Fourier filtering.
• To demonstrate that diffusion models can directly generate single-width, high-quality edge maps suitable for end-to-end downstream tasks.

Proposed method

• Adopts a latent-space diffusion probabilistic model (DPM) to reduce computational cost while maintaining high-resolution edge prediction.
• Employs uncertainty distillation to transfer gradients from pixel-level cross-entropy loss to the latent space, preserving annotator-robust uncertainty priors.
• Introduces a decoupled architecture to accelerate the denoising process by separating feature learning and refinement stages.
• Applies an adaptive Fourier filter to selectively adjust frequency components in the latent space, enhancing edge localization and sharpness.
• Uses a distillation-based training strategy to optimize latent parameters using standard pixel-level supervision, avoiding complex retraining.
• Leverages the iterative denoising process of DPMs to learn complex edge distributions directly at original image resolution, avoiding upsampling artifacts.

Experimental results

Research questions

• RQ1 Can a diffusion probabilistic model be effectively adapted for general edge detection to achieve both correctness and crispness?
• RQ2 How can uncertainty-aware supervision from pixel-level cross-entropy loss be preserved when training a diffusion model in latent space?
• RQ3 Can adaptive frequency filtering in the Fourier domain improve edge localization and sharpness in diffusion-based edge detection?
• RQ4 Does a decoupled architecture enable faster inference while maintaining high performance in diffusion-based edge detection?
• RQ5 Can diffusion models generate single-width, post-processing-free edge maps that outperform state-of-the-art CNN-based methods in both correctness and crispness?

Key findings

• On the NYUDv2 dataset, DiffusionEdge improves ODS, OIS (without post-processing), and AC by 30.2%, 28.1%, and 65.1% respectively compared to the second-best method.
• On the BIPED dataset, DiffusionEdge achieves an AC of 0.849, indicating near-perfect single-width edge predictions with minimal ambiguity.
• On Multicue, DiffusionEdge achieves an AC of 0.462, outperforming PiDiNet-based methods by up to 0.138 in crispness, even when using only random flipping for data augmentation.
• The method achieves superior qualitative results, with edges that are both highly accurate and sharply localized, closely matching ground truth boundaries.
• DiffusionEdge outperforms state-of-the-art CNN-based models like UAED and EDTER in both correctness and crispness, especially on challenging benchmarks with complex textures.
• The ablation study confirms that the adaptive Fourier filter and uncertainty distillation are critical for achieving high performance, with significant gains in AC and ODS metrics.