[Paper Review] Holistically-Nested Edge Detection

Saining Xie, Zhuowen Tu|arXiv (Cornell University)|Apr 24, 2015

Advanced Neural Network Applications36 references308 citations

TL;DR

Holistically-Nested Edge Detection (HED) is a deep learning-based edge detection framework that uses a fully convolutional neural network with deeply-supervised side outputs to learn hierarchical, multi-scale features for accurate image-to-image edge prediction. It achieves state-of-the-art performance on BSD500 (ODS F-score: 0.790) and NYU Depth (0.746) with inference speeds of 0.4 seconds per image on GPU, significantly outperforming prior CNN-based methods in both accuracy and efficiency.

ABSTRACT

We develop a new edge detection algorithm that tackles two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations (guided by deep supervision on side responses) that are important in order to approach the human ability resolve the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSD500 dataset (ODS F-score of .782) and the NYU Depth dataset (ODS F-score of .746), and do so with an improved speed (0.4 second per image) that is orders of magnitude faster than some recent CNN-based edge detection algorithms.

Motivation & Objective

• To address the long-standing challenge of edge and object boundary detection in natural images by learning rich, hierarchical visual representations.
• To improve performance and speed over existing CNN-based edge detection methods, which often suffer from high inference time or suboptimal accuracy.
• To enable holistic, image-to-image prediction using fully convolutional networks, avoiding patch-based or multi-stage processing.
• To leverage multi-scale and multi-level feature learning through deeply-supervised side outputs, enhancing edge localization and consistency.
• To demonstrate the effectiveness of HHA features (depth-encoded) in improving edge detection when combined with RGB input.

Proposed method

• HED employs a fully convolutional neural network (FCN) architecture inspired by the VGG-16 network, fine-tuned for end-to-end edge detection.
• The network integrates deeply-supervised side outputs at multiple convolutional layers (conv2, conv3, conv4, conv5) to guide feature learning and improve gradient flow.
• Each side output produces a coarse edge map, and the final prediction is obtained by fusing these responses using a learned fusion layer.
• The model is trained end-to-end with a multi-task loss combining supervision from all side outputs and the final fusion layer.
• For depth-enhanced detection, HED uses HHA features (horizontal disparity, height above ground, surface normal angle) to encode depth information, which are processed in parallel with RGB inputs.
• During inference, the average of the RGB and HHA predictions is used to produce the final edge map, achieving improved robustness and accuracy.

Experimental results

Research questions

• RQ1 Can a fully convolutional, deeply-supervised network architecture achieve state-of-the-art edge detection performance while maintaining high inference speed?
• RQ2 How does multi-scale and multi-level feature learning through side supervision improve edge localization and consistency compared to single-scale or non-supervised approaches?
• RQ3 To what extent can hand-crafted depth features (HHA) improve edge detection when fused with deep RGB features in a CNN-based framework?
• RQ4 Does end-to-end training with holistic image supervision lead to better generalization and edge consistency than patch-based or multi-stage methods?
• RQ5 How does the integration of multi-scale responses through a fusion layer compare to late fusion or early fusion strategies in terms of performance and robustness?

Key findings

• HED achieves an ODS F-score of 0.790 on the BSDS500 dataset, representing a significant improvement over prior state-of-the-art methods.
• On the NYU Depth dataset, HED achieves an ODS F-score of 0.746, demonstrating strong generalization to depth-aware edge detection.
• The model runs at 2.5 FPS on GPU (1 FPS with HHA fusion), and 12 seconds per image on CPU, making it practical for real-time applications.
• The use of HHA features improves performance over RGB-only detection, with the RGB-HHA fusion model achieving 0.746 ODS F-score on NYU Depth.
• Deep supervision significantly enhances performance, as shown by the drop in ODS F-score from 0.790 (with supervision) to 0.785 (without) on BSDS500.
• Late-merging of RGB and HHA predictions yields the highest average precision (AP = 0.840), outperforming early fusion or single-modality models.