[Paper Review] Simultaneous Edge Alignment and Learning

Zhiding Yu, Weiyang Liu+5 more|arXiv (Cornell University)|Aug 6, 2018

Medical Image Segmentation Techniques3 references4 citations

TL;DR

This paper proposes SEAL, an end-to-end deep learning framework that jointly optimizes edge alignment and detection by treating edge labels as latent variables in a probabilistic model. By formulating label refinement as a min-cost bipartite assignment problem during training, SEAL corrects misaligned annotations and produces sharper, more accurate edges, achieving state-of-the-art performance on SBD and Cityscapes benchmarks while improving noisy label quality.

ABSTRACT

Edge detection is among the most fundamental vision problems for its role in perceptual grouping and its wide applications. Recent advances in representation learning have led to considerable improvements in this area. Many state of the art edge detection models are learned with fully convolutional networks (FCNs). However, FCN-based edge learning tends to be vulnerable to misaligned labels due to the delicate structure of edges. While such problem was considered in evaluation benchmarks, similar issue has not been explicitly addressed in general edge learning. In this paper, we show that label misalignment can cause considerably degraded edge learning quality, and address this issue by proposing a simultaneous edge alignment and learning framework. To this end, we formulate a probabilistic model where edge alignment is treated as latent variable optimization, and is learned end-to-end during network training. Experiments show several applications of this work, including improved edge detection with state of the art performance, and automatic refinement of noisy annotations.

Motivation & Objective

• To address the critical challenge of label misalignment in edge detection, which degrades model performance despite advances in deep learning.
• To develop a unified framework that jointly learns edge detection and refines noisy, misaligned ground-truth annotations during training.
• To improve edge detection quality by modeling edge alignment as a latent variable optimization problem within a differentiable, end-to-end training process.
• To demonstrate that correcting label misalignment leads to sharper, more precise edge predictions and better generalization.
• To show that the proposed method outperforms existing state-of-the-art models on standard benchmarks like SBD and Cityscapes.

Proposed method

• Formulates edge alignment as a latent variable optimization problem within a probabilistic graphical model, treating ground-truth edges as unobserved variables to be inferred.
• Models the alignment process as a min-cost bipartite assignment problem between predicted and ground-truth edge pixels, enabling differentiable optimization.
• Integrates the alignment module into a fully convolutional network (FCN) backbone, enabling end-to-end training with backpropagation.
• Uses a differentiable relaxation of the discrete assignment problem to allow gradient flow during backpropagation, enabling joint optimization of detection and alignment.
• Applies reweighted cross-entropy loss with learned edge confidence to balance positive and negative samples, reducing false positives.
• Employs a two-stage training strategy: first, refine labels via the alignment module; second, fine-tune the full network with refined labels.

Experimental results

Research questions

• RQ1 Can joint learning of edge detection and label alignment improve edge detection performance beyond standard FCN-based models?
• RQ2 To what extent does correcting label misalignment enhance edge sharpness and reduce false positives in edge detection?
• RQ3 Can the proposed method automatically refine noisy or misaligned annotations without requiring re-annotation?
• RQ4 How does the performance of SEAL compare to state-of-the-art models under both standard and refined evaluation protocols?
• RQ5 Does the framework generalize across diverse datasets such as SBD and Cityscapes with varying annotation quality?

Key findings

• SEAL achieves a mean F-measure of 74.4 on the SBD test set under the standard benchmark, outperforming the previous state of the art (71.4) by a significant margin.
• On the re-annotated SBD test set with higher-quality labels, SEAL achieves a mean F-measure of 66.9 in the 'Thin' setting, outperforming CASENet-S (66.4) and demonstrating robustness to label noise.
• SEAL improves label quality, as shown by a 12.5% relative increase in F-measure over original annotations when evaluated against re-annotated ground truth, outperforming dense-CRF-based refinement.
• On the Cityscapes validation set, SEAL achieves superior performance across all categories, with a mean F-measure of 75.9, significantly outperforming CASENet and CASENet-S.
• Visualization shows that SEAL progressively tightens edge predictions around object boundaries, reducing thickness and false positives compared to baseline models.
• The method successfully reduces false positives near edges, even under reweighted loss settings, by correcting misaligned labels during training.