[Paper Review] Beyond the Pixel-Wise Loss for Topology-Aware Delineation

Agata Mosinska, Pablo Márquez-Neila+2 more|arXiv (Cornell University)|Dec 6, 2017

Cell Image Analysis Techniques23 references17 citations

TL;DR

This paper proposes a topology-aware loss and iterative refinement pipeline to improve deep learning-based delineation of curvilinear structures, such as roads and neuronal membranes. By leveraging high-level features from a pretrained VGG19 to measure topological consistency between predictions and ground truth, and applying parameter-shared iterative refinement, the method boosts segmentation accuracy by up to 30 percentage points over standard pixel-wise training.

ABSTRACT

Delineation of curvilinear structures is an important problem in Computer Vision with multiple practical applications. With the advent of Deep Learning, many current approaches on automatic delineation have focused on finding more powerful deep architectures, but have continued using the habitual pixel-wise losses such as binary cross-entropy. In this paper we claim that pixel-wise losses alone are unsuitable for this problem because of their inability to reflect the topological impact of mistakes in the final prediction. We propose a new loss term that is aware of the higher-order topological features of linear structures. We also introduce a refinement pipeline that iteratively applies the same model over the previous delineation to refine the predictions at each step while keeping the number of parameters and the complexity of the model constant. When combined with the standard pixel-wise loss, both our new loss term and our iterative refinement boost the quality of the predicted delineations, in some cases almost doubling the accuracy as compared to the same classifier trained with the binary cross-entropy alone. We show that our approach outperforms state-of-the-art methods on a wide range of data, from microscopy to aerial images.

Motivation & Objective

• Address the limitation of pixel-wise losses in deep learning for curvilinear structure delineation, which fail to account for topological errors like gaps or splits.
• Improve segmentation quality by incorporating higher-order topological constraints into the training objective.
• Develop a parameter-efficient iterative refinement pipeline that enhances predictions without increasing model complexity.
• Demonstrate state-of-the-art performance on diverse datasets including microscopy, aerial, and road images.
• Show that topological awareness leads to more structured, accurate, and robust predictions, especially in noisy or complex imaging conditions.

Proposed method

• Propose a topology-aware loss term that compares high-level feature maps from a pretrained VGG19 network to assess topological similarity between predicted and ground-truth linear structures.
• Use feature maps from the first three convolutional blocks of VGG19 to capture structural sensitivity to linear patterns, enabling detection of topological inconsistencies.
• Design an iterative refinement pipeline that applies the same trained model repeatedly to its own predictions, progressively closing gaps and refining topology.
• Share weights across all refinement stages to maintain constant model complexity and parameter count, even with multiple iterations.
• Combine the new topology loss with standard binary cross-entropy loss during training to jointly optimize for pixel accuracy and topological correctness.
• Apply the method end-to-end on U-Net and VGG-based architectures, demonstrating generalization across different network designs and data modalities.

Experimental results

Research questions

• RQ1 Can a loss function that explicitly encodes topological properties of curvilinear structures improve segmentation performance beyond pixel-wise supervision?
• RQ2 How does iterative refinement using a shared model architecture affect the quality and topological fidelity of predicted linear structures?
• RQ3 To what extent does the proposed method outperform state-of-the-art approaches on diverse datasets such as cracks, roads, and electron microscopy images?
• RQ4 Does the integration of topological supervision lead to measurable improvements in metrics sensitive to structure continuity and connectivity?
• RQ5 Can the method be applied effectively without architectural changes or additional parameters, especially in data-scarce biomedical settings?

Key findings

• The proposed topology-aware loss alone improves segmentation accuracy by up to 30 percentage points compared to standard binary cross-entropy training.
• When combined with iterative refinement, the method achieves a foreground-restricted random score of 0.981 on the ISBI EM challenge, ranking first among single-classifier methods.
• On the EM dataset, the method achieved 67.0% correct path detection, outperforming U-Net (51.5%) and Reg-AC (36.1%) in terms of topological correctness.
• Iterative refinement reduced the percentage of infeasible paths from 32.3% (OURS-NoRef) to 24.2% on the roads dataset, demonstrating improved structural consistency.
• The method significantly reduced false positives in background regions—such as mitochondria in neuronal membranes—while preserving and enhancing target structures.
• Qualitative results show that the method effectively closes small gaps in linear structures, preventing large-scale topological errors that pixel-wise losses fail to penalize.