[Paper Review] Label-efficient Segmentation via Affinity Propagation

Wentong Li, Yuqian Yuan+6 more|arXiv (Cornell University)|Oct 16, 2023

Advanced Computing and AlgorithmsSocial Sciences3 citations

TL;DR

This paper proposes APro, a label-efficient segmentation method that formulates affinity modeling as an affinity propagation process, using both global and local pairwise affinity terms to generate accurate soft pseudo-labels. By leveraging topology-aware graph propagation and an efficient O(N log N) implementation, APro achieves state-of-the-art performance across box-supervised instance segmentation, point/scribble-supervised semantic segmentation, and CLIP-guided semantic segmentation with significantly reduced computational cost.

ABSTRACT

Weakly-supervised segmentation with label-efficient sparse annotations has attracted increasing research attention to reduce the cost of laborious pixel-wise labeling process, while the pairwise affinity modeling techniques play an essential role in this task. Most of the existing approaches focus on using the local appearance kernel to model the neighboring pairwise potentials. However, such a local operation fails to capture the long-range dependencies and ignores the topology of objects. In this work, we formulate the affinity modeling as an affinity propagation process, and propose a local and a global pairwise affinity terms to generate accurate soft pseudo labels. An efficient algorithm is also developed to reduce significantly the computational cost. The proposed approach can be conveniently plugged into existing segmentation networks. Experiments on three typical label-efficient segmentation tasks, i.e. box-supervised instance segmentation, point/scribble-supervised semantic segmentation and CLIP-guided semantic segmentation, demonstrate the superior performance of the proposed approach.

Motivation & Objective

• To address the limitations of local appearance kernels in capturing long-range dependencies and object topology in weakly-supervised segmentation.
• To reduce computational cost in global affinity modeling while preserving fine-grained semantic relationships.
• To develop a plug-in component that enhances existing segmentation networks with minimal architectural changes.
• To improve pseudo-label quality for label-efficient segmentation tasks using both global and local affinity propagation.

Proposed method

• Formulates weakly-supervised segmentation as a unified affinity propagation process to iteratively refine soft pseudo-labels.
• Introduces a global pairwise affinity term using a topology-aware tree-based graph to model long-range dependencies beyond local neighborhoods.
• Develops a local pairwise affinity term to enforce spatial consistency and piecewise smoothness in predictions.
• Employs an efficient O(N log N) algorithm to replace the O(N²) brute-force computation, enabling practical deployment.
• Combines global and local affinity propagation in parallel or sequential cascades to generate complementary pseudo-labels.
• Integrates the APro module into existing segmentation networks for end-to-end training with sparse annotations.

Experimental results

Research questions

• RQ1 Can affinity propagation be effectively used to model both local and global pairwise relationships in weakly-supervised segmentation?
• RQ2 How does incorporating topology-aware graph structures improve long-range affinity modeling compared to local appearance kernels?
• RQ3 Can an efficient O(N log N) implementation maintain high accuracy while drastically reducing computational cost in global affinity propagation?
• RQ4 How do global and local affinity terms complement each other in generating high-quality soft pseudo-labels?
• RQ5 To what extent can APro be generalized across diverse label-efficient segmentation tasks such as box-supervised, point/scribble-supervised, and CLIP-guided segmentation?

Key findings

• The proposed APro method achieves 38.4% mask AP on the Pascal VOC dataset in box-supervised instance segmentation, outperforming prior methods.
• Using both global and local pairwise terms in parallel (GP-LP-P) yields the best performance, demonstrating their complementary nature.
• The efficient implementation reduces average runtime from 4,300 ms (O(N²)) to 0.8 ms (O(N log N)), enabling real-time inference.
• APro converges faster than MeanField-based methods, achieving 36.0% AP in 20 iterations compared to 100 iterations for comparable performance.
• The ablation study confirms that both global and local affinity terms are essential, with the global term contributing most to performance gain.
• The method generalizes well across three distinct label-efficient tasks, including CLIP-guided annotation-free semantic segmentation, showing strong transferability.