[Paper Review] Bootstrapping Semantic Segmentation with Regional Contrast

Shikun Liu, Shuaifeng Zhi+2 more|arXiv (Cornell University)|Apr 9, 2021

Domain Adaptation and Few-Shot LearningComputer Science32 references69 citations

TL;DR

ReCo introduces a pixel-level contrastive loss with active sampling to improve semantic segmentation in both supervised and semi-supervised settings, achieving sharper boundaries and strong label efficiency, especially with very few labels.

ABSTRACT

We present ReCo, a contrastive learning framework designed at a regional level to assist learning in semantic segmentation. ReCo performs semi-supervised or supervised pixel-level contrastive learning on a sparse set of hard negative pixels, with minimal additional memory footprint. ReCo is easy to implement, being built on top of off-the-shelf segmentation networks, and consistently improves performance in both semi-supervised and supervised semantic segmentation methods, achieving smoother segmentation boundaries and faster convergence. The strongest effect is in semi-supervised learning with very few labels. With ReCo, we achieve high-quality semantic segmentation models, requiring only 5 examples of each semantic class. Code is available at https://github.com/lorenmt/reco.

Motivation & Objective

• Motivate reducing labeling cost in semantic segmentation by leveraging regional contrast and hard pixels.
• Introduce ReCo, a pixel-level contrastive loss with adaptive sampling to learn from class relationships.
• Demonstrate that ReCo improves performance across semi-supervised and supervised settings on standard datasets.
• Show that ReCo accelerates convergence and yields sharper object boundaries.

Proposed method

• Attach a pixel-level representation head to the encoder for training (not used at inference).
• Define a ReCo loss that brings query pixel representations closer to their class mean (positive) and pushes them away from other classes (negatives) using a normalised dot-product similarity.
• Compute positives as the mean representation of each class within the batch; negatives are sampled from other classes using a learnable class-relationship graph G.
• Perform active sampling of queries by selecting pixels with low prediction confidence (hard queries).
• Perform active sampling of keys by constructing a per-batch pairwise class relationship graph G and sampling negatives from a softmax over class relationships.
• Train with a combined loss: L_total = L_supervised + L_reco (and L_unsupervised in semi-supervised settings via Mean Teacher).

Experimental results

Research questions

• RQ1 Can a region-focused contrastive loss improve pixel-level segmentation under limited labels?
• RQ2 How does adaptive sampling of hard queries and hard negatives affect segmentation performance and training efficiency?
• RQ3 Does ReCo consistently improve semi-supervised segmentation across different backbones and datasets?
• RQ4 What is the impact of ReCo on boundary sharpness and class confusion in segmentation outputs.

Key findings

• ReCo consistently improves performance for both semi-supervised and supervised segmentation methods across Cityscapes, Pascal VOC, and SUN RGB-D.
• The strongest gains occur in semi-supervised settings with very few labels, with up to 5–10% relative improvement in some cases when using ReCo with ClassMix.
• ReCo yields sharper object boundaries and better handling of high-frequency regions, especially near class boundaries and for small objects.
• Active sampling of hard queries and particularly hard negatives (guided by a batch-wise class relationship graph) yields meaningful gains and is more effective than random sampling.
• ReCo achieves competitive or state-of-the-art results on semi-supervised benchmarks (e.g., matching or surpassing PseudoSeg benchmarks with fewer labeled examples).
• The approach is lightweight to implement (adds a representation head during training only) and incurs minimal extra memory footprint.