[Paper Review] Show, Match and Segment: Joint Weakly Supervised Learning of Semantic Matching and Object Co-segmentation

Yunchun Chen, Yen‐Yu Lin+2 more|arXiv (Cornell University)|Jun 13, 2019

Advanced Neural Network Applications95 references4 citations

TL;DR

This paper proposes a weakly supervised, end-to-end trainable two-stream network that jointly optimizes semantic matching and object co-segmentation by leveraging cross-task consistency. By using predicted masks to suppress background clutter in matching and dense correspondences to enforce geometric consistency in segmentation, the method achieves state-of-the-art performance on five benchmarks with only image-level supervision.

ABSTRACT

We present an approach for jointly matching and segmenting object instances of the same category within a collection of images. In contrast to existing algorithms that tackle the tasks of semantic matching and object co-segmentation in isolation, our method exploits the complementary nature of the two tasks. The key insights of our method are two-fold. First, the estimated dense correspondence fields from semantic matching provide supervision for object co-segmentation by enforcing consistency between the predicted masks from a pair of images. Second, the predicted object masks from object co-segmentation in turn allow us to reduce the adverse effects due to background clutters for improving semantic matching. Our model is end-to-end trainable and does not require supervision from manually annotated correspondences and object masks. We validate the efficacy of our approach on five benchmark datasets: TSS, Internet, PF-PASCAL, PF-WILLOW, and SPair-71k, and show that our algorithm performs favorably against the state-of-the-art methods on both semantic matching and object co-segmentation tasks.

Motivation & Objective

• To address the limitations of isolated semantic matching and object co-segmentation methods that suffer from background clutter and incomplete object coverage.
• To exploit the complementary nature of semantic matching and object co-segmentation for improved performance in both tasks.
• To develop an end-to-end trainable model requiring only weak image-level supervision (i.e., image pairs with common objects) instead of costly manual annotations.
• To introduce a cross-network consistency loss that enforces geometric and appearance consistency between predicted masks and dense correspondences across image pairs.
• To improve robustness to intra-class variations, scale, pose, and viewpoint differences through joint optimization.

Proposed method

• The method employs a two-stream network: one stream predicts dense correspondence fields for semantic matching, and the other predicts foreground object masks for co-segmentation.
• A cross-network consistency loss is introduced to enforce that predicted masks from one stream are geometrically consistent with the dense correspondences from the other stream.
• For semantic matching, cycle-consistent losses are applied to improve geometric plausibility of predicted image transformations.
• For object co-segmentation, a perceptual contrastive loss enhances foreground appearance similarity across images while enforcing figure-ground dissimilarity using a cutoff threshold $m$.
• The model is trained end-to-end using only image-level supervision, with no need for annotated keypoint correspondences or object masks.
• Post-processing with DenseCRF, Otsu’s, or GrabCut is applied to refine segmentation masks, with minimal performance variation across methods.

Experimental results

Research questions

• RQ1 Can joint learning of semantic matching and object co-segmentation improve performance on both tasks compared to isolated approaches?
• RQ2 How can cross-task supervision from predicted masks and dense correspondences enhance robustness to background clutter and appearance variations?
• RQ3 To what extent does a cross-network consistency loss improve geometric and appearance consistency in weakly supervised settings?
• RQ4 How sensitive is the model to hyper-parameter choices such as the cutoff threshold $m$ in the perceptual contrastive loss?
• RQ5 Can a weakly supervised, end-to-end trainable framework achieve state-of-the-art results without manual annotations?

Key findings

• The proposed method achieves state-of-the-art performance on five benchmark datasets: TSS, Internet, PF-PASCAL, PF-WILLOW, and SPair-71k for both semantic matching and object co-segmentation.
• With the cutoff threshold $m$ set to 2, the perceptual contrastive loss achieves optimal performance, improving precision $\mathcal{P}$ and Jaccard index $\mathcal{J}$; performance degrades when $m$ is increased to 5 or 10.
• Turning off any individual loss component (e.g., cycle-consistent, perceptual contrastive, or cross-network consistency) leads to significant performance drops, confirming the contribution of each loss term.
• When the hyper-parameter for any loss is set too high (e.g., 1,000), that term dominates the training objective and causes performance degradation, indicating sensitivity to hyper-parameter tuning.
• The model processes each image in an average of 21 seconds on a GTX 1080 GPU, with total inference time of 280 minutes for 800 images from the TSS dataset.
• Post-processing with DenseCRF, Otsu’s, or GrabCut yields similar performance, indicating that the model’s segmentation output is robust and stable across refinement methods.