[Paper Review] Object cosegmentation using deep Siamese network

Prerana Mukherjee, Brejesh Lall+1 more|arXiv (Cornell University)|Mar 7, 2018

Advanced Neural Network Applications24 references18 citations

TL;DR

This paper proposes an end-to-end deep learning framework for object cosegmentation by formulating it as a clustering problem using a Siamese network to learn deep feature similarity across object proposals. The method achieves state-of-the-art performance on iCoseg and MSRC datasets, with a Jaccard similarity of 0.66 and mAP of 0.84 using MCG-based proposals, and generates visual summaries based on learned similarity scores.

ABSTRACT

Object cosegmentation addresses the problem of discovering similar objects from multiple images and segmenting them as foreground simultaneously. In this paper, we propose a novel end-to-end pipeline to segment the similar objects simultaneously from relevant set of images using supervised learning via deep-learning framework. We experiment with multiple set of object proposal generation techniques and perform extensive numerical evaluations by training the Siamese network with generated object proposals. Similar objects proposals for the test images are retrieved using the ANNOY (Approximate Nearest Neighbor) library and deep semantic segmentation is performed on them. Finally, we form a collage from the segmented similar objects based on the relative importance of the objects.

Motivation & Objective

• To address the challenge of segmenting similar objects across multiple images without prior knowledge of foreground objects.
• To exploit the commonness prior and joint processing of images to improve segmentation robustness under intra-class variations and noise.
• To develop an end-to-end pipeline that combines Siamese networks for similarity learning with fully convolutional networks for semantic segmentation.
• To generate a visual summary of co-segmented objects based on their relative similarity and importance.

Proposed method

• A Siamese network with shared weights is trained on pairs of object proposals using a contrastive loss function to learn discriminative deep features.
• The contrastive loss ensures that similar patches (same object class) are mapped closer in the embedding space, while dissimilar ones are pushed apart.
• Object proposals are generated using multiple techniques (EdgeBoxes, Selective Search, MCG, Salient Proposals) and fed into the trained Siamese network to extract 256D feature vectors.
• Similarity search is performed using the ANNOY library to retrieve the nearest neighbor proposals across test images based on Euclidean distance in the feature space.
• Semantic segmentation is applied to the retrieved similar proposals using a fully convolutional network (FCN) for pixel-level segmentation.
• A visual collage is constructed by placing the most similar objects at the center, preserving relative distances, with a synthetic background for visual coherence.

Experimental results

Research questions

• RQ1 Can a Siamese network effectively learn deep feature representations that encode the commonness prior for object cosegmentation?
• RQ2 How does the choice of object proposal generation method impact the performance of the cosegmentation pipeline?
• RQ3 Can the proposed method generalize to unseen object classes without fine-tuning, using only pre-trained features?
• RQ4 How effective is the visual collage generation in reflecting the relative similarity and importance of co-segmented objects?

Key findings

• The proposed method achieves a Jaccard similarity of 0.66 and mAP of 0.84 on the MSRC dataset when trained on 70% and tested on 100% of the data using MCG-based proposals.
• On the iCoseg dataset, the method achieves a Jaccard similarity of 0.654 and mAP of 0.81 when trained on 80% and tested on 100% using MCG proposals.
• The Siamese network trained on non-target classes (e.g., Pascal, animals) generalizes well to target classes (e.g., iCoseg, MSRC) with minimal fine-tuning, demonstrating strong zero-shot generalization.
• MCG-based object proposals yield the best performance across both datasets, outperforming EdgeBoxes, Selective Search, and saliency-based proposals.
• The visual collage generation effectively reflects the similarity hierarchy, with the most similar objects placed centrally and visually coherent arrangements.
• The inference time for feature extraction is under 100ms per image, making the method efficient for real-time applications.