[Paper Review] Toward unsupervised, multi-object discovery in large-scale image collections

Huy V. Vo, Patrick Pérez+1 more|arXiv (Cornell University)|Jul 6, 2020

Advanced Neural Network ApplicationsComputer Science42 references49 citations

TL;DR

The paper proposes an unsupervised pipeline using CNN-based region proposals and a regularized, scalable object discovery framework (rOSD) to discover multiple objects across large image collections.

ABSTRACT

This paper addresses the problem of discovering the objects present in a collection of images without any supervision. We build on the optimization approach of Vo et al. (CVPR'19) with several key novelties: (1) We propose a novel saliency-based region proposal algorithm that achieves significantly higher overlap with ground-truth objects than other competitive methods. This procedure leverages off-the-shelf CNN features trained on classification tasks without any bounding box information, but is otherwise unsupervised. (2) We exploit the inherent hierarchical structure of proposals as an effective regularizer for the approach to object discovery of Vo et al., boosting its performance to significantly improve over the state of the art on several standard benchmarks. (3) We adopt a two-stage strategy to select promising proposals using small random sets of images before using the whole image collection to discover the objects it depicts, allowing us to tackle, for the first time (to the best of our knowledge), the discovery of multiple objects in each one of the pictures making up datasets with up to 20,000 images, an over five-fold increase compared to existing methods, and a first step toward true large-scale unsupervised image interpretation.

Motivation & Objective

• Develop an unsupervised method to discover objects across large image collections without bounding-box supervision.
• Improve region proposals by leveraging CNN features trained on auxiliary classification tasks.
• Introduce a regularized OSD (rOSD) formulation that enables multi-object discovery within images.
• Propose a two-stage scalable approach to apply object discovery on datasets up to 20,000 images and beyond.

Proposed method

• Generate region proposals directly from CNN feature maps without bounding boxes by building a global saliency map and local maxima to form proposals.
• Introduce regularized OSD (rOSD) by constraining proposals grouped by their generating local maxima to at most one retained region per group.
• Improve scalability on large collections via a two-stage strategy: first select promising proposals per image, then run OSD on the full collection with the reduced proposal set.
• Adopt a two-stage large-scale variant, including pre-filtering neighborhoods and using proxy OSD before full dataset optimization.

Experimental results

Research questions

• RQ1 Can unsupervised CNN-based region proposals improve object discovery over traditional unsupervised proposals?
• RQ2 Does incorporating a group-wise constraint (one region per local-maximum group) boost multi-object discovery performance?
• RQ3 How well does the two-stage, large-scale strategy enable discovering multiple objects in images of very large datasets?
• RQ4 How do OSD and the proposed rOSD compare to state-of-the-art methods on standard benchmarks for single- and multi-object discovery?
• RQ5 What is the impact of using pre-trained CNN features without bounding boxes on discovery performance?

Key findings

• CNN-based region proposals outperform off-the-shelf unsupervised proposals in object discovery across several datasets.
• The regularized OSD (rOSD) significantly improves over the original OSD, enabling robust multi-object discovery.
• A two-stage large-scale approach enables applying object discovery to datasets up to 20,000 images while maintaining performance gains.
• On OD, VOC_6x2, VOC_all, and VOC12, rOSD yields competitive or superior results vs. the state of the art in multi-object discovery settings.
• Across large datasets, rOSD outperforms competitive methods in multi-object colocalization and discovery, with notable gains on VOC_all and VOC12.