[Paper Review] Fusing Saliency Maps with Region Proposals for Unsupervised Object Localization

Hakan Karaoğuz, Patric Jensfelt|arXiv (Cornell University)|Apr 11, 2018

Visual Attention and Saliency Detection18 references3 citations

TL;DR

This paper proposes a fully unsupervised object localization method that fuses deep learning-based saliency maps with class-agnostic region proposals to localize objects in single images without any category or instance-level supervision. The approach achieves state-of-the-art performance on benchmark datasets using only one image frame and demonstrates robustness in real-world conditions through a newly created, challenging dataset with variable lighting and viewpoints.

ABSTRACT

In this paper we address the problem of unsupervised localization of objects in single images. Compared to previous state-of-the-art method our method is fully unsupervised in the sense that there is no prior instance level or category level information about the image. Furthermore, we treat each image individually and do not rely on any neighboring image similarity. We employ deep-learning based generation of saliency maps and region proposals to tackle this problem. First salient regions in the image are determined using an encoder/decoder architecture. The resulting saliency map is matched with region proposals from a class agnostic region proposal network to roughly localize the candidate object regions. These regions are further refined based on the overlap and similarity ratios. Our experimental evaluations on a benchmark dataset show that the method gets close to current state-of-the-art methods in terms of localization accuracy even though these make use of multiple frames. Furthermore, we created a more challenging and realistic dataset with multiple object categories and varying viewpoint and illumination conditions for evaluating the method's performance in real world scenarios.

Motivation & Objective

• Address the challenge of unsupervised object localization in real-world robotics scenarios where labeled data is scarce and environmental conditions vary.
• Develop a method that localizes objects in single images without relying on neighboring image similarity or prior category information.
• Create a realistic, diverse dataset with multiple object categories under varying illumination and viewpoint conditions to better evaluate real-world performance.
• Improve robustness of unsupervised localization by combining semantic saliency maps with region proposals, avoiding reliance on supervised pretraining.

Proposed method

• Generate pixel-wise saliency maps using an encoder/decoder architecture trained for semantic segmentation to identify salient image regions.
• Generate region proposals using a class-agnostic region proposal network (RPN) to identify candidate object regions without category supervision.
• Fuse saliency maps and region proposals by matching high-overlap and high-similarity regions to refine candidate object locations.
• Apply non-maximum suppression (NMS) with a threshold of 0.05 to eliminate redundant proposals and improve localization precision.
• Refine object candidates using overlap and similarity ratios between saliency regions and proposals to enhance localization accuracy.
• Use a threshold of 300 pixels² for minimum saliency region size to filter out noise and small irrelevant regions.

Experimental results

Research questions

• RQ1 Can a fully unsupervised method localize objects in single images without any category or instance-level annotations?
• RQ2 How does the fusion of saliency maps and region proposals improve localization accuracy compared to using either component alone?
• RQ3 To what extent does the method generalize to real-world conditions with varying illumination and viewpoint changes?
• RQ4 How does the method perform in multi-object scenes, especially when objects are small or sparsely salient?
• RQ5 Can the proposed method achieve competitive performance compared to state-of-the-art methods that use multiple frames or weak supervision?

Key findings

• The method achieves localization accuracy comparable to state-of-the-art methods on the PASCAL VOC 2012 benchmark, despite using only a single image frame and no neighboring image similarity.
• On the KTH Handtool dataset, the plier category achieved the highest localization accuracy (63.7% CorLoc under artificial lighting with Camera 1), likely due to its textured surface.
• The method performs worse under natural lighting conditions for Camera 2 (14.3% for hammer), indicating sensitivity to low illumination and distance.
• Directional lighting improved screwdriver localization with Camera 2 (31.0% CorLoc), suggesting that lighting direction affects performance depending on object surface properties.
• The method struggles with multiple small objects, as seen in Table II (bottom row), where sparse salient regions are merged into a single large region due to fusion bias.
• Higher-resolution Camera 1 outperforms Camera 2 across all categories, with average CorLoc scores 10–20 percentage points higher, highlighting the impact of image quality and viewpoint.