[Paper Review] Move to See Better: Towards Self-Supervised Amodal Object Detection.

Zhaoyuan Fang, Ayush Jain+3 more|arXiv (Cornell University)|Nov 30, 2020

Advanced Neural Network Applications41 references6 citations

TL;DR

This paper proposes a self-supervised framework for amodal object detection that improves 2D object detectors in unseen scenes by leveraging multi-view RGB-D data from a moving agent in a 3D environment. By unprojecting confident 2D detections, performing unsupervised 3D segmentation, and reprojecting them as pseudo-labels, the method significantly boosts detector performance without human annotations, outperforming prior self-supervised approaches on both indoor and outdoor datasets.

ABSTRACT

Humans learn to better understand the world by moving around their environment to get more informative viewpoints of the scene. Most methods for 2D visual recognition tasks such as object detection and segmentation treat images of the same scene as individual samples and do not exploit object permanence in multiple views. Generalization to novel scenes and views thus requires additional training with lots of human annotations. In this paper, we propose a self-supervised framework to improve an object detector in unseen scenarios by moving an agent around in a 3D environment and aggregating multi-view RGB-D information. We unproject confident 2D object detections from the pre-trained detector and perform unsupervised 3D segmentation on the point cloud. The segmented 3D objects are then re-projected to all other views to obtain pseudo-labels for fine-tuning. Experiments on both indoor and outdoor datasets show that (1) our framework performs high-quality 3D segmentation from raw RGB-D data and a pre-trained 2D detector; (2) fine-tuning with self-supervision improves the 2D detector significantly where an unseen RGB image is given as input at test time; (3) training a 3D detector with self-supervision outperforms a comparable self-supervised method by a large margin.

Motivation & Objective

• To improve generalization of 2D object detectors to novel scenes and views without requiring extensive human annotations.
• To exploit object permanence across multiple viewpoints by treating scenes as multi-view sequences rather than isolated images.
• To develop a self-supervised framework that leverages 3D geometry and multi-view consistency to generate high-quality pseudo-labels for detector fine-tuning.
• To demonstrate that self-supervised 3D segmentation and pseudo-labeling can significantly improve 2D object detection performance in unseen scenarios.

Proposed method

• The framework uses a pre-trained 2D object detector to generate confident detections on RGB-D images from a moving agent in a 3D environment.
• Confident 2D detections are unprojected into 3D space to form initial 3D object proposals using depth information.
• Unsupervised 3D segmentation is performed on the point cloud to refine and group the unprojected detections into coherent 3D objects.
• The segmented 3D objects are reprojected into all other views to generate consistent pseudo-labels for self-supervised fine-tuning of the 2D detector.
• The self-supervised fine-tuning process leverages multi-view consistency to improve detector robustness and generalization to unseen scenes.
• The method trains a 3D detector using the generated pseudo-labels, achieving state-of-the-art performance compared to existing self-supervised methods.

Experimental results

Research questions

• RQ1 Can multi-view RGB-D data from a moving agent improve 2D object detector generalization in unseen scenes without human annotations?
• RQ2 How effective is unsupervised 3D segmentation of unprojected 2D detections in generating high-quality pseudo-labels for self-supervised learning?
• RQ3 To what extent does self-supervised fine-tuning with multi-view pseudo-labels improve 2D object detection performance on unseen RGB images?
• RQ4 How does the proposed method compare to existing self-supervised approaches in terms of 3D segmentation quality and detector accuracy?
• RQ5 Can the framework generalize across diverse indoor and outdoor environments with minimal supervision?

Key findings

• The proposed framework achieves high-quality 3D segmentation from raw RGB-D data and a pre-trained 2D detector, demonstrating strong geometric reasoning without supervision.
• Self-supervised fine-tuning significantly improves 2D object detector performance when tested on unseen RGB images, indicating strong generalization to novel views.
• The method outperforms a comparable self-supervised baseline in 3D detection, showing the effectiveness of multi-view pseudo-labeling via 3D segmentation.
• The framework generalizes well across both indoor and outdoor datasets, confirming its robustness to domain shifts.
• The use of object permanence across multiple views enables consistent pseudo-label generation, leading to improved detector accuracy without human-annotated data.