[Paper Review] P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding

Yunze Liu, Yi Li+4 more|arXiv (Cornell University)|Dec 24, 2020

Domain Adaptation and Few-Shot LearningComputer Science59 references34 citations

TL;DR

P4Contrast introduces a contrastive pretraining task on pairs of point-pixel pairs to fuse RGB and geometry for RGB-D scene understanding, yielding improved semantic segmentation and 3D object detection across ScanNet, SUN RGB-D, and 3RScan.

ABSTRACT

Self-supervised representation learning is a critical problem in computer vision, as it provides a way to pretrain feature extractors on large unlabeled datasets that can be used as an initialization for more efficient and effective training on downstream tasks. A promising approach is to use contrastive learning to learn a latent space where features are close for similar data samples and far apart for dissimilar ones. This approach has demonstrated tremendous success for pretraining both image and point cloud feature extractors, but it has been barely investigated for multi-modal RGB-D scans, especially with the goal of facilitating high-level scene understanding. To solve this problem, we propose contrasting "pairs of point-pixel pairs", where positives include pairs of RGB-D points in correspondence, and negatives include pairs where one of the two modalities has been disturbed and/or the two RGB-D points are not in correspondence. This provides extra flexibility in making hard negatives and helps networks to learn features from both modalities, not just the more discriminating one of the two. Experiments show that this proposed approach yields better performance on three large-scale RGB-D scene understanding benchmarks (ScanNet, SUN RGB-D, and 3RScan) than previous pretraining approaches.

Motivation & Objective

• Motivate self-supervised learning for dense RGB-D representations suitable for 3D scene understanding.
• Propose a novel pretraining task that uses pairs of point-pixel pairs to fuse RGB and geometry.
• Show that the approach yields state-of-the-art gains on multiple RGB-D benchmarks.

Proposed method

• Define point-pixel pairs as dense RGB-D tokens drawn from synchronized RGB and depth observations.
• Create two views of each scene and construct anchor/positive/negative point-pixel pairs, including disturbed pair negatives to force joint RGB-geometry learning.
• Use a PairInfoNCE loss to pull together anchor–positive and push apart anchor–negative point-pixel pairs.
• Employ a 2D-3D context backbone combining SR-UNet (3D) and FuseNet (2D) for fused RGB-D representations.
• Apply progressive hardness scheduling for partially disturbed negatives to balance learning difficulty.
• Train with RGB-D data augmentations, including point jittering for 3D and Gaussian noise for RGB.

Experimental results

Research questions

• RQ1 Can a contrastive objective over pairs of point-pixel pairs encourage better RGB-D fusion than single-modality or simple cross-modal contrasts?
• RQ2 Does a 2D-3D context backbone improve RGB-D feature learning compared to 3D-only or 2D-only baselines?
• RQ3 Do disturbed (partially negative) point-pixel pairs improve the learning of jointly informative RGB-D features?

Key findings

• P4Contrast improves downstream performance across three tasks: semantic segmentation on ScanNetV2 and 3RScan, plus 3D object detection on SUN RGB-D.
• On ScanNetV2 semantic segmentation, P4Contrast(2D-3D context) achieves 75.0 mIoU (K3) vs 72.1 baseline (K5) and 73.8 in some PointContrast variants.
• On 3RScan semantic segmentation, P4Contrast(2D-3D context) reaches 41.7 mIoU, up from 38.8 (PointContrast) and 37.3 (training from scratch).
• On SUN RGB-D 3D object detection, P4Contrast achieves 63.5 mAP@0.25, surpassing VoteNet, PointContrast, and ImVoteNet baselines.
• P4Contrast yields notable gains when fine-tuning with limited data, e.g., 4.5 mIoU improvement with only 10% of ScanNet training data.
• A 2D-3D context backbone with joint RGB-D fusion outperforms single-modality or simple RGB-augmented point approaches.