[Paper Review] Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs

Xinhang Song, Luis Herranz+1 more|arXiv (Cornell University)|Jan 21, 2018

Advanced Vision and ImagingComputer Science43 citations

TL;DR

This paper shows that learning depth features from scratch with weakly supervised patches and a depth-focused CNN outperforms transferring and fine-tuning RGB CNNs for RGB-D scene recognition, achieving state-of-the-art on NYU2 and SUN RGB-D.

ABSTRACT

Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN models and fine-tuning with the target RGB-D dataset. However, we show that this approach has the limitation of hardly reaching bottom layers, which is key to learn modality-specific features. In contrast, we focus on the bottom layers, and propose an alternative strategy to learn depth features combining local weakly supervised training from patches followed by global fine tuning with images. This strategy is capable of learning very discriminative depth-specific features with limited depth images, without resorting to Places-CNN. In addition we propose a modified CNN architecture to further match the complexity of the model and the amount of data available. For RGB-D scene recognition, depth and RGB features are combined by projecting them in a common space and further leaning a multilayer classifier, which is jointly optimized in an end-to-end network. Our framework achieves state-of-the-art accuracy on NYU2 and SUN RGB-D in both depth only and combined RGB-D data.

Motivation & Objective

• Motivate the challenges of RGB-D scene recognition due to limited depth data and modality gaps from RGB data.
• Investigate bottom-layer importance for inter-modal transfer from RGB to depth.
• Propose weakly supervised, patch-based pretraining and depth-specific CNN architectures.
• Develop an end-to-end RGB-D-CNN that fuses depth and RGB features in a shared space.

Proposed method

• Compare fine-tuning strategies that target top vs bottom layers and show bottom layers are crucial for depth transfer.
• Introduce weakly supervised pretraining on patches (WSP) to initialize depth CNNs without relying on Places-CNN.
• Propose a compact depth-CNN (D-CNN) with reduced kernels and three spatial pyramid pooling layers to manage parameter count.
• Train an RGB-D-CNN with fusion layers that project RGB and depth features into a common space for end-to-end learning.
• Evaluate using linear SVMs (and weighted SVM) on SUN RGB-D and NYUD2 with ablations and comparisons to RGB-based baselines.

Experimental results

Research questions

• RQ1 Can RGB-to-depth transfer learning effectively adapt bottom-layer representations for depth data?
• RQ2 Does training depth-specific CNNs from scratch with weak supervision yield better depth features than transferring RGB models?
• RQ3 How does a jointly trained RGB-D fusion network perform compared to single-modality nets on RGB-D scene recognition?
• RQ4 What architectural adjustments (kernel size, pooling, SPP) are beneficial for depth data with limited samples?

Key findings

• Fine-tuning only top layers on depth data is ineffective; bottom-layer fine-tuning yields substantial gains.
• Training depth models from scratch with patch-based weak supervision significantly improves depth features compared to full-image training.
• The proposed D-CNN with scaled kernels and SPP achieves strong depth-only performance and surpasses Places-CNN-based transfer baselines.
• A joint RGB-D-CNN with a fusion network outperforms separate RGB and depth pipelines on SUN RGB-D and NYUD2.
• Depth features learned directly from depth data are more discriminative and complementary to RGB features than transferred RGB features.