[论文解读] Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs

Xinhang Song, Luis Herranz+1 more|arXiv (Cornell University)|Jan 21, 2018

Advanced Vision and Imaging被引用 43

一句话总结

本论文显示，从头开始通过弱监督补丁学习深度特征和以深度为重点的CNN，在RGB-D场景识别中优于RGB CNN的迁移与微调，在NYU2和SUN RGB-D上达到最先进水平。

ABSTRACT

Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN models and fine-tuning with the target RGB-D dataset. However, we show that this approach has the limitation of hardly reaching bottom layers, which is key to learn modality-specific features. In contrast, we focus on the bottom layers, and propose an alternative strategy to learn depth features combining local weakly supervised training from patches followed by global fine tuning with images. This strategy is capable of learning very discriminative depth-specific features with limited depth images, without resorting to Places-CNN. In addition we propose a modified CNN architecture to further match the complexity of the model and the amount of data available. For RGB-D scene recognition, depth and RGB features are combined by projecting them in a common space and further leaning a multilayer classifier, which is jointly optimized in an end-to-end network. Our framework achieves state-of-the-art accuracy on NYU2 and SUN RGB-D in both depth only and combined RGB-D data.

研究动机与目标

• 由于深度数据有限且与RGB数据在模态上的差异，激发RGB-D场景识别的挑战。
• 研究从RGB到深度的跨模态迁移中底层表示的重要性。
• 提出基于补丁的弱监督预训练和深度特定的CNN架构。
• 开发一个端到端的RGB-D-CNN，在共享空间中融合深度和RGB特征。

提出的方法

• 比较针对顶部层与底部层的微调策略，并显示底部层对深度迁移至关重要。
• 引入基于补丁的弱监督预训练（WSP）以在不依赖Places-CNN的情况下初始化深度CNN。
• 提出一个紧凑的深度-CNN（D-CNN），通过减少卷积核数量和三个空间金字塔池化层来控制参数数量。
• 训练一个带融合层的RGB-D-CNN，将RGB和深度特征投射到公共空间以进行端到端学习。
• 使用线性SVM（以及加权SVM）在SUN RGB-D和NYUD2上进行评估，附带消融研究并与基于RGB的基线进行比较。

实验结果

研究问题

• RQ1 RGB到深度的迁移学习是否能有效地将底层表示迁移到深度数据？
• RQ2 从头开始用弱监督训练深度特定的CNN是否比迁移RGB模型得到更好的深度特征？
• RQ3 联合训练的RGB-D融合网络在RGB-D场景识别中的表现如何，相较于单模态网络？
• RQ4 针对样本有限的深度数据，哪些架构调整（卷积核大小、池化、SPP）是有益的？

主要发现

• 仅对深度数据微调顶层效果不佳；对底层进行微调能带来显著提升。
• 使用基于补丁的弱监督从头训练深度模型相比全图像训练显著改善深度特征。
• 所提出的D-CNN通过缩放卷积核和SPP实现了强劲的深度专用性能，超过基于Places-CNN的迁移基线。
• 一个带融合网络的联合RGB-D-CNN在SUN RGB-D和NYUD2上优于分离的RGB和深度流水线。
• 直接从深度数据学习的深度特征比迁移的RGB特征更具辨别力并且与RGB特征互补。