[Paper Review] 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

Angela Dai, Matthias Nießner|arXiv (Cornell University)|Mar 28, 2018

Advanced Vision and Imaging35 references4 citations

TL;DR

3DMV proposes a joint 3D-multi-view deep learning framework that fuses RGB and geometric features for 3D semantic scene segmentation in indoor RGB-D scans. By extracting 2D features from input images, backprojecting them into a 3D voxel grid via a differentiable layer, and combining them with 3D geometric features in an end-to-end network, the method achieves 75% mean intersection over union on the ScanNet benchmark—marking a +22.2% improvement over prior volumetric methods.

ABSTRACT

We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network. In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture. Rather than simply projecting color data into a volumetric grid and operating solely in 3D -- which would result in insufficient detail -- we first extract feature maps from associated RGB images. These features are then mapped into the volumetric feature grid of a 3D network using a differentiable backprojection layer. Since our target is 3D scanning scenarios with possibly many frames, we use a multi-view pooling approach in order to handle a varying number of RGB input views. This learned combination of RGB and geometric features with our joint 2D-3D architecture achieves significantly better results than existing baselines. For instance, our final result on the ScanNet 3D segmentation benchmark increases from 52.8\% to 75\% accuracy compared to existing volumetric architectures.

Motivation & Objective

• To improve 3D semantic scene segmentation in indoor environments by effectively combining RGB and geometric data.
• To address the limitation of existing volumetric networks that fail to leverage rich 2D image features due to low voxel resolution.
• To develop an end-to-end, joint 3D-multi-view architecture that learns synergistic representations from both modalities.
• To handle variable numbers of input RGB views using a multi-view pooling mechanism for robustness in sequential scanning scenarios.
• To outperform existing state-of-the-art volumetric and point-based methods on 3D semantic segmentation benchmarks.

Proposed method

• The method uses a 2D backbone network to extract high-resolution feature maps from input RGB images.
• These 2D features are downsampled via convolutional layers and then backprojected into a 3D volumetric grid using a differentiable differentiable backprojection layer.
• The 3D network processes both the backprojected 2D features and raw 3D geometric features (e.g., signed distance fields) in a shared 3D convolutional architecture.
• A multi-view pooling strategy aggregates features from multiple RGB views, enabling the network to handle variable input sequences from RGB-D scans.
• The entire network is trained end-to-end, allowing joint optimization of 2D and 3D feature learning.
• The fusion of 2D and 3D features is performed at intermediate layers (specifically, at the 2/3 layer), which was found to yield optimal performance.

Experimental results

Research questions

• RQ1 Can joint 2D-3D feature learning improve 3D semantic segmentation accuracy compared to using only 3D geometry or RGB data?
• RQ2 How does the integration of multi-view RGB features affect segmentation performance in dense 3D reconstruction scenarios?
• RQ3 What is the optimal fusion strategy (e.g., early, late, or intermediate) for combining 2D and 3D features in a 3D segmentation network?
• RQ4 To what extent do 2D and 3D features complement each other, and are they redundant or synergistic?
• RQ5 How does the number of input RGB views influence segmentation accuracy, and does performance saturate with more views?

Key findings

• The proposed 3DMV method achieves a mean intersection over union (mIoU) of 75% on the ScanNet 3D semantic segmentation benchmark, a significant improvement over the previous best volumetric method (52.8%).
• The method outperforms the best PointNet-based baseline by 14.8 percentage points in mIoU, demonstrating the effectiveness of joint 3D-multi-view learning.
• The 2/3 fusion strategy—where 2D and 3D features are combined at the 2/3 layer of the 3D network—yielded the best performance, outperforming early and late fusion variants.
• Adding more RGB views improves performance, but the gains diminish with each additional view, indicating diminishing returns beyond a certain number of inputs.
• The joint 3D-multi-view network learns orthogonal and complementary features: RGB and geometric features are not redundant but mutually reinforcing.
• The method generalizes well to 2D pixel-level segmentation tasks, achieving 71.2% accuracy on the NYU2 dataset when projecting 3D labels back to RGB frames.