[Paper Review] Bidirectional Projection Network for Cross Dimension Scene Understanding

Wenbo Hu, Hengshuang Zhao+3 more|arXiv (Cornell University)|Mar 26, 2021

3D Shape Modeling and Analysis70 references4 citations

TL;DR

This paper proposes Bidirectional Projection Network (BPNet), a novel end-to-end framework that enables bidirectional feature interaction between 2D images and 3D point clouds via a Bidirectional Projection Module (BPM) at multiple pyramid levels. By jointly optimizing 2D and 3D semantic segmentation, BPNet achieves state-of-the-art performance on the ScanNetV2 benchmark, outperforming single-modality baselines in both 2D and 3D mIoU scores.

ABSTRACT

2D image representations are in regular grids and can be processed efficiently, whereas 3D point clouds are unordered and scattered in 3D space. The information inside these two visual domains is well complementary, e.g., 2D images have fine-grained texture while 3D point clouds contain plentiful geometry information. However, most current visual recognition systems process them individually. In this paper, we present a \emph{bidirectional projection network (BPNet)} for joint 2D and 3D reasoning in an end-to-end manner. It contains 2D and 3D sub-networks with symmetric architectures, that are connected by our proposed \emph{bidirectional projection module (BPM)}. Via the \emph{BPM}, complementary 2D and 3D information can interact with each other in multiple architectural levels, such that advantages in these two visual domains can be combined for better scene recognition. Extensive quantitative and qualitative experimental evaluations show that joint reasoning over 2D and 3D visual domains can benefit both 2D and 3D scene understanding simultaneously. Our \emph{BPNet} achieves top performance on the ScanNetV2 benchmark for both 2D and 3D semantic segmentation. Code is available at \url{https://github.com/wbhu/BPNet}.

Motivation & Objective

• Address the limitation of unidirectional fusion in 2D-3D scene understanding by enabling bidirectional feature interaction.
• Leverage complementary information from 2D images (texture) and 3D point clouds (geometry) to improve joint scene recognition.
• Develop a scalable, end-to-end framework that integrates 2D and 3D reasoning at multiple architectural levels.
• Demonstrate the effectiveness of bidirectional feature flow in enhancing both 2D and 3D semantic segmentation performance.
• Validate generalization to 2.5D RGB-D data, showing applicability beyond pure 3D scenes.

Proposed method

• Design a symmetric U-Net-based architecture for both 2D and 3D branches to ensure balanced feature learning.
• Introduce the Bidirectional Projection Module (BPM) to establish learnable projection links between 2D and 3D feature maps at multiple pyramid levels.
• Perform bidirectional feature transfer: project 2D features into 3D space and vice versa using the BPM’s learned link matrix.
• Apply BPM at multiple decoder levels to enable coarse-to-fine fusion of complementary 2D and 3D features.
• Use multi-view 2D images and 3D point clouds as input, with depth maps converted to 3D point clouds for 3D processing.
• Train the entire network end-to-end using cross-entropy loss for both 2D and 3D semantic segmentation tasks.

Experimental results

Research questions

• RQ1 Can bidirectional feature interaction between 2D and 3D representations improve joint scene understanding beyond unidirectional fusion?
• RQ2 How does multi-level, coarse-to-fine feature fusion via BPM affect 2D and 3D semantic segmentation performance?
• RQ3 To what extent can 3D geometry improve 2D segmentation, and vice versa, in a joint learning framework?
• RQ4 Does the proposed BPNet generalize to 2.5D RGB-D data, such as NYUv2, beyond full 3D scenes?
• RQ5 What is the optimal number of 2D views and voxel size for maximizing joint 2D-3D performance?

Key findings

• BPNet achieves a 2D mIoU of 71.9% and 3D mIoU of 73.9% on ScanNetV2 using a 2cm voxel size, outperforming all single-modality and baseline joint methods.
• With only one 2D view, BPNet achieves the highest 2D mIoU (66.5%), indicating that minimal 2D input can still benefit from 3D guidance.
• 3D mIoU increases with more 2D views up to three, peaking at 70.6% with three views, suggesting optimal information gain before redundancy.
• Reducing voxel size from 5cm to 2cm improves 3D mIoU by 5.9 percentage points (from 68.0% to 72.1%) in the 3D-only baseline, and boosts 2D mIoU by 10.4 points when combined with BPNet.
• On NYUv2, BPNet achieves 73.5% dense pixel classification accuracy, surpassing 3DMV (71.2%) and other RGB-D baselines, demonstrating strong generalization to 2.5D data.
• Qualitative results show improved boundary sharpness in 2D segmentation and better distinction of shape-similar objects (e.g., wall vs. picture) in 3D segmentation due to bidirectional feature refinement.