[Paper Review] Attention-based Multi-modal Fusion Network for Semantic Scene Completion

Siqi Li, Changqing Zou+3 more|arXiv (Cornell University)|Mar 31, 2020

Advanced Vision and Imaging27 references4 citations

TL;DR

This paper proposes AMFNet, an end-to-end 3D convolutional network that improves semantic scene completion by fusing 2D semantic segmentation features with depth cues via a multi-modal attention mechanism. By leveraging 2D segmentation guidance and residual attention blocks, AMFNet achieves state-of-the-art performance, improving mIoU by 2.5% on SUNCG-RGBD and 2.6% on NYUv2 compared to prior methods.

ABSTRACT

This paper presents an end-to-end 3D convolutional network named attention-based multi-modal fusion network (AMFNet) for the semantic scene completion (SSC) task of inferring the occupancy and semantic labels of a volumetric 3D scene from single-view RGB-D images. Compared with previous methods which use only the semantic features extracted from RGB-D images, the proposed AMFNet learns to perform effective 3D scene completion and semantic segmentation simultaneously via leveraging the experience of inferring 2D semantic segmentation from RGB-D images as well as the reliable depth cues in spatial dimension. It is achieved by employing a multi-modal fusion architecture boosted from 2D semantic segmentation and a 3D semantic completion network empowered by residual attention blocks. We validate our method on both the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset and the results show that our method respectively achieves the gains of 2.5% and 2.6% on the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset against the state-of-the-art method.

Motivation & Objective

• To improve 3D semantic scene completion (SSC) by integrating 2D semantic segmentation as a supervisory signal.
• To address the sparsity of 3D scenes by focusing on reliable depth cues through attention mechanisms.
• To develop a multi-modal fusion framework that jointly optimizes 3D scene completion and semantic labeling.
• To demonstrate that 2D segmentation experience can effectively guide 3D completion, especially for object boundaries and small objects.

Proposed method

• A two-branch 3D volume network is designed, where one branch uses 2D semantic segmentation predictions to guide 3D feature learning.
• 2D-3D projection is applied to convert 2D semantic labels into an initial 3D semantic feature volume.
• A residual attention block (RAB) is introduced to dynamically emphasize meaningful voxels and suppress noise from empty regions.
• The network uses a multi-modal fusion architecture that combines RGB, depth, and HHA features for improved 3D representation.
• The 3D-guidance branch injects 2D segmentation results as spatial constraints to improve object boundary prediction.
• The model is trained end-to-end with cross-entropy and mean intersection-over-union (mIoU) losses for joint optimization.

Experimental results

Research questions

• RQ1 Can 2D semantic segmentation results improve 3D semantic scene completion performance?
• RQ2 How effective is attention-based feature refinement in handling sparse 3D voxel data?
• RQ3 Does leveraging 2D segmentation experience lead to better 3D completion and labeling than using only RGB-D inputs?
• RQ4 Can residual attention blocks enhance feature learning in 3D completion networks, especially for small or occluded objects?
• RQ5 What is the contribution of the 3D-guidance branch in improving object boundary accuracy?

Key findings

• AMFNet achieves a 2.5% mIoU improvement on the synthetic SUNCG-RGBD dataset compared to the state-of-the-art method.
• On the real NYUv2 dataset, AMFNet improves mIoU by 2.6% over the current SOTA method.
• Using ground-truth 2D segmentation labels as input to the 3D-guidance branch boosts performance by 3.8% on SUNCG-RGBD and 4.3% on NYUv2.
• The residual attention block (RAB) improves mIoU by 3.0% on SUNCG-RGBD and 2.9% on NYUv2 when compared to a baseline without attention.
• Ablation studies confirm that the 3D-guidance branch significantly improves boundary accuracy, especially for complex objects like beds and walls.
• The model with RAB better reconstructs small objects such as piles of paper on cabinets, which are often missed without attention.