[Paper Review] LMSCNet: Lightweight Multiscale 3D Semantic Completion

Luis Roldão, Raoul de Charette+1 more|arXiv (Cornell University)|Aug 24, 2020

3D Shape Modeling and Analysis51 references134 citations

TL;DR

LMSCNet proposes a lightweight, multiscale 3D semantic scene completion network that uses a 2D UNet backbone with 3D segmentation heads to achieve state-of-the-art performance on the SemanticKITTI benchmark while significantly reducing parameters and inference time. By leveraging 2D convolutions along the height axis and dilated 3D convolutions, it enables fast, coarse-scale inference—up to 372 FPS at 1:8 resolution—without sacrificing accuracy.

ABSTRACT

We introduce a new approach for multiscale 3Dsemantic scene completion from voxelized sparse 3D LiDAR scans. As opposed to the literature, we use a 2D UNet backbone with comprehensive multiscale skip connections to enhance feature flow, along with 3D segmentation heads. On the SemanticKITTI benchmark, our method performs on par on semantic completion and better on occupancy completion than all other published methods -- while being significantly lighter and faster. As such it provides a great performance/speed trade-off for mobile-robotics applications. The ablation studies demonstrate our method is robust to lower density inputs, and that it enables very high speed semantic completion at the coarsest level. Our code is available at https://github.com/cv-rits/LMSCNet.

Motivation & Objective

• Address the high computational cost and memory overhead of 3D CNNs in sparse 3D semantic scene completion.
• Enable real-time inference for mobile robotics and autonomous driving applications with sparse LiDAR inputs.
• Maintain high accuracy in both semantic labeling and scene completion despite using 2D convolutions instead of full 3D convolutions.
• Introduce a multiscale inference pipeline that supports fast, coarse-level completion without retraining.
• Demonstrate robustness to low-density LiDAR scans, simulating real-world sensor limitations.

Proposed method

• Uses a 2D UNet-style encoder-decoder backbone with 2D convolutions along the height axis to reduce computational cost.
• Employs 3D convolutional heads at the end of the network to predict dense 3D semantic labels from 2D feature maps.
• Integrates atrous spatial pyramid pooling (ASPP) blocks in the 3D heads to expand感受 field and enrich features.
• Applies multiscale skip connections across all decoder levels to enhance feature flow and contextual understanding.
• Enables multiscale inference by pruning deeper decoder branches, allowing fast coarse predictions (1:2, 1:4, 1:8) at reduced complexity.
• Uses sparse voxelized input from LiDAR scans without preprocessing (e.g., no TSDF or SDF conversion), preserving raw occupancy grids.

Experimental results

Research questions

• RQ1 Can a 2D-based backbone with 3D heads achieve competitive 3D semantic scene completion performance while being significantly faster and lighter than 3D-CNN baselines?
• RQ2 How does multiscale inference capability affect speed-accuracy trade-offs in 3D semantic completion?
• RQ3 To what extent is the model robust to low-density LiDAR inputs, simulating real-world sensor limitations?
• RQ4 Does the use of 2D convolutions along one axis compromise 3D spatial connectivity and overall performance?
• RQ5 Can a lightweight architecture outperform heavier models in both completion and semantic segmentation metrics?

Key findings

• LMSCNet achieves state-of-the-art performance on the SemanticKITTI benchmark with a 54.22% IoU on completion and 16.78% mIoU on semantics, outperforming prior methods in completion while matching or exceeding them in semantics.
• The model achieves 372 FPS at 1:8 resolution, which is 300x faster than TS3D+DNet+SATNet and 6x faster than SSCNet, demonstrating exceptional speed for coarse inference.
• Despite using 2D convolutions, the model maintains strong performance—mIoU drops only by 0.41% when ASPP is removed, and completion IoU drops by 0.68% when using a vanilla UNet decoder instead of the multiscale version.
• The model remains robust to low input density: with only 8-layer LiDAR input (2.10% density), it still recovers the general scene outline with reasonable accuracy.
• The full-scale LMSCNet has only 0.35M parameters and 72.6 GFLOPs, making it 5x lighter and 10x faster than SSCNet-full, while outperforming it in completion metrics.
• Ablation studies confirm that deconvolution layers contribute to performance (1.43% IoU drop without them), and the multiscale UNet decoder improves completion by 0.68% and semantics by 0.56%.