[Paper Review] Semi-supervised Implicit Scene Completion from Sparse LiDAR

Pengfei Li, Yongliang Shi+4 more|arXiv (Cornell University)|Nov 29, 2021

Remote Sensing and LiDAR Applications4 citations

TL;DR

This paper proposes a semi-supervised implicit scene completion method for sparse LiDAR point clouds by introducing learned shape embeddings as dense boundary constraints, enabling robust signed distance function learning without ground-truth SDF values in free space. The approach achieves 51.0% IoU on SemanticKITTI, significantly outperforming SIREN, and enables implicit semantic completion via two parallel heads.

ABSTRACT

Recent advances show that semi-supervised implicit representation learning can be achieved through physical constraints like Eikonal equations. However, this scheme has not yet been successfully used for LiDAR point cloud data, due to its spatially varying sparsity. In this paper, we develop a novel formulation that conditions the semi-supervised implicit function on localized shape embeddings. It exploits the strong representation learning power of sparse convolutional networks to generate shape-aware dense feature volumes, while still allows semi-supervised signed distance function learning without knowing its exact values at free space. With extensive quantitative and qualitative results, we demonstrate intrinsic properties of this new learning system and its usefulness in real-world road scenes. Notably, we improve IoU from 26.3% to 51.0% on SemanticKITTI. Moreover, we explore two paradigms to integrate semantic label predictions, achieving implicit semantic completion. Code and models can be accessed at https://github.com/OPEN-AIR-SUN/SISC.

Motivation & Objective

• To address the failure of semi-supervised SDF learning on sparse LiDAR data due to spatially varying sparsity and unreliable normal estimation.
• To develop a novel formulation that conditions implicit function learning on learned, shape-aware dense embeddings to stabilize training without requiring exact SDF values at free space.
• To enable implicit geometric and semantic scene completion in real-world road scenes using only sparse LiDAR inputs and limited supervision.
• To integrate semantic understanding into the implicit representation framework, achieving end-to-end implicit semantic completion.

Proposed method

• A hybrid architecture combines a sparse convolutional network for discriminative shape embedding learning from sparse point clouds and a generative model for predicting SDF values conditioned on these embeddings.
• Shape embeddings serve as dense, data-driven boundary values that encode both on-surface points (zeroth-order) and normal directions (first-order) constraints.
• The method uses trilinear or nearest-neighbor sampling to retrieve shape codes at input coordinates, enabling SDF prediction without explicit SDF supervision in free space.
• An Eikonal constraint is relaxed through the embedding-based supervision, reducing sensitivity to noisy or sparse off-surface anchors.
• Two paradigms are introduced for semantic completion: (1) a dense semantic head with K-NN mapping to implicit space, and (2) a parallel implicit generative head for semantic label fields.
• Positional encoding with high-level frequencies (e.g., L=10) is used to preserve fine geometric details in the SDF prediction.

Experimental results

Research questions

• RQ1 Can semi-supervised implicit learning be successfully applied to sparse LiDAR data, where prior methods like SIREN fail due to sparsity and unreliable normals?
• RQ2 How can shape embeddings be used to provide effective dense boundary constraints for SDF learning without requiring ground-truth SDF values in free space?
• RQ3 To what extent can learned shape embeddings improve the consistency and completeness of implicit scene representations in sparse LiDAR scenarios?
• RQ4 Can the proposed framework enable implicit semantic scene completion, and how does it compare to explicit semantic completion baselines?
• RQ5 Does the integration of positional encoding significantly affect the quality of implicit scene completion?

Key findings

• The method achieves 51.0% IoU on the SemanticKITTI benchmark, a substantial improvement over SIREN’s 26.3%.
• The ablation study shows that using positional encoding with level L=10 and including original coordinates yields the best performance (51.020 IoU), while omitting it drops IoU to 40.431.
• The t-SNE visualization of shape embeddings reveals clear clusters and sharp boundaries, indicating effective encoding of both on-surface and normal constraints.
• Implicit semantic completion via the SSC-A and SSC-B heads achieves mIoU of 20.2% and 18.0%, respectively, with a further boost to 23.4% when mapping JS3C-Net’s explicit predictions to the implicit space.
• Mesh reconstructions at arbitrary resolutions demonstrate the continuous nature of the learned SDF field, with consistent geometry even at higher resolutions.
• Qualitative results show that the method produces more complete and coherent reconstructions than SIREN, especially in sparse regions.