[Paper Review] Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion

Peng‐Shuai Wang, Yang Liu+1 more|arXiv (Cornell University)|Jun 6, 2020

3D Shape Modeling and Analysis45 references4 citations

TL;DR

This paper proposes a deep octree-based CNN with output-guided skip connections for 3D shape and scene completion, achieving state-of-the-art performance by combining efficient octree representation, very deep networks (up to 70 layers), and skip connections that preserve input geometry. The method outperforms prior approaches on benchmark datasets with improved accuracy and robustness to noise and incomplete inputs.

ABSTRACT

Acquiring complete and clean 3D shape and scene data is challenging due to geometric occlusion and insufficient views during 3D capturing. We present a simple yet effective deep learning approach for completing the input noisy and incomplete shapes or scenes. Our network is built upon the octree-based CNNs (O-CNN) with U-Net like structures, which enjoys high computational and memory efficiency and supports to construct a very deep network structure for 3D CNNs. A novel output-guided skip-connection is introduced to the network structure for better preserving the input geometry and learning geometry prior from data effectively. We show that with these simple adaptions -- output-guided skip-connection and deeper O-CNN (up to 70 layers), our network achieves state-of-the-art results in 3D shape completion and semantic scene computation.

Motivation & Objective

• Address the challenge of completing incomplete and noisy 3D shapes and scenes due to occlusion and limited capture views.
• Overcome the limitations of existing 3D CNNs, which suffer from high memory and computation costs, restricting network depth.
• Improve geometry preservation and learning of structural priors in 3D completion by designing a novel skip-connection mechanism.
• Achieve state-of-the-art performance in 3D shape and semantic scene completion using a simple yet effective deep network architecture.
• Enable end-to-end, single-forward-pass completion with high-quality output, avoiding expensive optimization or implicit function refinement.

Proposed method

• Utilizes an octree-based CNN framework (O-CNN) for efficient 3D representation, enabling high computational and memory efficiency.
• Employs a U-Net-like architecture with two deep residual networks: an encoder for feature extraction and a decoder for shape reconstruction.
• Introduces output-guided skip connections that link generated nodes in the decoder to their corresponding existing nodes in the input octree, not the encoder.
• The skip connections are conditioned on the output structure, ensuring geometric fidelity and noise robustness by preserving input features.
• Constructs very deep networks (up to 70 layers) using residual blocks, leveraging depth for improved feature learning without computational blowup.
• Applies standard training with SGD, weight decay, and learning rate decay, using Chamfer distance as the primary loss for optimization.

Experimental results

Research questions

• RQ1 Can a deep octree-based CNN with novel skip connections outperform existing 3D completion methods in terms of accuracy and robustness?
• RQ2 How does output-guided skip connection compare to standard skip connections in preserving input geometry and handling noise?
• RQ3 To what extent does increasing network depth (beyond 20–30 layers) improve 3D shape and scene completion performance with octree representations?
• RQ4 Can the proposed method generalize to tasks beyond standard completion, such as shape reconstruction from meso-skeletons?
• RQ5 Does the method achieve better results than implicit function-based or autoencoder-based approaches in a single forward pass without iterative optimization?

Key findings

• The proposed method achieves state-of-the-art performance on 3D shape completion benchmarks, outperforming existing methods including 3DEPN and SGCNet with similar depth and parameter count.
• The ablation study shows that removing full skip-connections (l₂ and l₃) improves performance, indicating that output-guided skip connections are more effective than standard U-Net skip connections.
• On the chair and plane datasets for meso-skeleton reconstruction, the median Chamfer distance is 1.04 and 5.55, respectively, outperforming P2P-Net (1.66 and 6.06).
• The output point clouds are regularly distributed and contain predicted normals, enabling high-quality mesh reconstruction via Poisson surface reconstruction—unlike scattered outputs from P2P-Net.
• The method enables direct, single-forward-pass completion without iterative optimization, unlike DeepSDF, which requires costly latent code optimization per shape.
• The network achieves high performance with up to 70 layers, demonstrating that deep architectures are feasible and beneficial in 3D CNNs when using efficient octree representations.