[Paper Review] Arbitrary Point Cloud Upsampling with Spherical Mixture of Gaussians

Anthony Dell’Eva, Marco Orsingher+1 more|arXiv (Cornell University)|Aug 10, 2022

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes APU-SMOG, a Transformer-based method for arbitrary point cloud upsampling that maps sparse inputs to a Spherical Mixture of Gaussians (SMOG) on the unit sphere, enabling sampling of any number of points at test time. The approach decouples network architecture from upsampling ratio, achieving state-of-the-art performance with a single model trained at a fixed ratio, including non-integer scaling factors.

ABSTRACT

Generating dense point clouds from sparse raw data benefits downstream 3D understanding tasks, but existing models are limited to a fixed upsampling ratio or to a short range of integer values. In this paper, we present APU-SMOG, a Transformer-based model for Arbitrary Point cloud Upsampling (APU). The sparse input is firstly mapped to a Spherical Mixture of Gaussians (SMOG) distribution, from which an arbitrary number of points can be sampled. Then, these samples are fed as queries to the Transformer decoder, which maps them back to the target surface. Extensive qualitative and quantitative evaluations show that APU-SMOG outperforms state-of-the-art fixed-ratio methods, while effectively enabling upsampling with any scaling factor, including non-integer values, with a single trained model. The code is available at https://github.com/apusmog/apusmog/

Motivation & Objective

• To address the limitation of existing point cloud upsampling models that require retraining for each fixed upsampling ratio.
• To enable arbitrary upsampling ratios, including non-integer values, with a single trained model.
• To decouple the upsampling process from the network architecture by using an intermediate probabilistic representation.
• To improve generalization and robustness to sparse and noisy inputs through a learnable spherical distribution and attention-based refinement.
• To achieve state-of-the-art performance on multiple benchmarks while maintaining flexibility in output resolution.

Proposed method

• The input sparse point cloud is encoded into a Spherical Mixture of Gaussians (SMOG) distribution on the unit sphere, where each point is associated with a mixture weight and a bivariate Gaussian in spherical coordinates.
• The parameters of the SMOG distribution (mean and variance in spherical coordinates) are predicted by a neural network, enabling a local, adaptive probabilistic representation of the input surface.
• An arbitrary number of points are sampled from the learned SMOG distribution at test time, allowing flexible upsampling to any desired resolution.
• The sampled points are fed as queries into a Transformer decoder that maps them back to the target 3D surface, decoupling the output resolution from the model architecture.
• A residual refinement block based on local self-attention is applied to correct outliers and improve geometric fidelity in the final output.
• The model is trained with a combination of reconstruction losses: a point-to-point distance loss (P2F), a Hausdorff distance loss (HD), and a projection loss (Π) to align Gaussian means with the surface.

Experimental results

Research questions

• RQ1 Can a single neural network model perform point cloud upsampling for any arbitrary upsampling ratio, including non-integer values, without retraining?
• RQ2 Does learning a Spherical Mixture of Gaussians as an intermediate representation enable better generalization and flexibility compared to fixed sampling strategies?
• RQ3 How effective is a Transformer-based decoder with attention-based refinement in reconstructing high-resolution point clouds from arbitrary samples of a learned distribution?
• RQ4 What is the contribution of each loss component (e.g., projection loss, reconstruction loss) to the final performance and robustness?
• RQ5 Can the method generalize to sparse inputs and noisy point clouds while preserving fine geometric details?

Key findings

• APU-SMOG achieves state-of-the-art performance on multiple benchmarks, with a Chamfer Distance (CD) of 0.528×10⁻³, Hausdorff Distance (HD) of 2.549×10⁻³, and P2F of 1.667×10⁻³ on the PU1K dataset.
• The ablation study confirms that the SMOG representation outperforms uniform sampling (FoldingNet-like) with a CD of 0.558×10⁻³, demonstrating the benefit of adaptive local distributions.
• Removing the residual refinement block increases HD to 8.536×10⁻³, indicating that it effectively reduces outliers and improves surface fidelity.
• The projection loss (ℒΠ) is more effective than the ACD loss for upsampling, as shown by a 2.694×10⁻³ P2F score when using ℒACD, compared to 1.667×10⁻³ with ℒΠ.
• The model generalizes well to sparse inputs (as low as 256 points) and noisy data, preserving fine structures like horse legs and car windows even under additive Gaussian noise (σ=0.02).
• The method successfully upsamples to non-integer ratios (e.g., r=2.34, r=4.93) with a single model trained at r=4, demonstrating true arbitrary upsampling capability.