[Paper Review] Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling

Wanquan Feng, Jin Li+3 more|arXiv (Cornell University)|Dec 8, 2021

3D Shape Modeling and Analysis4 citations

TL;DR

This paper introduces Neural Points, a novel point cloud representation where each point encodes a local continuous geometric shape using neural fields, enabling arbitrary-factored upsampling with superior detail preservation. By learning local isomorphisms between 2D parametric domains and 3D surface patches via deep features and shared neural fields, Neural Points achieve state-of-the-art performance in upsampling with strong generalization and robustness across diverse inputs and noise levels.

ABSTRACT

In this paper, we propose Neural Points, a novel point cloud representation and apply it to the arbitrary-factored upsampling task. Different from traditional point cloud representation where each point only represents a position or a local plane in the 3D space, each point in Neural Points represents a local continuous geometric shape via neural fields. Therefore, Neural Points contain more shape information and thus have a stronger representation ability. Neural Points is trained with surface containing rich geometric details, such that the trained model has enough expression ability for various shapes. Specifically, we extract deep local features on the points and construct neural fields through the local isomorphism between the 2D parametric domain and the 3D local patch. In the final, local neural fields are integrated together to form the global surface. Experimental results show that Neural Points has powerful representation ability and demonstrate excellent robustness and generalization ability. With Neural Points, we can resample point cloud with arbitrary resolutions, and it outperforms the state-of-the-art point cloud upsampling methods. Code is available at https://github.com/WanquanF/NeuralPoints.

Motivation & Objective

• To address the limited representation capacity of traditional discrete point clouds, which are constrained by finite resolution and cannot capture fine geometric details.
• To overcome the limitations of existing 'discrete-to-discrete' upsampling methods that fail to generalize across varying resolution factors and noisy inputs.
• To develop a continuous, resolution-free point cloud representation that preserves rich geometric details through local neural field modeling.
• To enable arbitrary-factored upsampling by constructing a global continuous surface from integrated local neural fields.
• To demonstrate robustness and generalization across diverse data, including real-scanned, noisy, and LiDAR data.

Proposed method

• Each point in the input point cloud is associated with a local 3D surface patch represented as a continuous bijective mapping from a 2D parametric domain via a neural field.
• Local deep features are extracted from each point and used as conditional inputs to the neural field network to encode local geometry.
• The neural field learns a local isomorphism between the 2D parametric domain and the 3D local surface patch, ensuring smooth and invertible mapping.
• A shared neural network is trained on high-resolution surfaces with rich geometric details, enabling generalization across diverse shapes.
• All local neural fields are integrated using a designed fusion strategy to reconstruct the global continuous surface.
• Arbitrary-resolution point clouds are generated by sampling the continuous Neural Points surface, enabling resolution-free upsampling.

Experimental results

Research questions

• RQ1 Can a point cloud representation that encodes local continuous geometric shapes via neural fields outperform traditional discrete representations in upsampling tasks?
• RQ2 Does the use of shared local neural fields conditioned on deep local features improve generalization and robustness across diverse and noisy inputs?
• RQ3 Can the proposed method achieve high-quality upsampling at very large factors (e.g., 256×) without degradation, unlike existing methods?
• RQ4 How does the Neural Points representation perform on real-world, noisy, and non-uniformly sampled data such as depth camera and LiDAR scans?
• RQ5 To what extent does the model generalize to unseen datasets without retraining?

Key findings

• Neural Points achieve state-of-the-art performance on standard upsampling benchmarks, outperforming methods like PU-GAN, PU-GCN, and Dis-PU in both quantitative metrics and visual quality.
• The method maintains high-quality results even at extreme upsampling factors (e.g., 256×), while baseline methods trained for 16× show significant degradation after two-stage application.
• On real captured data from an iPhone X depth sensor, Neural Points preserve fine geometric features and avoid artifacts, whereas Dis-PU fails to maintain geometry and PUGeo-Net introduces visible distortions.
• The model generalizes well to unseen datasets such as the PU-GAN benchmark without retraining, achieving the best performance in both quantitative and qualitative evaluations.
• Neural Points demonstrate strong robustness to Gaussian noise, maintaining smooth and accurate surfaces where other methods produce wrinkled or distorted results.
• The method achieves high-quality results on LiDAR scans from the KITTI dataset, confirming its applicability to real-world 3D sensing data.