[Paper Review] Fully-Convolutional Point Networks for Large-Scale Point Clouds

Dario Rethage, Johanna Wald+3 more|arXiv (Cornell University)|Aug 21, 2018

3D Shape Modeling and AnalysisEngineering18 references20 citations

TL;DR

This paper introduces Fully-Convolutional Point Networks (FCPN), a novel architecture that processes unorganized 3D point clouds by internally transforming them into organized voxel grids for efficient 3D convolutional processing. The method enables end-to-end, single-pass inference on large-scale point clouds (up to 200k points) with state-of-the-art performance in semantic segmentation and 3D scene captioning, while maintaining memory efficiency and spatially ordered output.

ABSTRACT

This work proposes a general-purpose, fully-convolutional network architecture for efficiently processing large-scale 3D data. One striking characteristic of our approach is its ability to process unorganized 3D representations such as point clouds as input, then transforming them internally to ordered structures to be processed via 3D convolutions. In contrast to conventional approaches that maintain either unorganized or organized representations, from input to output, our approach has the advantage of operating on memory efficient input data representations while at the same time exploiting the natural structure of convolutional operations to avoid the redundant computing and storing of spatial information in the network. The network eliminates the need to pre- or post process the raw sensor data. This, together with the fully-convolutional nature of the network, makes it an end-to-end method able to process point clouds of huge spaces or even entire rooms with up to 200k points at once. Another advantage is that our network can produce either an ordered output or map predictions directly onto the input cloud, thus making it suitable as a general-purpose point cloud descriptor applicable to many 3D tasks. We demonstrate our network's ability to effectively learn both low-level features as well as complex compositional relationships by evaluating it on benchmark datasets for semantic voxel segmentation, semantic part segmentation and 3D scene captioning.

Motivation & Objective

• To develop a general-purpose deep learning architecture capable of processing large-scale, unorganized 3D point clouds efficiently.
• To overcome the limitations of existing methods that either restrict input to organized grids or operate solely on unordered point sets.
• To enable end-to-end learning on raw point clouds without pre-processing or post-processing steps.
• To support both global and local feature learning across multiple spatial scales for diverse 3D understanding tasks.
• To demonstrate applicability beyond segmentation, including 3D scene captioning, by generating spatially coherent, viewpoint-aware descriptions.

Proposed method

• The network takes unorganized 3D point clouds as input and internally transforms them into structured 3D voxel grids for 3D convolutional processing.
• It employs a hybrid architecture: input is unstructured point cloud, but internal representations are organized via voxelization to enable efficient 3D convolutions.
• The network uses a fully-convolutional encoder-decoder structure with multi-scale abstraction and upsampling, preserving spatial resolution and context.
• Feature learning is performed through 3D convolutional layers followed by 1×1×1 convolutions to reduce dimensionality and enhance representational capacity.
• Nearest-neighbor interpolation in the latent space is used to propagate features back to the original point cloud resolution.
• For 3D captioning, the final layers are replaced with fully connected layers, while the semantic segmentation backbone is frozen and fine-tuned only on captioning heads.

Experimental results

Research questions

• RQ1 Can a fully-convolutional network be designed to process unstructured point clouds directly while leveraging the efficiency of 3D convolutions?
• RQ2 How does the proposed hybrid architecture compare to purely point-based or purely voxel-based networks in terms of performance and scalability?
• RQ3 Can the network generate spatially coherent, viewpoint-aware descriptions of 3D scenes, enabling higher-level scene understanding?
• RQ4 To what extent does the method scale to large-scale point clouds (e.g., entire rooms) with minimal memory overhead?
• RQ5 Does the internal organization of features via voxelization improve feature learning compared to direct point-based processing?

Key findings

• FCPN achieves state-of-the-art performance on the ShapeNet part segmentation benchmark, outperforming prior methods in 12 out of 16 categories with a mean mIoU of 84.0.
• On the ScanNet dataset for semantic voxel labeling, the network processes point clouds of up to 200k points in a single forward pass with only 9.1 seconds inference time on a Titan Xp.
• The method consumes only 40% more memory when processing point clouds five times larger in spatial extent and ten times larger in point count, demonstrating strong scalability.
• The network produces spatially ordered output, enabling successful 3D scene captioning with contextually relevant, viewpoint-aware descriptions.
• The model generalizes well across scales: it can be trained on small regions (e.g., 2.4×2.4×2.4 m) and applied to larger scenes during inference without retraining.
• The ablation study confirms that the internal voxelization and 3D convolutional processing significantly improve feature learning compared to point-only baselines.