[Paper Review] SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

Yifan Xu, Tianqi Fan+3 more|arXiv (Cornell University)|Mar 30, 2018

3D Shape Modeling and AnalysisEngineering19 references76 citations

TL;DR

SpiderCNN introduces SpiderConv, a point-set convolution that uses parameterized filters combining step and Taylor components to learn from irregular 3D point clouds, achieving state-of-the-art results on ModelNet40.

ABSTRACT

Deep neural networks have enjoyed remarkable success for various vision tasks, however it remains challenging to apply CNNs to domains lacking a regular underlying structures such as 3D point clouds. Towards this we propose a novel convolutional architecture, termed SpiderCNN, to efficiently extract geometric features from point clouds. SpiderCNN is comprised of units called SpiderConv, which extend convolutional operations from regular grids to irregular point sets that can be embedded in R^n, by parametrizing a family of convolutional filters. We design the filter as a product of a simple step function that captures local geodesic information and a Taylor polynomial that ensures the expressiveness. SpiderCNN inherits the multi-scale hierarchical architecture from classical CNNs, which allows it to extract semantic deep features. Experiments on ModelNet40 demonstrate that SpiderCNN achieves state-of-the-art accuracy 92.4% on standard benchmarks, and shows competitive performance on segmentation task.

Motivation & Objective

• Motivate deep learning directly on irregular 3D point clouds without voxelization or predefined grids.
• Propose SpiderConv as a convolution operator for point sets in R^n with learnable filters.
• Demonstrate that a multi-layer SpiderCNN can achieve high accuracy on 3D classification and segmentation tasks.
• Show that a combination of step-based geodesic information and Taylor expansion yields expressive filters.
• Compare against state-of-the-art methods on ModelNet40 and ShapeNet-Part to establish effectiveness.

Proposed method

• Define SpiderConv as a convolution on point sets with filters g_w supported in a ball around each point.
• Construct g_w as a product of a Step-based component g^Step and a Taylor-based component g^Taylor (g_w = g^Step_w^S * g^Taylor_w^T).
• Use a K-nearest neighbors (KNN) scheme to define locality and approximate Step with a linear mapping for efficiency.
• Parameterize g^Taylor with a order-3 Taylor expansion to capture local geometry (e.g., terms up to x^3, y^3, z^3 and cross terms).
• Train filters w via SGD with backpropagation; compute F ∗ g_w(p) as a sum over neighbors with learned weights.
• Adopt multi-channel, multi-layer SpiderCNN with top-k pooling to form global features for classification and per-point features for segmentation.

Experimental results

Research questions

• RQ1 Can SpiderConv effectively generalize convolution to irregular point clouds without voxelization?
• RQ2 Do parameterized filters combining step/geodesic information and Taylor expansions provide sufficient expressiveness for 3D geometric features?
• RQ3 How does SpiderCNN perform on standard 3D benchmarks for classification and segmentation compared to prior methods?
• RQ4 What architectural choices (K in KNN, number of Taylor terms, pooling strategy) maximize performance on ModelNet40 and ShapeNet-Part?

Key findings

• SpiderCNN with a 4-layer architecture achieves 92.4% accuracy on ModelNet40 when using 1024 points with normals.
• SpiderCNN+PointNet achieves 92.2% on ModelNet40, outperforming either method alone.
• On SHREC15, SpiderCNN (4-layer) reaches 95.8% accuracy, outperforming several baselines.
• In ShapeNet-Part segmentation, SpiderCNN attains a mean IoU of 85.24% across 16 categories, competitive with strong baselines.
• Top-2 pooling preserves more geometric detail than max-pooling, contributing to higher accuracy (92.4% vs 92.0% in 4-layer SpiderCNN).
• Experiments show MLP-based filters underperform compared to the Taylor+Step filter design (Taylor outperforming various MLP configurations).