[Paper Review] Recurrent Slice Networks for 3D Segmentation of Point Clouds

Qiangui Huang, Weiyue Wang+1 more|arXiv (Cornell University)|Feb 13, 2018

3D Shape Modeling and AnalysisEngineering7 references60 citations

TL;DR

RSNet introduces a lightweight local dependency module for point-cloud segmentation by projecting unordered points into an ordered slice sequence, applying bidirectional RNNs, and unpooling back to points; it achieves state-of-the-art results on S3DIS, ScanNet, and ShapeNet with efficient computation.

ABSTRACT

Point clouds are an efficient data format for 3D data. However, existing 3D segmentation methods for point clouds either do not model local dependencies \cite{pointnet} or require added computations \cite{kd-net,pointnet2}. This work presents a novel 3D segmentation framework, RSNet\footnote{Codes are released here https://github.com/qianguih/RSNet}, to efficiently model local structures in point clouds. The key component of the RSNet is a lightweight local dependency module. It is a combination of a novel slice pooling layer, Recurrent Neural Network (RNN) layers, and a slice unpooling layer. The slice pooling layer is designed to project features of unordered points onto an ordered sequence of feature vectors so that traditional end-to-end learning algorithms (RNNs) can be applied. The performance of RSNet is validated by comprehensive experiments on the S3DIS\cite{stanford}, ScanNet\cite{scannet}, and ShapeNet \cite{shapenet} datasets. In its simplest form, RSNets surpass all previous state-of-the-art methods on these benchmarks. And comparisons against previous state-of-the-art methods \cite{pointnet, pointnet2} demonstrate the efficiency of RSNets.

Motivation & Objective

• Motivate direct 3D segmentation on raw point clouds without voxelization or multi-view projections.
• Model local geometric dependencies efficiently to improve segmentation accuracy.
• Propose a lightweight local dependency module that preserves efficiency in time and memory.
• Validate RSNet on large-scale real-world and synthetic datasets to demonstrate state-of-the-art performance and efficiency.

Proposed method

• Introduce slice pooling to project unordered points to an ordered sequence of slice-level features.
• Apply a stack of bidirectional RNNs to model dependencies across slices.
• Use slice unpooling to map updated slice features back to individual points.
• Process three slicing directions (x, y, z) to capture anisotropic local context.
• Maintain linear time complexity O(n) with respect to the number of input points and O(1) w.r.t the slicing resolution r.
• Train with a baseline RSNet architecture featuring 1x1 convolutions for independent feature extraction and GRU-based RNNs by default (with ablations for LSTM/vanilla RNN).

Experimental results

Research questions

• RQ1 How can local geometric dependencies in point clouds be modeled efficiently without voxelization or dense 3D convolutions?
• RQ2 What is the impact of slice partitioning (resolution r) and block size on segmentation accuracy and computation?
• RQ3 Do slice pooling and slice unpooling enable effective use of RNNs to capture context across neighboring slices?
• RQ4 How does RSNet compare to state-of-the-art point-cloud segmentation methods in accuracy and efficiency on S3DIS, ScanNet, and ShapeNet?

Key findings

• RSNet achieves state-of-the-art performance on S3DIS and ScanNet without data augmentation, outperforming prior voxel-based and other point-cloud methods.
• RSNet consistently improves mean IoU and mean accuracy over previous methods on large-scale indoor scenes and synthetic data.
• The local dependency module (slice pooling + bidirectional RNNs + slice unpooling) is crucial for capturing local geometry and provides significant gains over PointNet-like baselines.
• RSNet offers favorable inference speed and memory usage, outperforming several PointNet++ configurations in speed and consuming less memory than traditional baselines.
• Ablation studies show a 2–3 cm slicing resolution (along all axes) with a 1m block size yields optimal performance on S3DIS, and GRU units outperform vanilla RNNs and LSTM in this setup.