[Paper Review] PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing

Hehe Fan, Yi Yang|arXiv (Cornell University)|Oct 18, 2019

3D Shape Modeling and AnalysisEngineering37 references81 citations

TL;DR

PointRNN processes moving 3D point clouds directly by using point coordinates and per-point states; it includes PointGRU and PointLSTM variants and demonstrates strong moving point cloud prediction on synthetic and real driving datasets.

ABSTRACT

In this paper, we introduce a Point Recurrent Neural Network (PointRNN) for moving point cloud processing. At each time step, PointRNN takes point coordinates $\boldsymbol{P} \in \mathbb{R}^{n imes 3}$ and point features $\boldsymbol{X} \in \mathbb{R}^{n imes d}$ as input ($n$ and $d$ denote the number of points and the number of feature channels, respectively). The state of PointRNN is composed of point coordinates $\boldsymbol{P}$ and point states $\boldsymbol{S} \in \mathbb{R}^{n imes d'}$ ($d'$ denotes the number of state channels). Similarly, the output of PointRNN is composed of $\boldsymbol{P}$ and new point features $\boldsymbol{Y} \in \mathbb{R}^{n imes d''}$ ($d''$ denotes the number of new feature channels). Since point clouds are orderless, point features and states from two time steps can not be directly operated. Therefore, a point-based spatiotemporally-local correlation is adopted to aggregate point features and states according to point coordinates. We further propose two variants of PointRNN, i.e., Point Gated Recurrent Unit (PointGRU) and Point Long Short-Term Memory (PointLSTM). We apply PointRNN, PointGRU and PointLSTM to moving point cloud prediction, which aims to predict the future trajectories of points in a set given their history movements. Experimental results show that PointRNN, PointGRU and PointLSTM are able to produce correct predictions on both synthetic and real-world datasets, demonstrating their ability to model point cloud sequences. The code has been released at \url{https://github.com/hehefan/PointRNN}.

Motivation & Objective

• Motivate moving point cloud processing beyond static point cloud analysis.
• Introduce a point-based RNN that preserves per-point structure by leveraging coordinates in both state and output.
• Propose two robust variants (PointGRU and PointLSTM) to address vanishing/exploding gradients in sequence modeling.
• Demonstrate the effectiveness of PointRNN family on synthetic and real-world driving datasets for future trajectory prediction.

Proposed method

• Define PointRNN that takes (P_t, X_t) as input and (P_{t-1}, S_{t-1}) as state, updating (P_t, S_t) and outputting (P_t, Y_t).
• Use a spatiotemporally-local correlation to aggregate X_t and S_{t-1} based on point coordinates via neighborhood queries (k-NN or ball query).
• Replace concatenation-based RNN operations with a point-based rnn function that pools neighbor-aggregated representations.
• Propose PointGRU and PointLSTM by substituting gated mechanisms with point-based rnn operations to mitigate gradient issues.
• Apply seq2seq architectures for moving point cloud prediction, predicting point displacements and thereby 3D scene flow.
• Evaluate with Moving MNIST point clouds and large autonomous driving datasets (Argoverse, nuScenes) using Chamfer Distance (CD) and Earth Mover’s Distance (EMD) as metrics.

Experimental results

Research questions

• RQ1 Can a point-wise recurrent unit effectively model dynamics in moving point clouds without voxelization or global pooling?
• RQ2 Do PointGRU and PointLSTM offer advantages over PointRNN in terms of stability and predictive accuracy for 3D trajectory prediction?
• RQ3 How do point-based locality and geometry preservation contribute to prediction quality on synthetic and real-world data?

Key findings

• PointRNN and its variants successfully predict future trajectories of points in moving point clouds on synthetic Moving MNIST data and real driving datasets.
• On Argoverse, advanced PointRNN achieves CD 0.2789 and EMD 0.8964, outperforming several baselines. On nuScenes, CD 0.0619 and EMD 0.3750 for PointRNN (ball query variant) are reported.
• Advanced architectures with ball query and hierarchical (multi-layer) designs improve accuracy and reduce FLOPs compared to basic models.
• Point-based models require significantly fewer FLOPs than voxel-based approaches, and hierarchical stacking yields notable gains in CD/EMD across settings.
• Ball query generally outperforms k-NN in predictive accuracy on the large driving datasets.