[Paper Review] CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting

Chaoyun Zhang, Marco Fiore+2 more|arXiv (Cornell University)|Jul 29, 2019

Air Quality Monitoring and Forecasting4 citations

TL;DR

CloudLSTM introduces a novel recurrent neural network architecture for spatiotemporal forecasting on irregular, geospatial point-cloud streams, using a Dynamic Point-cloud Convolution (DConv) operator that enables permutation-invariant, spatially aware feature learning without grid-based preprocessing. It achieves state-of-the-art performance on mobile traffic and air quality forecasting tasks, outperforming 13 benchmark models across multiple metrics without requiring data preprocessing.

ABSTRACT

This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources. We design a Dynamic Point-cloud Convolution (DConv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input. This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -- an important aspect in spatiotemporal predictive learning. The DConv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms. We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e., mobile service traffic forecasting and air quality indicator forecasting. Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of competitor neural network models.

Motivation & Objective

• To address the challenge of forecasting irregular, geospatial point-cloud data streams that lack grid structure and exhibit complex spatial correlations.
• To eliminate the need for preprocessing point-clouds into grids, which distorts spatial relationships and adds computational overhead.
• To develop a flexible, permutation-invariant neural operator that captures dynamic spatial dependencies over time in point-cloud sequences.
• To enable direct, end-to-end learning on raw point-cloud streams by integrating spatiotemporal feature extraction with recurrent architectures.
• To demonstrate superior forecasting performance on real-world applications such as mobile network traffic and urban air quality monitoring.

Proposed method

• Proposes a Dynamic Point-cloud Convolution (DConv) operator that performs spatial feature extraction over local neighborhoods of unordered, irregular point-clouds while preserving permutation invariance.
• DConv dynamically adapts to the spatial configuration of neighboring points at each time step, enabling deformable convolution over point-clouds without requiring fixed grid structures.
• Integrates DConv into standard RNN architectures (LSTM, GRU, RNN) and combines it with sequence-to-sequence learning and attention mechanisms for long-term forecasting.
• The model processes raw input sequences of point coordinates and time-series values, learning spatiotemporal representations directly from the geometric distribution of points.
• Uses a differentiable, learnable aggregation mechanism to compute local spatial features from k-nearest neighbors, with k configurable to balance complexity and performance.
• Employs a strict ablation protocol to isolate contributions of core components, including DConv, RNN type, and attention mechanism.

Experimental results

Research questions

• RQ1 Can a differentiable, permutation-invariant convolutional operator be designed to extract meaningful spatial features from irregular, unordered point-clouds in a spatiotemporal context?
• RQ2 Does direct modeling of point-cloud streams without grid-based preprocessing lead to improved forecasting accuracy compared to existing grid- or graph-based models?
• RQ3 How do architectural choices—such as the RNN variant (LSTM, GRU, RNN) and attention mechanism—affect performance in long-term point-cloud stream forecasting?
• RQ4 To what extent does the number of neighboring points (k) influence the predictive performance of the DConv-based model?
• RQ5 Can the proposed DConv operator be effectively integrated into standard sequence-to-sequence frameworks for end-to-end learning on point-cloud data streams?

Key findings

• CloudLSTM outperforms 13 benchmark models on mobile traffic forecasting, achieving the lowest MAE and RMSE and highest PSNR and SSIM across both urban test scenarios.
• On air quality forecasting, CloudLSTM reduces MAE by up to 12.2% and RMSE by up to 8.8% compared to ConvLSTM, demonstrating significant gains in accuracy.
• The DConv operator contributes more significantly to performance than the RNN architecture or attention mechanism, with the ranking of contributions being: core operator > RNN structure > attention.
• Forecasting performance of CloudLSTM is relatively insensitive to the number of neighbors (k), suggesting that small k values (e.g., k=3) are sufficient and practical.
• The attention mechanism provides a modest but consistent improvement, confirming its value in capturing long-range dependencies in sequence-to-sequence learning.
• CloudLSTM achieves state-of-the-art performance without any data preprocessing, proving the effectiveness of direct point-cloud stream modeling.