[论文解读] Traffic Prediction Based on Random Connectivity in Deep Learning with Long Short-Term Memory

Yuxiu Hua, Zhifeng Zhao+4 more|arXiv (Cornell University)|Nov 8, 2017

Traffic Prediction and Management Techniques参考文献 9被引用 4

一句话总结

本文提出随机连接长短期记忆网络（RCLSTM），一种稀疏连接的长短期记忆网络变体，通过随机连接神经元而非使用全连接结构来降低计算成本。尽管连接数减少35%，RCLSTM在长输入序列上的预测精度仍与传统LSTM相当或更优，表明稀疏性不会损害性能，甚至在特定条件下可提升性能。

ABSTRACT

Traffic prediction plays an important role in evaluating the performance of telecommunication networks and attracts intense research interests. A significant number of algorithms and models have been put forward to analyse traffic data and make prediction. In the recent big data era, deep learning has been exploited to mine the profound information hidden in the data. In particular, Long Short-Term Memory (LSTM), one kind of Recurrent Neural Network (RNN) schemes, has attracted a lot of attentions due to its capability of processing the long-range dependency embedded in the sequential traffic data. However, LSTM has considerable computational cost, which can not be tolerated in tasks with stringent latency requirement. In this paper, we propose a deep learning model based on LSTM, called Random Connectivity LSTM (RCLSTM). Compared to the conventional LSTM, RCLSTM makes a notable breakthrough in the formation of neural network, which is that the neurons are connected in a stochastic manner rather than full connected. So, the RCLSTM, with certain intrinsic sparsity, have many neural connections absent (distinguished from the full connectivity) and which leads to the reduction of the parameters to be trained and the computational cost. We apply the RCLSTM to predict traffic and validate that the RCLSTM with even 35% neural connectivity still shows a satisfactory performance. When we gradually add training samples, the performance of RCLSTM becomes increasingly closer to the baseline LSTM. Moreover, for the input traffic sequences of enough length, the RCLSTM exhibits even superior prediction accuracy than the baseline LSTM.

研究动机与目标

• 为解决传统LSTM在电信网络实时交通预测中的高计算成本问题。
• 探究LSTM中的稀疏连接是否能在降低模型复杂度的同时维持或提升预测性能。
• 评估神经连接密度和输入序列长度变化对预测精度的影响。
• 验证RCLSTM在减少参数量和训练开销的前提下，在真实交通数据预测中的有效性。

提出的方法

• RCLSTM用随机稀疏连接替代LSTM层中的全连接结构，其中每个神经元连接以固定概率建立。
• 模型采用三层堆叠架构处理序列化交通数据，以保留长程时间依赖性。
• 通过均匀随机分布形成神经连接，引入内在稀疏性，从而减少可训练参数数量。
• 使用标准损失函数（包括MSE和MAE）进行反向传播时间训练，并用于评估。
• 输入交通序列经过归一化处理并划分为训练集和测试集，采用可变长度滑动窗口以评估长期依赖学习能力。
• 性能通过MSE和MAE进行评估，并对连接比例和训练样本大小进行消融研究。

实验结果

研究问题

• RQ1 随机连接的LSTM架构是否能在显著降低计算成本的同时维持高预测精度？
• RQ2 RCLSTM实现与全连接LSTM相当性能所需的最低神经连接密度是多少？
• RQ3 输入交通序列长度如何影响RCLSTM相较于传统LSTM的预测精度？
• RQ4 当具备充足训练数据和长输入序列时，RCLSTM是否优于传统LSTM？

主要发现

• 仅使用35%神经连接的RCLSTM即可实现与全连接LSTM相当的预测精度，表明稀疏性不会降低性能。
• 随着训练样本量增加，35%连接的RCLSTM性能逐渐逼近传统LSTM，初始差距得以缩小。
• 在足够长的输入序列下，RCLSTM在MSE和MAE指标上均优于传统LSTM，展现出更强的长期依赖建模能力。
• 50%连接的RCLSTM性能与全连接LSTM相近，表明效率与精度之间存在显著权衡。
• 随着输入序列长度增加，RCLSTM的预测误差（MSE）波动更小且始终低于传统LSTM，表明其对长序列具有更强的鲁棒性。