[论文解读] Deep Recurrent Neural Networks for Time Series Prediction

Sharat C. Prasad, P. Vara Prasad|arXiv (Cornell University)|Jul 22, 2014

EEG and Brain-Computer Interfaces参考文献 12被引用 26

一句话总结

本文提出一种具有扩展时间反向传播的深度循环神经网络（RNN），用于建模时间序列中的高阶时序动态，实现端到端的特征提取与预测。该方法在癫痫发作预测中实现了超过99%的平均检测率，通过更深的网络结构和动态规划训练，展现出更优的泛化能力与误差降低效果。

ABSTRACT

Ability of deep networks to extract high level features and of recurrent networks to perform time-series inference have been studied. In view of universality of one hidden layer network at approximating functions under weak constraints, the benefit of multiple layers is to enlarge the space of dynamical systems approximated or, given the space, reduce the number of units required for a certain error. Traditionally shallow networks with manually engineered features are used, back-propagation extent is limited to one and attempt to choose a large number of hidden units to satisfy the Markov condition is made. In case of Markov models, it has been shown that many systems need to be modeled as higher order. In the present work, we present deep recurrent networks with longer backpropagation through time extent as a solution to modeling systems that are high order and to predicting ahead. We study epileptic seizure suppression electro-stimulator. Extraction of manually engineered complex features and prediction employing them has not allowed small low-power implementations as, to avoid possibility of surgery, extraction of any features that may be required has to be included. In this solution, a recurrent neural network performs both feature extraction and prediction. We prove analytically that adding hidden layers or increasing backpropagation extent increases the rate of decrease of approximation error. A Dynamic Programming (DP) training procedure employing matrix operations is derived. DP and use of matrix operations makes the procedure efficient particularly when using data-parallel computing. The simulation studies show the geometry of the parameter space, that the network learns the temporal structure, that parameters converge while model output displays same dynamic behavior as the system and greater than .99 Average Detection Rate on all real seizure data tried.

研究动机与目标

• 为解决浅层网络与马尔可夫模型在捕捉时间序列中高阶时序依赖关系方面的局限性。
• 在单一深度循环架构中实现时序特征与预测的端到端学习。
• 通过利用更深的网络结构，减少实现精确逼近所需的神经元数量。
• 通过动态规划与矩阵运算，开发一种高效的训练方法，实现可扩展的学习。
• 在真实生物医学时间序列中，特别是癫痫发作预测中，验证深度RNN的有效性。

提出的方法

• 所提出的模型采用具有多层隐藏层的深度循环神经网络，以建模浅层网络无法捕捉的复杂时序动态。
• 将时间反向传播扩展至超过一步，以捕捉长程依赖关系，从而改善高阶马尔可夫过程的建模。
• 基于动态规划推导出一种训练算法，利用矩阵运算高效计算长序列中的梯度。
• 该方法支持数据并行计算，提升了大规模时间序列数据训练的效率。
• 网络同时执行特征提取与预测，无需人工设计特征。
• 理论分析证明，增加网络深度或扩展反向传播范围可加速近似误差减少的速率。

实验结果

研究问题

• RQ1 具有扩展时间反向传播的深度循环网络是否能在建模高阶时序依赖关系方面优于浅层网络？
• RQ2 增加网络深度或反向传播长度在多大程度上可减少时间序列建模中的近似误差？
• RQ3 深度RNN中的端到端学习是否可消除生物医学时间序列预测中对手动特征工程的需求？
• RQ4 所提出的基于动态规划的训练方法在长序列中如何提升效率与可扩展性？
• RQ5 与现有方法相比，该深度RNN在癫痫发作预测中的实际性能如何？

主要发现

• 该深度RNN模型在所有测试的真实癫痫发作数据上均实现了超过99%的平均检测率，表现出极高的预测准确性。
• 理论分析证实，增加隐藏层或扩展反向传播范围可加速近似误差减少的速率。
• 仿真研究结果表明，该网络成功学习并复现了底层时序系统的动态行为。
• 训练过程中观察到参数收敛，表明在复杂参数空间中优化过程稳定。
• 结合矩阵运算的动态规划方法实现了高效训练，尤其在利用数据并行计算架构时优势显著。
• 该模型消除了对手动特征工程的需求，支持紧凑、低功耗的实现，适用于植入式设备。