[Paper Review] Fusion Recurrent Neural Network

Yiwen Sun, Yulu Wang+4 more|arXiv (Cornell University)|Jun 7, 2020

Traffic Prediction and Management Techniques47 references4 citations

TL;DR

This paper proposes Fusion Recurrent Neural Network (Fusion RNN), a simplified RNN architecture that replaces traditional gates and memory cells with a Fusion module for multi-round input-hidden state interaction and a Transport module for state propagation. On a large-scale real-world taxi trajectory dataset, Fusion RNN achieves state-of-the-art performance in estimated time of arrival (ETA) prediction, outperforming LSTM and GRU with a simpler structure.

ABSTRACT

Considering deep sequence learning for practical application, two representative RNNs - LSTM and GRU may come to mind first. Nevertheless, is there no chance for other RNNs? Will there be a better RNN in the future? In this work, we propose a novel, succinct and promising RNN - Fusion Recurrent Neural Network (Fusion RNN). Fusion RNN is composed of Fusion module and Transport module every time step. Fusion module realizes the multi-round fusion of the input and hidden state vector. Transport module which mainly refers to simple recurrent network calculate the hidden state and prepare to pass it to the next time step. Furthermore, in order to evaluate Fusion RNN's sequence feature extraction capability, we choose a representative data mining task for sequence data, estimated time of arrival (ETA) and present a novel model based on Fusion RNN. We contrast our method and other variants of RNN for ETA under massive vehicle travel data from DiDi Chuxing. The results demonstrate that for ETA, Fusion RNN is comparable to state-of-the-art LSTM and GRU which are more complicated than Fusion RNN.

Motivation & Objective

• To develop a simpler yet effective RNN architecture that avoids the complexity of gates and memory cells found in LSTM and GRU.
• To evaluate the sequence feature extraction capability of the proposed model on a real-world, large-scale spatiotemporal data mining task.
• To demonstrate that a minimal, non-gated RNN can achieve competitive performance compared to state-of-the-art RNN variants in practical applications.
• To explore the impact of fusion iteration count on model performance and robustness.

Proposed method

• Fusion RNN consists of two core modules: the Fusion module, which performs multi-round interaction between input and hidden state vectors, and the Transport module, which propagates the updated hidden state to the next time step.
• The Fusion module applies a learnable transformation to the concatenation of input and hidden state, followed by a non-linear activation, repeated for r iterations to refine the hidden representation.
• The Transport module uses a simple recurrent structure without gates, ensuring minimal parameter count and computational cost.
• The model is trained end-to-end using backpropagation with Adam optimizer and MAPE loss on a large-scale ETA prediction task from DiDi Chuxing’s 500M+ trajectory dataset.
• The core sequence feature extractor is modular and can be swapped with any RNN for fair comparison, enabling evaluation of Fusion RNN’s intrinsic capability.
• Hyperparameter analysis is conducted on the number of fusion iterations (r ∈ [1,5]) to study its impact on performance and robustness.

Experimental results

Research questions

• RQ1 Can a recurrent neural network without gates or memory cells achieve competitive performance in sequence modeling?
• RQ2 How does multi-round fusion of input and hidden state improve sequence representation compared to gated mechanisms?
• RQ3 What is the optimal number of fusion iterations for effective sequence learning in real-world spatiotemporal data?
• RQ4 Does a simpler RNN architecture outperform complex state-of-the-art models like LSTM and GRU in practical sequence prediction tasks?

Key findings

• Fusion RNN achieves a MAPE of 19.579% on the Beijing 2018 dataset, outperforming LSTM (19.598%) and GRU (19.673%) in terms of MAPE, MAE, and RMSE.
• The model improves upon LSTM by 0.09% in MAPE, 0.38% in MAE, and 0.21% in RMSE, demonstrating superior predictive accuracy with a simpler architecture.
• When r=2, Fusion RNN achieves the best performance, indicating that two rounds of fusion provide optimal balance between complexity and representational power.
• Fusion RNN maintains competitive performance across all values of r from 1 to 5, showing robustness to hyperparameter variation.
• The model outperforms non-deep learning baselines like route-ETA, confirming the effectiveness of data-driven learning on complex transportation systems.
• The results validate that effective information fusion can be achieved without explicit gating mechanisms, challenging the necessity of gates in RNNs.