[Paper Review] Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Jun‐Young Chung, Çaǧlar Gülçehre+2 more|arXiv (Cornell University)|Dec 11, 2014

Music and Audio Processing16 references10,749 citations

TL;DR

The paper compares LSTM, GRU, and tanh RNN units on sequence modeling tasks, showing that gated units outperform tanh, with GRU often matching or outperforming LSTM depending on the dataset.

ABSTRACT

In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM.

Motivation & Objective

• Motivate the evaluation of gated recurrent units (LSTM and GRU) versus traditional tanh units for sequence modeling.
• Assess whether gating improves performance, convergence speed, and generalization on music and speech datasets.
• Provide fair comparisons by matching model parameter counts across unit types and reporting training dynamics.

Proposed method

• Describe and implement three recurrent units: LSTM, GRU, and tanh RNN."
• Apply on polyphonic music datasets (Nottingham, JSB Chorales, MuseData, Piano-midi) and Ubisoft speech datasets (A and B).
• Use logistic sigmoid outputs for music models and mixture of Gaussians outputs for speech models."
• Control for overfitting by using comparable parameter counts across models and apply RMSProp with weight noise and gradient clipping.
• Evaluate using negative log-likelihood on training and test sets and analyze convergence via learning curves.

Experimental results

Research questions

• RQ1 Do gated units (LSTM and GRU) outperform the traditional tanh unit on sequence modeling tasks?
• RQ2 Between LSTM and GRU, which gated unit yields better performance, convergence, and generalization across datasets?
• RQ3 How do these units compare on polyphonic music modeling versus raw speech signal modeling?
• RQ4 Is GRU competitive with LSTM when model sizes are matched for a fair comparison?

Key findings

• Gated units outperform tanh RNNs on both music and speech datasets.
• GRU often outperforms LSTM on several music datasets and converges faster in training.
• On Ubisoft speech datasets, both LSTM and GRU outperform tanh, with LSTM best on Ubisoft A and GRU best on Ubisoft B."
• Convergence speed (updates and CPU time) is faster for GRU than LSTM on the music datasets, while tanh shows poor progress on Ubisoft datasets due to limited learning.
• The study could not conclusively declare a consistent winner between LSTM and GRU; effectiveness appears dataset-dependent.