[Paper Review] A comparison of LSTM and GRU networks for learning symbolic sequences

Roberto Cahuantzi, Xinye Chen+1 more|arXiv (Cornell University)|Jul 5, 2021

Neural Networks and Applications26 references27 citations

TL;DR

The paper empirically compares LSTM and GRU RNNs on learning symbolic sequences of varying complexity, finding that learning rate and unit count are crucial; GRUs excel on low-complexity sequences while LSTMs excel on high-complexity sequences.

ABSTRACT

We explore the architecture of recurrent neural networks (RNNs) by studying the complexity of string sequences it is able to memorize. Symbolic sequences of different complexity are generated to simulate RNN training and study parameter configurations with a view to the network's capability of learning and inference. We compare Long Short-Term Memory (LSTM) networks and gated recurrent units (GRUs). We find that an increase in RNN depth does not necessarily result in better memorization capability when the training time is constrained. Our results also indicate that the learning rate and the number of units per layer are among the most important hyper-parameters to be tuned. Generally, GRUs outperform LSTM networks on low-complexity sequences while on high-complexity sequences LSTMs perform better.

Motivation & Objective

• Investigate how RNN architectures memorize symbolic sequences of varying complexity.
• Assess the impact of hyper-parameters (learning rate, number of layers, units) on memorization performance.
• Compare LSTM and GRU performance in low- and high-complexity sequence tasks.
• Provide insights to guide hyper-parameter tuning for symbolic sequence learning tasks.

Proposed method

• Generate seed strings with controllable complexity using LZW-based complexity as a proxy for Kolmogorov complexity.
• Encode sequences with one-hot vectors and train RNNs to predict the next symbol in sliding windows.
• Compare LSTM and GRU units with varying layer counts and unit totals under different stopping criteria.
• Use Adam optimizer with specified learning rates and stop training when accuracy or loss criteria are met.
• Evaluate forecast accuracy using text similarity metrics (Damerau–Levenshtein and Jaro–Winkler) between predicted and validation strings.
• Provide publicly available code and libraries to reproduce experiments.

Experimental results

Research questions

• RQ1 How does the learning rate influence training efficiency and memorization accuracy for LSTM and GRU on symbolic sequences?
• RQ2 What is the effect of depth (number of layers) on memorization performance and training time for LSTM and GRU?
• RQ3 Do GRUs outperform LSTMs on low-complexity symbolic sequences and does this relationship invert for high-complexity sequences?
• RQ4 How do unit counts per layer affect performance and training time for both architectures across sequence complexities?

Key findings

• Learning rate around 0.01 yields the best training time across complexities.
• Single-layer RNNs with moderate units (around 100) often suffice for the tasks studied.
• GRUs outperform LSTMs on low-complexity sequences, while LSTMs outperform GRUs on high-complexity sequences.
• Increasing depth generally increases training time without clear gains in memorization accuracy.
• LSTMs are faster to train than GRUs on high-complexity sequences, while GRUs train faster on low-complexity sequences.
• Both architectures achieve high accuracy overall, but performance gaps emerge with varying sequence complexity.