[Paper Review] Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition

Haşim Sak, Andrew Senior+1 more|arXiv (Cornell University)|Feb 5, 2014

Speech Recognition and Synthesis15 references859 citations

TL;DR

This paper introduces novel LSTM-based RNN architectures with recurrent and non-recurrent projection layers to improve parameter efficiency and performance in large vocabulary speech recognition. The proposed models achieve state-of-the-art recognition accuracy with faster convergence than DNNs and standard LSTMs, especially on large output-scale tasks (e.g., 8000 context-dependent states).

ABSTRACT

Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully used for sequence labeling and sequence prediction tasks, such as handwriting recognition, language modeling, phonetic labeling of acoustic frames. However, in contrast to the deep neural networks, the use of RNNs in speech recognition has been limited to phone recognition in small scale tasks. In this paper, we present novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition. We train and compare LSTM, RNN and DNN models at various numbers of parameters and configurations. We show that LSTM models converge quickly and give state of the art speech recognition performance for relatively small sized models.

Motivation & Objective

• Address the scalability and parameter efficiency limitations of standard LSTM networks in large vocabulary speech recognition.
• Overcome the vanishing and exploding gradient problems inherent in conventional RNNs during sequence modeling.
• Improve recognition accuracy for large output spaces (e.g., 8000 context-dependent states) where standard LSTMs underperform.
• Demonstrate that LSTM-based models can outperform deep neural networks (DNNs) in large-scale speech recognition tasks.
• Design novel architectural components—recurrent and non-recurrent projection layers—to enhance model flexibility and performance without increasing recurrent connections.

Proposed method

• Introduce a recurrent projection layer between the LSTM layer and the output layer to reduce the number of recurrent parameters while preserving long-term dependencies.
• Add a non-recurrent projection layer after the LSTM layer to increase model capacity without adding recurrent connections, enabling greater flexibility in parameter allocation.
• Use hyperbolic tangent (tanh) activation for cell input and output units, and logistic sigmoid for input, output, and forget gates in the LSTM units.
• Apply peephole connections from memory cells to gates to improve timing precision in output generation.
• Implement connectionist temporal classification (CTC) for end-to-end training on unsegmented sequence data, enabling joint optimization of acoustic and language modeling.
• Use 25ms log-filterbank energy features (40-dimensional) as input, with output labels delayed by 5 frames to improve context awareness.

Experimental results

Research questions

• RQ1 Can LSTM-based RNN architectures with projection layers outperform standard LSTMs and DNNs in large vocabulary speech recognition tasks?
• RQ2 How do recurrent and non-recurrent projection layers affect parameter efficiency and recognition accuracy in large-scale acoustic models?
• RQ3 To what extent do the proposed architectures mitigate vanishing and exploding gradient problems in long sequence modeling?
• RQ4 Does the inclusion of both recurrent and non-recurrent projection layers lead to better performance than using only one type?
• RQ5 Can LSTM models achieve state-of-the-art performance in large vocabulary speech recognition where DNNs have previously dominated?

Key findings

• The proposed LSTM architectures with both recurrent and non-recurrent projection layers significantly outperform standard LSTM and DNN models, especially on large output-scale tasks (e.g., 8000 context-dependent states).
• For the 2000 context-dependent state model, the LSTM_c1024_r256 configuration achieved better word error rate (WER) than the DNN_10w5_5_512_lr256 model, demonstrating superiority over DNNs.
• The LSTM_c512 model with 512 memory cells achieved better performance than the DNN_10w5_2_864_lr256 model, highlighting the importance of depth in DNNs.
• The proposed architectures converged faster and showed more stable training compared to standard RNNs, which suffered from exploding gradients and instability.
• The LSTM network with both recurrent and non-recurrent projection layers (e.g., LSTM_1024_256) achieved higher frame accuracy than the standard LSTM with the same number of parameters (e.g., LSTM_1024).
• The results indicate that depth is crucial for DNNs, as increasing the number of hidden layers from 2 to 5 significantly improved performance, but even deep DNNs were outperformed by the proposed LSTM architectures.