[Paper Review] Multi-cell LSTM Based Neural Language Model

Thomas Cherian, Akshay Badola+1 more|arXiv (Cornell University)|Nov 15, 2018

Topic Modeling11 references4 citations

TL;DR

This paper proposes a multi-cell LSTM architecture that enhances standard LSTMs by incorporating multiple memory cells per node, using selection strategies like max or average to output a single value. The method achieves state-of-the-art performance on the Penn Treebank dataset, outperforming Zaremba et al.'s models with faster convergence and lower perplexity.

ABSTRACT

Language models, being at the heart of many NLP problems, are always of great interest to researchers. Neural language models come with the advantage of distributed representations and long range contexts. With its particular dynamics that allow the cycling of information within the network, `Recurrent neural network' (RNN) becomes an ideal paradigm for neural language modeling. Long Short-Term Memory (LSTM) architecture solves the inadequacies of the standard RNN in modeling long-range contexts. In spite of a plethora of RNN variants, possibility to add multiple memory cells in LSTM nodes was seldom explored. Here we propose a multi-cell node architecture for LSTMs and study its applicability for neural language modeling. The proposed multi-cell LSTM language models outperform the state-of-the-art results on well-known Penn Treebank (PTB) setup.

Motivation & Objective

• To explore the potential of multiple memory cells within a single LSTM node to improve long-range context modeling.
• To investigate effective selection mechanisms—such as max or average—for combining multiple memory cell values into a single output.
• To evaluate the performance of multi-cell LSTM on neural language modeling tasks, particularly on the Penn Treebank benchmark.
• To compare the proposed model against state-of-the-art LSTM-based language models, focusing on perplexity and training efficiency.
• To analyze the impact of architectural hyperparameters like embedding size and number of hidden layers on model performance.

Proposed method

• Proposes a multi-cell LSTM node architecture where each LSTM unit contains multiple memory cells instead of a single cell.
• Employs a selection mechanism (e.g., max or average) to combine values from multiple memory cells into a single output per node.
• Maintains standard RNN dynamics and backpropagation through time (BPTT) for training, ensuring compatibility with existing frameworks.
• Uses learning rate annealing and weight decay for regularization, with experiments on max-norm regularization showing no significant improvement.
• Trains models with varying hidden layer sizes, embedding dimensions, and number of layers to evaluate architectural impact.
• Employs cross-entropy loss with softmax output layer for language modeling, predicting the next word in a sequence.

Experimental results

Research questions

• RQ1 Can the integration of multiple memory cells per LSTM node improve language modeling performance compared to standard single-cell LSTMs?
• RQ2 Which selection strategy—maximum, average, or others—yields the best performance when combining multiple memory cell values?
• RQ3 How does the number of memory cells per node affect model convergence and perplexity on the Penn Treebank dataset?
• RQ4 Does the multi-cell LSTM architecture generalize well across different model sizes (small, medium, large) and hyperparameter settings?
• RQ5 Can the proposed model achieve lower perplexity and faster convergence than state-of-the-art models like Zaremba et al. (2014)?

Key findings

• The multi-cell LSTM model (MLSTM-LM) outperforms the state-of-the-art Zaremba et al. (2014) model on the Penn Treebank dataset, achieving a test perplexity of 77.12 with a two-hidden-layer architecture.
• The best-performing model (large MLSTM-LM with 1500 units and 1500-dimensional embeddings) achieved a test perplexity of 77.12, significantly lower than the baseline model.
• All MLSTM-LM models converged earlier than Zaremba et al.'s models, with saturation occurring around epoch 30 compared to epoch 55 for the baseline.
• The optimal performance was achieved with two hidden layers; performance degraded with four hidden layers, suggesting a need for better regularization in deeper architectures.
• Best results were obtained when the embedding size matched the number of hidden units (e.g., 1500 for 1500-unit models), with performance deteriorating when sizes were mismatched.
• Max-norm regularization and weight decay did not yield significant improvements, indicating limited benefit for this architecture under the tested settings.