[Paper Review] Long Short-Term Memory-Networks for Machine Reading

Jianpeng Cheng, Li Dong+1 more|arXiv (Cornell University)|Jan 25, 2016

Topic Modeling60 references191 citations

TL;DR

Introduces LSTMN, an LSTM augmented with an internal memory network and intra-attention to jointly memorize and relate tokens, improving language modeling, sentiment analysis, and natural language inference.

ABSTRACT

In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention. The reader extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell. This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens. The system is initially designed to process a single sequence but we also demonstrate how to integrate it with an encoder-decoder architecture. Experiments on language modeling, sentiment analysis, and natural language inference show that our model matches or outperforms the state of the art.

Motivation & Objective

• Develop a machine reading simulator that processes text incrementally and can perform shallow reasoning with memory and attention.
• Address memory compression and lack of structure handling in standard sequence models by integrating a memory network inside recurrence.
• Enable adaptive memory usage and relation discovery among tokens through neural attention within an LSTM framework.
• Demonstrate end-to-end training on multiple NLP tasks to match or surpass state-of-the-art models.

Proposed method

• Replace the LSTM memory cell with a memory network that stores contextual representations per input token.
• Use an attention mechanism to link the current token to past memories and compute adaptive summaries for memory and hidden states.
• Compute LSTM-like gating with the adaptive memory contents to update c_t and h_t (Equations 7–9).
• Optionally stack memory/hidden layers to form multi-hop or deep fusion variants for sequence-to-sequence tasks (Equations 10–16).
• Integrate intra-attention (within a sequence) and inter-attention (between sequences) for encoder–decoder setups (shallow and deep fusion, Equations 11–16).
• Train end-to-end on tasks like language modeling, sentiment analysis, and natural language inference, comparing to standard LSTMs and baselines.

Experimental results

Research questions

• RQ1 Can an LSTM enhanced with an internal memory network and attention better capture relations among tokens and handle longer sequences?
• RQ2 Does intra-attention over token memories improve representations for single-sequence tasks like language modeling?
• RQ3 Can LSTMN architectures be effectively combined with encoder–decoder models for two-sequence tasks such as translation or natural language inference?
• RQ4 Do LSTMN variants (single-layer vs multi-layer, shallow vs deep fusion) outperform traditional LSTM variants on standard NLP benchmarks?

Key findings

• Single-layer LSTMN achieves lower perplexity than KN5, RNN, and LSTM baselines on Penn Treebank.
• Three-layer LSTMN achieves the best perplexity among the tested deep architectures (102 for LSTMN 3).
• LSTMN outperforms standard LSTM baselines on sentiment classification, approaching state-of-the-art results.
• In SNLI-style natural language inference, LSTMN variants with shallow or deep fusion achieve competitive accuracy, with deep fusion yielding state-of-the-art performance among parameter-maired comparisons.
• Attention visualizations show the model learns meaningful, though undirected, lexical relations such as sits–at, everyone–is, and is–watching.
• The model demonstrates strong performance across language modeling, sentiment analysis, and natural language inference, validating the effectiveness of internal memory and intra-attentive reasoning.