[Paper Review] Neural Speed Reading with Structural-Jump-LSTM

Christian Hansen|arXiv (Cornell University)|Mar 20, 2019

Topic Modeling16 citations

TL;DR

This paper proposes Structural-Jump-LSTM, the first neural speed reading model that combines dynamic word-level skipping and punctuation-based jumping to reduce FLOP usage without sacrificing accuracy. It achieves up to 7.0x FLOP reduction on IMDB and DBPedia while maintaining or improving accuracy over vanilla LSTMs.

ABSTRACT

Recurrent neural networks (RNNs) can model natural language by sequentially 'reading' input tokens and outputting a distributed representation of each token. Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance. Efforts to speed up this inference, known as 'neural speed reading', either ignore or skim over part of the input. We present Structural-Jump-LSTM: the first neural speed reading model to both skip and jump text during inference. The model consists of a standard LSTM and two agents: one capable of skipping single words when reading, and one capable of exploiting punctuation structure (sub-sentence separators (,:), sentence end symbols (.!?), or end of text markers) to jump ahead after reading a word. A comprehensive experimental evaluation of our model against all five state-of-the-art neural reading models shows that Structural-Jump-LSTM achieves the best overall floating point operations (FLOP) reduction (hence is faster), while keeping the same accuracy or even improving it compared to a vanilla LSTM that reads the whole text.

Motivation & Objective

• To develop a neural speed reading model that reduces computational cost during inference without degrading model accuracy.
• To enable both word-level skipping and structure-aware jumping in recurrent neural networks for more efficient text processing.
• To dynamically determine skip and jump points based on linguistic structure (e.g., punctuation) rather than fixed budgets or heuristics.
• To outperform existing speed reading models in both FLOP reduction and accuracy retention across diverse NLP benchmarks.

Proposed method

• Introduces two agents: a skip agent that decides whether to skip a word after reading it, and a jump agent that uses punctuation to determine where to jump to next.
• Leverages punctuation markers (e.g., .!?;,:) to define dynamically spaced jumps to the next sub-sentence separator, sentence end, or end of text.
• Uses a standard LSTM as the core sequence encoder, with state updates only for processed words not skipped.
• Employs a reinforcement learning framework to train the skip and jump agents, optimizing for accuracy and FLOP reduction.
• Applies a differentiable Gumbel-Softmax sampling strategy to handle the discrete decisions of skipping and jumping during backpropagation.
• Treats the decision to skip or jump as a stochastic action, allowing end-to-end training with gradient-based optimization.

Experimental results

Research questions

• RQ1 Can a neural speed reading model simultaneously skip individual words and jump over unimportant text segments using structural cues?
• RQ2 Does combining word-level skipping with punctuation-based jumping lead to greater FLOP reduction than existing methods while maintaining or improving accuracy?
• RQ3 How does the performance of Structural-Jump-LSTM compare to state-of-the-art speed reading models in terms of FLOP reduction and accuracy?
• RQ4 Can the model generalize across diverse NLP tasks with varying text complexity and structure?
• RQ5 Does the use of dynamic, structure-based jumping improve training stability and inference efficiency compared to fixed-budget or block-based jumping methods?

Key findings

• Structural-Jump-LSTM achieves up to 7.0x FLOP reduction on the IMDB and DBPedia datasets while maintaining the same or higher accuracy than a vanilla LSTM.
• On the AG News dataset, the model reduces FLOPs by 2.4x and improves accuracy by 0.003 over the vanilla LSTM.
• The model reduces FLOPs by 4.1x on CBT-NE and 3.9x on CBT-CN, with accuracy improvements of 0.010 and 0.007 respectively.
• The model skips 70.7% of words and jumps 19.7% of the text on IMDB, significantly reducing the effective reading length.
• In all benchmarks, Structural-Jump-LSTM outperforms all five state-of-the-art speed reading models in terms of FLOP reduction and accuracy.
• The model demonstrates that combining dynamic skipping and jumping based on linguistic structure leads to superior efficiency and effectiveness compared to fixed-budget or block-based approaches.