[Paper Review] Neural Speed Reading via Skim-RNN

Minjoon Seo, Sewon Min+2 more|arXiv (Cornell University)|Nov 6, 2017

Topic Modeling19 references54 citations

TL;DR

Skim-RNN is an RNN variant that dynamically skim-feeds unimportant tokens using a smaller RNN, reducing computations while maintaining or improving accuracy, with controllable speed during inference and CPU-friendly latency benefits.

ABSTRACT

Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network (RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant input tokens. Skim-RNN gives computational advantage over an RNN that always updates the entire hidden state. Skim-RNN uses the same input and output interfaces as a standard RNN and can be easily used instead of RNNs in existing models. In our experiments, we show that Skim-RNN can achieve significantly reduced computational cost without losing accuracy compared to standard RNNs across five different natural language tasks. In addition, we demonstrate that the trade-off between accuracy and speed of Skim-RNN can be dynamically controlled during inference time in a stable manner. Our analysis also shows that Skim-RNN running on a single CPU offers lower latency compared to standard RNNs on GPUs.

Motivation & Objective

• Motivate efficient sequence processing by mimicking human speed reading to allocate less computation to less important tokens.
• Develop a two-path RNN (big read vs small skim) that can be used interchangeably with standard RNNs.
• Train the model end-to-end despite non-differentiable hard decisions via differentiable reparameterization.
• Demonstrate reduced floating-point operations and competitive accuracy across multiple NLP tasks.
• Show that skim decisions can be adjusted at inference time for dynamic speed-accuracy trade-offs.

Proposed method

• Introduce Skim-RNN with two RNN cells: a large 'read' cell and a smaller 'skim' cell sharing the same hidden state.
• Make a per-time-step hard decision to skim or read using a differentiable Gumbel-softmax reparameterization of a multinomial over two options.
• If skim is chosen, update only a subset of the hidden state with the small RNN and retain the remaining dimensions from the previous state.
• Train by minimizing the expected loss over decision sequences, using the Gumbel-softmax to enable backpropagation, plus a penalty encouraging skimming.
• Control skim rate and speed by adjusting the decision threshold and the auxiliary skim-penalty term during training.
• Skim-RNN outputs are compatible with standard RNN interfaces, enabling easy replacement of RNNs in existing models.

Experimental results

Research questions

• RQ1 Can Skim-RNN reduce computational cost (Flop-R) while maintaining or improving task accuracy across NLP tasks?
• RQ2 How does the skim decision impact performance compared to skipping tokens, and can skimming outperform skipping in practice?
• RQ3 Is the trade-off between speed and accuracy controllable at inference time without destabilizing training or inference?
• RQ4 Do CPU-based runtimes with Skim-RNN offer latency benefits over GPU-based standard RNNs in common NLP workloads?
• RQ5 How does Skim-RNN perform on text classification and question-answering benchmarks relative to LSTM-Jump and VCRNN?

Key findings

• Skim-RNN achieves substantial reductions in floating-point operations with competitive or improved accuracy compared to standard RNNs and several baselines.
• The skim strategy often yields better accuracy-speed trade-offs than token skipping, indicating benefits from paying attention to unimportant tokens.
• Inference-time control over skim threshold allows dynamic adjustment of speed vs. accuracy without retraining.
• CPU-based Skim-RNN can offer lower latency than GPU-based standard RNNs on several datasets, highlighting practical advantages for CPU-centric deployments.
• Skim-RNN maintains outputs at every time step, enabling easy integration into existing RNN-powered architectures while offering speedups up to several-fold depending on configuration.