[Paper Review] Recurrent Dropout Without Memory Loss

Stanislau Semeniuta, Aliaksei Severyn+1 more|arXiv (Cornell University)|Mar 16, 2016

Advanced Neural Network Applications23 references100 citations

TL;DR

Introduces a recurrent dropout method for RNNs that drops update vectors in gated architectures (LSTM/GRU) without losing long-term memory, achieving regularization benefits especially when combined with forward dropout.

ABSTRACT

This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to forward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in recurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for the most popular recurrent networks: vanilla RNNs, Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. Our experiments on three NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.

Motivation & Objective

• Motivate and address overfitting in RNNs, particularly LSTMs/GRUs, by regularizing recurrent connections.
• Propose a recurrent dropout technique that preserves long-term memory.
• Compare the proposed method with existing recurrent dropout approaches.
• Evaluate the method on language modeling, NER, and sentiment tasks across multiple datasets.
• Examine how sampling scheme (per-step vs per-sequence) interacts with recurrent dropout.

Proposed method

• Propose dropout applied to recurrent updates rather than hidden states to avoid memory loss in gated RNNs.
• Formulate dropout on the cell update vector g_t within LSTM/GRU equations while keeping memory paths intact.
• Allow per-step dropout mask sampling and compare to per-sequence sampling across tasks.
• Demonstrate memory-preserving dropout by contrasting with prior recurrent dropout schemes that drop hidden states or cell values.
• Investigate interaction with forward dropout and analyze convergence behavior through learning curves.

Experimental results

Research questions

• RQ1 How can dropout be applied to recurrent connections in LSTMs/GRUs without corrupting long-term memory?
• RQ2 What is the relationship between recurrent dropout and standard forward dropout in RNNs?
• RQ3 Should dropout masks be sampled per-step or per-sequence in recurrent dropout, and how does this affect performance?
• RQ4 Do recurrent dropout methods improve generalization across language modeling, NER, and sentiment tasks?
• RQ5 How does dropping the hidden state update vector compare to dropping hidden states themselves?

Key findings

• Recurrent dropout applied to the hidden state update vector in LSTMs/GRUs preserves memory and yields regularization benefits.
• Per-step dropout sampling generally matches or exceeds per-sequence sampling in effectiveness for the proposed method.
• Coupling recurrent dropout with forward dropout provides additional performance gains across several NLP benchmarks.
• The method improves perplexity in language modeling and F1 scores in NER when combined with forward dropout, and is particularly effective for LSTMs and GRUs.
• Dropout on update vectors avoids the memory-accumulation scaling issue that plagues dropout on hidden states or cell values.