[Paper Review] On Predictive Information in RNNs

Zhe Dong, Deniz Oktay+2 more|arXiv (Cornell University)|Oct 21, 2019

Neural Networks and Applications26 references4 citations

TL;DR

This paper investigates whether recurrent neural networks (RNNs) optimally compress past information while retaining predictive power for future sequences, finding that standard RNNs over-extract irrelevant past information. By injecting noise into the hidden state to constrain information, the authors improve RNN performance in terms of predictive accuracy, likelihood, and downstream classification—especially in low-data regimes—demonstrating that information bottleneck regularization enhances model efficiency and generalization.

ABSTRACT

Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. Empirically, we find that RNNs are suboptimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. We show that constraining past information by injecting noise into the hidden state can improve RNNs in several ways: optimality in the predictive information plane, sample quality, heldout likelihood, and downstream classification performance.

Motivation & Objective

• To determine whether RNNs trained with maximum likelihood optimally compress past inputs while retaining predictive information about the future, as seen in biological neurons.
• To evaluate the accuracy of mutual information estimation in RNNs and optimal models using held-out sets.
• To investigate whether constraining hidden state information via noise injection improves RNN performance on predictive and downstream tasks.
• To explore the trade-off between past information retention and future predictability in the information plane.
• To assess the impact of information constraints on sample quality, held-out likelihood, and classification accuracy in real-world sketch datasets.

Proposed method

• Measures predictive information as the mutual information between past and future sequence segments using variational mutual information estimators.
• Employs a noise injection strategy into the RNN hidden state to constrain the amount of past information retained, effectively regularizing the representation.
• Uses the Information Bottleneck Lagrangian to trace the optimal trade-off frontier between past and future information in the information plane.
• Trains class-conditional RNNs on the QuickDraw sketch dataset with varying dataset sizes and noise levels to evaluate generalization and performance.
• Evaluates model performance via held-out log-likelihood and downstream classification accuracy using a Naïve Bayes classifier.
• Validates mutual information estimates using held-out sets and compares RNNs against analytically derived optimal models for Gaussian processes.

Experimental results

Research questions

• RQ1 Do RNNs trained with maximum likelihood achieve optimal compression of past inputs while remaining maximally informative about the future?
• RQ2 How does noise injection into the RNN hidden state affect the balance between past information retention and future predictability?
• RQ3 To what extent does information constraint improve held-out likelihood and sample quality in low-data regimes?
• RQ4 Does limiting past information improve downstream classification performance on real-world sequential data?
• RQ5 How do different training objectives and architectural choices influence the position of RNNs in the information plane?

Key findings

• RNNs trained with maximum likelihood are suboptimal in the information plane, extracting more past information than necessary for future prediction.
• Injecting noise into the RNN hidden state improves held-out log-likelihood on small sketch datasets (e.g., 100 examples per class), indicating better generalization.
• Information constraint enhances downstream classification accuracy, particularly in low-data regimes, with improvements observed across 11 sketch classes.
• Even with up to 5,000 examples per class, RNNs can be compressed significantly (via noise injection) without loss in held-out likelihood, showing robustness to information reduction.
• The optimal trade-off between past and future information is not achieved by standard RNNs, but can be approached through explicit information bottleneck regularization.
• The study validates that mutual information estimation using held-out sets is critical for accurate assessment of information-theoretic performance in RNNs.