[Paper Review] Dynamic Bottleneck for Robust Self-Supervised Exploration

Chenjia Bai, Lingxiao Wang+5 more|arXiv (Cornell University)|Oct 20, 2021

Reinforcement Learning in Robotics58 references13 citations

TL;DR

This paper proposes Dynamic Bottleneck (DB), a self-supervised exploration method that learns dynamics-relevant representations via the information bottleneck principle to filter out noise. By maximizing mutual information between latent representations and next states while minimizing irrelevant state-action dependencies, DB-bonus measures intrinsic novelty based on information gain, outperforming state-of-the-art methods in noisy Atari environments with sparse rewards.

ABSTRACT

Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamics-irrelevant information, e.g., white-noise. To handle such dynamics-irrelevant information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based on the information-bottleneck principle. Based on the DB model, we further propose DB-bonus, which encourages the agent to explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. We evaluate the proposed method on Atari suits with dynamics-irrelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.

Motivation & Objective

• To address the instability of existing intrinsic curiosity and pseudo-count methods in environments with dynamics-irrelevant noise such as camera noise or random motion.
• To develop a representation learning method that captures only dynamics-relevant features by applying the information bottleneck principle.
• To design a novel intrinsic bonus—DB-bonus—that measures state-action novelty based on information gain in dynamics-relevant representations.
• To establish theoretical links between DB-bonus and UCB in linear MDPs and visiting counts in tabular MDPs.
• To evaluate the robustness and effectiveness of DB-bonus in self-supervised exploration under noisy dynamics, especially in sparse-reward settings like Montezuma’s Revenge.

Proposed method

• The DB model learns a latent representation $ Z_t $ of the state-action pair $ (S_t, A_t) $ using the information bottleneck principle to maximize $ I(Z_t; S_{t+1}) $, the mutual information between representation and next state.
• The model minimizes $ I([S_t, A_t]; Z_t) $, the mutual information between input and representation, to compress dynamics-irrelevant features.
• Contrastive learning is used to optimize the lower bound of $ I(Z_t; S_{t+1}) $, enabling the model to learn disentangled, dynamics-relevant representations.
• The DB-bonus is computed as the information gain of a state-action pair with respect to the learned DB representation, encouraging exploration of high-uncertainty transitions.
• The method is trained end-to-end using a dual objective: representation disentanglement via mutual information minimization and predictive performance via mutual information maximization.
• Theoretical connections are drawn between DB-bonus and UCB in linear MDPs and visiting counts in tabular MDPs, showing its alignment with provably efficient exploration.

Experimental results

Research questions

• RQ1 Can a representation learning method based on the information bottleneck principle effectively filter out dynamics-irrelevant noise in reinforcement learning?
• RQ2 How does the proposed DB-bonus compare to UCB and visiting count in terms of theoretical grounding and empirical performance?
• RQ3 Does the DB-bonus lead to more robust exploration than existing curiosity and pseudo-count methods in environments with injected noise?
• RQ4 Can the DB model learn meaningful, disentangled representations in complex environments like Montezuma’s Revenge despite sparse rewards?
• RQ5 How does the performance of DB-bonus compare to entropy-based exploration methods under noisy and sticky action conditions?

Key findings

• In Random-Box Atari environments with injected dynamics-irrelevant noise, DB-bonus outperforms several state-of-the-art exploration methods, demonstrating robustness to noise.
• The DB-bonus method enables the agent to pass half of the first room in Montezuma’s Revenge under self-supervised learning, where all baselines score zero.
• t-SNE visualization confirms that DB representations form meaningful, semantically clustered structures (e.g., jumping, descending, escaping), while raw states show no such clustering.
• The method shows strong performance in environments with sticky actions, indicating resilience to action perturbations.
• In contrast to entropy-based methods like RE3, which degrade significantly under noise due to increased state entropy, DB-bonus remains effective, highlighting its noise-robustness.
• Spike analysis in Gopher shows that high DB-bonus events correspond to rare, high-impact transitions (e.g., gopher making a hole), which are crucial for high reward acquisition.