[Paper Review] State Entropy Maximization with Random Encoders for Efficient Exploration

Younggyo Seo, Lili Chen+4 more|arXiv (Cornell University)|Feb 18, 2021

Reinforcement Learning in Robotics58 references22 citations

TL;DR

This paper proposes RE3, a compute-efficient exploration method that maximizes state entropy using a fixed random encoder to extract low-dimensional representations, enabling stable and scalable entropy estimation in high-dimensional observation spaces. RE3 significantly improves sample-efficiency in both model-free and model-based RL across DeepMind Control Suite, MiniGrid, and Atari benchmarks, outperforming methods that require learned representations or auxiliary models.

ABSTRACT

Recent exploration methods have proven to be a recipe for improving sample-efficiency in deep reinforcement learning (RL). However, efficient exploration in high-dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3), an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute-efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample-efficiency of both model-free and model-based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample-efficiency in downstream tasks. Source code and videos are available at https://sites.google.com/view/re3-rl.

Motivation & Objective

• To address the challenge of efficient exploration in high-dimensional observation spaces common in deep reinforcement learning.
• To enable stable and compute-efficient state entropy estimation without requiring representation learning or auxiliary models.
• To investigate whether randomly initialized encoders can provide meaningful representations for state entropy estimation.
• To improve sample-efficiency in both model-free and model-based RL algorithms through intrinsic reward shaping based on state entropy.
• To enable effective pre-training for downstream tasks by encouraging diverse behavior in sparse-reward environments.

Proposed method

• Utilizes a k-nearest neighbor (k-NN) entropy estimator in the latent space of a randomly initialized convolutional encoder to estimate state entropy.
• Fixes the encoder weights throughout training, eliminating the need for gradient updates or representation learning objectives.
• Employs the k-NN estimator on the fixed latent representations to compute intrinsic rewards proportional to state entropy.
• Integrates the intrinsic reward into standard RL algorithms (e.g., A2C, SAC) without modifying the policy or value function learning process.
• Leverages the inductive bias of convolutional architectures to ensure meaningful state similarity representations even without training.
• Applies the method across model-free (e.g., A2C, SAC), model-based (e.g., Dreamer), and on-policy settings with minimal hyperparameter tuning.

Experimental results

Research questions

• RQ1 Can a fixed, randomly initialized encoder provide a stable and meaningful representation space for state entropy estimation in high-dimensional environments?
• RQ2 Does maximizing state entropy via k-NN estimation in a fixed random encoder space improve sample-efficiency in deep RL?
• RQ3 How does RE3 compare to state-of-the-art exploration methods that rely on learned representations or auxiliary models?
• RQ4 Can RE3 enable effective pre-training for downstream tasks by encouraging diverse behavior in sparse-reward environments?
• RQ5 Is the method scalable and effective in visually complex environments such as Atari games?

Key findings

• RE3 achieves an average episode return of 0.49 at 2.4 million environment steps in the DoorKey-8x8 MiniGrid environment, outperforming A2C + ICM (0.20) and A2C + RND, which fail to learn non-trivial policies.
• In the MiniGrid Empty-16x16 environment, RE3 enables pre-training that improves downstream fine-tuning performance, with A2C + RE3 + PT showing superior sample-efficiency on DoorKey tasks.
• On Atari games, RE3 improves sample-efficiency over Rainbow and Rainbow + SE with contrastive learning, particularly on hard exploration games like Montezuma’s Revenge.
• RE3 outperforms methods requiring representation learning (e.g., contrastive learning) in MiniGrid, where data augmentation is incompatible with compact state embeddings.
• The method is computationally efficient, requiring no gradient updates for the encoder, making it scalable and practical for integration into existing RL pipelines.
• RE3 enables learning diverse behaviors without extrinsic rewards, demonstrating effective intrinsic exploration in large, empty state spaces.