[Paper Review] Redeeming Intrinsic Rewards via Constrained Optimization

Eric Chen, Zhang-Wei Hong+2 more|arXiv (Cornell University)|Nov 14, 2022

Reinforcement Learning in Robotics11 citations

TL;DR

This paper proposes Extrinsic-Intrinsic Policy Optimization (EIPO), a constrained optimization framework that dynamically balances intrinsic and extrinsic rewards in reinforcement learning. By formulating exploration as a constrained optimization problem, EIPO automatically suppresses intrinsic rewards when not needed and amplifies them during sparse-exploration phases, achieving consistent performance gains across 61 Atari games without manual hyperparameter tuning.

ABSTRACT

State-of-the-art reinforcement learning (RL) algorithms typically use random sampling (e.g., $ε$-greedy) for exploration, but this method fails on hard exploration tasks like Montezuma's Revenge. To address the challenge of exploration, prior works incentivize exploration by rewarding the agent when it visits novel states. Such intrinsic rewards (also called exploration bonus or curiosity) often lead to excellent performance on hard exploration tasks. However, on easy exploration tasks, the agent gets distracted by intrinsic rewards and performs unnecessary exploration even when sufficient task (also called extrinsic) reward is available. Consequently, such an overly curious agent performs worse than an agent trained with only task reward. Such inconsistency in performance across tasks prevents the widespread use of intrinsic rewards with RL algorithms. We propose a principled constrained optimization procedure called Extrinsic-Intrinsic Policy Optimization (EIPO) that automatically tunes the importance of the intrinsic reward: it suppresses the intrinsic reward when exploration is unnecessary and increases it when exploration is required. The results is superior exploration that does not require manual tuning in balancing the intrinsic reward against the task reward. Consistent performance gains across sixty-one ATARI games validate our claim. The code is available at https://github.com/Improbable-AI/eipo.

Motivation & Objective

• Address the inconsistency of intrinsic rewards in reinforcement learning, which can distract agents on easy tasks despite improving performance on hard exploration tasks.
• Overcome the limitations of fixed hyperparameter λ in combining intrinsic and extrinsic rewards, which often leads to suboptimal performance due to poor trade-off tuning.
• Develop a principled method that automatically adjusts the influence of intrinsic rewards based on the agent's current need for exploration.
• Ensure that the policy optimization remains aligned with the true extrinsic reward objective, avoiding bias from intrinsic reward signals.

Proposed method

• Formulate the RL objective as a constrained optimization problem where the extrinsic return is maximized subject to a constraint on intrinsic reward accumulation.
• Use a Lagrangian relaxation to derive a dual optimization problem that jointly updates the policy and the Lagrange multiplier, enabling adaptive control of intrinsic reward influence.
• Integrate the method with existing intrinsic reward functions such as RND and ICM, making it agnostic to the specific intrinsic reward mechanism.
• Apply the method within a policy gradient framework (e.g., PPO), allowing end-to-end training with both extrinsic and constrained intrinsic objectives.
• Dynamically adjust the importance of intrinsic rewards during training: suppress when extrinsic progress is sufficient, increase when novel state discovery is needed.
• Use a single hyperparameter (α) for the constraint threshold and another (β) for the Lagrange multiplier update, both tuned once per environment and reused across experiments.

Experimental results

Research questions

• RQ1 Can a constrained optimization framework automatically balance intrinsic and extrinsic rewards in RL without manual hyperparameter tuning?
• RQ2 Does EIPO outperform standard intrinsic reward methods and extrinsic-only baselines across diverse Atari environments with varying exploration difficulty?
• RQ3 How does EIPO perform when intrinsic rewards are misaligned with extrinsic rewards, such as in environments with misleading novelty signals?
• RQ4 Can EIPO maintain strong performance on dense-reward environments like MuJoCo, where intrinsic rewards are less critical?
• RQ5 Is the performance gain of EIPO robust across different intrinsic reward mechanisms (e.g., RND vs. ICM)?

Key findings

• EIPO-RND outperforms all baselines, including PPO with extrinsic rewards only and PPO with noisy exploration, on 61 Atari games, with a probability of improvement >0.5 against all baselines.
• In games like Montezuma’s Revenge, where sparse rewards make exploration critical, EIPO achieves significantly higher final scores than extrinsic-only methods.
• On easy exploration tasks such as James Bond, EIPO avoids the performance degradation seen in standard intrinsic reward methods by suppressing unnecessary exploration.
• EIPO-ICM successfully closes the performance gap with PPO in environments where ICM generates misleading intrinsic rewards (e.g., Kaboom), demonstrating robustness to reward misalignment.
• In MuJoCo environments with dense rewards, EIPO matches or exceeds PPO and RND performance, confirming it does not degrade performance in easy settings.
• The method achieves consistent gains without re-tuning hyperparameters per environment, validating its adaptability and generalization across tasks.