[Paper Review] Rethinking Value Function Learning for Generalization in Reinforcement Learning

Seungyong Moon, JunYeong Lee+1 more|arXiv (Cornell University)|Oct 18, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Delayed-Critic Policy Gradient (DCPG), a model-free policy gradient algorithm that improves observational generalization in reinforcement learning by training the value network less frequently than the policy network, reducing memorization and enhancing sample efficiency. It further introduces a unified self-supervised dynamics learning task using a single discriminator to jointly optimize forward and inverse dynamics, achieving state-of-the-art performance on the Procgen benchmark with a 202.2 ± 10.2 PPO-normalized test score.

ABSTRACT

Our work focuses on training RL agents on multiple visually diverse environments to improve observational generalization performance. In prior methods, policy and value networks are separately optimized using a disjoint network architecture to avoid interference and obtain a more accurate value function. We identify that a value network in the multi-environment setting is more challenging to optimize and prone to memorizing the training data than in the conventional single-environment setting. In addition, we find that appropriate regularization on the value network is necessary to improve both training and test performance. To this end, we propose Delayed-Critic Policy Gradient (DCPG), a policy gradient algorithm that implicitly penalizes value estimates by optimizing the value network less frequently with more training data than the policy network. This can be implemented using a single unified network architecture. Furthermore, we introduce a simple self-supervised task that learns the forward and inverse dynamics of environments using a single discriminator, which can be jointly optimized with the value network. Our proposed algorithms significantly improve observational generalization performance and sample efficiency on the Procgen Benchmark.

Motivation & Objective

• To address observational overfitting in multi-environment reinforcement learning, where value networks tend to memorize training data.
• To improve training and test performance in visually diverse environments by reducing value network memorization through regularization.
• To develop a unified, single-network architecture that jointly optimizes policy, value, and dynamics networks without architectural decoupling.
• To enhance sample efficiency and generalization via a self-supervised dynamics learning objective using a single discriminator.
• To demonstrate that delayed value updates and dynamics learning are synergistic for improved performance on the Procgen benchmark.

Proposed method

• Proposes DCPG, a policy gradient algorithm that trains the value network less frequently than the policy network, implicitly penalizing value estimates and reducing memorization.
• Uses a single unified network architecture, avoiding the need for disjoint policy and value networks to prevent interference.
• Introduces a self-supervised task that jointly learns forward and inverse dynamics using a single discriminator, with a combined loss function.
• Employs a dual objective: a clipped surrogate policy objective for policy updates and a delayed value function update to stabilize training.
• Applies a dynamics loss that encourages the discriminator to correctly classify real transitions (st, at, st+1) and counterfactual transitions (st, ˆat, st+1) or (st, at, ˆst+1).
• Uses a delayed critic update schedule where the value network is updated less frequently but with more training data than the policy network.

Experimental results

Research questions

• RQ1 Does reducing the update frequency of the value network mitigate memorization in multi-environment RL?
• RQ2 Can a single unified network architecture effectively train policy, value, and dynamics networks without interference?
• RQ3 Does joint learning of forward and inverse dynamics using a single discriminator improve state representation and generalization?
• RQ4 Is the performance gain from delayed value updates synergistic with self-supervised dynamics learning?
• RQ5 Can DCPG achieve state-of-the-art observational generalization and sample efficiency on the Procgen benchmark?

Key findings

• DCPG achieved a PPO-normalized test score of 184.5 ± 5.2 on the Procgen benchmark, significantly outperforming PPO (160.3 ± 6.3) and PPG (171.7 ± 4.9).
• The variant DDCPG, combining DCPG with joint forward and inverse dynamics learning, achieved a 202.2 ± 10.2 PPO-normalized test score, setting a new SOTA on Procgen.
• The number of out-of-distribution (OOD) actions classified as valid by the forward dynamics discriminator dropped from 7.05 ± 1.47 (DCPG+F) to 0.74 ± 0.75 (DDCPG), showing improved action-aware dynamics understanding.
• Joint dynamics learning with a single discriminator reduced OOD action misclassification compared to using separate discriminators, indicating better use of action information.
• The combination of delayed value updates and dynamics learning produced synergistic improvements, with DDCPG outperforming both DCPG and DPPG in test performance.
• Statistical analysis using RLiable confirmed that DDCPG significantly outperforms PPG across all metrics (median, IQM, mean) with 95% confidence intervals.