[Paper Review] DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction

Aviral Kumar, Abhishek Gupta+1 more|arXiv (Cornell University)|Mar 16, 2020

Reinforcement Learning in RoboticsComputer Science51 references36 citations

TL;DR

The paper identifies the absence of corrective feedback in bootstrapped RL methods and proposes DisCor, a distribution-correction re-weighting strategy, to improve stability and performance especially in multi-task and noisy-reward settings.

ABSTRACT

Deep reinforcement learning can learn effective policies for a wide range of tasks, but is notoriously difficult to use due to instability and sensitivity to hyperparameters. The reasons for this remain unclear. When using standard supervised methods (e.g., for bandits), on-policy data collection provides "hard negatives" that correct the model in precisely those states and actions that the policy is likely to visit. We call this phenomenon "corrective feedback." We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from this corrective feedback, and training on the experience collected by the algorithm is not sufficient to correct errors in the Q-function. In fact, Q-learning and related methods can exhibit pathological interactions between the distribution of experience collected by the agent and the policy induced by training on that experience, leading to potential instability, sub-optimal convergence, and poor results when learning from noisy, sparse or delayed rewards. We demonstrate the existence of this problem, both theoretically and empirically. We then show that a specific correction to the data distribution can mitigate this issue. Based on these observations, we propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training, resulting in substantial improvements in a range of challenging RL settings, such as multi-task learning and learning from noisy reward signals. Blog post presenting a summary of this work is available at: https://bair.berkeley.edu/blog/2020/03/16/discor/.

Motivation & Objective

• Investigate why bootstrapped value targets fail to benefit from corrective feedback in ADP-based RL.
• Theoretically and empirically demonstrate instability and sub-optimal convergence due to data-distribution–value-function interactions.
• Develop a practical data-distribution correction method to restore corrective feedback and stabilize learning.
• Show that DisCor improves performance, particularly in multi-task and noisy-reward scenarios.

Proposed method

• Analyze corrective feedback concepts using a bandit-like intuition and formal definitions.
• Derive an optimal data distribution p_k that maximizes corrective feedback under Bellman updates.
• Propose tractable surrogates for Q*-related quantities and re-weight replay-buffer samples using importance weights.
• Introduce a practical weight function w_k(s,a) proportional to exp(-gamma [P^{pi_{k-1}} Δ_{k-1}](s,a)/tau).
• Train a secondary model Δ_phi to estimate bootstrap/backup error Δ_k for weighting and error modeling.
• Provide algorithm DisCor that integrates weighted Bellman backups with a secondary Δ model on top of standard DQN/SAC frameworks.

Experimental results

Research questions

• RQ1 What mechanisms cause corrective feedback to be absent in bootstrapped RL methods?
• RQ2 How can the data distribution be corrected to maximize corrective feedback during training?
• RQ3 Does re-weighting transitions by an optimal distribution improve stability and performance in practice?
• RQ4 How does DisCor perform in challenging settings like multi-task RL and learning from noisy rewards?

Key findings

• Corrective feedback can be absent in ADP methods, leading to suboptimal convergence and instability even with replay buffers.
• An optimal training distribution p_k assigns higher probability to high-Bellman-error regions while considering proximity to Q*, mitigated via tractable surrogates.
• Re-weighting replay-buffer transitions with weights w_k based on estimated corrective potential reduces error accumulation and stabilizes learning.
• DisCor improves performance in challenging settings, notably achieving around 50% higher final success on MT10 multi-task benchmarks compared to SAC (under the reported results).
• The approach is compatible with standard ADP-based deep RL algorithms such as DQN and SAC, and supports learning from noisy reward signals and multi-task scenarios.