[Paper Review] Improved Corruption Robust Algorithms for Episodic Reinforcement Learning

Yifang Chen, Simon S. Du+1 more|arXiv (Cornell University)|Feb 13, 2021

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper proposes novel corruption-robust algorithms for episodic reinforcement learning that achieve strictly improved regret bounds by depending on fine-grained corruption magnitudes (reward and transition corruptions) rather than just the number of corrupted episodes. It introduces a reward-free exploration subroutine and two meta-algorithms that achieve additive dependence on corruption levels, marking the first such result in tabular RL with provable optimality.

ABSTRACT

We study episodic reinforcement learning under unknown adversarial corruptions in both the rewards and the transition probabilities of the underlying system. We propose new algorithms which, compared to the existing results in (Lykouris et al., 2020), achieve strictly better regret bounds in terms of total corruptions for the tabular setting. To be specific, firstly, our regret bounds depend on more precise numerical values of total rewards corruptions and transition corruptions, instead of only on the total number of corrupted episodes. Secondly, our regret bounds are the first of their kind in the reinforcement learning setting to have the number of corruptions show up additively with respect to $\min\{\sqrt{T}, ext{PolicyGapComplexity}\}$ rather than multiplicatively. Our results follow from a general algorithmic framework that combines corruption-robust policy elimination meta-algorithms, and plug-in reward-free exploration sub-algorithms. Replacing the meta-algorithm or sub-algorithm may extend the framework to address other corrupted settings with potentially more structure.

Motivation & Objective

• To address the limitations of prior work in adversarial corruption robustness in episodic reinforcement learning.
• To develop algorithms that depend on precise magnitudes of reward and transition corruptions rather than just the number of corrupted episodes.
• To achieve regret bounds with additive rather than multiplicative dependence on corruption levels, resolving an open problem in the field.
• To provide a general framework combining corruption-robust policy elimination and reward-free exploration for broader applicability.

Proposed method

• Proposes EstAll, a corruption-robust reward-free exploration algorithm that returns (ε + (C^p + C^r)ε²)-close policy value estimates.
• Introduces BARBAR-RL, a meta-algorithm using EstAll that achieves Õ(min{√T, PolicyGapComplexity} + (1 + C^p)(C^p + C^r)) regret under non-adaptive adversaries.
• Develops BrutePolicyElimination-RL, a meta-algorithm with Õ(√T + (C^p + C^r)²) regret under adaptive adversaries who observe the policy before corrupting.
• Employs a general framework combining corruption-robust meta-algorithms with plug-in reward-free exploration subroutines for improved robustness.
• Uses state-action visitation frequency and Q-value estimation error as core estimators in the reward-free exploration phase.
• Applies theoretical bounds on value function differences via total variation distance between transition kernels and reward functions.

Experimental results

Research questions

• RQ1 Can regret bounds in episodic RL be made to depend additively on corruption magnitudes rather than multiplicatively?
• RQ2 Is it possible to achieve additive dependence on total corruptions C^p and C^r in tabular RL, even when the number of corrupted episodes is unknown?
• RQ3 Can reward-free exploration be made robust to adversarial corruptions in both rewards and transitions?
• RQ4 Can policy-elimination style algorithms achieve instance-dependent regret bounds in the presence of corruptions?
• RQ5 Can the framework be extended to other corrupted RL settings with additional structural assumptions?

Key findings

• The proposed BARBAR-RL meta-algorithm achieves regret Õ(min{√T, PolicyGapComplexity} + (1 + C^p)(C^p + C^r)) under non-adaptive adversaries, with additive dependence on corruption magnitudes.
• The BrutePolicyElimination-RL meta-algorithm achieves Õ(√T + (C^p + C^r)²) regret under adaptive adversaries, providing the first additive dependence on corruption levels in tabular RL.
• EstAll ensures (ε + (C^p + C^r)ε²)-close policy value estimates with sample complexity Õ(log|Π|/ε²) when C^p ≤ Õ(1/ε), otherwise signals corruption levels.
• The regret bounds are the first to depend on the actual magnitudes of reward (C^r) and transition (C^p) corruptions rather than just the number of corrupted episodes.
• The framework demonstrates that additive corruption dependence is achievable in tabular RL, resolving an open problem posed by Lykouris et al. (2020).
• Replacing EstAll with standard reward-free exploration algorithms leads to worse T-dependence, indicating EstAll’s superior robustness in corrupted settings.