[Paper Review] A Model Selection Approach for Corruption Robust Reinforcement Learning

Chen-Yu Wei, Christoph Dann+1 more|arXiv (Cornell University)|Oct 7, 2021

Advanced Bandit Algorithms Research68 references4 citations

TL;DR

This paper proposes a model selection framework for reinforcement learning that achieves optimal regret bounds in the presence of adversarial corruption in both transitions and rewards, without prior knowledge of the corruption level. By leveraging regret-balancing across base algorithms tuned to different corruption levels, the method attains a worst-case optimal regret of $\widetilde{\mathcal{O}}(\min\{1/\Delta, \sqrt{T}\} + C)$ for tabular MDPs and $\widetilde{\mathcal{O}}(\sqrt{(1+C)T})$ for linear MDPs, resolving an open problem on corruption-robust RL.

ABSTRACT

We develop a model selection approach to tackle reinforcement learning with adversarial corruption in both transition and reward. For finite-horizon tabular MDPs, without prior knowledge on the total amount of corruption, our algorithm achieves a regret bound of $\widetilde{\mathcal{O}}(\min\{\frac{1}Δ, \sqrt{T}\}+C)$ where $T$ is the number of episodes, $C$ is the total amount of corruption, and $Δ$ is the reward gap between the best and the second-best policy. This is the first worst-case optimal bound achieved without knowledge of $C$, improving previous results of Lykouris et al. (2021); Chen et al. (2021); Wu et al. (2021). For finite-horizon linear MDPs, we develop a computationally efficient algorithm with a regret bound of $\widetilde{\mathcal{O}}(\sqrt{(1+C)T})$, and another computationally inefficient one with $\widetilde{\mathcal{O}}(\sqrt{T}+C)$, improving the result of Lykouris et al. (2021) and answering an open question by Zhang et al. (2021b). Finally, our model selection framework can be easily applied to other settings including linear bandits, linear contextual bandits, and MDPs with general function approximation, leading to several improved or new results.

Motivation & Objective

• To address the open problem of achieving optimal regret in reinforcement learning when both transitions and rewards are adversarially corrupted, without prior knowledge of the total corruption amount C.
• To develop a model selection framework that adaptively selects among base algorithms designed for different hypothetical corruption levels.
• To achieve worst-case optimal regret bounds that scale optimally with both T (episodes) and C (corruption), matching information-theoretic lower bounds.
• To extend the framework to linear MDPs and other settings like linear bandits and contextual bandits, yielding improved or new theoretical guarantees.

Proposed method

• The method uses a meta-algorithm, COBE (Corruption-robustness through Balancing and Elimination), which maintains multiple instances of a base algorithm, each tuned to a different estimated corruption level.
• It applies regret balancing via carefully chosen importance weights $\alpha_i$ to ensure that the performance of each base algorithm is kept comparable, preventing any single one from being overly penalized.
• The algorithm dynamically increases the estimated corruption level $k$ after each early termination of a base algorithm instance, indicating that the current estimate was too low.
• The framework relies on a subroutine, BASIC, which performs model selection over a hierarchy of corruption hypotheses using a stopping condition based on cumulative regret and confidence bounds.
• It uses a parameterized regret bound of the form $\mathcal{R}(t,\theta) = \sqrt{\beta_1 t} + \beta_2 \theta + \beta_3$ to model the performance of base algorithms under different corruption assumptions.
• The method ensures that the overall regret is within a poly-logarithmic factor of the best base algorithm in hindsight, even when the true corruption level is unknown.

Experimental results

Research questions

• RQ1 Can we achieve a regret bound that is optimal in both T and C for tabular MDPs when the total corruption C is unknown?
• RQ2 Is it possible to design a model selection framework that adaptively identifies the correct corruption level without prior knowledge, while maintaining near-optimal regret?
• RQ3 Can the proposed approach be extended to linear MDPs to achieve optimal regret bounds of the form $\widetilde{\mathcal{O}}(\sqrt{(1+C)T})$?
• RQ4 Does the model selection framework yield improved or new results in related settings such as linear bandits and contextual bandits?
• RQ5 Can the framework achieve instance-dependent regret bounds of the form $\widetilde{\mathcal{O}}(\min\{1/\Delta, \sqrt{T}\} + C)$ without prior knowledge of C?

Key findings

• For finite-horizon tabular MDPs, the proposed algorithm achieves a regret bound of $\widetilde{\mathcal{O}}(\min\{1/\Delta, \sqrt{T}\} + C)$, which is the first worst-case optimal bound without prior knowledge of C.
• The bound matches the information-theoretic lower bound established by Wu et al. (2021), resolving a long-standing open problem in corruption-robust RL.
• For finite-horizon linear MDPs, the algorithm achieves a regret bound of $\widetilde{\mathcal{O}}(\sqrt{(1+C)T})$ with a computationally efficient variant, improving upon prior work.
• The framework answers an open question posed by Zhang et al. (2021) regarding optimal dependence on C in linear MDPs.
• The method generalizes to linear bandits, linear contextual bandits, and MDPs with general function approximation, yielding improved or new regret bounds in these settings.
• The model selection approach ensures that the regret is within a poly-logarithmic factor of the best base algorithm in hindsight, even when the true corruption level is unknown.