[Paper Review] Provably Efficient Online Agnostic Learning in Markov Games.

Yi Tian, Yuanhao Wang+2 more|arXiv (Cornell University)|Oct 28, 2020

Reinforcement Learning in Robotics22 references5 citations

TL;DR

This paper presents the first provably efficient online agnostic learning algorithm for Markov games with unobserved opponent actions, achieving a sublinear regret bound of $\tilde{\mathcal{O}}(K^{3/4})$ after $K$ episodes. The method is independent of opponents' action space size, improving upon prior work by an exponential factor even when actions are observable.

ABSTRACT

We study online agnostic learning, a problem that arises in episodic multi-agent reinforcement learning where the actions of the opponents are unobservable. We show that in this challenging setting, achieving sublinear regret against the best response in hindsight is statistically hard. We then consider a weaker notion of regret, and present an algorithm that achieves after $K$ episodes a sublinear $ ilde{\mathcal{O}}(K^{3/4})$ regret. This is the first sublinear regret bound (to our knowledge) in the online agnostic setting. Importantly, our regret bound is independent of the size of the opponents' action spaces. As a result, even when the opponents' actions are fully observable, our regret bound improves upon existing analysis (e.g., (Xie et al., 2020)) by an exponential factor in the number of opponents.

Motivation & Objective

• To address the challenge of online agnostic learning in episodic multi-agent reinforcement learning where opponents' actions are unobservable.
• To analyze the statistical hardness of achieving sublinear regret against the best response in hindsight under unobserved opponent behavior.
• To propose a weaker regret notion that enables provably sublinear regret in the online agnostic setting.
• To design an algorithm with regret independent of the size of opponents' action spaces, improving scalability.
• To achieve improved regret bounds compared to prior work, especially when opponents' actions are observable.

Proposed method

• Introduce a weaker regret notion to circumvent the statistical hardness of achieving sublinear regret against the best response in hindsight.
• Design an algorithm that achieves $\tilde{\mathcal{O}}(K^{3/4})$ regret after $K$ episodes, leveraging a novel analysis framework.
• Decouple the regret bound from the size of opponents' action spaces through a structured exploration and estimation mechanism.
• Use online learning techniques adapted to Markov games with partial observability of opponent actions.
• Apply concentration inequalities and regret decomposition to derive the final bound.
• Ensure the regret bound remains sublinear and independent of the number of opponents or their action sets.

Experimental results

Research questions

• RQ1 Is sublinear regret against the best response in hindsight achievable in online agnostic learning with unobserved opponent actions?
• RQ2 Can a weaker regret notion enable provably sublinear regret in this setting?
• RQ3 Can the regret bound be made independent of the size of opponents' action spaces?
• RQ4 How does the proposed algorithm's regret compare to existing methods when opponents' actions are observable?
• RQ5 What is the optimal achievable regret bound in the online agnostic learning setting for Markov games?

Key findings

• The paper establishes that achieving sublinear regret against the best response in hindsight is statistically hard in the online agnostic setting with unobserved opponent actions.
• The proposed algorithm achieves a sublinear regret bound of $\tilde{\mathcal{O}}(K^{3/4})$ after $K$ episodes, which is the first such result in this setting.
• The regret bound is independent of the size of the opponents' action spaces, enabling scalability to large or unknown action sets.
• Even when opponents' actions are observable, the regret bound improves upon prior work (e.g., Xie et al., 2020) by an exponential factor in the number of opponents.
• The method provides a provably efficient solution to online agnostic learning in Markov games under partial observability.
• The analysis demonstrates that a weaker regret notion is sufficient to achieve sublinear regret while maintaining independence from action space size.