[Paper Review] Stochastic matrix games with bandit feedback.

Brendan O’Donoghue, Tor Lattimore+1 more|arXiv (Cornell University)|Jun 9, 2020

Advanced Bandit Algorithms Research29 references4 citations

TL;DR

This paper introduces and analyzes UCB and K-learning variants for stochastic matrix games with bandit feedback, where players face an unknown payoff matrix and only observe noisy payoffs. It establishes regret bounds that hold even against adversarial opponents, showing these algorithms significantly outperform standard bandit methods and Thompson sampling, which fails catastrophically in this setting.

ABSTRACT

We study a version of the classical zero-sum matrix game with unknown payoff matrix and bandit feedback, where the players only observe each others actions and a noisy payoff. This generalizes the usual matrix game, where the payoff matrix is known to the players. Despite numerous applications, this problem has received relatively little attention. Although adversarial bandit algorithms achieve low regret, they do not exploit the matrix structure and perform poorly relative to the new algorithms. The main contributions are regret analyses of variants of UCB and K-learning that hold for any opponent, e.g., even when the opponent adversarially plays the best response to the learner's mixed strategy. Along the way, we show that Thompson fails catastrophically in this setting and provide empirical comparison to existing algorithms.

Motivation & Objective

• To address the gap in research on zero-sum matrix games with unknown payoff matrices and bandit feedback, where only noisy payoffs and actions are observed.
• To develop learning algorithms that exploit the matrix structure of the game, unlike standard adversarial bandit methods.
• To provide theoretical regret bounds that hold for any opponent, including those playing best responses to the learner’s strategy.
• To demonstrate the failure of Thompson sampling in this setting and compare performance empirically with existing algorithms.

Proposed method

• Proposes variants of UCB and K-learning tailored for matrix games under bandit feedback, incorporating payoff estimation and confidence bounds.
• Uses a matrix-structured learning framework where the learner maintains estimates of the payoff matrix and updates them based on observed noisy rewards.
• Applies confidence interval techniques to balance exploration and exploitation, ensuring low regret even under adversarial opponent behavior.
• Introduces a novel regret analysis that holds uniformly over all opponent strategies, including worst-case best-response play.
• Employs empirical evaluation to compare the proposed algorithms against Thompson sampling and standard bandit algorithms.

Experimental results

Research questions

• RQ1 Can UCB and K-learning variants achieve low regret in matrix games with unknown payoff matrices and bandit feedback?
• RQ2 How do these algorithms perform against adversarial opponents who best-respond to the learner’s mixed strategy?
• RQ3 Why does Thompson sampling fail catastrophically in this matrix game setting with bandit feedback?
• RQ4 To what extent do structure-aware algorithms outperform general-purpose adversarial bandit algorithms in this context?

Key findings

• The proposed UCB and K-learning variants achieve sublinear regret even when the opponent plays the best response to the learner’s mixed strategy.
• These algorithms significantly outperform standard adversarial bandit algorithms by exploiting the underlying matrix structure of the game.
• Thompson sampling is shown to fail catastrophically in this setting, leading to unbounded regret in the worst case.
• Empirical results confirm that the proposed methods achieve lower regret and better convergence than existing algorithms.