[Paper Review] No Internal Regret via Neighborhood Watch

Dean P. Foster, Alexander Rakhlin|arXiv (Cornell University)|Aug 30, 2011

Advanced Bandit Algorithms Research9 references3 citations

TL;DR

This paper introduces the Neighborhood Watch algorithm, which achieves $O(\sqrt{T})$ internal and external regret in finite partial-monitoring games under the local observability condition, even against non-stochastic (adversarial) opponents. The method uses a two-level exploration strategy over action neighborhoods, leveraging unbiased loss estimation via the local observability condition to ensure regret bounds matching the optimal rate for bandit feedback settings.

ABSTRACT

We present an algorithm which attains O(\sqrt{T}) internal (and thus external) regret for finite games with partial monitoring under the local observability condition. Recently, this condition has been shown by (Bartok, Pal, and Szepesvari, 2011) to imply the O(\sqrt{T}) rate for partial monitoring games against an i.i.d. opponent, and the authors conjectured that the same holds for non-stochastic adversaries. Our result is in the affirmative, and it completes the characterization of possible rates for finite partial-monitoring games, an open question stated by (Cesa-Bianchi, Lugosi, and Stoltz, 2006). Our regret guarantees also hold for the more general model of partial monitoring with random signals.

Motivation & Objective

• To close the open problem of characterizing regret rates in finite partial-monitoring games under non-i.i.d. (adversarial) opponents.
• To prove that the $O(\sqrt{T})$ regret rate, previously known for i.i.d. opponents under local observability, also holds for non-stochastic adversaries.
• To extend the result to the more general model of partial monitoring with random signals.
• To establish that local observability fully characterizes the $O(\sqrt{T})$ regret rate for both internal and external regret, completing the classification of possible rates.

Proposed method

• Introduces a notion of local internal regret, where a player does not benefit by switching to a neighboring action in the neighborhood graph defined by opponent's mixed strategy best responses.
• Uses a two-level algorithm: first randomly select a neighborhood, then select an action within it, ensuring a flow condition that links sub-algorithm regret to global regret.
• Employs unbiased loss estimation via the local observability condition, which guarantees that $\ell_i - \ell_j \in \text{Im}(S_{(i,j)}^\top)$ for neighboring actions $i,j$, enabling reliable comparison of neighboring actions.
• Constructs estimation vectors $v_{(i,j)}$ such that $\ell_j - \ell_i = S_{(i,j)}^\top v_{(i,j)}$, allowing the algorithm to estimate relative losses of neighboring actions from observed signals.
• Applies a modified version of the Hedge algorithm with separate learning rate $\eta$ and exploration parameter $\gamma$, achieving $O(\sqrt{T})$ regret in expectation with $\gamma = 0$ and high probability with $\gamma \propto T^{-1/2}$.
• Adapts the analysis to random signals by replacing deterministic signal matrices $S$ with random matrices $\Xi$, where $\mathbb{E}[\Xi_i] = \Xi_i$, and maintains the same regret guarantees under the same local observability condition.

Experimental results

Research questions

• RQ1 Can the $O(\sqrt{T})$ regret rate for external regret, known under local observability for i.i.d. opponents, be extended to non-stochastic (adversarial) opponents?
• RQ2 Does the local observability condition fully characterize the $O(\sqrt{T})$ regret rate for both internal and external regret in finite partial-monitoring games?
• RQ3 Can the $O(\sqrt{T})$ regret guarantee be extended to the case of random signals, rather than deterministic feedback?
• RQ4 Is there a general algorithmic framework that achieves $O(\sqrt{T})$ internal regret under local observability, even in adversarial settings?

Key findings

• The Neighborhood Watch algorithm achieves $O(\sqrt{T})$ external regret against a non-stochastic opponent under the local observability condition, confirming the conjecture by Bartók, Pál, and Szepesvári.
• The same algorithm achieves $O(\sqrt{T})$ internal regret, completing the characterization of possible regret rates for finite partial-monitoring games.
• The local observability condition is both necessary and sufficient for $O(\sqrt{T})$ regret, as it enables unbiased estimation of relative losses between neighboring actions.
• The analysis extends to random signals: under the same local observability condition, the algorithm maintains $O(\sqrt{T})$ regret in expectation with $\gamma = 0$ and high probability with $\gamma \propto T^{-1/2}$.
• The regret bound is tight: it matches the known optimal rate for bandit feedback games (where $L = H$), and no faster rate is possible under the same condition.
• The result resolves an open question posed by Cesa-Bianchi, Lugosi, and Stoltz regarding the characterization of regret rates in partial-monitoring games.