[Paper Review] Efficient and Robust Algorithms for Adversarial Linear Contextual Bandits

Gergely Neu, Julia Olkhovskaya|arXiv (Cornell University)|Feb 1, 2020

Advanced Bandit Algorithms Research51 references4 citations

TL;DR

This paper proposes two efficient algorithms—RealLinExp3 and RobustLinExp3—for adversarial linear contextual bandits with i.i.d. contexts. RealLinExp3 achieves a regret bound of $\widetilde{O}(\sqrt{KdT})$, matching the best-known lower bound, while RobustLinExp3 ensures robustness to model misspecification with a bound of $\widetilde{O}((Kd)^{1/3}T^{2/3}) + \varepsilon\sqrt{d}T$, marking the first such guarantees for this setting.

ABSTRACT

We consider an adversarial variant of the classic $K$-armed linear contextual bandit problem where the sequence of loss functions associated with each arm are allowed to change without restriction over time. Under the assumption that the $d$-dimensional contexts are generated i.i.d.~at random from a known distributions, we develop computationally efficient algorithms based on the classic Exp3 algorithm. Our first algorithm, RealLinExp3, is shown to achieve a regret guarantee of $\widetilde{O}(\sqrt{KdT})$ over $T$ rounds, which matches the best available bound for this problem. Our second algorithm, RobustLinExp3, is shown to be robust to misspecification, in that it achieves a regret bound of $\widetilde{O}((Kd)^{1/3}T^{2/3}) + \varepsilon \sqrt{d} T$ if the true reward function is linear up to an additive nonlinear error uniformly bounded in absolute value by $\varepsilon$. To our knowledge, our performance guarantees constitute the very first results on this problem setting.

Motivation & Objective

• To close the gap in the literature by designing computationally efficient algorithms with strong performance guarantees for adversarial linear contextual bandits.
• To address the limitations of existing algorithms that assume fixed or stochastically generated loss functions and are sensitive to model misspecification.
• To develop methods that maintain strong regret bounds even when the true reward function deviates slightly from the assumed linear model.
• To enable practical deployment of contextual bandit algorithms in real-world settings where loss functions may be adversarial and models are likely misspecified.
• To establish the first formal performance guarantees for adversarial linear contextual bandits under i.i.d. context assumptions.

Proposed method

• The approach reduces the contextual bandit problem to a set of auxiliary bandit problems with fixed action sets by exploiting the i.i.d. nature of contexts and statistical independence between contexts and losses.
• RealLinExp3 uses the Exp3 algorithm with context-specific loss estimation via unbiased estimators, enabling efficient regret minimization over $T$ rounds.
• RobustLinExp3 incorporates a robustness term that accounts for nonlinear errors bounded by $\varepsilon$, using a modified loss estimation strategy to maintain performance under misspecification.
• Matrix Geometric Resampling (MGR) is introduced as a novel, computationally efficient method for estimating inverse covariance matrices in loss estimation, avoiding the high cost of direct inversion.
• The analysis relies on Lemma 4.1, which relates the regret in the contextual setting to the expected regret in the auxiliary bandit problems, enabling reduction to standard bandit analysis.
• The algorithms are designed to be computationally efficient and scalable, with theoretical guarantees derived under mild assumptions on context distribution and loss structure.

Experimental results

Research questions

• RQ1 Can we design a computationally efficient algorithm for adversarial linear contextual bandits that achieves minimax-optimal regret under i.i.d. context assumptions?
• RQ2 Can we achieve robust performance guarantees when the true reward function is linear up to an $\varepsilon$-bounded nonlinear error?
• RQ3 Is it possible to maintain strong regret bounds in the presence of adversarial loss sequences while still leveraging contextual information?
• RQ4 Can we develop a practical method for estimating inverse covariance matrices in adversarial linear bandits without incurring high computational cost?
• RQ5 What is the fundamental trade-off between regret performance and robustness to model misspecification in this setting?

Key findings

• RealLinExp3 achieves a regret bound of $\widetilde{O}(\sqrt{KdT})$ over $T$ rounds, which matches the best-known lower bound for this problem, establishing minimax optimality.
• RobustLinExp3 achieves a regret bound of $\widetilde{O}((Kd)^{1/3}T^{2/3}) + \varepsilon\sqrt{d}T$ when the true reward function is linear up to an $\varepsilon$-bounded error, providing the first such robustness guarantee for this setting.
• The proposed algorithms are computationally efficient and do not require prior knowledge of the loss sequence, making them suitable for real-time decision-making applications.
• The Matrix Geometric Resampling (MGR) technique provides a scalable alternative to direct inverse covariance estimation, with connections to gradient descent dynamics in stochastic optimization.
• Theoretical analysis shows that sublinear regret is impossible without assumptions on context generation; the i.i.d. assumption is shown to be both natural and necessary for tractability.
• The reduction to auxiliary bandit problems via Lemma 4.1 enables a clean analysis and opens a new direction for contextual bandit algorithms based on loss function estimation rather than policy class optimization.