[Paper Review] An Empirical Process Approach to the Union Bound: Practical Algorithms for Combinatorial and Linear Bandits

Julian Katz-Samuels, Lalit Jain+2 more|arXiv (Cornell University)|Jun 21, 2020

Advanced Bandit Algorithms ResearchDecision Sciences31 references20 citations

TL;DR

This paper introduces a novel empirical process-based approach to the union bound in pure-exploration linear and combinatorial bandits, replacing explicit union bounds over arms with a design based on Gaussian width. The proposed algorithm achieves near-optimal sample complexity that scales with the geometry of the arm set, avoiding the $\log(|\mathcal{Z}|)$ factor common in prior work, and provides the first fixed-budget algorithm for transductive linear bandits with matching guarantees up to logarithmic factors.

ABSTRACT

This paper proposes near-optimal algorithms for the pure-exploration linear bandit problem in the fixed confidence and fixed budget settings. Leveraging ideas from the theory of suprema of empirical processes, we provide an algorithm whose sample complexity scales with the geometry of the instance and avoids an explicit union bound over the number of arms. Unlike previous approaches which sample based on minimizing a worst-case variance (e.g. G-optimal design), we define an experimental design objective based on the Gaussian-width of the underlying arm set. We provide a novel lower bound in terms of this objective that highlights its fundamental role in the sample complexity. The sample complexity of our fixed confidence algorithm matches this lower bound, and in addition is computationally efficient for combinatorial classes, e.g. shortest-path, matchings and matroids, where the arm sets can be exponentially large in the dimension. Finally, we propose the first algorithm for linear bandits in the the fixed budget setting. Its guarantee matches our lower bound up to logarithmic factors.

Motivation & Objective

• To eliminate the $\log(|\mathcal{Z}|)$ factor in sample complexity bounds that arises from naive union bounds over arms in pure-exploration linear bandits.
• To develop a computationally efficient experimental design for combinatorial bandits (e.g., shortest paths, matchings, matroids) with exponentially large arm sets.
• To establish a novel lower bound based on Gaussian width that characterizes the fundamental sample complexity limit.
• To provide the first fixed-budget algorithm for transductive linear bandits with tight theoretical guarantees.
• To demonstrate empirical superiority over existing methods, especially in high-dimensional and combinatorial settings.

Proposed method

• Uses the TIS inequality to model suprema of Gaussian processes and derive an experimental design minimizing expected suprema of deviations.
• Defines a new experimental design objective based on the Gaussian width of the arm set $\mathcal{Z}$, replacing worst-case variance minimization (e.g., G-optimal design).
• Solves a relaxed optimization problem over distributions $\lambda \in \Delta$ to minimize $\mathbb{E}_{\eta \sim N(0,I)}[\max_{z \in \mathcal{Z}_k} (\hat{z}_k - z)^\top A(\lambda)^{-1/2} \eta]$, focusing on differences from the current best arm.
• Implements a two-step approximation: minimizing both the expected max deviation and the maximum norm in the $A(\lambda)^{-1}$-metric, then averaging the resulting designs.
• Uses a stochastic mirror descent algorithm with $\lambda \in \Delta$ to compute the design, with 2,000 samples to estimate Gaussian width.
• Adapts the algorithm to the fixed-budget setting by replacing the confidence-based stopping rule with a budget-constrained recommendation.

Experimental results

Research questions

• RQ1 Can the $\log(|\mathcal{Z}|)$ factor in sample complexity be eliminated by replacing explicit union bounds with a geometric measure of the arm set?
• RQ2 Is Gaussian width a fundamental quantity governing the sample complexity of pure-exploration linear bandits?
• RQ3 Can a computationally efficient algorithm be designed for combinatorial bandits that scales with the geometry of the arm set rather than its size?
• RQ4 What is the optimal experimental design for fixed-budget linear bandits, and how does it compare to fixed-confidence guarantees?
• RQ5 How does the proposed algorithm perform empirically compared to state-of-the-art baselines in combinatorial and transductive settings?

Key findings

• The fixed-confidence algorithm's sample complexity matches the new lower bound based on Gaussian width, up to a constant factor, and avoids the $\log(|\mathcal{Z}|)$ penalty.
• For combinatorial classes like shortest paths, matchings, and matroids, the algorithm is computationally efficient despite exponentially large arm sets.
• In biclique detection experiments, the proposed algorithm (Peace) required over 3× fewer samples than DisRegion when the gap between the best and second-best biclique was small ($h = 0.2$).
• The fixed-budget algorithm achieves performance matching the lower bound up to a $\log(d)$ factor in most standard settings.
• The algorithm outperforms existing methods by an order of magnitude in several combinatorial bandit settings, particularly when the gap between optimal and suboptimal arms is small.
• Using only the top-$k$ arm differences in the design computation leads to a provably valid and efficient approximation, with no loss in correctness.