[Paper Review] An Efficient Pessimistic-Optimistic Algorithm for Stochastic Linear Bandits with General Constraints

Xin Liu, Bin Li+2 more|arXiv (Cornell University)|Feb 10, 2021

Advanced Bandit Algorithms ResearchDecision Sciences44 references17 citations

TL;DR

This paper proposes a pessimistic-optimistic algorithm for stochastic linear bandits with general nonlinear constraints, leveraging a primal-dual framework to achieve $ tilde{ m O}igl((K^{0.75}/ar{\delta} + d) mspace{2pt}\sqrt{\tau}\bigr)$ regret and zero constraint violation after a constant time $\tau^\prime$, independent of the horizon $T$. The method ensures computational efficiency by decoupling primal and dual updates, with dual complexity scaling only with the number of constraints, not the action or feature space sizes.

ABSTRACT

This paper considers stochastic linear bandits with general nonlinear constraints. The objective is to maximize the expected cumulative reward over horizon $T$ subject to a set of constraints in each round $τ\leq T$. We propose a pessimistic-optimistic algorithm for this problem, which is efficient in two aspects. First, the algorithm yields $ ilde{\cal O}\left(\left(\frac{K^{0.75}}δ+d ight)\sqrtτ ight)$ (pseudo) regret in round $τ\leq T,$ where $K$ is the number of constraints, $d$ is the dimension of the reward feature space, and $δ$ is a Slater's constant; and zero constraint violation in any round $τ>τ',$ where $τ'$ is independent of horizon $T.$ Second, the algorithm is computationally efficient. Our algorithm is based on the primal-dual approach in optimization and includes two components. The primal component is similar to unconstrained stochastic linear bandits (our algorithm uses the linear upper confidence bound algorithm (LinUCB)). The computational complexity of the dual component depends on the number of constraints, but is independent of the sizes of the contextual space, the action space, and the feature space. Thus, the overall computational complexity of our algorithm is similar to that of the linear UCB for unconstrained stochastic linear bandits.

Motivation & Objective

• To address stochastic linear bandits under general nonlinear constraints, particularly in settings requiring safety, fairness, or budget adherence.
• To design an algorithm that maintains low regret while ensuring zero constraint violation after a finite time independent of the horizon $T$.
• To ensure computational efficiency by making the dual update complexity depend only on the number of constraints, not on the size of the action or feature space.
• To generalize existing bandit formulations by introducing anytime cumulative constraints that must be satisfied in every round, not just at the end of the horizon.

Proposed method

• The algorithm employs a primal-dual approach, where the primal component uses a linear upper confidence bound (LinUCB) style update for reward maximization.
• The dual component maintains Lagrangian multipliers that are updated based on cumulative constraint violations, with update complexity depending only on the number of constraints $K$.
• A pessimistic-optimistic strategy balances exploration and constraint satisfaction by incorporating confidence bounds on both rewards and cost estimates.
• The algorithm uses a time-varying learning rate $\epsilon_t = \frac{4d\log(1+T)}{\sqrt{t}}$ and a value parameter $V_t = \frac{\delta d\sqrt{t}\log(1+T)}{4}$ to control regret and constraint violation.
• It introduces a novel Lyapunov-like function to analyze the evolution of constraint violations and derive bounds on expected cumulative cost.
• The method ensures zero constraint violation after $\tau^\prime = \frac{64d^2\log^2(1+T)}{\delta^2}$ by maintaining $\epsilon_t \leq \delta/2$.

Experimental results

Research questions

• RQ1 Can we design a computationally efficient algorithm for stochastic linear bandits with general nonlinear constraints that ensures zero constraint violation after a finite time?
• RQ2 What regret bound can be achieved under anytime cumulative constraints, and how does it scale with the number of constraints $K$, dimension $d$, and Slater’s constant $\delta$?
• RQ3 How can the primal-dual framework be adapted to maintain both low regret and strong constraint satisfaction in a bandit setting with unknown cost functions?
• RQ4 Can the dual update complexity be decoupled from the size of the action and feature spaces, enabling scalability to large contextual problems?

Key findings

• The algorithm achieves a regret bound of $\mathcal{R}(\tau) = \tilde{\mathcal{O}}\bigl(\bigl(\frac{K^{0.75}}{\delta} + d\bigr)\sqrt{\tau}\bigr)$ in round $\tau$, which is sublinear and nearly optimal.
• Constraint violation is zero for all $\tau > \tau^\prime = \frac{64d^2\log^2(1+T)}{\delta^2}$, where $\tau^\prime$ is independent of the horizon $T$.
• The expected cumulative cost violation $\mathbb{E}[Q(\tau)]$ is bounded by $\mathcal{O}\bigl(\frac{d^2\log^2(1+T)}{\delta^2} + \frac{1}{\delta}\bigr)$, ensuring long-term feasibility.
• The computational complexity is comparable to unconstrained LinUCB, as the dual update depends only on $K$, not on the size of the action or feature space.
• The algorithm maintains feasibility through a Lyapunov function analysis, proving that constraint violations are driven to zero after a constant time.
• The method is robust to non-linear cost functions and does not require linearity in the cost structure, broadening applicability to real-world problems like healthcare and budgeted recommendations.