[Paper Review] Unifying the stochastic and the adversarial Bandits with Knapsack

Anshuka Rangi, Massimo Franceschetti+1 more|arXiv (Cornell University)|Oct 23, 2018

Advanced Bandit Algorithms Research15 references3 citations

TL;DR

This paper proposes EXP3.BwK and EXP3++.BwK, novel algorithms for the adversarial Bandits with Knapsack (BwK) problem that achieve order-optimal regret without requiring the restrictive assumption that rewards exceed costs. The algorithms unify performance across both stochastic and adversarial settings, providing $O(\sqrt{KB})$ regret in the adversarial case and $O(\log B)$ regret in the stochastic case, with an additional $\log B$ factor in the latter under EXP3++.BwK.

ABSTRACT

This paper investigates the adversarial Bandits with Knapsack (BwK) online learning problem, where a player repeatedly chooses to perform an action, pays the corresponding cost, and receives a reward associated with the action. The player is constrained by the maximum budget $B$ that can be spent to perform actions, and the rewards and the costs of the actions are assigned by an adversary. This problem has only been studied in the restricted setting where the reward of an action is greater than the cost of the action, while we provide a solution in the general setting. Namely, we propose EXP3.BwK, a novel algorithm that achieves order optimal regret. We also propose EXP3++.BwK, which is order optimal in the adversarial BwK setup, and incurs an almost optimal expected regret with an additional factor of $\log(B)$ in the stochastic BwK setup. Finally, we investigate the case of having large costs for the actions (i.e., they are comparable to the budget size $B$), and show that for the adversarial setting, achievable regret bounds can be significantly worse, compared to the case of having costs bounded by a constant, which is a common assumption within the BwK literature.

Motivation & Objective

• Address the lack of algorithms that perform well in both stochastic and adversarial Bandits with Knapsack (BwK) settings.
• Remove the restrictive assumption in prior work that rewards must exceed costs in adversarial BwK.
• Provide a unified algorithmic framework that achieves order-optimal regret in both stochastic and adversarial BwK.
• Analyze the impact of large action costs (comparable to budget B) on regret bounds in adversarial BwK.
• Establish theoretical regret guarantees that hold under general reward-cost settings, without physical or dimensional comparability assumptions.

Proposed method

• Extend the EXP3 algorithm to the BwK setting by introducing EXP3.BwK, which uses a time-varying learning rate $\gamma_t = 0.5\sqrt{c_{\text{min}}^2 \log K / Kt}$ to balance exploration and exploitation under budget constraints.
• Introduce EXP3++.BwK as a refined variant that improves regret bounds in the stochastic setting by incorporating a more aggressive exploration schedule.
• Use a confidence-based exploration strategy with $\epsilon_t(i) \leq 0.5c_{\text{min}}\sqrt{\log K / tK}$ to control estimation error in loss and reward estimates.
• Apply concentration inequalities and martingale arguments to bound the expected regret via decomposition into estimation error, exploration, and learning rate terms.
• Leverage the structure of the knapsack constraint by defining $\tau(E)$ as the stopping time when the budget is exhausted, and analyze regret up to $T = \max\{T(i^*), \tau(E)\}$.
• Derive regret bounds using a combination of Chernoff-type bounds and summation inequalities, particularly bounding terms like $\sum_t \exp(-\gamma_t \tilde{\Delta}_t(i))$ via integration and asymptotic analysis.

Experimental results

Research questions

• RQ1 Can a single algorithm achieve order-optimal regret in both stochastic and adversarial BwK without assuming rewards exceed costs?
• RQ2 What is the impact of large action costs (comparable to the budget B) on the achievable regret in adversarial BwK?
• RQ3 How can the EXP3 framework be adapted to handle knapsack constraints while maintaining optimal regret guarantees?
• RQ4 Can the regret in the stochastic BwK setting be improved to $O(\log B)$ while preserving $O(\sqrt{KB})$ regret in the adversarial case?
• RQ5 What are the theoretical limits of regret when costs are not bounded by a constant, but scale with the budget B?

Key findings

• EXP3.BwK achieves $O(\sqrt{KB})$ expected regret in the adversarial BwK setting, which is order-optimal and does not require the reward-greater-than-cost assumption.
• EXP3++.BwK achieves $O(\log B)$ expected regret in the stochastic BwK setting with an additional $\log B$ factor, making it almost optimal.
• The regret bound for EXP3.BwK is $O\left(\sqrt{BK\log K / c_{\text{min}}^3}\right)$, showing dependence on the minimum cost $c_{\text{min}}$.
• When costs are large (comparable to budget B), the achievable regret bounds in the adversarial BwK setting can be significantly worse than in the bounded-cost case.
• The analysis shows that the assumption of bounded costs (common in prior work) is critical—without it, regret guarantees degrade substantially.
• Theoretical bounds are derived using concentration inequalities and summation of exponential terms, with key steps relying on Lemma 10 and bounds on $\sum_t \exp(-\gamma_t \tilde{\Delta}_t(i))$.