[Paper Review] Algorithms with logarithmic or sublinear regret for constrained contextual bandits

Huasen Wu, R. Srikant+2 more|arXiv (Cornell University)|Dec 7, 2015

Advanced Bandit Algorithms Research25 references23 citations

TL;DR

This paper proposes UCB-ALP, a novel algorithm for constrained contextual bandits with time and budget constraints, combining adaptive linear programming with upper-confidence-bound exploration. It achieves logarithmic regret in known and unknown context distribution settings, marking the first such result in this problem class.

ABSTRACT

We study contextual bandits with budget and time constraints, referred to as constrained contextual bandits. The time and budget constraints significantly complicate the exploration and exploitation tradeoff because they introduce complex coupling among contexts over time. To gain insight, we first study unit-cost systems with known context distribution. When the expected rewards are known, we develop an approximation of the oracle, referred to Adaptive-Linear-Programming (ALP), which achieves near-optimality and only requires the ordering of expected rewards. With these highly desirable features, we then combine ALP with the upper-confidence-bound (UCB) method in the general case where the expected rewards are unknown a priori. We show that the proposed UCB-ALP algorithm achieves logarithmic regret except for certain boundary cases. Further, we design algorithms and obtain similar regret bounds for more general systems with unknown context distribution and heterogeneous costs. To the best of our knowledge, this is the first work that shows how to achieve logarithmic regret in constrained contextual bandits. Moreover, this work also sheds light on the study of computationally efficient algorithms for general constrained contextual bandits.

Motivation & Objective

• Address the challenge of contextual bandits under time and budget constraints, which complicate the exploration-exploitation tradeoff due to temporal coupling.
• Develop an efficient algorithm that achieves near-optimal performance when expected rewards are known, using only reward ordering via Adaptive-Linear-Programming (ALP).
• Extend the ALP approach to unknown reward settings by integrating it with the UCB method, ensuring logarithmic regret in general cases.
• Generalize the framework to systems with unknown context distributions and heterogeneous costs, maintaining strong regret bounds.
• Provide the first theoretical guarantee of logarithmic regret in constrained contextual bandits, advancing computationally efficient algorithm design.

Proposed method

• Introduce Adaptive-Linear-Programming (ALP) as an approximation to the oracle policy, relying only on the ordering of expected rewards for near-optimal performance.
• Combine ALP with the UCB method to handle unknown expected rewards, enabling exploration while maintaining regret guarantees.
• Design a UCB-ALP algorithm that dynamically balances exploration and exploitation under time and budget constraints.
• Extend the framework to heterogeneous cost systems by adapting the ALP formulation to account for varying context costs.
• Use confidence bounds to guide action selection, ensuring that suboptimal actions are explored only logarithmically often.
• Prove regret bounds by analyzing the coupling between context sequences and constraint violations over time.

Experimental results

Research questions

• RQ1 Can logarithmic regret be achieved in contextual bandits with time and budget constraints, despite complex temporal coupling?
• RQ2 How can an efficient algorithm be designed when expected rewards are unknown, while still maintaining strong regret performance?
• RQ3 To what extent can the ALP approximation be extended to systems with heterogeneous costs and unknown context distributions?
• RQ4 What is the impact of constraint coupling on the exploration-exploitation tradeoff in contextual bandits?
• RQ5 Can a computationally efficient algorithm be developed that achieves logarithmic regret in general constrained contextual bandit problems?

Key findings

• The UCB-ALP algorithm achieves logarithmic regret in constrained contextual bandits with known context distributions and known expected rewards.
• The ALP method provides near-optimal performance using only the ordering of expected rewards, significantly reducing computational complexity.
• The proposed framework generalizes to systems with unknown context distributions and heterogeneous costs, maintaining logarithmic regret bounds.
• This work establishes the first theoretical guarantee of logarithmic regret in the constrained contextual bandit setting.
• The results demonstrate that computationally efficient algorithms with strong regret performance are achievable even under complex coupling constraints.
• Boundary cases exist where logarithmic regret does not hold, but these are explicitly characterized and excluded from the main regret analysis.