[Paper Review] Polynomial-time Algorithms for Combinatorial Pure Exploration with Full-bandit Feedback

Yuko Kuroki, Liyuan Xu+3 more|arXiv (Cornell University)|Jan 1, 2019

Advanced Bandit Algorithms Research20 references3 citations

TL;DR

This paper presents polynomial-time algorithms for stochastic combinatorial pure exploration under full-bandit feedback, where only the sum of rewards across arms is observed. It introduces a novel approximation algorithm for 0-1 quadratic maximization and achieves provably efficient sample complexity with theoretical guarantees, enabling scalable top-k selection in combinatorial bandit settings.

ABSTRACT

We study the problem of stochastic combinatorial pure exploration (CPE), where an agent sequentially pulls a set of single arms (a.k.a. a super arm) and tries to find the best super arm. Among a variety of problem settings of the CPE, we focus on the full-bandit setting, where we cannot observe the reward of each single arm, but only the sum of the rewards. Although we can regard the CPE with full-bandit feedback as a special case of pure exploration in linear bandits, an approach based on linear bandits is not computationally feasible since the number of super arms may be exponential. In this paper, we first propose a polynomial-time bandit algorithm for the CPE under general combinatorial constraints and provide an upper bound of the sample complexity. Second, we design an approximation algorithm for the 0-1 quadratic maximization problem, which arises in many bandit algorithms with confidence ellipsoids. Based on our approximation algorithm, we propose novel bandit algorithms for the top-k selection problem, and prove that our algorithms run in polynomial time. Finally, we conduct experiments on synthetic and real-world datasets, and confirm the validity of our theoretical analysis in terms of both the computation time and the sample complexity.

Motivation & Objective

• To address the computational infeasibility of existing linear bandit approaches in combinatorial pure exploration with exponential super-arm spaces.
• To design a polynomial-time bandit algorithm for combinatorial pure exploration under general combinatorial constraints with full-bandit feedback.
• To develop an efficient approximation algorithm for 0-1 quadratic maximization, a key subroutine in confidence ellipsoid-based bandit methods.
• To enable scalable and provably efficient top-k selection in combinatorial bandit problems under full-bandit feedback.
• To validate the theoretical sample complexity and computation time through experiments on synthetic and real-world datasets.

Proposed method

• Proposes a polynomial-time bandit algorithm for combinatorial pure exploration under general combinatorial constraints using full-bandit feedback.
• Introduces a novel approximation algorithm for the 0-1 quadratic maximization problem, which arises in confidence ellipsoid computation.
• Leverages the approximation algorithm to design efficient bandit algorithms for the top-k selection problem.
• Employs confidence ellipsoids and sample complexity analysis to ensure convergence to the optimal super arm.
• Designs a sampling strategy that balances exploration and exploitation under partial feedback.
• Employs theoretical analysis to derive an upper bound on the sample complexity of the proposed algorithm.

Experimental results

Research questions

• RQ1 Can we design a polynomial-time algorithm for combinatorial pure exploration under full-bandit feedback with provable sample complexity guarantees?
• RQ2 How can we efficiently solve the 0-1 quadratic maximization problem that arises in confidence ellipsoid-based bandit algorithms?
• RQ3 Can the proposed approximation algorithm be integrated into bandit algorithms to ensure both efficiency and correctness in top-k selection?
• RQ4 Does the proposed method achieve low sample complexity while maintaining polynomial-time computation in practice?
• RQ5 How do the theoretical sample complexity bounds compare to empirical performance on synthetic and real-world datasets?

Key findings

• The proposed algorithm achieves a provable upper bound on sample complexity for combinatorial pure exploration under full-bandit feedback.
• The approximation algorithm for 0-1 quadratic maximization enables polynomial-time computation in settings where exact solutions are infeasible.
• The resulting bandit algorithms for top-k selection run in polynomial time and maintain theoretical guarantees.
• Experiments confirm that the theoretical sample complexity bounds align with empirical performance on both synthetic and real-world datasets.
• The method significantly reduces computation time compared to naive linear bandit approaches, especially in high-dimensional combinatorial spaces.
• The algorithm demonstrates scalability and efficiency in practice, validating the theoretical analysis of sample complexity and runtime.