[Paper Review] Bandit Algorithms for Prophet Inequality and Pandora's Box

Khashayar Gatmiry, Thomas Keßelheim+2 more|arXiv (Cornell University)|Nov 16, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces bandit algorithms for the Prophet Inequality and Pandora's Box problems under limited feedback, where only the reward of the chosen action is observed. It achieves near-optimal regret bounds of $\widetilde{O}(\mathsf{poly}(n)\sqrt{T})$ by maintaining confidence intervals on optimal policy indices and designing learnable regret upper bounds through a novel exploration-exploitation trade-off mechanism.

ABSTRACT

The Prophet Inequality and Pandora's Box problems are fundamental stochastic problem with applications in Mechanism Design, Online Algorithms, Stochastic Optimization, Optimal Stopping, and Operations Research. A usual assumption in these works is that the probability distributions of the $n$ underlying random variables are given as input to the algorithm. Since in practice these distributions need to be learned, we initiate the study of such stochastic problems in the Multi-Armed Bandits model. In the Multi-Armed Bandits model we interact with $n$ unknown distributions over $T$ rounds: in round $t$ we play a policy $x^{(t)}$ and receive a partial (bandit) feedback on the performance of $x^{(t)}$. The goal is to minimize the regret, which is the difference over $T$ rounds in the total value of the optimal algorithm that knows the distributions vs. the total value of our algorithm that learns the distributions from the partial feedback. Our main results give near-optimal $ ilde{O}(\mathsf{poly}(n)\sqrt{T})$ total regret algorithms for both Prophet Inequality and Pandora's Box. Our proofs proceed by maintaining confidence intervals on the unknown indices of the optimal policy. The exploration-exploitation tradeoff prevents us from directly refining these confidence intervals, so the main technique is to design a regret upper bound that is learnable while playing low-regret Bandit policies.

Motivation & Objective

• To address the gap in online stochastic optimization where underlying distributions are unknown and must be learned from limited feedback.
• To study the Prophet Inequality and Pandora’s Box problems in the Multi-Armed Bandits framework with only reward feedback.
• To design algorithms that achieve near-optimal regret despite limited feedback, minimizing the difference between learned policy and optimal policy with known distributions.
• To establish theoretical lower bounds showing sublinear regret is impossible under adversarial inputs.
• To unify online learning across feedback models by enabling low-regret performance with minimal feedback.

Proposed method

• Maintain confidence intervals on the unknown indices of the optimal policy to guide exploration.
• Design a regret upper bound function that is learnable while playing low-regret bandit policies.
• Use an interval-shrinking algorithm to refine estimates of optimal thresholds in both Prophet Inequality and Pandora’s Box settings.
• Apply a doubling framework to ensure low-regret performance across rounds with adaptive exploration.
• Introduce a moving difference and swapping difference analysis to bound policy deviation in Pandora’s Box with fixed or dynamic orders.
• Convert the learned policy into the optimal policy in polynomial time by iteratively refining constraints and confidence bounds.

Experimental results

Research questions

• RQ1 Can near-optimal regret be achieved in the Prophet Inequality problem when only bandit feedback (reward only) is available?
• RQ2 Can the same regret bounds be extended to the Pandora’s Box problem under bandit feedback with unknown distributions?
• RQ3 What is the fundamental limit of regret in these problems under adversarial input sequences with limited feedback?
• RQ4 How can confidence intervals on optimal policy indices be maintained and refined under exploration-exploitation trade-offs in bandit feedback?
• RQ5 Can a single algorithm achieve near-optimal performance across different feedback models by minimizing feedback requirements?

Key findings

• The paper achieves a total regret of $\widetilde{O}(\mathsf{poly}(n)\sqrt{T})$ for both the Prophet Inequality and Pandora’s Box problems under bandit feedback.
• For Prophet Inequality with $n=2$, the algorithm uses an interval-shrinking approach to maintain confidence in the optimal threshold, achieving $\widetilde{O}(\sqrt{T})$ regret.
• In Pandora’s Box, the algorithm bounds both moving difference and swapping difference through dynamic constraint updates, enabling efficient policy conversion.
• A lower bound of $\Omega(\sqrt{T})$ is established for stochastic inputs, showing the regret bound is tight up to logarithmic factors.
• For adversarial inputs, the paper proves $\Omega(T)$ regret is unavoidable, even under full feedback, showing sublinear regret is impossible.
• The results resolve open questions on sublinear regret in online learning for Pandora’s Box under adversarial inputs.