[Paper Review] Distribution oblivious, risk-aware algorithms for multi-armed bandits with unbounded rewards

Anmol Kagrecha, Jayakrishnan Nair+1 more|arXiv (Cornell University)|Jun 1, 2019

Advanced Bandit Algorithms Research16 citations

TL;DR

This paper proposes distribution-oblivious, risk-aware algorithms for best-arm identification in multi-armed bandits with unbounded rewards, optimizing a trade-off between expected reward and Conditional Value at Risk (CVaR). It introduces a novel CVaR estimator and concentration inequality for heavy-tailed distributions, achieving provable error bounds without prior knowledge of reward distribution moments or suboptimality gaps.

ABSTRACT

Classical multi-armed bandit problems use the expected value of an arm as a metric to evaluate its goodness. However, the expected value is a risk-neutral metric. In many applications like finance, one is interested in balancing the expected return of an arm (or portfolio) with the risk associated with that return. In this paper, we consider the problem of selecting the arm that optimizes a linear combination of the expected reward and the associated Conditional Value at Risk (CVaR) in a fixed budget best-arm identification framework. We allow the reward distributions to be unbounded or even heavy-tailed. For this problem, our goal is to devise algorithms that are entirely distribution oblivious, i.e., the algorithm is not aware of any information on the reward distributions, including bounds on the moments/tails, or the suboptimality gaps across arms. In this paper, we provide a class of such algorithms with provable upper bounds on the probability of incorrect identification. In the process, we develop a novel estimator for the CVaR of unbounded (including heavy-tailed) random variables and prove a concentration inequality for the same, which could be of independent interest. We also compare the error bounds for our distribution oblivious algorithms with those corresponding to standard non-oblivious algorithms. Finally, numerical experiments reveal that our algorithms perform competitively when compared with non-oblivious algorithms, suggesting that distribution obliviousness can be realised in practice without incurring a significant loss of performance.

Motivation & Objective

• To address the limitation of classical multi-armed bandit algorithms that rely on risk-neutral expected value metrics in applications where risk management is critical.
• To develop algorithms that are entirely distribution oblivious—requiring no prior knowledge of reward distribution moments, tail behavior, or suboptimality gaps—while still ensuring strong performance guarantees.
• To optimize a linear combination of expected reward and Conditional Value at Risk (CVaR) in a fixed-budget best-arm identification framework.
• To establish provable upper bounds on the probability of incorrect arm identification under unbounded or heavy-tailed reward distributions.

Proposed method

• Design a novel estimator for CVaR of unbounded and heavy-tailed random variables, enabling risk-aware decision-making without distributional assumptions.
• Prove a concentration inequality for the proposed CVaR estimator, which is essential for deriving finite-sample performance guarantees.
• Construct a class of distribution-oblivious algorithms that balance exploration and exploitation based on the estimated CVaR and expected reward, without using any distributional information.
• Use a fixed-budget framework to ensure that the algorithm selects the best arm with high probability, even when reward distributions are unknown and potentially heavy-tailed.
• Integrate the CVaR estimator into a bandit learning framework that dynamically allocates pulls to arms based on risk-adjusted performance estimates.
• Derive theoretical upper bounds on the probability of incorrect identification that depend only on the problem's inherent difficulty, not on distributional parameters.

Experimental results

Research questions

• RQ1 Can we design distribution-oblivious algorithms for best-arm identification that effectively balance expected reward and risk, as measured by CVaR, under unbounded reward distributions?
• RQ2 What is a robust and provably accurate estimator for CVaR when reward distributions are unbounded or heavy-tailed and no distributional information is available?
• RQ3 How do the error bounds of distribution-oblivious algorithms compare to those of non-oblivious algorithms that use knowledge of moments or tail behavior?
• RQ4 Can distribution-oblivious algorithms achieve competitive performance in practice despite lacking distributional assumptions?
• RQ5 What concentration properties does the proposed CVaR estimator exhibit under general unbounded distributions?

Key findings

• The proposed CVaR estimator is the first to provide provable concentration bounds for unbounded, potentially heavy-tailed distributions without requiring knowledge of moments or tail decay rates.
• The paper establishes theoretical upper bounds on the probability of incorrect identification that hold without any assumptions on the reward distribution beyond existence of the first moment.
• Numerical experiments show that the distribution-oblivious algorithms perform competitively with non-oblivious counterparts, indicating that distribution ignorance does not incur a significant performance penalty.
• The concentration inequality for the CVaR estimator is of independent interest and may be applicable beyond the bandit setting.
• The error bounds for the proposed algorithms are comparable in order of magnitude to those of non-oblivious algorithms, demonstrating that distribution obliviousness is feasible without sacrificing theoretical guarantees.
• The framework successfully integrates risk-awareness (via CVaR) into the best-arm identification problem while maintaining distribution obliviousness, a novel contribution in the bandit literature.