[Paper Review] Modeling Human Decision-making in Generalized Gaussian Multi-armed Bandits

Paul Reverdy, Vaibhav Srivastava+1 more|arXiv (Cornell University)|Jul 23, 2013

Advanced Bandit Algorithms Research47 references4 citations

TL;DR

This paper proposes the Upper Credible Limit (UCL) algorithm for Bayesian multi-armed bandit problems with Gaussian rewards, modeling human decision-making under uncertainty. It proves logarithmic cumulative expected regret for deterministic and stochastic UCL variants, demonstrates improved performance with informative priors and correlation structure, and validates human behavior using empirical data, extending to transition costs and graph-structured arms via block and graphical block UCL algorithms with similar regret bounds.

ABSTRACT

We present a formal model of human decision-making in explore-exploit tasks using the context of multi-armed bandit problems, where the decision-maker must choose among multiple options with uncertain rewards. We address the standard multi-armed bandit problem, the multi-armed bandit problem with transition costs, and the multi-armed bandit problem on graphs. We focus on the case of Gaussian rewards in a setting where the decision-maker uses Bayesian inference to estimate the reward values. We model the decision-maker's prior knowledge with the Bayesian prior on the mean reward. We develop the upper credible limit (UCL) algorithm for the standard multi-armed bandit problem and show that this deterministic algorithm achieves logarithmic cumulative expected regret, which is optimal performance for uninformative priors. We show how good priors and good assumptions on the correlation structure among arms can greatly enhance decision-making performance, even over short time horizons. We extend to the stochastic UCL algorithm and draw several connections to human decision-making behavior. We present empirical data from human experiments and show that human performance is efficiently captured by the stochastic UCL algorithm with appropriate parameters. For the multi-armed bandit problem with transition costs and the multi-armed bandit problem on graphs, we generalize the UCL algorithm to the block UCL algorithm and the graphical block UCL algorithm, respectively. We show that these algorithms also achieve logarithmic cumulative expected regret and require a sub-logarithmic expected number of transitions among arms. We further illustrate the performance of these algorithms with numerical examples. NB: Appendix G included in this version details minor modifications that correct for an oversight in the previously-published proofs. The remainder of the text reflects the published work.

Motivation & Objective

• To formalize human decision-making in explore-exploit tasks using Bayesian multi-armed bandit frameworks.
• To develop and analyze the Upper Credible Limit (UCL) algorithm for standard, transition-cost, and graph-structured bandits with Gaussian rewards.
• To demonstrate that good priors and correlation assumptions significantly improve performance, even over short time horizons.
• To validate the stochastic UCL algorithm against empirical human behavioral data from controlled experiments.
• To extend the UCL framework to block and graphical block UCL algorithms for structured decision problems with sub-logarithmic transition costs.

Proposed method

• Uses Bayesian inference to model prior knowledge on mean rewards, with uninformative or informative priors.
• Proposes the deterministic UCL algorithm that selects the arm with the highest upper credible limit of the posterior mean.
• Introduces the stochastic UCL algorithm as a randomized variant to better capture human exploration behavior.
• Extends the UCL framework to block UCL for multi-armed bandits with transition costs and graphical block UCL for graph-structured arms.
• Applies a corrected proof technique using a tail bound from Garivier and Moulines to fix an oversight in earlier proofs regarding selection dependence.
• Empirically validates model performance by comparing stochastic UCL with human subject data from Princeton University IRB-approved experiments.

Experimental results

Research questions

• RQ1 How can human decision-making in explore-exploit tasks be formally modeled using Bayesian multi-armed bandits?
• RQ2 What is the theoretical performance of the UCL algorithm in terms of cumulative expected regret under uninformative and informative priors?
• RQ3 To what extent do prior knowledge and correlation assumptions among arms improve decision-making performance?
• RQ4 Can the stochastic UCL algorithm effectively capture human behavioral patterns in bandit tasks?
• RQ5 How do the block UCL and graphical block UCL algorithms perform in structured environments with transition costs or graph constraints?

Key findings

• The deterministic UCL algorithm achieves logarithmic cumulative expected regret, which is optimal for uninformative priors.
• The stochastic UCL algorithm captures human decision-making behavior effectively when calibrated with appropriate parameters.
• Good priors and correlation assumptions reduce regret significantly, even over short time horizons, improving performance beyond standard algorithms.
• The block UCL and graphical block UCL algorithms achieve logarithmic cumulative expected regret and require a sub-logarithmic expected number of transitions among arms.
• Corrected proofs using Garivier and Moulines' tail bound validate the original regret bounds, with revised bounds showing $\mathbb{E}[n_i^T] \leq \frac{8a\sigma_s^2}{\Delta_i^2}\log T + o(\log T)$ for deterministic UCL.
• Empirical data from human experiments confirm that the stochastic UCL model with tuned parameters closely matches observed human behavior.