[Paper Review] Cheap Bandits

Manjesh K. Hanawal, Venkatesh Saligrama+2 more|arXiv (Cornell University)|Jun 15, 2015

Advanced Bandit Algorithms Research30 references20 citations

TL;DR

This paper introduces CheapUCB, a bandit algorithm that leverages smooth reward structures over a graph to minimize sensing cost while matching the regret performance of existing methods. By exploiting group-level average reward observations—cheaper than individual action sensing—it achieves linear cost scaling and establishes a $ O( sqrt{dT})$ lower bound on regret for spectral bandits on graphs with effective dimension $d$.

ABSTRACT

We consider stochastic sequential learning problems where the learner can observe the extit{average reward of several actions}. Such a setting is interesting in many applications involving monitoring and surveillance, where the set of the actions to observe represent some (geographical) area. The importance of this setting is that in these applications, it is actually extit{cheaper} to observe average reward of a group of actions rather than the reward of a single action. We show that when the reward is extit{smooth} over a given graph representing the neighboring actions, we can maximize the cumulative reward of learning while extit{minimizing the sensing cost}. In this paper we propose CheapUCB, an algorithm that matches the regret guarantees of the known algorithms for this setting and at the same time guarantees a linear cost again over them. As a by-product of our analysis, we establish a $\Omega(\sqrt{dT})$ lower bound on the cumulative regret of spectral bandits for a class of graphs with effective dimension $d$.

Motivation & Objective

• To address sequential learning problems where observing group averages of actions is cheaper than observing individual actions.
• To model reward smoothness over a graph structure representing neighboring actions in surveillance and monitoring applications.
• To design an algorithm that minimizes sensing cost while maintaining optimal regret performance.
• To establish a fundamental lower bound on regret for spectral bandits on graphs with effective dimension $d$.

Proposed method

• Model the reward as smooth over a graph, where neighboring actions have similar rewards.
• Use group-level average reward observations as feedback, which are cheaper to obtain than individual action rewards.
• Design CheapUCB, a UCB-based algorithm that selects groups of actions to observe based on upper confidence bounds.
• Leverage spectral properties of the graph to bound regret and ensure cost efficiency.
• Analyze regret under smoothness assumptions using graph effective dimension $d$.
• Derive a $ Omega(\sqrt{dT})$ lower bound on cumulative regret for a class of graphs with effective dimension $d$.

Experimental results

Research questions

• RQ1 Can we design a bandit algorithm that reduces sensing cost by using group average rewards without sacrificing regret performance?
• RQ2 How does reward smoothness over a graph affect the trade-off between sensing cost and regret in sequential learning?
• RQ3 What is the fundamental limit on regret for spectral bandits on graphs with effective dimension $d$?
• RQ4 Can we achieve linear cost scaling in sensing while matching the regret guarantees of standard bandit algorithms?

Key findings

• CheapUCB achieves the same regret guarantees as standard bandit algorithms while reducing sensing cost to linear scaling.
• The algorithm exploits reward smoothness over a graph to make efficient group observations, significantly lowering operational cost.
• A $ Omega(\sqrt{dT})$ lower bound on cumulative regret is established for spectral bandits on graphs with effective dimension $d$.
• The lower bound confirms that the regret scaling of CheapUCB is information-theoretically optimal for this class of graphs.
• The results show that group observation is not only cost-effective but also theoretically sound under smooth reward assumptions.
• The analysis demonstrates that spectral properties of the graph, particularly its effective dimension, directly govern the fundamental limits of regret in this setting.