[Paper Review] Multi-Armed Bandits for Correlated Markovian Environments with Smoothed Reward Feedback

Tanner Fiez, Shreyas Sekar+1 more|arXiv (Cornell University)|Mar 11, 2018

Advanced Bandit Algorithms Research35 references3 citations

TL;DR

This paper proposes EpochUCB and EpochGreedy algorithms for multi-armed bandits in correlated Markovian environments with smoothed reward feedback, where the decision-maker lacks state information and observes only batched, averaged rewards per epoch. The algorithms achieve sublinear regret by using arm-dependent, linearly increasing epoch lengths, overcoming the linear regret suffered by UCB and ε-greedy in such settings.

ABSTRACT

We study a multi-armed bandit problem in a dynamic environment where arm rewards evolve in a correlated fashion according to a Markov chain. Different than much of the work on related problems, in our formulation a learning algorithm does not have access to either a priori information or observations of the state of the Markov chain and only observes smoothed reward feedback following time intervals we refer to as epochs. We demonstrate that existing methods such as UCB and $\varepsilon$-greedy can suffer linear regret in such an environment. Employing mixing-time bounds on Markov chains, we develop algorithms called EpochUCB and EpochGreedy that draw inspiration from the aforementioned methods, yet which admit sublinear regret guarantees for the problem formulation. Our proposed algorithms proceed in epochs in which an arm is played repeatedly for a number of iterations that grows linearly as a function of the number of times an arm has been played in the past. We analyze these algorithms under two types of smoothed reward feedback at the end of each epoch: a reward that is the discount-average of the discounted rewards within an epoch, and a reward that is the time-average of the rewards within an epoch.

Motivation & Objective

• To address multi-armed bandit problems where rewards evolve via a correlated Markov chain, without access to the underlying state or immediate feedback.
• To model real-world systems where reward feedback is batch-processed and delayed, such as digital platforms and e-commerce.
• To demonstrate that standard bandit algorithms like UCB and ε-greedy suffer linear regret in this setting due to failure to account for state correlation and delayed feedback.
• To develop new algorithms that achieve sublinear regret despite limited feedback and no state observation.
• To analyze performance under two types of smoothed feedback: discount-averaged and time-averaged rewards within epochs.

Proposed method

• Algorithms operate in epochs where each arm is played repeatedly for a duration that grows linearly with the number of past plays of that arm.
• Epoch length is determined by the cumulative number of times an arm has been selected, ensuring longer exploration per arm over time.
• Smoothed reward feedback is computed as either the time-average or discount-average of instantaneous rewards within each epoch.
• Theoretical analysis leverages mixing-time bounds on Markov chains to control the deviation of observed rewards from stationary distribution rewards.
• Regret bounds are derived by decomposing the regret into components related to mixing time and exploration-exploitation trade-off.
• The algorithms are designed to stabilize the state distribution toward stationarity, enabling reliable reward estimation despite delayed and smoothed feedback.

Experimental results

Research questions

• RQ1 Can standard bandit algorithms like UCB and ε-greedy achieve sublinear regret in a correlated Markovian environment with only smoothed, batched feedback?
• RQ2 What is the impact of fixed versus increasing epoch lengths on regret in such environments?
• RQ3 How can sublinear regret be guaranteed when the decision-maker has no access to the underlying Markov state and only observes averaged rewards?
• RQ4 Under what conditions do smoothed reward feedback models (time-average vs. discount-average) enable effective learning in correlated Markovian bandits?
• RQ5 Can mixing-time properties of Markov chains be exploited to bound the regret penalty in non-stationary, correlated reward settings?

Key findings

• Standard UCB and ε-greedy algorithms suffer linear regret in the proposed correlated Markovian environment with smoothed feedback, due to failure to stabilize the state distribution.
• EpochUCB and EpochGreedy achieve sublinear regret by using arm-dependent, linearly increasing epoch lengths that allow the system to approach stationarity.
• The regret bounds scale with the mixing time of the underlying Markov chain, demonstrating that slower mixing leads to higher regret.
• The analysis shows that constant-length epochs lead to linear regret, justifying the need for adaptive epoch growth in such settings.
• The smoothed reward feedback model—whether time-average or discount-average—enables reliable estimation of stationary rewards when combined with sufficiently long epochs.
• Theoretical guarantees are established for both discount-averaged and time-averaged feedback, showing that sublinear regret is achievable under both formulations.