[Paper Review] Episodic Multi-armed Bandits

Cem Tekin, Mihaela van der Schaar|arXiv (Cornell University)|Aug 4, 2015

Advanced Bandit Algorithms Research15 references3 citations

TL;DR

This paper introduces episodic multi-armed bandits (eMAB), a new online learning framework where agents select action sequences across episodes, receive feedback after each action, and can terminate an episode via a 'stop' action to reveal terminal rewards and step-level costs. The proposed FeedBAL algorithm achieves logarithmic regret with high probability, growing as O(log n) in the number of episodes, with polynomial dependence on steps, actions, and states.

ABSTRACT

We introduce a new class of reinforcement learning methods referred to as {\em episodic multi-armed bandits} (eMAB). In eMAB the learner proceeds in {\em episodes}, each composed of several {\em steps}, in which it chooses an action and observes a feedback signal. Moreover, in each step, it can take a special action, called the $stop$ action, that ends the current episode. After the $stop$ action is taken, the learner collects a terminal reward, and observes the costs and terminal rewards associated with each step of the episode. The goal of the learner is to maximize its cumulative gain (i.e., the terminal reward minus costs) over all episodes by learning to choose the best sequence of actions based on the feedback. First, we define an {\em oracle} benchmark, which sequentially selects the actions that maximize the expected immediate gain. Then, we propose our online learning algorithm, named {\em FeedBack Adaptive Learning} (FeedBAL), and prove that its regret with respect to the benchmark is bounded with high probability and increases logarithmically in expectation. Moreover, the regret only has polynomial dependence on the number of steps, actions and states. eMAB can be used to model applications that involve humans in the loop, ranging from personalized medical screening to personalized web-based education, where sequences of actions are taken in each episode, and optimal behavior requires adapting the chosen actions based on the feedback.

Motivation & Objective

• To address sequential decision-making problems where feedback is delayed until episode termination, and action order affects outcomes.
• To model real-world applications such as personalized education and healthcare, where treatment or teaching sequences are evaluated only after completion.
• To design an online learning algorithm that adapts to feedback during episodes and competes with a benchmark that selects actions myopically based on current feedback.
• To establish theoretical regret bounds that grow logarithmically in the number of episodes and polynomially in the number of steps, actions, and states.

Proposed method

• Formalizes eMAB as a framework where learning proceeds in episodes, each consisting of sequential actions, intermediate feedback, and a 'stop' action that triggers terminal reward and cost revelation.
• Defines a benchmark that, at each step, selects the action maximizing immediate expected gain based on current feedback, serving as the performance target.
• Proposes FeedBack Adaptive Learning (FeedBAL), an online algorithm that uses feedback to adaptively select actions within each episode to minimize regret relative to the benchmark.
• Employs high-probability confidence bounds on expected gains to guide action selection and ensure robustness to uncertainty.
• Derives regret bounds using concentration inequalities, showing that expected regret grows logarithmically in the number of episodes and polynomially in system size.
• Demonstrates that eMAB generalizes existing models like online adaptive submodular maximization and bridges standard MAB with full MDP-based reinforcement learning.

Experimental results

Research questions

• RQ1 Can an online learning algorithm be designed to effectively learn optimal action sequences in episodic settings with delayed feedback?
• RQ2 How can regret be bounded in a framework where rewards are only revealed after episode termination and intermediate feedback is used to guide decisions?
• RQ3 What is the theoretical performance limit of an algorithm competing against a myopic benchmark that adapts its action choice based on real-time feedback?
• RQ4 How does the regret scale with the number of episodes, steps per episode, and number of actions or states in the system?
• RQ5 In what ways does eMAB generalize or differ from standard multi-armed bandits, reinforcement learning in MDPs, and online convex optimization?

Key findings

• The FeedBAL algorithm achieves regret that grows logarithmically in the number of episodes, specifically O(log n), with high probability.
• The regret bound has polynomial dependence on the number of steps, actions, and states, making it scalable to moderately large systems.
• The theoretical analysis proves that the expected regret of FeedBAL is bounded with high probability, ensuring reliable performance guarantees.
• eMAB generalizes existing models such as online adaptive submodular maximization and provides a middle ground between standard MAB and full MDP-based RL.
• The framework is well-suited for applications involving human-in-the-loop decision making, such as personalized education and healthcare, where sequences of actions are evaluated only after completion.
• The model distinguishes itself from OCO and SCO by measuring regret against an adaptive benchmark that changes per episode based on feedback, rather than a fixed action.