[Paper Review] Rising Multi-Armed Bandits with Known Horizons

Seockbean Song, Chenyu Gan+4 more|arXiv (Cornell University)|Feb 11, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

The paper introduces CURE-UCB, a horizon-aware algorithm for Rising Multi-Armed Bandits (RMAB) with known horizons, and provides theoretical regret guarantees plus extensive empirical validation showing its superiority over horizon-agnostic baselines.

ABSTRACT

The Rising Multi-Armed Bandit (RMAB) framework models environments where expected rewards of arms increase with plays, which models practical scenarios where performance of each option improves with the repeated usage, such as in robotics and hyperparameter tuning. For instance, in hyperparameter tuning, the validation accuracy of a model configuration (arm) typically increases with each training epoch. A defining characteristic of RMAB is em horizon-dependent optimality: unlike standard settings, the optimal strategy here shifts dramatically depending on the available budget $T$. This implies that knowledge of $T$ yields significantly greater utility in RMAB, empowering the learner to align its decision-making with this shifting optimality. However, the horizon-aware setting remains underexplored. To address this, we propose a novel CUmulative Reward Estimation UCB (CURE-UCB) that explicitly integrates the horizon. We provide a rigorous analysis establishing a new regret upper bound and prove that our method strictly outperforms horizon-agnostic strategies in structured environments like ``linear-then-flat'' instances. Extensive experiments demonstrate its significant superiority over baselines.

Motivation & Objective

• Highlight horizon-dependent optimality in finite-horizon RMAB settings.
• Propose and formalize the CURE-UCB algorithm that leverages known horizon to estimate cumulative potential.
• prove dominance over horizon-agnostic methods in structured environments and establish regret bounds for general concave rising environments.
• Validate the approach through synthetic experiments and a real-world online model selection task (IMDB).

Proposed method

• Define RMAB with concave rising reward functions and horizon-dependent optimal policy (single-arm play maximization of cumulative reward).
• Introduce CURE-UCB which builds a horizon-adaptive index B_i(t) that estimates bounded cumulative reward over the remaining horizon.
• Decompose B_i(t) into recent average, estimated future gain, and an exploration bonus with a sliding window h_i.
• Provide initialization by sampling each arm twice, then iterative arm selection by argmax of B_i(t).
• Present theoretical results: strict dominance in Linear-Then-Flat (LTF) settings (Theorem 5.2) and a general regret upper bound for concave rising environments (Theorem 5.4).
• Compare with horizon-agnostic (R-ed-UCB) and non-stationary baselines, illustrating horizon-aware benefits.

Experimental results

Research questions

• RQ1 How does explicit horizon knowledge affect optimal sequencing of arm pulls in RMAB?
• RQ2 Can a horizon-aware estimator outperform horizon-agnostic methods across structured and general rising reward settings?
• RQ3 What are the theoretical guarantees (dominance and regret bounds) of horizon-aware strategies in RMAB?
• RQ4 Do empirical results on synthetic (LTF and concave) and real-world tasks support horizon-aware advantages?

Key findings

• CURE-UCB consistently achieves lower cumulative regret than baselines across horizons in both LTF and concave settings.
• In structured LTF environments, CURE-UCB strictly dominates horizon-agnostic R-ed-UCB (Theorem 5.2).
• The approach yields a general regret upper bound for concave rising environments (Theorem 5.4).
• Empirical results show CURE-UCB ranks best or near best across synthetic benchmarks and online model selection tasks (IMDB).
• The horizon-adaptive index B_i(t) effectively identifies when to invest in growth versus exploit immediate returns.