[论文解读] Continuous-time multi-armed bandits under random intervention times

Kei Noba, José Luis Pérez+2 more|arXiv (Cornell University)|Mar 4, 2026

Advanced Bandit Algorithms Research被引用 0

一句话总结

这篇论文给出连续时间多臂老虎机在随机更新时的显式吉廷斯指数表征，包括 Lévy 驱动的臂和指数互到情形，并证明了吉廷斯策略的最优性。

ABSTRACT

This paper examines multi-armed bandits in which actions are taken at random discrete times. The model consists of $J$ independent arms. When an arm is operated, it must remain active for a random duration, modeled by the inter-arrival time of a (possibly arm-dependent) renewal process. For arms evolving as a Lévy process, we provide an explicit characterization of the Gittins index, which is known to yield an optimal strategy. Furthermore, when the inter-arrival times are exponential and the arms evolve as either a spectrally negative Lévy process, a reflected spectrally negative Lévy process, or a diffusion process, the Gittins index is explicitly characterized in terms of the scale function or diffusion characteristics, respectively. Numerical experiments are performed to support the theoretical results.

研究动机与目标

• Motivate the study of multi-armed bandits where actions are taken at random times and arms stay active for random renewals.
• Provide an explicit Gittins index characterization for arms evolving as Lévy processes.
• Derive explicit Gittins index expressions under exponential renewal times for spectrally negative Lévy, reflected spectrally negative Lévy, and diffusion arms.
• Show asymptotic and convergence results linking exponential-renewal indices to classical continuous-time indices.

提出的方法

• Define a multi-armed bandit with J independent arms and arm-specific renewal times.
• Formulate the discounted reward and the Gittins index as an optimal stopping problem for each arm.
• Derive a general Gittins index expression for Lévy-driven arms using fluctuation theory.
• Obtain explicit index formulas under exponential inter-arrival times for spectrally negative Lévy, reflected spectrally negative Lévy, and diffusion processes via scale functions or diffusion characteristics.
• Prove asymptotic behavior and convergence results, including weak convergence of measures mu^λ to mu^∞ and index convergence.

实验结果

研究问题

• RQ1 How can the Gittins index be explicitly characterized when arms follow general Lévy processes with random renewal times?
• RQ2 What are the explicit forms of the Gittins index under exponential renewal times for spectrally negative Lévy, reflected spectrally negative Lévy, and diffusion arms?
• RQ3 How does the Gittins index behave as the renewal rate grows large, and does it converge to the classical continuous-time index?
• RQ4 Does the Gittins index policy remain optimal under arm-dependent renewal times?
• RQ5 How can fluctuation theory and scale functions be leveraged to compute the index in these settings?

主要发现

• The Gittins index strategy is optimal for the continuous-time bandit with random intervention times.
• An explicit Gittins index characterization is obtained for general Lévy-process arms.
• With exponential renewal times, closed-form index expressions are derived for spectrally negative, reflected spectrally negative, and diffusion arms in terms of scale functions or diffusion data.
• The index converges to the continuous-time limit as the exponential rate increases, via a weak convergence result mu^λ ⇒ mu^∞.
• Asymptotic behavior shows the index tends to the reward function as the renewal rate tends to zero.
• Numerical experiments are provided to support the theoretical results.