[Paper Review] Online Learning with Predictable Sequences

Alexander Rakhlin, Karthik Sridharan|arXiv (Cornell University)|Aug 18, 2012

Advanced Bandit Algorithms Research16 references148 citations

TL;DR

This paper introduces online learning algorithms that leverage predictable sequences—where outcomes follow a known trend plus noise—to achieve tighter regret bounds than worst-case guarantees. By incorporating prior knowledge of the predictable process, the method adapts to benign sequences while maintaining robustness against adversarial noise, achieving improved performance in settings like time series and stock prediction.

ABSTRACT

We present methods for online linear optimization that take advantage of benign (as opposed to worst-case) sequences. Specifically if the sequence encountered by the learner is described well by a known "predictable process", the algorithms presented enjoy tighter bounds as compared to the typical worst case bounds. Additionally, the methods achieve the usual worst-case regret bounds if the sequence is not benign. Our approach can be seen as a way of adding prior knowledge about the sequence within the paradigm of online learning. The setting is shown to encompass partial and side information. Variance and path-length bounds can be seen as particular examples of online learning with simple predictable sequences. We further extend our methods and results to include competing with a set of possible predictable processes (models), that is "learning" the predictable process itself concurrently with using it to obtain better regret guarantees. We show that such model selection is possible under various assumptions on the available feedback. Our results suggest a promising direction of further research with potential applications to stock market and time series prediction.

Motivation & Objective

• Address the limitation of standard online learning methods that provide only worst-case regret bounds, which are often pessimistic for regular or structured sequences.
• Develop computationally feasible algorithms that achieve tighter regret bounds when the observed sequence is approximately predictable, i.e., follows a known process plus noise.
• Maintain standard worst-case regret guarantees when the sequence is not predictable, ensuring robustness.
• Extend the framework to allow model selection among multiple candidate predictable processes, learning the best one concurrently with optimization.
• Enable application to partial and side-information settings, including online convex optimization and bandit problems.

Proposed method

• Model the sequence as a predictable process $M_t$ plus adversarial noise, where deviations from $M_t$ are bounded by $σ_t$.
• Use a symmetrization argument and constrained adversary analysis to derive tighter regret bounds based on the total variance $\sum_{t=1}^T \sigma_t^2$.
• Apply the doubling trick to avoid requiring knowledge of $\sum_{t=1}^T \sigma_t^2$ in advance, enabling adaptive regret bounds.
• Adapt the method to online convex optimization via reduction to linear optimization, using self-concordant barriers for efficient computation.
• Extend to multi-armed bandits by reducing the problem to a linear bandit over the simplex, using unbiased estimators from observed rewards.
• Use the SCRiBLe algorithm with a self-concordant barrier on the simplex to derive regret bounds in the bandit setting, achieving $O(\eta^{-1} \log dT)$ dependence on time and dimension.

Experimental results

Research questions

• RQ1 Can online learning algorithms achieve tighter regret bounds when the sequence is approximately predictable, i.e., follows a known trend with bounded noise?
• RQ2 How can prior knowledge of a predictable process be integrated into online learning without sacrificing worst-case robustness?
• RQ3 Is it possible to simultaneously learn which predictable process (model) best fits the data while achieving low regret?
• RQ4 Can the framework be extended to partial and side-information settings, such as online convex optimization and bandit feedback?
• RQ5 What are the tightest achievable regret bounds when the predictable process is unknown but candidates are available for selection?

Key findings

• The proposed algorithm achieves regret bounds of the form $O\left(\sum_{t=1}^T \sigma_t^2\right)^{1/2}$ when the sequence is close to a predictable process, significantly improving over worst-case $O(\sqrt{T})$ bounds.
• The method maintains standard worst-case regret bounds of $O(\sqrt{T})$ when no predictable structure exists, ensuring robustness.
• Model selection across multiple predictable processes is possible, with regret scaling as $O(\eta^{-1} \log(dT))$ for the multi-armed bandit setting.
• The algorithm is adaptive: it does not require prior knowledge of $\sum_{t=1}^T \sigma_t^2$, thanks to the doubling trick.
• The framework extends to online convex optimization via reduction to linear optimization with self-concordant barriers.
• For multi-armed bandits, the algorithm achieves expected regret bounded by $\frac{1}{1 - 4\eta s d^2} \left( \inf_j \sum_{t=1}^T \langle e_j, x_t \rangle + d \eta^{-1} \log(dT) \right)$, demonstrating practical applicability.