[Paper Review] Online Forecasting of Total-Variation-bounded Sequences

Dheeraj Baby, Yu-Xiang Wang|arXiv (Cornell University)|Jan 1, 2019

Stochastic processes and financial applications8 citations

TL;DR

This paper presents the first polynomial-time algorithm for online forecasting of total-variation-bounded sequences with unknown variation $C_n$, achieving a cumulative square error of $\tilde{O}(n^{1/3}C_n^{2/3}\sigma^{4/3} + C_n^2)$ with high probability. It leverages Haar wavelet structure and adaptivity to unknown smoothness, matching a tight lower bound up to logarithmic factors.

ABSTRACT

We consider the problem of online forecasting of sequences of length $n$ with total-variation at most $C_n$ using observations contaminated by independent $\sigma$-subgaussian noise. We design an $O(n\log n)$-time algorithm that achieves a cumulative square error of $ ilde{O}(n^{1/3}C_n^{2/3}\sigma^{4/3} + C_n^2)$ with high probability. We also prove a lower bound that matches the upper bound in all parameters (up to a $\log(n)$ factor). To the best of our knowledge, this is the first **polynomial-time** algorithm that achieves the optimal $O(n^{1/3})$ rate in forecasting total variation bounded sequences and the first algorithm that **adapts to unknown** $C_n$.Our proof techniques leverage the special localized structure of Haar wavelet basis and the adaptivity to unknown smoothness parameters in the classical wavelet smoothing [Donoho et al., 1998]. We also compare our model to the rich literature of dynamic regret minimization and nonstationary stochastic optimization, where our problem can be treated as a special case. We show that the workhorse in those settings --- online gradient descent and its variants with a fixed restarting schedule --- are instances of a class of **linear forecasters** that require a suboptimal regret of $ ilde{\Omega}(\sqrt{n})$. This implies that the use of more adaptive algorithms is necessary to obtain the optimal rate.

Motivation & Objective

• To design a computationally efficient online forecasting algorithm for sequences with bounded total variation.
• To achieve optimal prediction error rates in the presence of subgaussian noise when the total-variation bound $C_n$ is unknown.
• To close the gap between optimal theoretical rates and practical, polynomial-time algorithms in this setting.
• To demonstrate that standard online gradient descent methods are suboptimal for this problem due to their linear forecasting structure.

Proposed method

• The algorithm uses the localized structure of the Haar wavelet basis to efficiently represent and estimate total-variation-bounded sequences.
• It applies adaptive thresholding in the Haar wavelet domain to balance estimation error and noise sensitivity.
• The method incorporates a data-driven selection of regularization parameters based on noise level $\sigma$ and sequence length $n$, enabling adaptation to unknown $C_n$.
• It achieves $O(n\log n)$ runtime by exploiting sparsity in the Haar transform of bounded-variation signals.
• The analysis combines concentration inequalities with wavelet-specific properties to bound the cumulative square error.
• The algorithm's design is contrasted with linear forecasters like online gradient descent, which are shown to incur suboptimal $\tilde{\Omega}(\sqrt{n})$ regret.

Experimental results

Research questions

• RQ1 Can an efficient, polynomial-time algorithm achieve the optimal $n^{1/3}$ rate for online forecasting of total-variation-bounded sequences with unknown $C_n$?
• RQ2 What is the fundamental limit of performance for online forecasting under total-variation constraints with subgaussian noise?
• RQ3 Why do standard linear forecasters such as online gradient descent fail to achieve the optimal rate in this setting?
• RQ4 How can wavelet-based methods be leveraged to achieve both computational efficiency and statistical optimality?
• RQ5 To what extent does adaptivity to unknown smoothness parameters improve forecasting performance in this context?

Key findings

• The proposed algorithm achieves a cumulative square error of $\tilde{O}(n^{1/3}C_n^{2/3}\sigma^{4/3} + C_n^2)$ with high probability, matching the information-theoretic lower bound up to logarithmic factors.
• The algorithm runs in $O(n\log n)$ time, making it the first polynomial-time method to achieve the optimal $n^{1/3}$ rate for this problem.
• The method adapts to unknown $C_n$ without prior knowledge, which is a key advantage over prior approaches requiring $C_n$ as input.
• The analysis shows that online gradient descent and similar linear forecasters incur a suboptimal regret of $\tilde{\Omega}(\sqrt{n})$, proving they cannot achieve the optimal rate.
• The use of Haar wavelets enables both computational efficiency and statistical optimality by exploiting localized signal structure.
• The work establishes a tight connection between wavelet smoothing techniques and optimal online forecasting, extending classical results to the online setting.