[Paper Review] Online Learning for Distribution-Free Prediction

Dave Zachariah, Petre Stoica+1 more|arXiv (Cornell University)|Mar 15, 2017

Gaussian Processes and Bayesian Inference39 references3 citations

TL;DR

This paper proposes an online, distribution-free learning method for prediction that uses covariance-fitting to estimate hyperparameters in linear regression and combiner-type predictors. The approach enables linear-time, constant-memory online updates, avoids local minima, prunes irrelevant features, and supports efficient, distribution-free prediction intervals via split conformal inference.

ABSTRACT

We develop an online learning method for prediction, which is important in problems with large and/or streaming data sets. We formulate the learning approach using a covariance-fitting methodology, and show that the resulting predictor has desirable computational and distribution-free properties: It is implemented online with a runtime that scales linearly in the number of samples; has a constant memory requirement; avoids local minima problems; and prunes away redundant feature dimensions without relying on restrictive assumptions on the data distribution. In conjunction with the split conformal approach, it also produces distribution-free prediction confidence intervals in a computationally efficient manner. The method is demonstrated on both real and synthetic datasets.

Motivation & Objective

• Address the challenge of scalable, real-time prediction in large or growing datasets with streaming data.
• Develop a learning method that avoids restrictive distributional assumptions on input data.
• Enable online, incremental learning with constant memory and linear computational cost per update.
• Support tractable, distribution-free prediction intervals without relying on parametric assumptions.
• Automatically prune irrelevant feature dimensions without prior knowledge of data distribution.

Proposed method

• Formulates predictor learning via a covariance-fitting criterion that generalizes existing methods to non-zero mean structures.
• Applies cyclic minimization to solve the hyperparameter estimation problem, ensuring monotonic decrease in the cost function.
• Derives closed-form update rules for weights in both linear regression (Lr) and linear combiner (Lc) forms using recursive updates.
• Introduces a two-phase update strategy: one for features with zero prior weight (early stage), another for active features with sparsity-inducing penalties.
• Uses the split conformal framework to generate distribution-free prediction intervals with minimal computational overhead.
• Employs recursive computation of key statistics (e.g., $oldsymbol{ ho}$, $oldsymbol{ ho}$, $oldsymbol{ ho}$) to maintain constant memory and linear runtime.

Experimental results

Research questions

• RQ1 Can a distribution-free online learning method be designed that scales linearly with data size and uses constant memory?
• RQ2 How can hyperparameter learning be made robust to unknown data distributions while avoiding local minima?
• RQ3 Can feature pruning be achieved automatically without assuming specific data distributions or sparsity structures?
• RQ4 Can distribution-free prediction intervals be efficiently computed in an online setting using this method?
• RQ5 How does the proposed online method compare in performance and scalability to offline cross-validation on real and synthetic data?

Key findings

• The proposed method achieves online learning with linear runtime and constant memory, enabling deployment on large or streaming datasets.
• The algorithm avoids local minima due to the convexity of the cost function and cyclic minimization, ensuring convergence to a global solution.
• Feature pruning occurs naturally through the regularization structure, suppressing irrelevant dimensions without prior distributional assumptions.
• The method supports computationally efficient, distribution-free prediction intervals via integration with the split conformal framework.
• Empirical evaluation on real and synthetic datasets shows competitive predictive performance compared to offline cross-validation, with significant gains in computational efficiency.
• The method is robust to unknown data distributions and maintains stable performance across diverse data regimes.