[Paper Review] Online Active Linear Regression via Thresholding

Carlos Riquelme, Ramesh Johari+1 more|arXiv (Cornell University)|Feb 9, 2016

Machine Learning and Algorithms4 citations

TL;DR

This paper proposes a threshold-based online active learning algorithm for linear regression that selects the most informative data points by measuring their norm in a transformed space, significantly reducing mean squared error (MSE) compared to random sampling. The method achieves near-optimal performance in low- and high-dimensional settings, with theoretical guarantees and strong empirical robustness even under nonlinearity and high dimensionality.

ABSTRACT

We consider the problem of online active learning to collect data for regression modeling. Specifically, we consider a decision maker with a limited experimentation budget who must efficiently learn an underlying linear population model. Our main contribution is a novel threshold-based algorithm for selection of most informative observations; we characterize its performance and fundamental lower bounds. We extend the algorithm and its guarantees to sparse linear regression in high-dimensional settings. Simulations suggest the algorithm is remarkably robust: it provides significant benefits over passive random sampling in real-world datasets that exhibit high nonlinearity and high dimensionality --- significantly reducing both the mean and variance of the squared error.

Motivation & Objective

• Address the challenge of efficient data collection in online regression when labeling is costly and the budget for labeling is limited.
• Develop a computationally efficient algorithm that selects the most informative observations in real-time to minimize prediction error.
• Extend the algorithm to high-dimensional sparse linear models using adaptive thresholding and l1-regularization to maintain performance.
• Provide theoretical guarantees on performance, including upper and lower bounds on mean squared error (MSE), under well-specified linear models.
• Demonstrate robustness of the method in practice, especially in settings with nonlinearity, high dimensionality, and model misspecification.

Proposed method

• Use a thresholding rule based on the norm of covariate vectors in a space transformed by the estimated covariance matrix to select informative observations.
• Apply the algorithm sequentially in an online fashion, where each new observation is evaluated for labeling based on whether its norm exceeds a precomputed threshold.
• Introduce an adaptive extension that first learns the sparsity pattern of the true model using an initial budget before applying active learning to the relevant subspace.
• Leverage l1-regularization (e.g., Lasso) in high-dimensional settings to stabilize estimation and maintain performance despite noisy or irrelevant features.
• Use a theoretical framework to derive high-probability upper bounds on MSE and compare them to fundamental lower bounds to assess near-optimality.
• Implement and evaluate the algorithm using both synthetic data with controlled nonlinearity and real-world datasets such as Combined Cycle Power Plant and Protein Structure.

Experimental results

Research questions

• RQ1 Can a simple thresholding rule based on covariate norms achieve significant improvements in prediction accuracy over passive random sampling in online linear regression?
• RQ2 How does the performance of active learning degrade as the dimensionality of the feature space increases, and can this be mitigated in high-dimensional sparse settings?
• RQ3 To what extent does the algorithm remain effective when the underlying data-generating process exhibits nonlinearity, even though the model is assumed linear?
• RQ4 What is the theoretical relationship between the upper bound on MSE from the proposed algorithm and the fundamental lower bound on achievable performance?
• RQ5 Can the thresholding-based active learning approach maintain its benefits when combined with regularization techniques like ridge or Lasso in high-dimensional models?

Key findings

• The threshold-based active learning algorithm significantly reduces both the mean and variance of the prediction error compared to random sampling, even in high-dimensional and nonlinear settings.
• In low-dimensional settings with Gaussian covariates, the algorithm achieves an MSE bound of $ \sigma^2 d^2 / [kd + 2( u - 1)k \log k] $, improving over the passive baseline $ \sigma^2 d / k $ when $ k = \Omega(\exp(d)) $ or $ \nu = \Omega(d) $.
• The algorithm maintains strong performance under regularization: for ridge and Lasso estimators, the MSE of the thresholding-based selection is consistently lower than random sampling, with variance reduced to less than 30% in some cases.
• Even in the presence of nonlinearity (e.g., quadratic terms), the active learning approach outperforms random sampling up to a certain level of nonlinearity, after which random sampling becomes more effective.
• The method remains robust in real-world datasets such as Combined Cycle Power Plant and Protein Structure, where it reduces MSE even after including second-order interactions.
• The adaptive extension that learns the sparsity pattern first and then applies active learning to the relevant subspace achieves substantial gains in high-dimensional sparse models, with theoretical guarantees provided in Theorem 3.3.