[Paper Review] Adaptive Feature Selection: Computationally Efficient Online Sparse Linear Regression under RIP

Satyen Kale, Zohar Karnin+2 more|arXiv (Cornell University)|Jun 14, 2017

Sparse and Compressive Sensing Techniques19 references3 citations

TL;DR

This paper presents computationally efficient online sparse linear regression algorithms under the restricted isometry property (RIP), enabling sublinear regret for both realizable and agnostic settings. By leveraging weakly supermodular function properties and online optimization techniques like BEXP and VAW, the method adaptively selects informative features in real time, achieving regret bounds that scale sublinearly with time T while maintaining polynomial-time complexity under RIP assumptions.

ABSTRACT

Online sparse linear regression is an online problem where an algorithm repeatedly chooses a subset of coordinates to observe in an adversarially chosen feature vector, makes a real-valued prediction, receives the true label, and incurs the squared loss. The goal is to design an online learning algorithm with sublinear regret to the best sparse linear predictor in hindsight. Without any assumptions, this problem is known to be computationally intractable. In this paper, we make the assumption that data matrix satisfies restricted isometry property, and show that this assumption leads to computationally efficient algorithms with sublinear regret for two variants of the problem. In the first variant, the true label is generated according to a sparse linear model with additive Gaussian noise. In the second, the true label is chosen adversarially.

Motivation & Objective

• To address the computational intractability of online sparse linear regression under standard assumptions.
• To design polynomial-time algorithms with sublinear regret for online sparse linear regression.
• To establish that the restricted isometry property (RIP) enables tractable solutions in both realizable and agnostic settings.
• To enable adaptive feature selection by dynamically choosing the most informative features at each round under budget constraints.
• To resolve the open problem of what assumptions make online sparse regression computationally feasible.

Proposed method

• The method assumes that the design matrix of feature vectors satisfies the restricted isometry property (RIP) for all sufficiently large t.
• It models the prediction error as a weakly supermodular set function g(S), which allows for greedy approximation via the BEXP algorithm.
• The algorithm divides the sequence into mini-batches and applies the VAW algorithm within each batch to minimize cumulative loss.
• It uses a two-stage approach: first, BEXP selects a k₁-sparse subset of features per batch based on expected error reduction; second, VAW learns the optimal weights on the selected features.
• The regret bound is derived by combining the approximation error of the greedy selection (via weak supermodularity) and the online learning regret of VAW.
• The analysis leverages concentration bounds and properties of the restricted condition number to control error propagation across batches.

Experimental results

Research questions

• RQ1 Can online sparse linear regression be made computationally efficient under a natural assumption like RIP?
• RQ2 Does the RIP assumption suffice to achieve sublinear regret in both realizable and agnostic settings?
• RQ3 Can adaptive feature selection be efficiently integrated into online sparse regression under bounded feature query budgets?
• RQ4 Is there a way to achieve regret sublinear in T while maintaining polynomial-time complexity?
• RQ5 Can the RIP assumption be weakened without losing computational tractability?

Key findings

• The proposed algorithm achieves regret bounded by O(κ²k√(d k₁ log(d) T / k₀)) + O(k₁ log T) under the RIP assumption, with sublinear growth in T.
• The regret bound is sublinear in T when k₁ = Ω(κ²k log T), ensuring the algorithm's performance approaches that of the best k-sparse predictor.
• The method guarantees that the expected cumulative loss is within O(κ²k√(d k₁ log(d) T / k₀)) of the optimal k-sparse predictor, with additional O(k₁ log T) loss from online learning.
• The analysis shows that the weak supermodularity of the error function under RIP enables effective greedy feature selection with bounded approximation error.
• The algorithm adaptsively selects different sparse feature subsets at each round, solving the adaptive feature selection problem efficiently.
• The results extend to the realizable setting under weaker conditions like irrepresentability, suggesting broader applicability beyond strict RIP.