[Paper Review] Truncated Linear Regression in High Dimensions

Constantinos Daskalakis, Dhruv Rohatgi+1 more|arXiv (Cornell University)|Jul 29, 2020

Sparse and Compressive Sensing Techniques20 references4 citations

TL;DR

This paper proposes a computationally and statistically efficient method for high-dimensional truncated linear regression using Stochastic Gradient Descent (SGD) on a truncated LASSO formulation. It achieves an optimal ℓ₂ reconstruction error of O(√(k log n)/m) for recovering k-sparse n-dimensional vectors x*, under mild assumptions on the truncation set and design matrix, generalizing prior work on sparse regression and truncated regression separately.

ABSTRACT

As in standard linear regression, in truncated linear regression, we are given access to observations $(A_i, y_i)_i$ whose dependent variable equals $y_i= A_i^{ m T} \cdot x^* + η_i$, where $x^*$ is some fixed unknown vector of interest and $η_i$ is independent noise; except we are only given an observation if its dependent variable $y_i$ lies in some "truncation set" $S \subset \mathbb{R}$. The goal is to recover $x^*$ under some favorable conditions on the $A_i$'s and the noise distribution. We prove that there exists a computationally and statistically efficient method for recovering $k$-sparse $n$-dimensional vectors $x^*$ from $m$ truncated samples, which attains an optimal $\ell_2$ reconstruction error of $O(\sqrt{(k \log n)/m})$. As a corollary, our guarantees imply a computationally efficient and information-theoretically optimal algorithm for compressed sensing with truncation, which may arise from measurement saturation effects. Our result follows from a statistical and computational analysis of the Stochastic Gradient Descent (SGD) algorithm for solving a natural adaptation of the LASSO optimization problem that accommodates truncation. This generalizes the works of both: (1) [Daskalakis et al. 2018], where no regularization is needed due to the low-dimensionality of the data, and (2) [Wainright 2009], where the objective function is simple due to the absence of truncation. In order to deal with both truncation and high-dimensionality at the same time, we develop new techniques that not only generalize the existing ones but we believe are of independent interest.

Motivation & Objective

• Address the challenge of recovering high-dimensional, sparse vectors from truncated linear regression data where only responses within a truncation set S are observed.
• Overcome the limitations of standard linear regression and LASSO in high-dimensional (m ≪ n) and truncated settings.
• Develop a method that is both computationally efficient and statistically optimal under sparsity and truncation.
• Generalize prior results on sparse regression (Wainwright) and truncated regression (Daskalakis et al.) to handle both high-dimensionality and truncation simultaneously.

Proposed method

• Adapt the standard LASSO optimization to incorporate truncation by modifying the objective function to account for the likelihood of observations within the truncation set S.
• Apply Stochastic Gradient Descent (SGD) to solve the truncated LASSO problem, enabling scalability to high-dimensional settings.
• Use an evaluation oracle and inverse function approximation to enable efficient sampling from truncated Gaussian distributions.
• Leverage concentration and anti-concentration properties of Gaussian design matrices and noise to ensure convergence and error bounds.
• Establish that the truncation set S permits a constant fraction of samples to survive, ensuring sufficient information for recovery.
• Prove that the SGD algorithm on the truncated LASSO achieves optimal statistical error by combining sparsity-inducing ℓ₁ regularization with likelihood-based truncation correction.

Experimental results

Research questions

• RQ1 Can we achieve optimal ℓ₂ reconstruction error in high-dimensional truncated linear regression when m ≪ n and x* is k-sparse?
• RQ2 Is there a computationally efficient algorithm that simultaneously handles sparsity and truncation in linear regression?
• RQ3 How does the interplay between truncation and high-dimensionality affect the statistical and computational complexity of parameter recovery?
• RQ4 Can SGD on a truncated LASSO formulation achieve the information-theoretically optimal error rate under mild assumptions on the truncation set and design matrix?
• RQ5 What novel analytical techniques are required to handle both truncation and sparsity beyond existing methods in sparse regression or truncated regression?

Key findings

• The proposed SGD-based method on the truncated LASSO achieves an ℓ₂ reconstruction error of O(√(k log n)/m), which is information-theoretically optimal.
• The method is computationally efficient and runs in polynomial time, even when m ≪ n.
• The analysis generalizes prior results: it recovers the optimal error rate from Wainwright (2009) for high-dimensional sparse regression and the truncation-robustness of Daskalakis et al. (2022) for low-dimensional truncated regression.
• The truncation set S must allow a constant fraction of samples to survive, ensuring sufficient signal for recovery.
• The method relies on novel techniques for approximate sampling from truncated Gaussians and inverse function approximation with high accuracy.
• The error bound scales with √(k log n)/m, reflecting the trade-off between sparsity, dimensionality, and sample size, and matches the known minimax lower bound.