[Paper Review] Distributed High-dimensional Regression Under a Quantile Loss Function

Xi Chen, Weidong Liu+2 more|arXiv (Cornell University)|Jun 13, 2019

Statistical Methods and InferenceMathematics40 references32 citations

TL;DR

The paper develops a distributed estimator for high-dimensional regression with heavy-tailed noise via quantile loss, transforming QR to a penalized least squares form to enable efficient communication and support recovery.

ABSTRACT

This paper studies distributed estimation and support recovery for high-dimensional linear regression model with heavy-tailed noise. To deal with heavy-tailed noise whose variance can be infinite, we adopt the quantile regression loss function instead of the commonly used squared loss. However, the non-smooth quantile loss poses new challenges to high-dimensional distributed estimation in both computation and theoretical development. To address the challenge, we transform the response variable and establish a new connection between quantile regression and ordinary linear regression. Then, we provide a distributed estimator that is both computationally and communicationally efficient, where only the gradient information is communicated at each iteration. Theoretically, we show that, after a constant number of iterations, the proposed estimator achieves a near-oracle convergence rate without any restriction on the number of machines. Moreover, we establish the theoretical guarantee for the support recovery. The simulation analysis is provided to demonstrate the effectiveness of our method.

Motivation & Objective

• Motivate robust estimation for high-dimensional data with heavy-tailed noise in distributed settings.
• Propose a new distributed estimator that connects quantile regression to ordinary least squares for computational efficiency.
• Achieve near-oracle convergence rates with constant-number iterations.
• Establish theoretical guarantees for support recovery in a distributed QR context.
• Demonstrate computational and communication efficiency through a gradient-based coordination scheme.

Proposed method

• Transform quantile regression into a penalized least squares problem using a pseudo-response to enable Lasso mechanics.
• Develop a distributed approximate Newton method that communicates only p+1 dimensional gradients per iteration.
• Use kernel density estimation to estimate the QR density at zero as part of the transformation.
• Iteratively update with a sequence of density estimates and gradient aggregates to refine the estimator.
• Avoid communicating full covariance matrices by leveraging local computations and aggregated gradient-like terms.
• Provide an initial estimator from a local QR solve on one machine and refine it through multiple distributed rounds.

Experimental results

Research questions

• RQ1 Can high-dimensional regression with heavy-tailed noise be efficiently estimated in a distributed setting using quantile loss?
• RQ2 Does transforming QR into a penalized least squares problem enable sparse recovery and near-oracle rates with limited communication?
• RQ3 What are the convergence guarantees and support recovery conditions for the proposed distributed estimator under minimal assumptions?
• RQ4 How many iterations are needed to reach near-oracle performance in a distributed QR framework?
• RQ5 How can one design communication to rely on gradient information rather than full matrices without sacrificing accuracy?

Key findings

• A distributed estimator with a pseudo-response reduces QR to a squared loss problem, enabling efficient Lasso-type estimation.
• The method achieves near-oracle convergence rates after a constant number of iterations, without restricting the number of machines.
• Support recovery guarantees are established, with beta-min conditions that improve with iterations.
• The algorithm communicates only p+1 dimensional gradients per iteration, avoiding transmission of large matrices.
• The approach accommodates very heavy-tailed noise without requiring finite variance assumptions.
• Initial and iterative steps leverage kernel density estimation and local-remote gradient aggregation to ensure robustness and sparsity.