[Paper Review] Fast Exact Conformalization of Lasso using Piecewise Linear Homotopy

Jing Lei|arXiv (Cornell University)|Aug 1, 2017

Statistical Methods and Inference3 citations

TL;DR

This paper proposes a fast, exact algorithm for conformal prediction in high-dimensional Lasso regression by leveraging piecewise linear homotopy to track Lasso solutions under perturbations of a single training point. The method computes exact prediction sets by identifying interval endpoints through a single Lasso fit, achieving significant computational gains over grid-based methods while preserving valid coverage under model misspecification.

ABSTRACT

Conformal prediction is a general method that converts almost any point predictor to a prediction set. The resulting set keeps good statistical properties of the original estimator under standard assumptions, and guarantees valid average coverage even when the model is misspecified. A main challenge in applying conformal prediction in modern applications is efficient computation, as it generally requires an exhaustive search over the entire output space. In this paper we develop an exact and computationally efficient conformalization of the Lasso and elastic net. The method makes use of a novel piecewise linear homotopy of the Lasso solution under perturbation of a single input sample point. As a by-product, we provide a simpler and better justified online Lasso algorithm, which may be of independent interest. Our derivation also reveals an interesting accuracy-stability trade-off in conformal inference, which is analogous to the bias-variance trade-off in traditional parameter estimation. The practical performance of the new algorithm is demonstrated using both synthetic and real data examples.

Motivation & Objective

• To address the high computational cost of conformal prediction in high-dimensional settings, particularly for Lasso regression.
• To develop an exact and efficient method for computing conformal prediction sets without relying on grid search or sample splitting.
• To establish a novel homotopy-based framework that tracks Lasso solutions under perturbation of a single data point.
• To reveal and analyze the accuracy-stability trade-off in conformal inference, analogous to the bias-variance trade-off.
• To provide a more interpretable and computationally efficient online Lasso algorithm as a by-product.

Proposed method

• The method uses a piecewise linear homotopy of the Lasso solution path under perturbation of a single training sample’s response value.
• It explicitly tracks the re-fitted residuals as a piecewise linear function of the candidate response value, enabling exact identification of prediction set boundaries.
• The algorithm derives the end points of prediction intervals by analyzing changes in active set and residual behavior along the homotopy path.
• It leverages the symmetry and exchangeability properties of conformal prediction to ensure valid average coverage under minimal assumptions.
• The approach avoids repeated model fitting by computing the entire homotopy path from a single initial Lasso fit on the original data.
• The method is extended to elastic net and can be adapted to other sparse estimation problems such as generalized Lasso and graphical Lasso.

Experimental results

Research questions

• RQ1 Can conformal prediction for Lasso be computed exactly and efficiently without grid search or sample splitting?
• RQ2 How does the piecewise linear homotopy of the Lasso solution under response perturbation enable exact computation of conformal prediction sets?
• RQ3 What is the relationship between residual stability and the width of conformal prediction intervals in high-dimensional regression?
• RQ4 How does the choice of regularization parameter affect the trade-off between accuracy, stability, and computational efficiency in conformal inference?
• RQ5 Can the homotopy framework be used to derive a more efficient and stable online Lasso algorithm?

Key findings

• The Lasso conformal prediction set is a union of intervals, and the algorithm computes the exact endpoints of these intervals using a single Lasso fit on the original data.
• The method reduces computational cost from O(100×) to near-linear in the number of candidate values, achieving significant speedups over grid-based approaches.
• The algorithm maintains valid average coverage under model misspecification, even when the true model is dense or the design matrix is ill-conditioned.
• Empirical results on diabetes and Boston housing data show that the method achieves near-ideal coverage with minimal variability across 100 sample splits.
• Smaller regularization parameters tend to yield shorter prediction intervals in the Boston housing data, indicating high predictability and low need for shrinkage.
• The inclusion of quadratic terms improves prediction performance in the Boston housing data, reducing interval length substantially, while having minimal effect in the diabetes data.