[Paper Review] Constructing confidence sets after lasso selection by randomized estimator augmentation

Seunghyun Min, Qing Zhou|arXiv (Cornell University)|Apr 16, 2019

Statistical Methods and Inference15 references4 citations

TL;DR

This paper proposes a novel method for constructing valid joint confidence sets after lasso selection using randomized estimator augmentation and Markov chain Monte Carlo (MCMC) sampling from the conditional distribution of the response given the lasso active set. By incorporating randomization over estimated mean vectors, the approach enables accurate, low-volume confidence sets with guaranteed coverage, outperforming state-of-the-art methods in both coverage and efficiency.

ABSTRACT

Although a few methods have been developed recently for building confidence intervals after model selection, how to construct confidence sets for joint post-selection inference is still an open question. In this paper, we develop a new method to construct confidence sets after lasso variable selection, with strong numerical support for its accuracy and effectiveness. A key component of our method is to sample from the conditional distribution of the response $y$ given the lasso active set, which, in general, is very challenging due to the tiny probability of the conditioning event. We overcome this technical difficulty by using estimator augmentation to simulate from this conditional distribution via Markov chain Monte Carlo given any estimate $ ildeμ$ of the mean $μ_0$ of $y$. We then incorporate a randomization step for the estimate $ ildeμ$ in our sampling procedure, which may be interpreted as simulating from a posterior predictive distribution by averaging over the uncertainty in $μ_0$. Our Monte Carlo samples offer great flexibility in the construction of confidence sets for multiple parameters. Extensive numerical results show that our method is able to construct confidence sets with the desired coverage rate and, moreover, that the diameter and volume of our confidence sets are substantially smaller in comparison with a state-of-the-art method.

Motivation & Objective

• To address the open problem of constructing joint confidence sets for multiple parameters after data-driven lasso model selection.
• To overcome the computational challenge of sampling from the conditional distribution of the response given the lasso active set, which has low probability in high dimensions.
• To develop a flexible, scalable framework for post-selection inference that maintains valid coverage while minimizing confidence set volume.
• To extend beyond individual parameter inference to simultaneous inference on arbitrary subsets of selected variables.
• To improve numerical stability and efficiency compared to existing methods like Lee et al. (2016) by averaging over uncertainty in the mean estimate via randomization.

Proposed method

• Uses estimator augmentation to simulate from the conditional distribution of the response $ y $ given the lasso active set $ A $, enabling MCMC sampling under complex constraints.
• Employs a two-stage MCMC procedure: first, sample from the conditional distribution of $ y $ given $ A $ and a fixed estimate $ ilde{ u} $, then randomize over $ ilde{ u} $ to approximate the posterior predictive distribution.
• Introduces a randomization step over the estimated mean $ ilde{ u} $, which allows for averaging over uncertainty in $ u_0 $, improving robustness and coverage.
• Constructs confidence sets by leveraging Monte Carlo samples from the joint conditional distribution, enabling flexible inference on arbitrary subsets $ u_B $ of the selected parameters.
• Uses a Metropolis-Hastings algorithm with tailored proposals that respect the polyhedral constraints induced by the lasso active set and variable selection thresholds.
• Handles variable active set changes during MCMC by using truncated normal proposals for coefficients and dynamic reconfiguration of the active set $ ilde{eta}_j $.

Experimental results

Research questions

• RQ1 Can valid joint confidence sets be constructed after lasso selection when the active set is data-driven and high-dimensional?
• RQ2 How can one efficiently sample from the conditional distribution of $ y $ given the lasso active set, a distribution with very low probability in high dimensions?
• RQ3 Can randomized estimator augmentation improve the numerical stability and efficiency of post-selection inference compared to existing methods?
• RQ4 Does the proposed method yield confidence sets with better coverage and smaller volume than state-of-the-art approaches like Lee et al. (2016)?
• RQ5 Can the method be generalized to handle group structures or non-lasso selection procedures?

Key findings

• The proposed method achieves the nominal coverage rate for joint confidence sets, even in high-dimensional settings with $ p > n $.
• Numerical results show that the diameter and volume of the confidence sets are substantially smaller than those produced by Lee et al. (2016), indicating improved precision.
• The method maintains valid coverage even when the selected model is not the true model, by conditioning only on the active set and not assuming a true linear model.
• Randomization over the estimated mean $ ilde{ u} $ significantly improves numerical stability and reduces the variance of the confidence set estimates.
• The MCMC sampler successfully navigates the complex, irregular support of the conditional distribution by dynamically adjusting the active set and using truncated proposals.
• The method generalizes naturally to arbitrary subsets $ B eq ext{all} $, avoiding the overly conservative family-wise error rate control of simultaneous individual intervals.