[Paper Review] MAGIC: a general, powerful and tractable method for selective inference

Xiaoying Tian, Nan Bi+1 more|arXiv (Cornell University)|Jul 9, 2016

Statistical Methods and Inference18 references13 citations

TL;DR

MAGIC is a general, powerful, and tractable method for selective inference after solving convex optimization problems with smooth loss and ℓ₁ penalties. By incorporating randomization and reparametrizing constraints into a canonical sampling space, MAGIC enables efficient Monte Carlo sampling for valid inference, significantly improving statistical power over prior methods while maintaining computational tractability across Lasso, logistic Lasso, and graphical models.

ABSTRACT

Selective inference is a recent research topic that tries to perform valid inference after using the data to select a reasonable statistical model. We propose MAGIC, a new method for selective inference that is general, powerful and tractable. MAGIC is a method for selective inference after solving a convex optimization problem with smooth loss and $\ell_1$ penalty. Randomization is incorporated into the optimization problem to boost statistical power. Through reparametrization, MAGIC reduces the problem into a sampling problem with simple constraints. MAGIC applies to many $\ell_1$ penalized optimization problem including the Lasso, logistic Lasso and neighborhood selection in graphical models, all of which we consider in this paper.

Motivation & Objective

• To address the challenge of valid statistical inference after data-dependent model selection, particularly in high-dimensional settings.
• To develop a general method applicable to a wide range of ℓ₁-penalized optimization problems beyond specific cases like Lasso.
• To improve statistical power in selective inference by introducing randomization into the optimization process.
• To enhance computational tractability by transforming complex constraint sets into simpler, canonical sampling spaces.
• To enable the construction of valid confidence intervals and p-values for selected variables after model selection.

Proposed method

• MAGIC introduces a randomized convex optimization problem by adding a random vector ωᵀβ to the objective, which enhances statistical power.
• The method reparametrizes the selective inference problem into a sampling problem over a canonical constraint set, avoiding repeated boundary computation.
• It uses a Gibbs sampler to jointly sample data S, estimated coefficients β̂, and subgradients ẑ, with Metropolis-Hastings steps for S and β̂, and truncated G distributions for ẑ.
• For Lasso, it samples the response y conditionally on the active set, using a proposal that preserves the active set and uses a rotation-based step for y.
• For neighborhood selection, it samples data columns one at a time, conditioning on norms and pairwise inner products, and uses spherical sampling to preserve constraints.
• Theoretical justification is provided via a change-of-variables formula that yields the exact density of the selective distribution, enabling valid inference.

Experimental results

Research questions

• RQ1 Can a general method for selective inference be developed that applies to arbitrary smooth loss functions and exponential family distributions?
• RQ2 How can randomization in the optimization problem improve statistical power in selective inference?
• RQ3 Can the computational burden of sampling from complex constraint sets be reduced through reparametrization?
• RQ4 To what extent does MAGIC outperform existing methods in terms of power and interval coverage for selected variables?
• RQ5 Can MAGIC be effectively applied to diverse problems such as Lasso, logistic Lasso, and Gaussian graphical models?

Key findings

• MAGIC achieves significantly higher statistical power than previous methods such as those in [6] and [13], particularly in simulation studies.
• The method enables valid inference for a broad class of ℓ₁-penalized models, including Lasso, logistic Lasso, and neighborhood selection in Gaussian graphical models.
• By reparametrizing the constraint set into a canonical form, MAGIC avoids the need to compute boundaries at each MCMC step, greatly improving computational efficiency.
• Theoretical results establish the exact density of the selective distribution via a change-of-variables formula, ensuring valid inference under the randomized framework.
• The Gibbs sampler with Metropolis-Hastings and truncated sampling steps effectively explores the constrained posterior, enabling accurate p-values and confidence intervals.
• MAGIC supports the construction of confidence intervals for selected variables, a capability lacking in some competing methods like FDR-based approaches.