[Paper Review] The Accessible Lasso Models

Amir Sepehri, Naftali Harris|arXiv (Cornell University)|Jan 12, 2015

Statistical Methods and Inference12 references3 citations

TL;DR

This paper provides a geometric characterization of the set of accessible lasso models—those signed coefficient configurations that can be selected by the lasso for some response vector y—by showing they correspond to faces of the convex hull of the design matrix and its negation. The key result is a direct proof of model selection inconsistency when the true model size exceeds half the number of observations, regardless of the regularization parameter λ.

ABSTRACT

A new line of research on the lasso exploits the beautiful geometric fact that the lasso fit is the residual from projecting the response vector $y$ onto a certain convex polytope. This geometric picture also allows an exact geometric description of the set of accessible lasso models for a given design matrix, that is, which configurations of the signs of the coefficients it is possible to realize with some choice of $y$. In particular, the accessible lasso models are those that correspond to a face of the convex hull of all the feature vectors together with their negations. This convex hull representation then permits the enumeration and bounding of the number of accessible lasso models, which in turn provides a direct proof of model selection inconsistency when the size of the true model is greater than half the number of observations.

Motivation & Objective

• To characterize which signed models (support and sign configurations) can be selected by the lasso for a given design matrix.
• To establish a geometric correspondence between accessible lasso models and faces of the convex hull of the design matrix and its negative.
• To bound the number of accessible lasso models using polytope combinatorics, particularly the upper bound theorem.
• To provide a direct geometric proof of model selection inconsistency when the true model size exceeds half the number of observations.
• To demonstrate that model selection inconsistency is inherent to the design matrix geometry and independent of the choice of regularization parameter λ.

Proposed method

• Reformulate the lasso optimization problem using a non-negative parameterization with X = [X₀, -X₀] and β = (β₀⁺, β₀⁻), ensuring at most one of each pair is non-zero.
• Define accessible models S as those for which the set A_S = {y : supp(β(y)) = S} is non-empty, corresponding to regions in response space.
• Use the KKT conditions to show that the null model polytope A_∅ is the polar dual of the convex hull CH(X), enabling geometric analysis.
• Establish that each accessible signed model S corresponds to a face F_S of CH(X), and that A_S is the relative interior of the cone generated by F_S and X_S.
• Apply the upper bound theorem from convex polytope theory to bound the number of faces of CH(X) of each dimension, thereby bounding the number of accessible models.
• Derive model selection inconsistency by showing that the number of accessible models of size k is strictly less than the number of possible signed models when k > n/2, making selection of the true model impossible with high probability.

Experimental results

Research questions

• RQ1 Which signed models can be selected by the lasso for a given design matrix X?
• RQ2 How many distinct signed models are accessible to the lasso, and how does this number depend on the design matrix?
• RQ3 Why is model selection inconsistent when the true model size exceeds half the number of observations?
• RQ4 Does the choice of regularization parameter λ affect the set of accessible models?
• RQ5 Can the geometric structure of the convex hull of X and -X fully characterize the lasso's model selection behavior?

Key findings

• The set of accessible lasso models corresponds exactly to the faces of the convex hull of the design matrix and its negation, CH(X).
• The number of accessible models of size k is bounded above by the upper bound theorem for convex polytopes, which limits the number of k-dimensional faces of CH(X).
• When the true model size k exceeds n/2, the number of accessible models is strictly less than the number of possible signed models of size k, making selection of the true model impossible.
• Model selection inconsistency is not due to the choice of λ but is a geometric property of the design matrix, holding for all λ > 0.
• Simulations confirm that lasso performance degrades significantly when k ≈ n/2, with selection error increasing sharply, consistent with the theoretical bound.
• Even under correlated design matrices, the selection error increases slightly with correlation, but the core inconsistency result remains robust.