[Paper Review] More Powerful and General Selective Inference for Stepwise Feature Selection using the Homotopy Continuation Approach

Kazuya Sugiyama, Vo Nguyen Le Duy+1 more|arXiv (Cornell University)|Dec 25, 2020

Statistical Methods and Inference44 references4 citations

TL;DR

This paper proposes a homotopy continuation-based selective inference method for stepwise feature selection that overcomes the power loss from over-conditioning in traditional polytope-based approaches. By tracking selection events through continuous path-following in data space, the method enables more powerful and general inference for complex algorithms like forward-backward SFS with AIC-based stopping, maintaining high statistical power even under increased algorithmic complexity.

ABSTRACT

Conditional selective inference (SI) has been actively studied as a new statistical inference framework for data-driven hypotheses. The basic idea of conditional SI is to make inferences conditional on the selection event characterized by a set of linear and/or quadratic inequalities. Conditional SI has been mainly studied in the context of feature selection such as stepwise feature selection (SFS). The main limitation of the existing conditional SI methods is the loss of power due to over-conditioning, which is required for computational tractability. In this study, we develop a more powerful and general conditional SI method for SFS using the homotopy method which enables us to overcome this limitation. The homotopy-based SI is especially effective for more complicated feature selection algorithms. As an example, we develop a conditional SI method for forward-backward SFS with AIC-based stopping criteria and show that it is not adversely affected by the increased complexity of the algorithm. We conduct several experiments to demonstrate the effectiveness and efficiency of the proposed method.

Motivation & Objective

• To address the power loss in existing conditional selective inference methods due to over-conditioning on selection events.
• To develop a more general and powerful inference framework applicable to complex feature selection algorithms such as forward-backward SFS with AIC-based stopping.
• To enable computationally tractable inference by leveraging the homotopy continuation method to track selection events across data perturbations.
• To demonstrate that the proposed method maintains high statistical power even when selection complexity increases.

Proposed method

• Uses the homotopy continuation method to trace the path of selected features as the response vector varies along a linear trajectory in the sample space.
• Identifies the truncation region in the sample space where the same feature selection outcome is preserved, enabling conditional inference on the selected features.
• Characterizes the selection event not by a single polytope but through continuous path tracking, avoiding the need for excessive conditioning on signs or selection history.
• Applies the method to forward-backward SFS with AIC-based stopping criteria, showing it remains effective despite algorithmic complexity.
• Derives the conditional sampling distribution of the test statistic by restricting the data space to the region where the same hypothesis is selected, using piecewise-quadratic functions for validation error in cross-validation settings.
• Employs a minimum-conditioning approach by conditioning only on the final selection event, avoiding additional constraints on intermediate models or signs.

Experimental results

Research questions

• RQ1 Can selective inference for stepwise feature selection be made more powerful by reducing over-conditioning?
• RQ2 Can the homotopy continuation method effectively characterize selection events in complex feature selection algorithms like forward-backward SFS with AIC-based stopping?
• RQ3 Does the proposed method maintain high statistical power when the selection process involves multiple stages and criteria?
• RQ4 How does the proposed method compare to polytope-based approaches in terms of power and computational feasibility under increasing algorithmic complexity?

Key findings

• The homotopy-based selective inference method achieves higher statistical power than polytope-based methods by minimizing over-conditioning.
• The method successfully handles complex feature selection algorithms such as forward-backward SFS with AIC-based stopping, maintaining high power without adverse effects from increased complexity.
• Experiments show that the true positive rate (TPR) decreases as the number of candidate models in cross-validation increases, due to more restrictive conditioning, which is consistent with the observed increase in confidence interval length.
• The truncation region for the conditional sampling distribution is derived as the intersection of intervals where the selected model remains optimal, using piecewise-quadratic functions of the data perturbation parameter.
• The method demonstrates computational feasibility and statistical validity in cross-validation settings by conditioning only on the final selection outcome, avoiding unnecessary constraints on intermediate models.