[Paper Review] Improving knockoffs with conditional calibration

Yixiang Luo, William Fithian+1 more|arXiv (Cornell University)|Aug 19, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces the calibrated knockoff (cKnockoff) procedure, a method that uniformly improves the power of the knockoff filter for false discovery rate (FDR) control in high-dimensional linear models by incorporating a fallback testing framework via conditional calibration. It significantly enhances performance in sparse settings—especially when the number of non-null variables is near the threshold $1/\alpha$—while maintaining finite-sample FDR control and outperforming both knockoffs and Benjamini-Hochberg in challenging scenarios.

ABSTRACT

The knockoff filter of Barber and Candes (arXiv:1404.5609) is a flexible framework for multiple testing in supervised learning models, based on introducing synthetic predictor variables to control the false discovery rate (FDR). Using the conditional calibration framework of Fithian and Lei (arXiv:2007.10438), we introduce the calibrated knockoff procedure, a method that uniformly improves the power of any fixed-X or model-X knockoff procedure. We show theoretically and empirically that the improvement is especially notable in two contexts where knockoff methods can be nearly powerless: when the rejection set is small, and when the structure of the design matrix in fixed-X knockoffs prevents us from constructing good knockoff variables. In these contexts, calibrated knockoffs even outperform competing FDR-controlling methods like the (dependence-adjusted) procedure Benjamini-Hochberg in many scenarios.

Motivation & Objective

• To address the 'threshold phenomenon' in knockoff filters, where power drops sharply when the number of non-null variables is less than $1/\alpha$, making knockoffs nearly powerless in sparse settings.
• To develop a method that uniformly improves the power of any knockoff procedure while preserving finite-sample FDR control.
• To provide a robust alternative to both knockoffs and Benjamini-Hochberg in settings with low signal strength or high correlation in the design matrix.
• To enable practical deployment of knockoff-based inference in real-world data with limited non-null variables, such as in HIV drug resistance studies.

Proposed method

• The cKnockoff procedure augments the rejection set of a base knockoff procedure by applying a 'fallback test' to variables not selected by knockoffs.
• It uses the conditional calibration framework of Fithian and Lei (2022) to set the significance levels of fallback tests without violating FDR control.
• The fallback tests are conditionally calibrated on the knockoff statistics, ensuring that the overall procedure maintains finite-sample FDR control at level $\alpha$.
• The method is implemented for fixed-$X$ knockoffs using efficient computational tricks, including parallelization and optimized test statistics.
• For model-$X$ knockoffs, the method requires minor modifications and different computational strategies due to the assumption on the joint distribution of predictors.
• The procedure is designed to be strictly more powerful than the original knockoff filter in every problem instance under the same data and assumptions.

Experimental results

Research questions

• RQ1 Can a method be developed that uniformly improves the power of the knockoff filter while maintaining finite-sample FDR control?
• RQ2 How does the performance of knockoffs degrade when the number of non-null variables is close to $1/\alpha$, and can this be mitigated?
• RQ3 Can fallback testing, calibrated conditionally on knockoff statistics, enhance discovery power without inflating FDR?
• RQ4 In sparse settings with high correlation or low signal, does cKnockoff outperform both knockoffs and Benjamini-Hochberg?
• RQ5 Can the calibrated knockoff procedure be efficiently implemented in practice, especially in high-dimensional settings like HIV drug resistance analysis?

Key findings

• cKnockoff uniformly improves power over the original knockoff filter in all problem instances, with the largest gains observed in sparse settings where knockoffs are nearly powerless.
• In the HIV drug resistance dataset with $\alpha = 0.05$, knockoffs rejected almost no variables, while cKnockoff and cKnockoff* made a substantial number of discoveries, significantly outperforming knockoffs.
• cKnockoff achieved higher replicability with the TSM panel than knockoffs, identifying more mutations known to be associated with drug resistance.
• For $\alpha = 0.2$, cKnockoff maintained high power even when knockoffs were no longer ineffective, demonstrating robustness across FDR levels.
• The computational cost of cKnockoff is only a small multiple of knockoffs, and it can be accelerated via parallelization, making it practical for real-world use.
• In settings with low signal and high correlation, cKnockoff outperformed both knockoffs and Benjamini-Hochberg, especially when BH had high false discovery proportions.