[Paper Review] Causal Feature Selection via Orthogonal Search

Ashkan Soleymani, Anant Raj+3 more|arXiv (Cornell University)|Jul 6, 2020

Bayesian Modeling and Causal Inference4 citations

TL;DR

This paper proposes a novel causal feature selection method using orthogonal search and Double Machine Learning principles to identify direct causal parents of a response variable from purely observational data. The approach enables polynomial-time computation, handles nonlinear and cyclic relationships, and provides theoretical guarantees, significantly outperforming existing methods in high-dimensional settings.

ABSTRACT

The problem of inferring the direct causal parents of a response variable among a large set of explanatory variables is of high practical importance in many disciplines. However, established approaches often scale at least exponentially with the number of explanatory variables, are difficult to extend to nonlinear relationships, and are difficult to extend to cyclic data. Inspired by {\em Debiased} machine learning methods, we study a one-vs.-the-rest feature selection approach to discover the direct causal parent of the response. We propose an algorithm that works for purely observational data while also offering theoretical guarantees, including the case of partially nonlinear relationships possibly under the presence of cycles. As it requires only one estimation for each variable, our approach is applicable even to large graphs. We demonstrate significant improvements compared to established approaches.

Motivation & Objective

• To address the challenge of identifying direct causal parents of a response variable in high-dimensional, nonlinear, and potentially cyclic systems using only observational data.
• To overcome limitations of existing causal discovery methods, including exponential scaling, reliance on interventional data, and poor performance in high-dimensional settings.
• To develop a scalable and theoretically grounded method that maintains low bias and high statistical power for feature selection in complex structural equation models.
• To extend Double Machine Learning principles to a one-vs.-the-rest feature selection framework for causal discovery.
• To provide asymptotic theoretical guarantees on the consistency and selection accuracy of the proposed method.

Proposed method

• The method employs a one-vs.-the-rest approach, treating each covariate as a potential direct cause and estimating its causal effect via orthogonalized estimating equations.
• It leverages Double Machine Learning to debias the estimation of each coefficient in the linear structural equation model, reducing regularization-induced bias.
• For each variable, the method estimates the conditional mean of the response given all other variables, using machine learning models to control for confounding.
• The orthogonalization step ensures that the estimator of the coefficient of interest is asymptotically normal and approximately unbiased, even in high-dimensional settings.
• The algorithm is computationally efficient, requiring only one estimation per variable, enabling scalability to large graphs.
• Theoretical guarantees are derived under assumptions of no direct influence of the response on covariates (NFD) and linear direct effects (LDC), even with nonlinear and cyclic structures.

Experimental results

Research questions

• RQ1 Can a one-vs.-the-rest feature selection approach based on Double Machine Learning principles effectively identify direct causal parents in purely observational, high-dimensional data?
• RQ2 Does the proposed method maintain statistical consistency and low bias in the presence of nonlinear relationships and cyclic confounding structures?
• RQ3 Can the method achieve polynomial-time complexity while preserving theoretical guarantees in high-dimensional settings?
• RQ4 How does the method compare to standard regularization techniques like Lasso in terms of selection accuracy and bias in high-dimensional causal feature selection?
• RQ5 To what extent does the method generalize to real-world datasets with complex, nonlinear, and cyclic dependencies?

Key findings

• The proposed method achieves significant improvements in causal feature selection accuracy compared to established approaches, particularly in high-dimensional and nonlinear settings.
• The method demonstrates strong empirical performance, correctly identifying key clinical features such as patient age quantile and blood gas parameters as direct causes of patient admission to semi-intensive care units in 1000 runs.
• In 1000 repeated runs, patient age quantile was predicted as a direct cause in 100% of cases for both semi-intensive and intensive care unit outcomes, indicating high stability and reliability.
• Features such as pO2, ctO2, and pH venous blood gas analysis were predicted as direct causes in over 90% of runs, aligning with clinical expectations.
• The method successfully identified biologically plausible direct causes in complex medical datasets, including markers like creatinine, lactate dehydrogenase, and transaminases, with high prediction frequency.
• The approach outperforms standard Lasso and unregularized regression baselines in terms of selection accuracy and bias reduction, especially in high-dimensional regimes with limited samples.