[Paper Review] Anchor regression: heterogeneous data meets causality

Dominik Rothenhaüsler, Nicolai Meinshausen+2 more|arXiv (Cornell University)|Jan 18, 2018

Bayesian Modeling and Causal Inference43 references67 citations

TL;DR

Anchor regression introduces a regression method using exogenous anchors to interpolate between OLS and IV, providing distributional robustness against shifts and improving replicability under heterogeneous data.

ABSTRACT

We consider the problem of predicting a response variable from a set of covariates on a data set that differs in distribution from the training data. Causal parameters are optimal in terms of predictive accuracy if in the new distribution either many variables are affected by interventions or only some variables are affected, but the perturbations are strong. If the training and test distributions differ by a shift, causal parameters might be too conservative to perform well on the above task. This motivates anchor regression, a method that makes use of exogeneous variables to solve a relaxation of the causal minimax problem by considering a modification of the least-squares loss. The procedure naturally provides an interpolation between the solutions of ordinary least squares and two-stage least squares. We prove that the estimator satisfies predictive guarantees in terms of distributional robustness against shifts in a linear class; these guarantees are valid even if the instrumental variables assumptions are violated. If anchor regression and least squares provide the same answer (anchor stability), we establish that OLS parameters are invariant under certain distributional changes. Anchor regression is shown empirically to improve replicability and protect against distributional shifts.

Motivation & Objective

• Motivate prediction under training-test distribution shifts and data heterogeneity.
• Define anchor regression to trade off predictive performance on observational data and perturbed data.
• Link anchor regression to causal concepts and instrumental variables while relaxing IV assumptions.
• Provide computationally simple estimator with theoretical robustness guarantees under shift interventions.

Proposed method

• Define the population anchor regression objective that penalizes projection residuals and controls projection of residuals onto the anchor space (Eq. 4).
• Provide a finite-sample plug-in estimator that solves a transformed LS problem (Eq. 5).
• Show that the estimator interpolates between partialling out, OLS, and IV as gamma varies (Eq. 7).
• Relate anchor regression to k-class estimators and to IV under instrumental variables assumptions.
• Allow high-dimensional extensions with sparsity via an l1 penalty and discuss practical computation.
• Explain that gamma tunes robustness to shift interventions and that larger gamma emphasizes invariance.

Experimental results

Research questions

• RQ1 How can we achieve predictive robustness to distributional shifts via anchor-based perturbations?
• RQ2 How does anchor regression relate to and interpolate between PA, OLS, and IV under different gamma values?
• RQ3 Under what conditions do anchor regression coefficients coincide with OLS and what does that imply for invariance and replicability?
• RQ4 Can anchor regression provide distributionally robust guarantees when anchors are invalid instruments?

Key findings

• Anchor regression provides predictive guarantees that are robust against a class of shift interventions in a linear setting.
• The method interpolates between PA, OLS, and IV as gamma varies, ultimately connecting to IV as gamma→∞ under certain identifiability conditions.
• If anchor regression and ordinary least squares give the same coefficients (anchor stability), OLS parameters remain invariant under certain distributional changes.
• Empirically, anchor regression improves replicability and protects against distributional shifts on test data across heterogeneous training groups.
• The approach remains useful even when anchors are not valid instruments, by exploiting invariance properties for robust prediction.