[Paper Review] Near-Optimal Linear Regression under Distribution Shift

Qi Lei, Wei Hu+1 more|arXiv (Cornell University)|Jun 23, 2021

Machine Learning and Algorithms4 citations

TL;DR

This paper proposes near-minimax linear estimators for linear regression under distribution shift, using labeled source data and unlabeled target data. By adapting regularization via estimated input covariances, the method achieves risk within a constant factor of the minimax optimal risk across covariate and model shift settings, significantly outperforming ridge regression in theory and practice.

ABSTRACT

Transfer learning is essential when sufficient data comes from the source domain, with scarce labeled data from the target domain. We develop estimators that achieve minimax linear risk for linear regression problems under distribution shift. Our algorithms cover different transfer learning settings including covariate shift and model shift. We also consider when data are generated from either linear or general nonlinear models. We show that linear minimax estimators are within an absolute constant of the minimax risk even among nonlinear estimators for various source/target distributions.

Motivation & Objective

• Address the challenge of linear regression when target domain data is scarce but source domain data is abundant, under distribution shift.
• Develop estimators that achieve near-minimax risk for linear regression under covariate shift and model shift.
• Characterize the conditions under which linear estimators can match the performance of nonlinear estimators up to a constant factor.
• Provide theoretical guarantees on risk bounds under standard data concentration assumptions.
• Demonstrate the superiority of data-dependent regularization over standard ridge regression in the presence of distribution shift.

Proposed method

• Propose a two-stage meta-algorithm that leverages labeled source data and unlabeled target data to estimate optimal linear estimators.
• Use empirical covariance matrices from unlabeled target data to adapt regularization, enabling robustness to input distribution shifts.
• Formulate the risk minimization problem as a matrix optimization over linear estimators, with bias-variance decomposition to analyze worst-case risk.
• Establish high-probability concentration bounds on estimated covariances using subgaussian assumptions on input features.
• Prove that the estimator's risk is within a constant factor of the minimax risk, even among nonlinear estimators, under commuting second-moment matrices.
• Derive theoretical separation results showing improvement over ridge regression by a factor of $\tilde{O}(d^{-1/4})$.

Experimental results

Research questions

• RQ1 Can linear estimators achieve near-minimax risk in linear regression under distribution shift?
• RQ2 How does data-dependent regularization based on target-domain covariances improve performance compared to fixed regularization like ridge regression?
• RQ3 Under what conditions do linear estimators match the performance of nonlinear estimators up to a constant factor?
• RQ4 What is the theoretical gap between ridge regression and the minimax optimal estimator under covariate shift?
• RQ5 How does the proposed method perform when the source and target input covariances commute?

Key findings

• The proposed estimator achieves worst-case risk within a universal constant factor of the minimax risk among all linear estimators, even under general nonlinear data models.
• When the second-moment matrices of source and target domains commute, the estimator achieves near-minimax risk among all possible estimators, including nonlinear ones.
• The method provides a theoretical improvement over ridge regression by a multiplicative factor of $\tilde{O}(d^{-1/4})$ in risk under covariate shift.
• Theoretical analysis confirms that data-dependent regularization, informed by unlabeled target data, is essential to mitigate performance degradation due to distribution shift.
• Empirical results validate that the proposed estimator significantly outperforms ridge regression and heuristic baselines in various distribution shift scenarios.
• The risk bounds are robust under standard subgaussian assumptions on input features, with high-probability concentration of estimated covariances.