[Paper Review] Pseudo-Labeling for Kernel Ridge Regression under Covariate Shift

Kaizheng Wang|arXiv (Cornell University)|Feb 20, 2023

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a pseudo-labeling method for kernel ridge regression under covariate shift, where unlabeled target data and labeled source data are used to train an imputation model and multiple candidate models. By using pseudo-labels from the imputation model to select the best-performing candidate, the method achieves minimax optimal error rates up to a logarithmic factor, adapting to the target distribution and covariate shift structure without requiring labeled target data.

ABSTRACT

We develop and analyze a principled approach to kernel ridge regression under covariate shift. The goal is to learn a regression function with small mean squared error over a target distribution, based on unlabeled data from there and labeled data that may have a different feature distribution. We propose to split the labeled data into two subsets, and conduct kernel ridge regression on them separately to obtain a collection of candidate models and an imputation model. We use the latter to fill the missing labels and then select the best candidate accordingly. Our non-asymptotic excess risk bounds demonstrate that our estimator adapts effectively to both the structure of the target distribution and the covariate shift. This adaptation is quantified through a notion of effective sample size that reflects the value of labeled source data for the target regression task. Our estimator achieves the minimax optimal error rate up to a polylogarithmic factor, and we find that using pseudo-labels for model selection does not significantly hinder performance.

Motivation & Objective

• To address the challenge of training a regression model with small mean squared error over a target distribution when only unlabeled target data and labeled source data are available.
• To develop a principled method for model selection under covariate shift, where traditional validation techniques fail due to missing labels in the target distribution.
• To show that pseudo-labeling via an imputation model can enable effective model selection without significant performance degradation, even when the imputation model is imperfect.
• To establish non-asymptotic excess risk bounds that demonstrate adaptivity to the structure of the target distribution and the degree of covariate shift.
• To resolve the chicken-and-egg problem in model selection by proving that the overhead from pseudo-labels is negligible compared to their benefit in selection accuracy.

Proposed method

• The labeled source data are split into two disjoint subsets: one for training an imputation model and the other for training a collection of candidate kernel ridge regression models with different regularization parameters.
• An imputation model is trained on a subset of the source data to predict labels for the unlabeled target data, generating pseudo-labels for model evaluation.
• Candidate models are trained on the remaining source data using kernel ridge regression with varying penalty parameters.
• Model selection is performed via hold-out validation using the pseudo-labeled target data, effectively reducing the problem to standard KRR with validation.
• A novel bias-variance decomposition is introduced to analyze the imputation model's impact on model selection, showing that even noisy pseudo-labels can be useful.
• Theoretical analysis establishes that the final estimator achieves minimax optimal error rates up to a logarithmic factor under mild regularity conditions on the kernel and distributions.

Experimental results

Research questions

• RQ1 Can pseudo-labeling be used effectively for model selection in kernel ridge regression when labels are missing in the target distribution?
• RQ2 Does the use of pseudo-labels from an imputation model trained on a different distribution (source) lead to a valid and adaptive estimator for the target distribution?
• RQ3 What is the theoretical impact of imputation error on model selection performance, and can it be bounded in a way that ensures minimax optimality?
• RQ4 How does the proposed method adapt to unknown covariate shift and the intrinsic smoothness of the target regression function?
• RQ5 Can the method achieve minimax optimal error rates despite using synthetic labels and a potentially suboptimal imputation model?

Key findings

• The proposed estimator achieves the minimax optimal error rate for kernel ridge regression under covariate shift up to a logarithmic factor, even when the imputation model is trained on a different distribution.
• The method is robust to imputation error: the overhead from using pseudo-labels is negligible compared to the mean squared error of the imputation model, enabling faithful model selection.
• Theoretical analysis reveals a novel bias-variance decomposition specific to model selection under pseudo-labeling, showing that high-error pseudo-labels can still yield optimal model selection.
• The approach adapts to the unknown structure of the target distribution and the degree of covariate shift, without requiring prior knowledge of the shift or the true regression function.
• The method achieves high-probability excess risk bounds that depend on the complexity of the RKHS and the concentration of the covariate shift, demonstrating strong finite-sample performance.
• Empirical validation via theoretical bounds confirms that the selected model performs nearly as well as if hold-out validation were available on the target data.