[Paper Review] Transfer Learning for Nonparametric Regression: Non-asymptotic Minimax Analysis and Adaptive Procedure

Tommaso Cai, Hongming Pu|arXiv (Cornell University)|Jan 22, 2024

Statistical Methods and Inference4 citations

TL;DR

This paper develops a minimax-optimal transfer learning method for nonparametric regression under the posterior drift model, introducing a confidence thresholding estimator that achieves near-optimal convergence rates. It establishes theoretical guarantees for auto-smoothing and super-acceleration effects and proposes an adaptive algorithm that performs well across diverse smoothness and bias strength regimes.

ABSTRACT

Transfer learning for nonparametric regression is considered. We first study the non-asymptotic minimax risk for this problem and develop a novel estimator called the confidence thresholding estimator, which is shown to achieve the minimax optimal risk up to a logarithmic factor. Our results demonstrate two unique phenomena in transfer learning: auto-smoothing and super-acceleration, which differentiate it from nonparametric regression in a traditional setting. We then propose a data-driven algorithm that adaptively achieves the minimax risk up to a logarithmic factor across a wide range of parameter spaces. Simulation studies are conducted to evaluate the numerical performance of the adaptive transfer learning algorithm, and a real-world example is provided to demonstrate the benefits of the proposed method.

Motivation & Objective

• To establish the non-asymptotic minimax risk for transfer learning in nonparametric regression under the posterior drift model.
• To develop a novel estimator—confidence thresholding—that achieves minimax optimal risk up to a logarithmic factor.
• To investigate the unique phenomena of auto-smoothing and super-acceleration in transfer learning compared to standard nonparametric regression.
• To propose a data-driven adaptive procedure that achieves minimax risk across a wide range of smoothness and bias strength parameters.
• To validate the method through simulations and a real-world application in air quality prediction.

Proposed method

• Proposes a confidence thresholding estimator that combines target and source domain data by thresholding based on confidence in source function estimates.
• Analyzes the minimax risk in the posterior drift model where the difference between target and source functions is bounded in $L_1$ by a polynomial bias of order $\epsilon$.
• Derives non-asymptotic minimax lower and upper bounds for the risk, showing that the estimator achieves the optimal rate up to a logarithmic factor.
• Introduces an adaptive algorithm that selects tuning parameters based on data-driven criteria to achieve near-optimality across varying smoothness and bias strength.
• Uses local polynomial regression and bandwidth selection techniques to estimate the source function and calibrate the thresholding rule.
• Employs a two-stage estimation strategy: first estimate the source function, then use it to improve target function estimation via thresholding.

Experimental results

Research questions

• RQ1 What is the non-asymptotic minimax risk for transfer learning in nonparametric regression under the posterior drift model?
• RQ2 How does transfer learning induce auto-smoothing and super-acceleration compared to standard nonparametric regression?
• RQ3 Can a data-driven adaptive procedure achieve minimax optimality across a broad class of smoothness and bias strength parameters?
• RQ4 What is the optimal rate of convergence for the transfer learning estimator, and how does it depend on sample sizes, smoothness, and bias strength?
• RQ5 How does the confidence thresholding estimator compare empirically to baseline methods in simulation and real-world settings?

Key findings

• The minimax risk for transfer learning in the posterior drift model is bounded below by $ C_L \cdot \left( n_{\max}^{-\frac{2\beta_{\max}}{2\beta_{\max}+d}} + (\epsilon \wedge n_Q^{-\frac{\beta_Q}{2\beta_Q + d}}) \cdot n_Q^{-\frac{\beta_Q}{2\beta_Q + d}} + \frac{1}{n_Q} \right) $, which captures the trade-off between source and target data.
• The confidence thresholding estimator achieves the minimax optimal rate up to a logarithmic factor, demonstrating its theoretical efficiency.
• The paper identifies two unique phenomena: auto-smoothing, where the estimator naturally adapts to the smoothness of the target function, and super-acceleration, where the convergence rate exceeds that of standard nonparametric regression.
• The adaptive procedure achieves minimax risk up to a logarithmic factor across a wide range of smoothness parameters $\beta_Q$, $\beta_P$, and bias strength $\epsilon$.
• Simulation studies show that the adaptive algorithm outperforms baseline methods in terms of mean squared error across various configurations.
• A real-world application to air quality prediction demonstrates the practical benefits of transfer learning using the proposed method, especially when source and target functions are similar but not identical.