[Paper Review] On Hypothesis Transfer Learning of Functional Linear Models

Haotian Lin, Matthew Reimherr|arXiv (Cornell University)|Jun 9, 2022

Face and Expression Recognition4 citations

TL;DR

This paper proposes a novel transfer learning framework for functional linear regression using Reproducing Kernel Hilbert Space (RKHS) norms to measure task relatedness and enable structural interpretation of transferred information. It introduces two algorithms—targeted transfer learning and robust aggregation across sources—and establishes minimax optimal prediction risk rates, proving statistically provable gains from transfer learning under functional data settings.

ABSTRACT

We study the transfer learning (TL) for the functional linear regression (FLR) under the Reproducing Kernel Hilbert Space (RKHS) framework, observing that the TL techniques in existing high-dimensional linear regression are not compatible with the truncation-based FLR methods, as functional data are intrinsically infinite-dimensional and generated by smooth underlying processes. We measure the similarity across tasks using RKHS distance, allowing the type of information being transferred to be tied to the properties of the imposed RKHS. Building on the hypothesis offset transfer learning paradigm, two algorithms are proposed: one conducts the transfer when positive sources are known, while the other leverages aggregation techniques to achieve robust transfer without prior information about the sources. We establish asymptotic lower bounds for this learning problem and show that the proposed algorithms enjoy a matching upper bound. These analyses provide statistical insights into factors that contribute to the dynamics of the transfer. We also extend the results to functional generalized linear models. The effectiveness of the proposed algorithms is demonstrated via extensive synthetic data as well as real-world data applications.

Motivation & Objective

• Address the challenge of limited training samples in functional linear regression by leveraging related source tasks through transfer learning.
• Develop a theoretically grounded method to quantify task relatedness using RKHS norms, enabling structural interpretation of transferred information.
• Propose two algorithms: one for targeted transfer when source tasks are known, and another for robust aggregation when source identities are unknown.
• Establish minimax optimal convergence rates for prediction risk, providing theoretical justification for statistical gains in transfer learning.
• Demonstrate the method's effectiveness on synthetic and real financial data, validating both theoretical claims and practical utility.

Proposed method

• Measure relatedness between target and source functional linear models using RKHS norm, which encodes structural similarity in the coefficient functions.
• Propose a targeted transfer learning algorithm that combines the target model with a weighted sum of source models based on their RKHS-relatedness.
• Design a robust transfer learning algorithm that aggregates results from multiple source models using exponential weighting, minimizing risk from negative transfer.
• Use roughness regularization and Tikhonov-type estimators to ensure smoothness and stability in coefficient function estimation.
• Apply theoretical tools such as Fano’s lemma and Varshamov-Gilbert construction to derive minimax lower bounds on prediction risk.
• Leverage eigen-decomposition of covariance operators and concentration inequalities to bound estimation error and derive convergence rates.

Experimental results

Research questions

• RQ1 How can we formally quantify the relatedness between functional linear models in a way that supports interpretable information transfer?
• RQ2 What is the optimal rate of convergence for prediction risk in functional linear regression under transfer learning?
• RQ3 How can we design a robust transfer learning algorithm that performs well even when source tasks are not perfectly aligned with the target?
• RQ4 What structural and statistical factors determine whether a source task improves or harms the target model’s performance?
• RQ5 Can theoretical prediction risk bounds be derived that prove the statistical benefit of transfer learning in functional data settings?

Key findings

• The proposed transfer learning algorithms achieve minimax optimal prediction risk rates, proving that statistical gains from transfer learning are mathematically provable under the functional linear model framework.
• The RKHS norm-based relatedness measure enables structural interpretation of which features or patterns are being transferred between models.
• The robust aggregation algorithm significantly reduces the risk of negative transfer by combining multiple source models through exponential weighting, outperforming naive averaging.
• Empirical results on synthetic and real financial data show consistent performance gains across various smoothness levels and temperature settings in the exponential weighting scheme.
• Theoretical analysis reveals that the eigenvalue decay rate of the covariance operators and the alignment of source and target coefficient functions are key factors influencing transfer effectiveness.
• The convergence rate of the prediction risk is shown to be optimal, with the rate depending on the smoothness of the coefficient function and the number of available source samples.