[Paper Review] Robust angle-based transfer learning in high dimensions

Tian Gu, Yi Han+1 more|arXiv (Cornell University)|Oct 23, 2022

Cancer-related molecular mechanisms research5 citations

TL;DR

This paper proposes angle-based transfer learning (angleTL), a robust, closed-form method for high-dimensional regression that leverages pre-trained source model parameters without requiring individual-level data. By exploiting the angular similarity between source and target model coefficients, angleTL adaptively improves prediction accuracy, prevents negative transfer, and outperforms benchmark methods in both simulations and real-world genetic risk prediction across biobanks.

ABSTRACT

Transfer learning aims to improve the performance of a target model by leveraging data from related source populations, which is known to be especially helpful in cases with insufficient target data. In this paper, we study the problem of how to train a high-dimensional ridge regression model using limited target data and existing regression models trained in heterogeneous source populations. We consider a practical setting where only the parameter estimates of the fitted source models are accessible, instead of the individual-level source data. Under the setting with only one source model, we propose a novel flexible angle-based transfer learning (angleTL) method, which leverages the concordance between the source and the target model parameters. We show that angleTL unifies several benchmark methods by construction, including the target-only model trained using target data alone, the source model fitted on source data, and distance-based transfer learning method that incorporates the source parameter estimates and the target data under a distance-based similarity constraint. We also provide algorithms to effectively incorporate multiple source models accounting for the fact that some source models may be more helpful than others. Our high-dimensional asymptotic analysis provides interpretations and insights regarding when a source model can be helpful to the target model, and demonstrates the superiority of angleTL over other benchmark methods. We perform extensive simulation studies to validate our theoretical conclusions and show the feasibility of applying angleTL to transfer existing genetic risk prediction models across multiple biobanks.

Motivation & Objective

• To address the challenge of training high-dimensional regression models with limited target data when only pre-trained source model parameters are available.
• To develop a method that prevents negative transfer when source and target populations are heterogeneous, especially when individual-level source data is unavailable.
• To unify and improve upon existing benchmark transfer learning methods by leveraging the concordance between source and target model parameters through an angle-based similarity measure.
• To extend the framework to multiple source models by incorporating a weighted aggregation strategy that accounts for source model utility.
• To provide theoretical justification and empirical validation of angleTL across diverse simulation settings and real-world biobank data applications.

Proposed method

• The method introduces a novel penalty term based on the cosine similarity (angle) between the target coefficient vector and source model estimates, promoting alignment when source and target parameters are concordant.
• The objective function combines the standard least squares loss with a regularization term: $\lambda\|\boldsymbol{\beta}\|_{2}^{2} - 2\eta\hat{\mathbf{w}}^{\top}\boldsymbol{\beta}$, where $\hat{\mathbf{w}}$ is the source model estimate and $\eta$ controls the influence of the source.
• A closed-form solution is derived, enabling efficient computation without iterative optimization or access to individual-level data.
• For multiple sources, the method aggregates source models via a weighted average that accounts for their relative similarity to the target, with weights determined by angular concordance.
• Theoretical analysis under high-dimensional asymptotics provides conditions under which source models are beneficial, identifying key factors such as signal strength, noise level, and dimensionality.
• The approach is extended to classification by adapting the similarity measure to fitted outcomes or rank-based comparisons, maintaining the angle-based regularization framework.

Experimental results

Research questions

• RQ1 Under what conditions is a pre-trained source model beneficial for improving target model performance in high-dimensional settings with limited data?
• RQ2 How can transfer learning be made robust to population heterogeneity, especially when source and target models are poorly aligned?
• RQ3 What is the impact of signal strength, noise level, and model dimensionality on the performance of angle-based transfer learning?
• RQ4 How can multiple source models be optimally combined when their utility for the target varies?
• RQ5 Can angleTL prevent negative transfer when incorporating information from dissimilar source populations?

Key findings

• angleTL consistently outperforms benchmark methods, including target-only and source-only models, across a wide range of simulation settings, especially when source and target parameters are moderately aligned.
• The method prevents negative transfer by down-weighting sources with low angular concordance, ensuring performance does not degrade when source data is irrelevant or heterogeneous.
• Theoretical analysis shows that angleTL's performance is adaptive to signal strength and dimensionality, with improved estimation when the target signal is strong and the ratio of dimensionality to sample size is moderate.
• In real-world applications, angleTL successfully transfers genetic risk prediction models across multiple biobanks, demonstrating feasibility and robustness in practice.
• The closed-form solution enables efficient implementation without requiring iterative communication or individual-level data, making it suitable for privacy-preserving settings.
• Extension to classification tasks via outcome-based similarity measures shows promise, with simulation results confirming the method's flexibility beyond linear regression.