[論文レビュー] Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing

Daniel Boharon, Yehuda Dar|arXiv (Cornell University)|Feb 18, 2026

Domain Adaptation and Few-Shot Learning被引用数 0

ひとこと要約

This paper extends linear transfer learning to multiple overparameterized pretrained models, analyzes when more models help, and proposes a debiasing technique to counter overparameterization bias to achieve consistent transfer performance.

ABSTRACT

We study transfer learning for a linear regression task using several least-squares pretrained models that can be overparameterized. We formulate the target learning task as optimization that minimizes squared errors on the target dataset with penalty on the distance of the learned model from the pretrained models. We analytically formulate the test error of the learned target model and provide the corresponding empirical evaluations. Our results elucidate when using more pretrained models can improve transfer learning. Specifically, if the pretrained models are overparameterized, using sufficiently many of them is important for beneficial transfer learning. However, the learning may be compromised by overparameterization bias of pretrained models, i.e., the minimum $\ell_2$-norm solution's restriction to a small subspace spanned by the training examples in the high-dimensional parameter space. We propose a simple debiasing via multiplicative correction factor that can reduce the overparameterization bias and leverage more pretrained models to learn a target predictor.

研究の動機と目的

• Motivate and analyze transfer learning for linear regression when multiple pretrained models are available.
• Characterize how overparameterization of pretrained models affects transfer benefits and when more models help or hurt.
• Develop a debiasing technique to mitigate overparameterization bias and leverage many pretrained models effectively.

提案手法

• Formulate target task learning as minimizing squared error with a penalty on the distance to pretrained models (Eq. 6).
• Derive the closed-form target solution (Eq. 7) under reasonable assumptions.
• Use random matrix theory to analyze the asymptotic test error (Theorem 4.4, Eqs. 8–11).
• Characterize the impact of the number of pretrained models m and their parameterization on transfer performance (Theorem 5.2, Corollary 5.1).
• Propose an overparameterization debiasing by scaling task relation operators to reduce bias (Section 5.4).
• Discuss consistency/instability: underparameterized pretrained models yield consistent transfer as m grows, while overparameterized models can be inconsistent without debiasing (Theorems 5.4).

実験結果

リサーチクエスチョン

• RQ1 複数の事前学習モデルを用いたターゲット線形回帰タスクはどれくらい有益か?
• RQ2 事前学習モデルの過parameterizationは、モデル数を増やすことで得られる利得にどのように影響するか?
• RQ3 複数の過parameterizedな事前学習モデルを使用する際に、整合性を回復し転移学習を改善するデバイアス除去法は機能するか?
• RQ4 複数の事前学習モデルを組み合わせる際のソース-ターゲットタスクの関係演算子の相互作用はどうなるか？

主な発見

• 複数の事前学習モデルは単一モデルの転移学習を上回り、特定のパラメータ化領域ではネガティブ転移を解決することさえある。
• 多くのモデルを追加して得られる利得はモデル数が多くなると弱まることがあるが、過parameterizationバイアスを除去するデバイアス除去により有用性が延長される。
• 過parameterizationバイアスはmが大きくなると転移予測子の一貫性を損なう原因となるが、デバイアス除去を適用すれば一貫性を回復できる。
• Assumption 3.1の下で閉形式のTL解が存在し、単一モデルの効果をまとめる行列（Gamma_TL,infty項）を介して洞察を与える。
• 最適な転移ハイパーパラメータは事前学習モデル数とデータ規模に比例して拡大し、屈折回帰をベースラインとして結びつく（Theorem D.1 and Corollary 5.1）。
• 提案されたデバイアス除去法は、ソースの過parameterizationレベルの逆数で関係演算子をスケールし、一貫性を改善し、より多くの事前学習モデルを有効活用する。