[论文解读] Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation

Kakei Yamamoto, Martin J. Wainwright|arXiv (Cornell University)|Mar 26, 2026

Domain Adaptation and Few-Shot Learning被引用 0

一句话总结

本论文提出 Residual-as-Teacher (RaT) 方法，利用教师估计学生预测残差而非直接模仿输出，降低教师偏差并具备有利的统计与计算性质。给出非渐近的超额风险界、与标准软匹配的分离以及收敛性保障，并在协变量漂移下给出实证验证。

ABSTRACT

We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model. A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM). This approach directly propagates any systematic bias or mis-specification present in the teacher, thereby degrading the student's predictions. We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions. Our analysis shows how the student can thereby emulate a proximal gradient scheme for solving an oracle optimization problem, and this provably reduces the effect of teacher bias. For general student--teacher pairs, we establish non-asymptotic excess risk bounds for any RaT fixed point, along with convergence guarantees for the student-teacher iterative scheme. For kernel-based student--teacher pairs, we prove a sharp separation: the RaT method achieves the minimax-optimal rate, while the SM method incurs constant prediction error for any sample size. Experiments on both synthetic data and ImageNette classification under covariate shift corroborate our theoretical findings.

研究动机与目标

• Motivate and formalize bias propagation in student–teacher estimation, where direct soft matching inherits teacher bias.
• Introduce RaT as an alternative that uses the teacher to estimate residuals and refine the student.
• Provide non-asymptotic risk bounds and convergence guarantees for RaT.
• Show a sharp separation between RaT and standard soft-matching (SM) in kernel settings.
• Validate theory with experiments on synthetic data and covariate-shifted ImageNette.

提出的方法

• Define the RaT procedure as a fixed-point operator combining a proximal student update with residual-based teacher estimation.
• Use a residual regression step where the teacher is trained to predict the student’s residuals and applied to target covariates.
• Relate RaT to proximal gradient updates and to oracle estimands with a penalty: f† = argmin_f L̄_m(f) + Pen(f).
• Establish a Picard-iteration scheme to compute RaT fixed points.
• Derive non-asymptotic excess risk bounds (Theorem 1) and a separation result against SM (Theorem 2).
• Provide convergence guarantees for the iterative RaT algorithm (Theorem 3).

实验结果

研究问题

• RQ1 Can RaT reduce the impact of teacher bias compared to direct soft matching (SM) in student–teacher estimation?
• RQ2 What are the statistical properties (excess risk, convergence) of RaT fixed points relative to the oracle estimand f†?
• RQ3 How does RaT perform under covariate shift, especially for kernel-based student–teacher pairs?
• RQ4 What are the computational guarantees for the RaT iterative algorithm and its convergence behavior?
• RQ5 Do experiments on synthetic data and covariate-shifted ImageNette corroborate the theoretical guarantees?

主要发现

• RaT achieves non-asymptotic excess risk bounds relative to the oracle estimand, linking performance to the teacher-induced gradient accuracy.
• For kernel-based pairs, RaT attains minimax-optimal rates while SM incurs constant prediction error under the same conditions.
• RaT exhibits a pronounced performance gap over SM due to its residual-focused guidance, as shown in theory and experiments.
• The RaT iterative scheme (Picard updates) converges under mild conditions and yields fixed points independent of the chosen stepsize.
• Experiments on synthetic data and covariate-shifted ImageNette support the theoretical advantages of RaT over SM.