[论文解读] Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting

Yu-Chen Den, Kuan‐Yu Chen+2 more|arXiv (Cornell University)|Mar 17, 2026

Stock Market Forecasting Methods被引用 0

一句话总结

引入 TIPS，一种蒸馏框架，通过综合多种归纳偏置（因果性、局部性、周期性）到单一 Transformer，以在制度切换下提升金融时间序列预测，并实现更低的推理成本的State-of-the-art结果。

ABSTRACT

Transformer-based models have been widely adopted for time-series forecasting due to their high representational capacity and architectural flexibility. However, many Transformer variants implicitly assume stationarity and stable temporal dynamics -- assumptions routinely violated in financial markets characterized by regime shifts and non-stationarity. Empirically, state-of-the-art time-series Transformers often underperform even vanilla Transformers on financial tasks, while simpler architectures with distinct inductive biases, such as CNNs and RNNs, can achieve stronger performance with substantially lower complexity. At the same time, no single inductive bias dominates across markets or regimes, suggesting that robust financial forecasting requires integrating complementary temporal priors. We propose TIPS (Transformer with Inductive Prior Synthesis), a knowledge distillation framework that synthesizes diverse inductive biases -- causality, locality, and periodicity -- within a unified Transformer. TIPS trains bias-specialized Transformer teachers via attention masking, then distills their knowledge into a single student model with regime-dependent alignment across inductive biases. Across four major equity markets, TIPS achieves state-of-the-art performance, outperforming strong ensemble baselines by 55%, 9%, and 16% in annual return, Sharpe ratio, and Calmar ratio, while requiring only 38% of the inference-time computation. Further analyses show that TIPS generates statistically significant excess returns beyond both vanilla Transformers and its teacher ensembles, and exhibits regime-dependent behavioral alignment with classical architectures during their profitable periods. These results highlight the importance of regime-dependent inductive bias utilization for robust generalization in non-stationary financial time series.

研究动机与目标

• 在金融时间序列预测中由于制度切换和非平稳性，迫切需要自适应的归纳偏置以提升表现。
• 证明天真地将多种偏置合并相较于偏置专门化模型或集成会降低性能。
• 提出并验证 TIPS，一种基于蒸馏的框架，将多样化偏置综合到单一 transformer 中。
• 表明 TIPS 在主要股票市场实现最先进的性能，同时降低推理成本。

提出的方法

• 通过注意力屏蔽和输入设计，训练编码不同先验（因果性、局部性、周期性）的偏置专门化 Transformer 教师。
• 从七位教师（六位偏置特定加上 Vanilla Transformer）构建 Bias Teacher Ensemble，以捕捉多样化的先验。
• 用强正则化将集合预测蒸馏到单一学生 Transformer，避免僵化的模仿。
• 构建带温度缩放的软集成目标，并应用标签平滑以改善校准。
• 在不受约束的注意力下训练学生，以综合先验，并使用随机权重平均（SWA）提升鲁棒性。
• 提供分析以显示偏置在制度条件依赖下的激活与统计性超额收益。

Figure 1 . Performance–efficiency trade-off across generic time-series models, financial forecasting models, and classical architectures evaluated across multiple equity markets. The figure highlights substantial variation in performance and computational cost across model families, with TIPS achiev

实验结果

研究问题

• RQ1 多样的归纳偏置能否提升 Transformer 在非平稳金融数据上的鲁棒性？
• RQ2 与偏置专门化或集成相比，简单合并多种偏置是否会降低性能？
• RQ3 蒸馏后的学生是否能有效综合多种先验，同时维持高效推理？
• RQ4 偏置先验是否在制度条件下按市场有利条件激活？
• RQ5 TIPS 在多大程度上提供统计显著的超额收益，超越基线模型？

主要发现

• TIPS 在四大主要股票市场中实现最强的整体性能，平均夏普比率和年化收益均优于基线模型。
• Bias Teacher Ensemble（通过注意力屏蔽）超过经典架构和通用 SOTA 模型的集成，显示了在无架构异质性情况下偏置编码的有效性。
• 蒸馏到单一学生相较于偏置集合带来显著提升，并将推理时间缩短约7倍，使单一模型也具备集成级鲁棒性。
• 消融实验显示正则化组件（低温蒸馏、标签平滑、SWA）共同必要，以实现有效的偏置综合。
• 分析表明 TIPS 相对于普通 Transformer 具有统计显著的阿尔法收益，表明从归纳偏置综合中提取了有益信号。

Figure 2 . Overview of the TIPS training framework. (a) Bias-specialized Transformer (TFM) teachers are constructed via different attention masks or positional biases (Colors indicate where the masks and biases are applied). (b) Teachers are trained independently for ranking prediction. (c) Teacher