[Paper Review] Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting

Yu-Chen Den, Kuan‐Yu Chen+2 more|arXiv (Cornell University)|Mar 17, 2026

Stock Market Forecasting Methods0 citations

TL;DR

Introduces TIPS, a distillation framework that synthesizes multiple inductive biases (causality, locality, periodicity) into a single Transformer to improve financial time series forecasting under regime shifts, achieving state-of-the-art results with lower inference cost.

ABSTRACT

Transformer-based models have been widely adopted for time-series forecasting due to their high representational capacity and architectural flexibility. However, many Transformer variants implicitly assume stationarity and stable temporal dynamics -- assumptions routinely violated in financial markets characterized by regime shifts and non-stationarity. Empirically, state-of-the-art time-series Transformers often underperform even vanilla Transformers on financial tasks, while simpler architectures with distinct inductive biases, such as CNNs and RNNs, can achieve stronger performance with substantially lower complexity. At the same time, no single inductive bias dominates across markets or regimes, suggesting that robust financial forecasting requires integrating complementary temporal priors. We propose TIPS (Transformer with Inductive Prior Synthesis), a knowledge distillation framework that synthesizes diverse inductive biases -- causality, locality, and periodicity -- within a unified Transformer. TIPS trains bias-specialized Transformer teachers via attention masking, then distills their knowledge into a single student model with regime-dependent alignment across inductive biases. Across four major equity markets, TIPS achieves state-of-the-art performance, outperforming strong ensemble baselines by 55%, 9%, and 16% in annual return, Sharpe ratio, and Calmar ratio, while requiring only 38% of the inference-time computation. Further analyses show that TIPS generates statistically significant excess returns beyond both vanilla Transformers and its teacher ensembles, and exhibits regime-dependent behavioral alignment with classical architectures during their profitable periods. These results highlight the importance of regime-dependent inductive bias utilization for robust generalization in non-stationary financial time series.

Motivation & Objective

• Motivate the need for adaptive inductive biases in financial time series forecasting due to regime shifts and non-stationarity.
• Demonstrate that naïve multi-bias merging degrades performance compared to bias-specialized models or ensembles.
• Propose and validate TIPS, a distillation-based framework that synthesizes diverse biases into a single transformer.
• Show that TIPS achieves state-of-the-art performance across major equity markets while reducing inference cost.

Proposed method

• Train bias-specialized Transformer teachers encoding distinct priors (causality, locality, periodicity) via attention masking and input design.
• Form a Bias Teacher Ensemble from seven teachers (six bias-specific plus vanilla Transformer) to capture diverse priors.
• Distill ensemble predictions into a single student Transformer using aggressive regularization to avoid rigid imitation.
• Construct soft ensemble targets with temperature scaling and apply label smoothing to improve calibration.
• Train the student with unconstrained attention to synthesize priors, using Stochastic Weight Averaging for robustness.
• Provide analyses to show regime-dependent activation of biases and statistical excess returns.

Figure 1 . Performance–efficiency trade-off across generic time-series models, financial forecasting models, and classical architectures evaluated across multiple equity markets. The figure highlights substantial variation in performance and computational cost across model families, with TIPS achiev

Experimental results

Research questions

• RQ1 Can diverse inductive biases improve robustness of Transformers on non-stationary financial data?
• RQ2 Does naïve merging of multiple biases degrade performance compared to specialization or ensembling?
• RQ3 Can a distilled student effectively synthesize multiple priors while maintaining inference efficiency?
• RQ4 Do bias priors activate regime-specifically, aligning with profitable market conditions?
• RQ5 To what extent does TIPS provide statistically significant excess returns beyond baseline models?

Key findings

• TIPS delivers strongest overall performance across four major equity markets, with superior average Sharpe Ratio and annual return compared to baselines.
• The Bias Teacher Ensemble (via attention masking) outperforms ensembles of classical architectures and generic SOTA models, showing effectiveness of bias encoding without architectural heterogeneity.
• Distillation into a single student yields substantial gains over the bias ensemble and reduces inference time by a factor of about 7, enabling ensemble-level robustness with a single model.
• Ablation shows regularization components (low-temp distillation, label smoothing, SWA) are jointly necessary for effective bias synthesis.
• Analysis reveals TIPS achieves statistically significant alpha beyond vanilla Transformers, indicating beneficial signal extraction from inductive bias synthesis.

Figure 2 . Overview of the TIPS training framework. (a) Bias-specialized Transformer (TFM) teachers are constructed via different attention masks or positional biases (Colors indicate where the masks and biases are applied). (b) Teachers are trained independently for ranking prediction. (c) Teacher