[Paper Review] Minusformer: Improving Time Series Forecasting by Progressively Learning Residuals

Daojun Liang, Haixia Zhang+3 more|arXiv (Cornell University)|Feb 4, 2024

Time Series Analysis and Forecasting4 citations

TL;DR

Minusformer proposes a novel Transformer-based architecture for time series forecasting that replaces additive attention with subtraction-based residual learning to combat overfitting. By progressively learning residuals through a dual-stream highway mechanism, it achieves state-of-the-art performance with an average 11.9% improvement across datasets, while enhancing interpretability and robustness to depth and hyperparameters.

ABSTRACT

In this paper, we find that ubiquitous time series (TS) forecasting models are prone to severe overfitting. To cope with this problem, we embrace a de-redundancy approach to progressively reinstate the intrinsic values of TS for future intervals. Specifically, we introduce a dual-stream and subtraction mechanism, which is a deep Boosting ensemble learning method. And the vanilla Transformer is renovated by reorienting the information aggregation mechanism from addition to subtraction. Then, we incorporate an auxiliary output branch into each block of the original model to construct a highway leading to the ultimate prediction. The output of subsequent modules in this branch will subtract the previously learned results, enabling the model to learn the residuals of the supervision signal, layer by layer. This designing facilitates the learning-driven implicit progressive decomposition of the input and output streams, empowering the model with heightened versatility, interpretability, and resilience against overfitting. Since all aggregations in the model are minus signs, which is called Minusformer. Extensive experiments demonstrate the proposed method outperform existing state-of-the-art methods, yielding an average performance improvement of 11.9% across various datasets.The code has been released at https://github.com/Anoise/Minusformer.

Motivation & Objective

• To address the pervasive problem of overfitting in deep time series forecasting models, especially in high-dimensional attribute spaces.
• To improve model generalization and resilience against overfitting without sacrificing predictive performance.
• To enable implicit, progressive decomposition of input and output signals through a learning-driven residual mechanism.
• To enhance model interpretability by aligning block-level outputs with the final prediction structure.
• To design a deep, hyperparameter-insensitive architecture that maintains performance even at extreme depth.

Proposed method

• The model replaces standard additive attention in Transformers with subtraction-based aggregation, forming a 'Minusformer' architecture where all information flows use minus signs.
• An auxiliary output stream is introduced in each block, where subsequent outputs subtract previously learned results to learn residuals of the supervision signal.
• The dual-stream design enables implicit, progressive decomposition of both input and output sequences through residual learning across layers.
• The architecture supports deep stacking (up to 16 blocks) without overfitting, due to the de-redundancy effect of subtraction-based learning.
• The method is compatible with various attention mechanisms (e.g., Prob-Attention, Period-Attention), allowing plug-and-play integration for performance gains.
• Model interpretability is enhanced by aligning block outputs with the final prediction shape, enabling visualization of learned components at each layer.

Figure 1: Comparison of the proposed Minusformer and other latest advanced models. The results (MSE) are averaged across all prediction lengths. The numerical suffix after the model indicates the input length of the model. Minusformer is configured with two versions of input length in order to align

Experimental results

Research questions

• RQ1 Can replacing additive attention with subtraction-based aggregation reduce overfitting in time series forecasting models?
• RQ2 Does progressive residual learning through a dual-stream highway mechanism improve generalization and performance?
• RQ3 Can the Minusformer architecture maintain strong performance when deeply stacked, unlike standard Transformers?
• RQ4 To what extent does the model’s design enhance interpretability of learned components across layers?
• RQ5 How does Minusformer perform when integrated with novel attention mechanisms like Prob-Attention or Flow-Attention?

Key findings

• Minusformer achieves an average performance improvement of 11.9% across diverse real-world time series datasets compared to existing state-of-the-art methods.
• On the Electricity and Weather datasets, Prob-Attention integrated into Minusformer reduced MSE by 46% and 61%, respectively, setting new SOTA results.
• The model maintains strong performance even when deepened to 16 blocks, with no sign of overfitting, unlike standard iTransformer which overfits at 8 blocks.
• Visualizations show that shallow blocks learn salient patterns in low-dimensional settings, while deeper blocks absorb residual components as model capacity increases.
• The model is less sensitive to hyperparameter tuning, as shown in ablation studies in Appendix B.
• Attention visualization confirms that each block learns meaningful, evolving patterns, with deeper blocks capturing finer-grained residuals.

Figure 2: Generalization of the model when time series are aggregated in different directions. The experiment was conducted utilizing Transformer with 4 blocks (baseline) on the Traffic dataset.