[Paper Review] Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting Problem

Peiwang Tang, Xianchao Zhang|arXiv (Cornell University)|Jan 4, 2023

Time Series Analysis and Forecasting4 citations

TL;DR

Infomaxformer proposes a maximum entropy-based self-attention mechanism and Keys/Values Distilling to reduce Transformer complexity from 𝒪(L²) to 𝒪(L), enabling long time-series forecasting with improved accuracy. It combines seasonal-trend decomposition with sparse attention to capture long-range dependencies, outperforming SOTA models on multiple long-sequence datasets.

ABSTRACT

The Transformer architecture yields state-of-the-art results in many tasks such as natural language processing (NLP) and computer vision (CV), since the ability to efficiently capture the precise long-range dependency coupling between input sequences. With this advanced capability, however, the quadratic time complexity and high memory usage prevents the Transformer from dealing with long time-series forecasting problem (LTFP). To address these difficulties: (i) we revisit the learned attention patterns of the vanilla self-attention, redesigned the calculation method of self-attention based the Maximum Entropy Principle. (ii) we propose a new method to sparse the self-attention, which can prevent the loss of more important self-attention scores due to random sampling.(iii) We propose Keys/Values Distilling method motivated that a large amount of feature in the original self-attention map is redundant, which can further reduce the time and spatial complexity and make it possible to input longer time-series. Finally, we propose a method that combines the encoder-decoder architecture with seasonal-trend decomposition, i.e., using the encoder-decoder architecture to capture more specific seasonal parts. A large number of experiments on several large-scale datasets show that our Infomaxformer is obviously superior to the existing methods. We expect this to open up a new solution for Transformer to solve LTFP, and exploring the ability of the Transformer architecture to capture much longer temporal dependencies.

Motivation & Objective

• Address the quadratic time and memory complexity of vanilla Transformers in long time-series forecasting (LTFP).
• Overcome performance degradation from random sampling in sparse attention mechanisms.
• Reduce redundant features in self-attention maps through a novel Keys/Values Distilling mechanism.
• Improve modeling of complex time-series patterns by integrating seasonal-trend decomposition with Transformer architecture.
• Enable effective long-sequence input (up to 1440 steps) while maintaining high predictive accuracy.

Proposed method

• Reformulate self-attention using the Maximum Entropy Principle to guide sparse attention computation, reducing complexity while preserving key dependencies.
• Introduce a sparsity mechanism based on entropy maximization (MEA) that selects dominant queries instead of random sampling, minimizing feature loss.
• Propose Keys/Values Distilling to compress redundant attention representations, reducing both time and memory complexity to 𝒪(L).
• Integrate time-series decomposition (TSD) with encoder-decoder architecture to separate and model seasonal and trend components explicitly.
• Combine MEA with TSD to enable long-sequence modeling with high accuracy, avoiding performance drops from sparsity.

Figure 1. Infomaxformer architecture

Experimental results

Research questions

• RQ1 Can the Maximum Entropy Principle be used to design a more effective and efficient sparse attention mechanism for long time-series forecasting?
• RQ2 How can redundant features in self-attention maps be effectively removed without sacrificing predictive performance?
• RQ3 Does combining time-series decomposition with sparse attention improve long-range dependency modeling in Transformers?
• RQ4 Can the proposed method scale to ultra-long input sequences (e.g., 1440 steps) while maintaining or improving accuracy?
• RQ5 How does the performance of MEA compare to standard attention and other sparse attention mechanisms under memory and sequence length constraints?

Key findings

• Infomaxformer reduces time and memory complexity from 𝒪(L²) to 𝒪(L), enabling processing of ultra-long sequences up to 1440 steps.
• On the ETTh1 dataset, Infomaxformer achieves an MSE of 0.573 and MAE of 0.585 at a sequence length of 1440, outperforming baselines.
• Ablation studies confirm that removing any component—MEA, TSD, or Keys/Values Distilling—leads to significant performance degradation.
• The combination of TSD and MEA outperforms SDB with AC in Autoformer, especially in long-sequence forecasting (e.g., 192-step prediction).
• The sampling factor U has a minimal impact on performance when sufficiently large, indicating robustness to variance estimation in MEA.
• The model maintains stable performance across varying sampling factors, unlike Informer, due to a more informed initial sampling strategy.

(a) Softmax scores at Head1@Encoder layer