[Paper Review] BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis

Zelin Ni, Hang Yu+3 more|arXiv (Cornell University)|Oct 31, 2023

Stock Market Forecasting Methods12 citations

TL;DR

BasisFormer learns a data-driven, interpretable basis via self-supervised contrastive learning and uses a Coef module with bidirectional cross-attention to align time series with basis vectors, then forecasts by aggregating future-basis vectors. It achieves state-of-the-art gains on six datasets for univariate and multivariate forecasting.

ABSTRACT

Bases have become an integral part of modern deep learning-based models for time series forecasting due to their ability to act as feature extractors or future references. To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. However, current state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this challenge, we propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. This architecture comprises three components: First, we acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Next, we design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, we present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. Through extensive experiments on six datasets, we demonstrate that BasisFormer outperforms previous state-of-the-art methods by 11.04\% and 15.78\% respectively for univariate and multivariate forecasting tasks. Code is available at: \url{https://github.com/nzl5116190/Basisformer}

Motivation & Objective

• Motivate the use of learnable bases that are tailored to a dataset of time series.
• Learn a basis via adaptive self-supervised learning treating historical and future views as two views.
• Develop a Coef module with bidirectional cross-attention to measure time-series–basis similarity.
• Design a Forecast module to aggregate future-basis vectors using learned coefficients for accurate predictions.
• Demonstrate end-to-end training and show state-of-the-art improvements on six real-world datasets.

Proposed method

• Basis learning via adaptive self-supervised contrastive learning between historical and future views of each time series.
• Coef module using bidirectional cross-attention to compute similarity coefficients between time series and basis vectors in the historical view.
• Forecast module projecting and aggregating future-basis vectors weighted by the coefficients to predict future values.
• Loss design combining: 1) prediction MSE, 2) InfoNCE alignment loss between historical and future view coefficients, 3) a smoothness regularization on the learned basis over time, plus a timestamp-conditioned four-layer MLP basis generator.
• End-to-end training of BasisFormer integrating Basis, Coef, and Forecast components.

Experimental results

Research questions

• RQ1 Can a learnable and interpretable basis outperform fixed or covariate-based bases in time series forecasting?
• RQ2 How can we robustly learn basis–time series associations that are consistent across historical and future views?
• RQ3 What is the impact of the number of basis vectors and multi-head attention on forecasting accuracy?
• RQ4 How important are the self-supervised alignment and smoothness losses for basis quality and forecast performance?

Key findings

• BasisFormer achieves up to 11.04% improvement over SOTA in univariate forecasting and 15.78% in multivariate forecasting.
• Learnable bases significantly outperform fixed sine/cosine bases and covariate embeddings on the evaluated datasets.
• Increasing the number of heads in the Coef module generally helps up to a point, after which performance may decline.
• InfoNCE-based alignment loss and the smoothness loss in the Basis module both contribute positively, with their combination yielding the best results.
• The model yields strong performance, notably on highly periodic datasets like Traffic, suggesting effective learning of dataset-specific bases.