[Paper Review] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

Gerald Woo, Chenghao Liu+3 more|arXiv (Cornell University)|Feb 3, 2022

Time Series Analysis and Forecasting86 citations

TL;DR

CoST learns disentangled trend and seasonal representations via time- and frequency-domain contrastive learning, then uses a simple regressor, achieving state-of-the-art forecasting performance on multivariate and univariate long-sequence benchmarks.

ABSTRACT

Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -- we justify such a paradigm from a causal perspective. Following this principle, we propose a new time series representation learning framework for time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. It is also robust to various choices of backbone encoders, as well as downstream regressors. Code is available at https://github.com/salesforce/CoST.

Motivation & Objective

• Motivate a representation-learning paradigm for time series forecasting that separates seasonal and trend components from observed data.
• Propose CoST to learn disentangled seasonal-trend representations using contrastive losses in time and frequency domains.
• Demonstrate robustness and superior forecasting performance across real-world multivariate and univariate datasets.

Proposed method

• Represent time series as a sum of trend, seasonal, and error components to justify disentangled representations from a causal perspective.
• Introduce a backbone encoder f_b to map X (h x m) to latent V (h x d).
• Trend Feature Disentangler (TFD): a mixture of auto-regressive 1D causal convolutions with varying kernel sizes (2^i) + average pooling, trained with a time-domain MoCo-style contrastive loss (L_time).
• Seasonal Feature Disentangler (SFD): apply FFT to obtain frequency-domain features, pass through a learnable per-frequency complex-valued linear layer, then inverse FFT to obtain V^(S), trained with a frequency-domain contrastive loss (L_amp + L_phase).
• Overall objective: L = L_time + (alpha/2) (L_amp + L_phase).
• Demonstrate robustness to backbones and downstream regressors via extensive ablations.

Experimental results

Research questions

• RQ1 Can disentangled seasonal-trend representations improve long-sequence time series forecasting compared to end-to-end methods?
• RQ2 Do time-domain and frequency-domain contrastive losses yield robust trend and seasonal representations under distribution shifts in the error component?
• RQ3 How do architectural choices (TFD/SFD) and the learnable Fourier layer impact predictive accuracy and generalization?

Key findings

• CoST achieves state-of-the-art forecasting performance, with notable improvements (e.g., 21.3% reduction in MSE on multivariate benchmarks) over competitive representation-learning baselines.
• CoST is robust to different backbone encoders (TCN, LSTM, Transformer) and downstream regressors (ridge, linear, kernel ridge).
• Ablations show that both trend and seasonal components contribute to gains, and their combination yields the best results.
• A frequency-domain contrastive loss enables effective learning of discriminative seasonal representations without needing explicit knowledge of seasonal periods.
• Empirical analysis and case study indicate clear separability and clusterability of learned seasonal and trend representations.