[Paper Review] TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning

Jiexi Liu, Songcan Chen|arXiv (Cornell University)|Dec 25, 2023

Time Series Analysis and Forecasting4 citations

TL;DR

TimesURL proposes a self-supervised contrastive learning framework for universal time series representation learning by introducing frequency-temporal augmentation (FTAug), double Universums (temporal- and instance-wise) as hard negatives, and joint optimization with time reconstruction. It achieves state-of-the-art performance across six downstream tasks, including forecasting, imputation, classification, anomaly detection, and transfer learning.

ABSTRACT

Learning universal time series representations applicable to various types of downstream tasks is challenging but valuable in real applications. Recently, researchers have attempted to leverage the success of self-supervised contrastive learning (SSCL) in Computer Vision(CV) and Natural Language Processing(NLP) to tackle time series representation. Nevertheless, due to the special temporal characteristics, relying solely on empirical guidance from other domains may be ineffective for time series and difficult to adapt to multiple downstream tasks. To this end, we review three parts involved in SSCL including 1) designing augmentation methods for positive pairs, 2) constructing (hard) negative pairs, and 3) designing SSCL loss. For 1) and 2), we find that unsuitable positive and negative pair construction may introduce inappropriate inductive biases, which neither preserve temporal properties nor provide sufficient discriminative features. For 3), just exploring segment- or instance-level semantics information is not enough for learning universal representation. To remedy the above issues, we propose a novel self-supervised framework named TimesURL. Specifically, we first introduce a frequency-temporal-based augmentation to keep the temporal property unchanged. And then, we construct double Universums as a special kind of hard negative to guide better contrastive learning. Additionally, we introduce time reconstruction as a joint optimization objective with contrastive learning to capture both segment-level and instance-level information. As a result, TimesURL can learn high-quality universal representations and achieve state-of-the-art performance in 6 different downstream tasks, including short- and long-term forecasting, imputation, classification, anomaly detection and transfer learning.

Motivation & Objective

• Address the challenge of learning universal time series representations applicable to diverse downstream tasks.
• Overcome limitations of existing self-supervised contrastive learning methods in time series, which often introduce inappropriate inductive biases via naive data augmentations.
• Improve contrastive learning by incorporating high-quality hard negative samples tailored to temporal structure.
• Capture both segment-level and instance-level semantics to enable task-agnostic pre-training for multi-task generalization.
• Develop a unified framework that jointly optimizes contrastive learning and time reconstruction for richer representation learning.

Proposed method

• Proposes FTAug (Frequency-Temporal Augmentation) that combines cropping and frequency mix to preserve temporal and spectral characteristics.
• Introduces double Universums—separately constructed temporal- and instance-wise Universums—as a form of hard negative samples to enhance discriminative learning.
• Employs a dual contrastive learning mechanism (DualCon) that synthesizes representations from both Universum types to improve feature discrimination.
• Integrates time reconstruction as a joint objective with contrastive learning to capture fine-grained (segment-level) and coarse-grained (instance-level) semantics.
• Uses a standard encoder backbone to extract representations from augmented time series, followed by contrastive and reconstruction heads.
• Applies a multi-head contrastive loss that leverages both positive pairs from FTAug and negative pairs from the double Universum structure.

Experimental results

Research questions

• RQ1 Can frequency-temporal augmentation preserve temporal properties better than standard CV/NLP-style augmentations in time series?
• RQ2 How does the inclusion of task-specific hard negative samples (via Universums) improve contrastive learning for universal representation?
• RQ3 To what extent does joint optimization with time reconstruction enhance representation quality across different levels of granularity?
• RQ4 Can a single pre-trained model achieve state-of-the-art performance across diverse downstream tasks without task-specific fine-tuning?
• RQ5 What is the relative contribution of each component (FTAug, double Universum, reconstruction) to the overall performance?

Key findings

• TimesURL achieves state-of-the-art performance on six downstream tasks: short- and long-term forecasting, imputation, classification, anomaly detection, and transfer learning.
• On the Yahoo and KPI anomaly detection benchmarks, TimesURL achieves F1 scores of 0.750 and 0.688 respectively, outperforming prior methods like TS2Vec (0.745 and 0.677).
• Ablation studies show that removing any component—frequency mixing, instance Universum, temporal Universum, double Universum, or time reconstruction—leads to a performance drop of 1.8% to 4.3% on average across 30 UEA datasets.
• The double Universum design significantly improves performance over single Universum variants, confirming the importance of capturing both temporal and instance-level discriminative signals.
• Transfer learning results show that representations learned by TimesURL achieve an average accuracy of 0.895 on CinCECGTorso and 0.864 on CBF, outperforming no-transfer baselines in some cases.
• The joint optimization of contrastive learning and time reconstruction leads to better generalization, as evidenced by improved performance across all evaluated tasks.