[Paper Review] Time-Series Representation Learning via Temporal and Contextual Contrasting

Emadeldeen Eldele, Mohamed Ragab+5 more|arXiv (Cornell University)|Jun 26, 2021

Time Series Analysis and ForecastingComputer Science25 references35 citations

TL;DR

TS-TCC is an unsupervised time-series representation learning framework that uses two correlated augmented views and novel temporal and contextual contrastive modules to produce representations that enable linear-classifier performance comparable to supervised models, with strong few-shot and transfer learning results.

ABSTRACT

Learning decent representations from unlabeled time-series data with temporal dynamics is a very challenging task. In this paper, we propose an unsupervised Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC), to learn time-series representation from unlabeled data. First, the raw time-series data are transformed into two different yet correlated views by using weak and strong augmentations. Second, we propose a novel temporal contrasting module to learn robust temporal representations by designing a tough cross-view prediction task. Last, to further learn discriminative representations, we propose a contextual contrasting module built upon the contexts from the temporal contrasting module. It attempts to maximize the similarity among different contexts of the same sample while minimizing similarity among contexts of different samples. Experiments have been carried out on three real-world time-series datasets. The results manifest that training a linear classifier on top of the features learned by our proposed TS-TCC performs comparably with the supervised training. Additionally, our proposed TS-TCC shows high efficiency in few-labeled data and transfer learning scenarios. The code is publicly available at https://github.com/emadeldeen24/TS-TCC.

Motivation & Objective

• Motivate learning meaningful representations from unlabeled time-series data with temporal dynamics.
• Develop two correlated augmentations (weak and strong) to create diverse views for contrastive learning.
• Introduce a temporal contrasting module with cross-view prediction to capture robust temporal dependencies.
• Add a contextual contrasting module to enhance discriminative power of learned representations.
• Demonstrate effectiveness across multiple real-world time-series tasks and transfer scenarios.

Proposed method

• Create two views x^s (strong) and x^w (weak) via time-series specific augmentations (permutation-jitter, jitter-scale).
• Encode views with a 3-block CNN encoder to obtain latent z_t for each timestep.
• Temporal Contrasting (TC): use autoregressive transformer to predict future z from past z, cross-view (c_t^s predicts z_{t+k}^w and c_t^w predicts z_{t+k}^s).
• Compute TC losses L_TC^s and L_TC^w using a log-bilinear cross-view prediction with a shared W_k mapping from context to feature space.
• Contextual Contrasting (CC): apply a nonlinear projection to contexts, then maximize agreement between same-sample contexts across views while minimizing agreement with other samples (negative samples) using a contrastive loss.
• Combine losses: L = λ1*(L_TC^s + L_TC^w) + λ2*L_CC, train end-to-end in PyTorch with Transformer-based autoregressive model.

Experimental results

Research questions

• RQ1 Can unsupervised TS-TCC learn representations from unlabeled time-series data that are competitive with supervised training when used with a linear classifier?
• RQ2 Do temporal cross-view predictions and contextual contrasting improve representation quality and transferability across different time-series domains?
• RQ3 How do time-series-specific augmentations and dual views impact downstream performance in HAR, sleep staging, epilepsy prediction, and fault diagnosis?
• RQ4 What is the impact of few-shot labeled data and transfer learning on the effectiveness of TS-TCC?

Key findings

• TS-TCC achieves competitive or superior linear evaluation performance compared to supervised training across HAR, Sleep-EDF, Epilepsy, and Fault Diagnosis datasets.
• In few-shot and transfer learning scenarios, TS-TCC significantly outperforms supervised baselines when labeled data are scarce and improves cross-domain transferability.
• Ablation studies show cross-view temporal prediction and contextual contrasting both contribute to performance gains, with stronger gains when using both augmentations.
• Temporal features (via the cross-view prediction) are particularly beneficial, and the proposed augmentations outperform single-augmentation setups.
• TS-TCC demonstrates strong transfer during cross-domain fault diagnosis, outperforming supervised pretraining in the majority of tested scenarios.
• Overall, a linear classifier on TS-TCC features matches or nears supervised performance, confirming effective unsupervised representation learning.