[Paper Review] Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Xiang Zhang, Ziyuan Zhao+2 more|ArXiv.org|Jun 17, 2022

EEG and Brain-Computer InterfacesNeuroscience127 citations

TL;DR

introduces Time-Frequency Consistency (TF-C) for self-supervised pre-training on time series by aligning time-based and frequency-based embeddings, improving transfer to unseen target datasets.

ABSTRACT

Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequency-based representations of the same example are located close together in the time-frequency space. To this end, we posit that time-frequency consistency (TF-C) -- embedding a time-based neighborhood of an example close to its frequency-based neighborhood -- is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. Code and datasets: https://github.com/mims-harvard/TFC-pretraining.

Motivation & Objective

• Motivate robust pre-training for time series despite domain shifts between pre-training and target data.
• Propose TF-C as a generalizable pre-training principle that does not require target-domain data during pre-training.
• Develop a decomposable model with time and frequency encoders and cross-space projectors to enforce TF-C.
• Introduce time-based and frequency-based contrastive losses and a consistency (triplet-like) loss to fuse representations.
• Demonstrate transfer gains across diverse datasets and tasks compared to state-of-the-art baselines.

Proposed method

• Define two parallel encoders for time and frequency domains, with cross-space projections to a shared time–frequency space.
• Apply a bank of time-based augmentations and a frequency-augmentation strategy that perturbs the spectrum.
• Use NT-Xent style contrastive losses to align time-based and frequency-based representations separately.
• Introduce a triplet-inspired consistency loss to encourage closeness between time–frequency representations across domains.
• Combine losses into a TF-C objective that balances contrastive and consistency terms (L_TF-C = λ(L_T + L_F) + (1−λ)L_C).

Experimental results

Research questions

• RQ1 Can time-based and frequency-based representations of the same time series be aligned in a shared latent space without target-domain data?
• RQ2 Do frequency-domain augmentations and a TF-C consistency objective improve transfer to unseen target datasets?
• RQ3 How does TF-C perform relative to state-of-the-art self-supervised baselines on diverse time-series tasks?
• RQ4 Is the TF-C pre-training beneficial in both one-to-one and one-to-many transfer settings?
• RQ5 What is the effect of the proposed frequency perturbations on representation robustness?

Key findings

• TF-C outperforms all baselines by 15.4% in average F1 score in one-to-one transfer settings.
• TF-C improves precision by 8.4% in challenging one-to-many transfer settings.
• The approach shows strong transfer across eight datasets including EEG, EMG, ECG, gait, and vibration signals.
• The model uses a four-component architecture (time encoder, frequency encoder, and two cross-space projectors) to embed into a shared time–frequency space.
• Frequency-domain augmentations are effective and novel in contrastive learning for time series.