[Paper Review] Large Scale Time-Series Representation Learning via Simultaneous Low and High Frequency Feature Bootstrapping

Vandan Gorade, Azad Singh+1 more|arXiv (Cornell University)|Apr 24, 2022

Time Series Analysis and Forecasting4 citations

TL;DR

This paper proposes a non-contrastive self-supervised learning framework for time-series representation learning that simultaneously captures low- and high-frequency features using a siamese network with shared weights and dual feature extraction heads. By applying identical augmentations to two views of raw data and leveraging temporal convolutional networks (TCNs) with varying receptive fields, the method achieves state-of-the-art performance on five real-world datasets, outperforming contrastive and other self-supervised approaches while using only 10% of labeled data.

ABSTRACT

Learning representation from unlabeled time series data is a challenging problem. Most existing self-supervised and unsupervised approaches in the time-series domain do not capture low and high-frequency features at the same time. Further, some of these methods employ large scale models like transformers or rely on computationally expensive techniques such as contrastive learning. To tackle these problems, we propose a non-contrastive self-supervised learning approach efficiently captures low and high-frequency time-varying features in a cost-effective manner. Our method takes raw time series data as input and creates two different augmented views for two branches of the model, by randomly sampling the augmentations from same family. Following the terminology of BYOL, the two branches are called online and target network which allows bootstrapping of the latent representation. In contrast to BYOL, where a backbone encoder is followed by multilayer perceptron (MLP) heads, the proposed model contains additional temporal convolutional network (TCN) heads. As the augmented views are passed through large kernel convolution blocks of the encoder, the subsequent combination of MLP and TCN enables an effective representation of low as well as high-frequency time-varying features due to the varying receptive fields. The two modules (MLP and TCN) act in a complementary manner. We train an online network where each module learns to predict the outcome of the respective module of target network branch. To demonstrate the robustness of our model we performed extensive experiments and ablation studies on five real-world time-series datasets. Our method achieved state-of-art performance on all five real-world datasets.

Motivation & Objective

• To address the limitation of existing self-supervised time-series methods in simultaneously capturing low- and high-frequency temporal patterns.
• To develop a cost-effective, non-contrastive learning framework that avoids computationally expensive contrastive learning and large models like transformers.
• To improve representation generalization by bootstrapping latent features through two augmented views of the same time series using the same augmentation family.
• To demonstrate robust performance across diverse real-world time-series datasets with minimal labeled data.

Proposed method

• The framework employs a siamese neural network with two weight-sharing branches, each processing a different augmented view of the same raw time series input.
• Each branch is followed by a low-frequency feature bootstrapping (LFFB) and a high-frequency feature bootstrapping (HFFB) module, using multilayer perceptrons (MLP) and temporal convolutional networks (TCN), respectively.
• Augmentations are sampled from a single family (jitter, permute, rotate), ensuring consistency and avoiding negative pair computation.
• The TCN heads use dilated convolutions with varying kernel sizes and dilation rates to capture long-range dependencies across multiple temporal scales.
• A balanced loss function combines supervisory signals from both LFFB and HFFB modules, with a learnable balance factor λ to weight contributions.
• The model is trained end-to-end using only positive pairs, avoiding contrastive learning's computational burden.

Figure 1: Given an input sample x, model creates two augmented versions $x_{1}\sim t$ and $x_{2}\sim t^{{}^{\prime}}$ , where $t$ and $t^{{}^{\prime}}$ are sampled from a set $T$ of augmentations. $x_{1}$ and $x_{2}$ pass through the backbone encoders parameterized by $\theta$ and $\epsilon$ which g

Experimental results

Research questions

• RQ1 Can a non-contrastive self-supervised learning approach effectively learn complementary low- and high-frequency representations in time-series data without relying on negative pairs?
• RQ2 How does the choice of augmentation strategy—specifically, using the same augmentation family for both views—affect representation quality and model generalization?
• RQ3 What is the optimal configuration of TCN kernel size and dilation rate relative to the backbone encoder for capturing multi-scale temporal patterns?
• RQ4 To what extent can the proposed method generalize across diverse time-series datasets with minimal labeled data?

Key findings

• The proposed method achieves state-of-the-art performance on five real-world time-series datasets, including HAR, ECG, and EEG, outperforming existing contrastive and non-contrastive methods.
• On the HAR dataset, the model achieves a Macro-F1 score of 93.28 ± 0.36 when using a TCN kernel size smaller than the backbone kernel and dilation rate greater than the TCN kernel size.
• Performance drops significantly when the balance factor λ is set to extreme values (e.g., λ = 500), with a Macro-F1 score of 87.56 ± 13.26, indicating sensitivity to λ selection.
• The model achieves close-to-supervised performance using only 10% of the labeled data, demonstrating strong data efficiency.
• Ablation studies confirm that using the same augmentation family for both views yields better performance than mixed augmentation strategies.
• The optimal configuration is found to be when the TCN kernel size is larger than the backbone kernel and the dilation rate exceeds the TCN kernel size, maximizing feature diversity.