[Paper Review] VIbCReg: Variance-Invariance-better-Covariance Regularization for Self-Supervised Learning on Time Series

Dae-Soo Lee, Erlend Aune|arXiv (Cornell University)|Sep 2, 2021

Anomaly Detection Techniques and Applications32 references4 citations

TL;DR

This paper proposes VIbCReg, a variance-invariance-better-covariance regularization method for self-supervised time series representation learning. By improving VICReg's covariance term and introducing an IterNorm layer in the projection head, VIbCReg accelerates convergence and achieves state-of-the-art performance on linear evaluation and fine-tuning benchmarks using the UCR archive and PTB-XL ECG dataset.

ABSTRACT

Self-supervised learning for image representations has recently had many breakthroughs with respect to linear evaluation and fine-tuning evaluation. These approaches rely on both cleverly crafted loss functions and training setups to avoid the feature collapse problem. In this paper, we improve on the recently proposed VICReg paper, which introduced a loss function that does not rely on specialized training loops to converge to useful representations. Our method improves on a covariance term proposed in VICReg, and in addition we augment the head of the architecture by an IterNorm layer that greatly accelerates convergence of the model. Our model achieves superior performance on linear evaluation and fine-tuning evaluation on a subset of the UCR time series classification archive and the PTB-XL ECG dataset.

Motivation & Objective

• To address feature collapse in self-supervised time series representation learning without relying on complex training loops.
• To improve upon the VICReg method by refining its covariance regularization term for better feature diversity and invariance.
• To accelerate model convergence through architectural enhancement using an IterNorm layer in the projection head.
• To achieve superior linear evaluation and fine-tuning performance on standard time series benchmarks.

Proposed method

• The method introduces a modified covariance regularization term that enhances feature variance and invariance in self-supervised contrastive learning.
• It incorporates an IterNorm layer in the projection head to stabilize and accelerate training dynamics.
• The loss function retains VICReg's core contrastive objective but improves the balance between variance, invariance, and covariance terms.
• The approach is trained end-to-end with standard data augmentations, avoiding specialized training protocols.
• The model is evaluated on linear probing and fine-tuning protocols to assess representation quality.

Experimental results

Research questions

• RQ1 Can a refined covariance regularization term improve representation quality in self-supervised time series learning?
• RQ2 Does the addition of an IterNorm layer significantly accelerate convergence in time series SSL?
• RQ3 How does VIbCReg compare to VICReg in linear evaluation and fine-tuning performance on time series benchmarks?
• RQ4 Can the method generalize across diverse time series datasets such as UCR and PTB-XL?

Key findings

• VIbCReg achieves state-of-the-art performance on linear evaluation and fine-tuning benchmarks using the UCR time series classification archive.
• The method demonstrates faster convergence compared to baseline methods due to the integration of the IterNorm layer.
• The improved covariance regularization leads to more robust and diverse feature representations.
• The model outperforms VICReg on both the UCR and PTB-XL ECG datasets in downstream evaluation tasks.