[Paper Review] Contrastive Neural Processes for Self-Supervised Learning

Konstantinos Kallidromitis, Denis Gudovskiy+3 more|arXiv (Cornell University)|Oct 24, 2021

Music and Audio Processing34 references4 citations

TL;DR

This paper proposes Contrastive Neural Processes (ContrNP), a self-supervised learning framework that combines neural processes for time series forecasting with contrastive learning to generate data-augmented representations without manual augmentation design. It achieves state-of-the-art performance across industrial, medical, and audio datasets, improving accuracy by over 10% on ECG data and matching fully supervised learning with only 10% labeled data.

ABSTRACT

Recent contrastive methods show significant improvement in self-supervised learning in several domains. In particular, contrastive methods are most effective where data augmentation can be easily constructed e.g. in computer vision. However, they are less successful in domains without established data transformations such as time series data. In this paper, we propose a novel self-supervised learning framework that combines contrastive learning with neural processes. It relies on recent advances in neural processes to perform time series forecasting. This allows to generate augmented versions of data by employing a set of various sampling functions and, hence, avoid manually designed augmentations. We extend conventional neural processes and propose a new contrastive loss to learn times series representations in a self-supervised setup. Therefore, unlike previous self-supervised methods, our augmentation pipeline is task-agnostic, enabling our method to perform well across various applications. In particular, a ResNet with a linear classifier trained using our approach is able to outperform state-of-the-art techniques across industrial, medical and audio datasets improving accuracy over 10% in ECG periodic data. We further demonstrate that our self-supervised representations are more efficient in the latent space, improving multiple clustering indexes and that fine-tuning our method on 10% of labels achieves results competitive to fully-supervised learning.

Motivation & Objective

• To address the challenge of designing effective data augmentations in time series self-supervised learning, where standard image-based augmentations do not apply.
• To develop a task-agnostic representation learning framework that leverages neural processes for forecasting and contrastive learning for representation alignment.
• To improve representation quality in the latent space, enabling better clustering and downstream transfer learning.
• To demonstrate that a contrastive loss combined with neural process forecasting can outperform existing self-supervised methods on diverse real-world time series benchmarks.

Proposed method

• The method uses neural processes to generate multiple views of the same time series by applying different sampling functions, effectively creating data augmentations without manual design.
• A dual objective loss is introduced: a contrastive loss to minimize distance between positive views and maximize distance to negative views, and a regression loss to improve forecasting performance.
• The model employs out-of-context sampling to increase regularization and improve the quality of the generated augmentations.
• The framework is trained end-to-end using a contrastive loss that encourages disentangled, class-aware representations in the latent space.
• It is compatible with any neural network architecture, such as ResNet, for downstream classification.
• The approach is agnostic to data modality and is particularly effective for multivariate, imbalanced time series data.

Experimental results

Research questions

• RQ1 Can a self-supervised learning framework based on neural processes generate effective data augmentations without relying on handcrafted transformations in time series data?
• RQ2 How does combining contrastive learning with neural process forecasting improve representation quality in the latent space compared to existing methods?
• RQ3 To what extent can the proposed method generalize across diverse time series applications such as healthcare, industrial monitoring, and audio?
• RQ4 Can the learned representations achieve performance competitive with fully supervised learning using only a small fraction of labeled data?
• RQ5 How does the method compare in terms of clustering quality and representation disentanglement to state-of-the-art self-supervised baselines?

Key findings

• ContrNP outperforms all self-supervised baselines on the Atrial Fibrillation (AFDB) dataset, achieving 94.2% accuracy with a linear classifier, a 10.6% improvement over the next best method (SimCLR at 82.3%).
• On the ECG periodic data, ContrNP improves accuracy by over 10% compared to state-of-the-art methods, demonstrating strong performance on challenging medical time series.
• ContrNP achieves superior clustering performance, with a Silhouette score of 0.36 ± 0.07 on AFDB and a Davies–Bouldin index of 0.89 ± 0.15 on Urban8K, outperforming all baselines.
• Fine-tuning on just 10% of the labels yields results competitive with fully supervised learning, particularly on the Urban8K audio dataset where it reaches 95.4% accuracy.
• The method produces better-separated representations in the latent space, as shown by t-SNE visualizations, indicating improved disentanglement and class separation.
• ContrNP achieves state-of-the-art results across industrial, medical, and audio time series datasets, demonstrating broad applicability and robustness.