[Paper Review] Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding

Sana Tonekaboni, Danny Eytan+1 more|arXiv (Cornell University)|Jun 1, 2021

Time Series Analysis and ForecastingComputer Science40 references104 citations

TL;DR

Introduces Temporal Neighborhood Coding (TNC), a self-supervised framework for learning transferable representations of non-stationary multivariate time series by contrasting neighboring vs non-neighboring windows with PU-weighted loss.

ABSTRACT

Time series are often complex and rich in information but sparsely labeled and therefore challenging to model. In this paper, we propose a self-supervised framework for learning generalizable representations for non-stationary time series. Our approach, called Temporal Neighborhood Coding (TNC), takes advantage of the local smoothness of a signal's generative process to define neighborhoods in time with stationary properties. Using a debiased contrastive objective, our framework learns time series representations by ensuring that in the encoding space, the distribution of signals from within a neighborhood is distinguishable from the distribution of non-neighboring signals. Our motivation stems from the medical field, where the ability to model the dynamic nature of time series data is especially valuable for identifying, tracking, and predicting the underlying patients' latent states in settings where labeling data is practically impossible. We compare our method to recently developed unsupervised representation learning approaches and demonstrate superior performance on clustering and classification tasks for multiple datasets.

Motivation & Objective

• Motivate unsupervised representation learning for complex, sparsely labeled time series, especially in healthcare.
• Define temporal neighborhoods with stationary properties to capture non-stationary dynamics.
• Develop a debiased contrastive objective with PU-learning-based weighting to handle sampling bias in negatives.
• Demonstrate that TNC representations are generalizable and transferable across tasks and datasets.

Proposed method

• Represent time series as X in R^{D x T} and define windows W_t of length delta.
• Define temporal neighborhood N_t as windows with centroids t* ~ N(t, eta * delta) and estimate eta via Augmented Dickey-Fuller test.
• Use an encoder Enc to map W_t to Z_t and a discriminator D(Z_t, Z) to decide if two representations are from the same neighborhood.
• Train with a debiased contrastive loss that pulls Z_t with neighbors closer and pushes Z_t away from non-neighbors, applying PU-weight w_t to non-neighbors.
• Account for sampling bias by treating non-neighbors as unlabeled samples with positive weight and negative weight combinations (PU learning).
• The framework is encoder-architecture agnostic and can use simple RNN or CNN-based encoders; the discriminator is a lightweight binary classifier.

Experimental results

Research questions

• RQ1 Can Temporal Neighborhood Coding learn representations that distinguish temporally proximal windows from distant ones in non-stationary time series?
• RQ2 Do TNC representations cluster according to latent states and support downstream tasks like clustering and classification across diverse datasets?
• RQ3 How does TNC compare to CPC and Triplet-Loss with PU-weighting in terms of clustering quality and classification performance?
• RQ4 Can the neighborhood size be automatically adapted per window using stationarity tests (ADF) to balance informative neighborhoods vs state changes?

Key findings

• TNC yields superior clustering quality (higher Silhouette, lower Davies-Bouldin) across simulated, ECG waveform, and HAR datasets compared with CPC, Triplet-Loss, and K-means baseline.
• TNC achieves classification performance close to supervised models, with AUPRC and accuracy substantially higher than CPC and Triplet-Loss on several datasets.
• TNC representations reveal clear trajectories that change at state transitions, capturing non-stationary dynamics in the latent space.
• Temporal neighborhoods estimated via ADF enable robust, per-window adaptation of neighborhood width, balancing information and state changes.