[Paper Review] Self-Supervised Time Series Representation Learning by Inter-Intra Relational Reasoning

Haoyi Fan, Fengbin Zhang+1 more|arXiv (Cornell University)|Nov 27, 2020

Time Series Analysis and Forecasting52 references23 citations

TL;DR

SelfTime proposes a self-supervised time series representation learning framework that jointly models inter-sample and intra-temporal relations to capture global and local structural patterns. By leveraging relational reasoning over sampled positive/negative samples and multi-scale time-piece relations, it achieves state-of-the-art performance on multiple time series classification benchmarks, outperforming prior methods by up to 20.2% in transfer learning settings.

ABSTRACT

Self-supervised learning achieves superior performance in many domains by extracting useful representations from the unlabeled data. However, most of traditional self-supervised methods mainly focus on exploring the inter-sample structure while less efforts have been concentrated on the underlying intra-temporal structure, which is important for time series data. In this paper, we present SelfTime: a general self-supervised time series representation learning framework, by exploring the inter-sample relation and intra-temporal relation of time series to learn the underlying structure feature on the unlabeled time series. Specifically, we first generate the inter-sample relation by sampling positive and negative samples of a given anchor sample, and intra-temporal relation by sampling time pieces from this anchor. Then, based on the sampled relation, a shared feature extraction backbone combined with two separate relation reasoning heads are employed to quantify the relationships of the sample pairs for inter-sample relation reasoning, and the relationships of the time piece pairs for intra-temporal relation reasoning, respectively. Finally, the useful representations of time series are extracted from the backbone under the supervision of relation reasoning heads. Experimental results on multiple real-world time series datasets for time series classification task demonstrate the effectiveness of the proposed method. Code and data are publicly available at https://haoyfan.github.io/.

Motivation & Objective

• To address the limitation of existing self-supervised methods that focus primarily on inter-sample structure while neglecting intra-temporal dependencies in time series.
• To develop a generalizable framework that captures both global sample-level and local temporal-level structural patterns in unlabeled time series.
• To improve representation learning by designing pretext tasks that reason over multiple levels of relational structure.
• To evaluate the transferability and robustness of learned representations across diverse real-world time series datasets.

Proposed method

• The framework uses a shared backbone network to extract features from time series inputs and their augmented views.
• Inter-sample relation is modeled by sampling positive (augmented view) and negative (distant sample) pairs relative to a given anchor.
• Intra-temporal relation is captured by sampling time-piece segments from the same anchor and constructing multi-scale temporal relations based on temporal distance.
• Two dedicated relation reasoning heads are used: one for quantifying inter-sample similarity and another for intra-temporal temporal pattern reasoning.
• The model is trained end-to-end using contrastive loss to optimize the relational predictions, with representations extracted from the shared backbone.
• A multi-scale temporal relation sampling strategy enables modeling of short-, middle-, and long-term temporal dependencies.

Experimental results

Research questions

• RQ1 Can modeling both inter-sample and intra-temporal relations improve self-supervised time series representation learning?
• RQ2 How does the inclusion of multi-scale temporal relations affect the quality of learned representations?
• RQ3 Does the proposed method generalize across diverse time series domains and outperform existing self-supervised baselines?
• RQ4 To what extent do the learned representations transfer to downstream classification tasks with limited labeled data?
• RQ5 How do different data augmentation and relation sampling strategies impact the final performance?

Key findings

• SelfTime achieves new state-of-the-art performance on multiple time series classification benchmarks, outperforming existing self-supervised methods.
• On the IWS → DLD transfer task, SelfTime improves by 20.2% over Deep InfoMax and 6.81% over Relation.
• In the UGLA → CricketX transfer setting, SelfTime achieves a 4.73% improvement over Relation and 9.06% over Deep InfoMax.
• t-SNE visualization confirms that SelfTime learns more semantically consistent and well-clustered representations than baselines.
• The method demonstrates strong transferability, with consistent gains across all evaluated source-target domain transfer scenarios.
• Ablation studies confirm that both inter-sample and intra-temporal relation reasoning contribute significantly to performance, with the latter being particularly effective for capturing local temporal patterns.