[论文解读] Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

Wenrui Zhang, L. Yang+2 more|arXiv (Cornell University)|May 20, 2022

Time Series Analysis and Forecasting被引用 10

一句话总结

本文提出跨重建变压器（CRT），一种自监督时间序列表征学习方法，通过使用Transformer编码器在时域与频域之间进行跨域重建。通过应用课程学习的掩码重建与实例判别，CRT在多个真实世界数据集上实现最先进性能，相较于现有方法提升2%–9%，通过增强时间-频谱相关性的建模。

ABSTRACT

Unsupervised/self-supervised representation learning in time series is critical since labeled samples are usually scarce in real-world scenarios. Existing approaches mainly leverage the contrastive learning framework, which automatically learns to understand the similar and dissimilar data pairs. Nevertheless, they are restricted to the prior knowledge of constructing pairs, cumbersome sampling policy, and unstable performances when encountering sampling bias. Also, few works have focused on effectively modeling across temporal-spectral relations to extend the capacity of representations. In this paper, we aim at learning representations for time series from a new perspective and propose Cross Reconstruction Transformer (CRT) to solve the aforementioned problems in a unified way. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we transform time series into the frequency domain and randomly drop certain parts in both time and frequency domains. Dropping can maximally preserve the global context compared to cropping and masking. Then a transformer architecture is utilized to adequately capture the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, which is called Dropped Temporal-Spectral Modeling. To discriminate the representations in global latent space, we propose Instance Discrimination Constraint to reduce the mutual information between different time series and sharpen the decision boundaries. Additionally, we propose a specified curriculum learning strategy to optimize the CRT, which progressively increases the dropping ratio in the training process.

研究动机与目标

• 解决现有自监督时间序列方法忽略频谱信息且因掩码导致分布偏移的局限性。
• 通过跨域重建显式建模时间-频谱相关性，提升表征学习能力。
• 消除对比学习中对复杂采样策略的依赖，并避免基于重建方法的分布偏移。
• 开发一种鲁棒的端到端框架，通过课程学习与实例判别提升下游性能。

提出的方法

• 通过快速傅里叶变换获取频域表征，包括幅度与相位，以保留频谱信息。
• 通过在时域与频域中随机掩码补丁，引入分布偏移，同时保留全局上下文，提出一种跨域丢弃-重建任务。
• 共享的Transformer编码器处理来自两个域的表征，通过联合建模时间与频谱特征，学习跨域依赖性。
• 模型从掩码表征中重建原始时域与频域数据，实现自监督预训练。
• 采用课程学习策略，在预训练过程中逐步增加丢弃比率，以提升鲁棒性与泛化能力。
• 应用实例判别约束（IDC），最小化不同样本间的互信息，从而在潜在空间中锐化决策边界。

Figure 1: Comparisons with previous self-supervised methods. “Encode” is the encoder producing the representations of time series. “Temporal” and “Spectral” are time-domain and frequency-domain data respectively. Note that our CRT utilizes both temporal and spectral information, and captures cross-d

实验结果

研究问题

• RQ1 时域与频域之间的跨域重建是否能提升自监督时间序列表征学习？
• RQ2 建模时间-频谱相关性如何增强特征可分性与下游性能？
• RQ3 结合渐进式掩码的课程学习是否能提升预训练中的鲁棒性与泛化能力？
• RQ4 所提出的实例判别约束与对比学习相比，在稳定性与性能上表现如何？
• RQ5 在频域中保留相位信息在多大程度上能提升重建质量与表征质量？

主要发现

• CRT在三个基准数据集上均实现最先进性能，在分类准确率上较现有方法提升2%至9%。
• 在PTB-XL数据集上，CRT在'_MI'类别上达到98.94% ± 0.22的AUC，显著优于F2T（87.30%）与T2F（94.24%）方法。
• 消融实验表明，课程学习与实例判别约束均显著提升模型性能，IDC有效降低样本间的互信息。
• 跨域重建框架（CRT）始终优于单向方法（T2F与F2T），尤其在具有周期性结构的复杂信号（如心电图）上表现更优。
• 模型展现出强大的泛化能力，在70%丢弃率下仍保持高重建质量，表明其对全局模式的学习具有鲁棒性。
• 在频域中包含相位信息相比仅使用幅度的方法，能生成更准确、更具信息量的表征。

Figure 2: The framework of our CRT. For each time series, we transform it into the frequency domain and calculate the magnitude and phase as explained in Section III-B . We then slice the data and randomly drop some sliced patches. The remaining patches are projected as three types of embeddings thr