[Paper Review] Unsupervised Scalable Representation Learning for Multivariate Time Series

Jean-Yves Franceschi, Aymeric Dieuleveut+1 more|arXiv (Cornell University)|Jan 30, 2019

Time Series Analysis and ForecastingComputer Science40 references187 citations

TL;DR

Defines a scalable unsupervised method to learn universal embeddings for multivariate time series using a causal dilated CNN encoder trained with a time-based triplet loss and negative sampling.

ABSTRACT

Time series constitute a challenging data type for machine learning algorithms, due to their highly variable lengths and sparse labeling in practice. In this paper, we tackle this challenge by proposing an unsupervised method to learn universal embeddings of time series. Unlike previous works, it is scalable with respect to their length and we demonstrate the quality, transferability and practicability of the learned representations with thorough experiments and comparisons. To this end, we combine an encoder based on causal dilated convolutions with a novel triplet loss employing time-based negative sampling, obtaining general-purpose representations for variable length and multivariate time series.

Motivation & Objective

• Develop a general-purpose, unsupervised representation learning method for multivariate time series that handles variable lengths and is scalable.
• Learn fixed-length embeddings from variable-length inputs without supervision.
• Demonstrate universality and transferability of the representations across datasets and tasks.
• Showcase applicability to classification and regression, including long time series and sparse labeling scenarios.

Proposed method

• Use an encoder based on stacks of causal dilated convolutions to map variable-length time series to fixed-length embeddings.
• Introduce a time-based triplet loss with negative sampling inspired by word2vec to learn similarity between subseries without labels.
• Train decoder-free encoder with fixed-length output via global max pooling and a linear projection.
• Allow inputs of varying length by training on a range of subseries lengths to produce transferable representations.
• Provide an efficient training procedure with complexity O(K * c(f)) focused on per-term backpropagation to save memory.
• Evaluate representations by training simple classifiers (SVM) on top of embeddings and by performing transfer experiments.

Experimental results

Research questions

• RQ1 Can unsupervised training of a scalable encoder produce universal, transferable representations for multivariate time series?
• RQ2 Do time-based triplet losses with negative sampling enable effective discrimination of similar vs. dissimilar subseries without labels?
• RQ3 Are fixed-length embeddings from variable-length time series useful for downstream tasks like classification and regression across diverse datasets?
• RQ4 How does the proposed method compare to other unsupervised and supervised approaches on standard time series benchmarks?
• RQ5 Can the representations scale to long time series and multivariate data while remaining efficient?

Key findings

• The method yields high-quality representations that are transferable across datasets.
• It outperforms concurrent unsupervised methods on several UCR datasets and approaches state-of-the-art supervised methods in some settings.
• Representations enable effective sparsely labeled learning, often surpassing fully supervised baselines when labels are scarce.
• The learned embeddings are useful for non-classification tasks such as regression on long time series.
• The encoder can handle multivariate inputs by adjusting the first layer and achieves competitive results on the UEA multivariate archive.
• On a long real-world time series, representations substantially reduce inference-time data size with minimal loss in predictive performance.