[Paper Review] Towards a universal neural network encoder for time series

Joan Serrà, Santiago Pascual+1 more|arXiv (Cornell University)|May 10, 2018

Time Series Analysis and Forecasting28 references89 citations

TL;DR

A convolutional time-series encoder with a time-wise attention mechanism learns fixed-length representations that generalize to unseen data types, enabling competitive classification with minimal adaptation. Pre-trained encoders offer compact, fast representations usable with simple classifiers.

ABSTRACT

We study the use of a time series encoder to learn representations that are useful on data set types with which it has not been trained on. The encoder is formed of a convolutional neural network whose temporal output is summarized by a convolutional attention mechanism. This way, we obtain a compact, fixed-length representation from longer, variable-length time series. We evaluate the performance of the proposed approach on a well-known time series classification benchmark, considering full adaptation, partial adaptation, and no adaptation of the encoder to the new data type. Results show that such strategies are competitive with the state-of-the-art, often outperforming conceptually-matching approaches. Besides accuracy scores, the facility of adaptation and the efficiency of pre-trained encoders make them an appealing option for the processing of scarcely- or non-labeled time series.

Motivation & Objective

• Motivate representation learning for time series with variable lengths and scarce labels.
• Develop a universal encoder that generalizes to unseen data types without full re-training.
• Evaluate adaptation strategies: no adaptation, partial adaptation, and full adaptation.
• Assess the trade-offs between representation size, accuracy, and efficiency.
• Compare encoder-based approaches against state-of-the-art baselines on a large time series benchmark.

Proposed method

• A CNN-based encoder with three convolutional blocks and two 2-factor max-pooling layers.
• An attention mechanism over half of the filters via a time-wise softmax to produce fixed-length representations.
• Instance normalization and PReLU activations to stabilize training.
• A final fully-connected layer to produce a k-dimensional representation.
• Multi-head training using a per-dataset classification head to map representations to labels during training.
• Evaluation with varying adaptation scenarios: no adaptation (1NN on representations), mapping adaptation (LR/SVM), and full adaptation (encoder fine-tuning).

Experimental results

Research questions

• RQ1 Can a universal time series encoder trained on multiple datasets generalize to unseen data types without substantial adaptation?
• RQ2 How do different adaptation strategies affect accuracy when transferring representations to new time-series datasets?
• RQ3 What is the impact of representation size (k) on performance under various adaptation regimes?
• RQ4 Do encoder-based representations achieve competitive performance compared to state-of-the-art baselines on the UEA/UCR benchmark?
• RQ5 Is the learned representation useful with simple classifiers such as 1NN, LR, and SVM?

Key findings

• Encoder-based representations with 1NN, LR, or SVM are competitive with or outperform traditional distance-based baselines.
• Adaptation of the encoder (ADAPT) yields competitive performance with state-of-the-art methods, often close to or surpassing several baselines.
• Pre-trained encoders (Encoder-NEW) outperform training from scratch on the target task (Encoder-N EW).
• Smaller fixed representations (k) hurt no-adaptation and mapping adaptation, but have limited impact under full adaptation.
• Encoder-ADAPT and Encoder-NEW reduce training complexity while providing strong performance compared to ensemble baselines like COTE and HIVE-COTE.
• Representation size: no consistent gain from increasing k under full adaptation, but larger k helps when no adaptation is used.