[Paper Review] TimeNet: Pre-trained deep recurrent neural network for time series classification

Pankaj Malhotra, Vishnu Tv+3 more|arXiv (Cornell University)|Jun 23, 2017

Time Series Analysis and Forecasting21 references113 citations

TL;DR

TimeNet trains an unsupervised sequence auto-encoder to produce fixed-length embeddings from varying-length time series and uses the encoder as a generic off-the-shelf feature extractor for time series classification, outperforming domain-specific encoders and DTW-based methods on many UCR datasets.

ABSTRACT

Inspired by the tremendous success of deep Convolutional Neural Networks as generic feature extractors for images, we propose TimeNet: a deep recurrent neural network (RNN) trained on diverse time series in an unsupervised manner using sequence to sequence (seq2seq) models to extract features from time series. Rather than relying on data from the problem domain, TimeNet attempts to generalize time series representation across domains by ingesting time series from several domains simultaneously. Once trained, TimeNet can be used as a generic off-the-shelf feature extractor for time series. The representations or embeddings given by a pre-trained TimeNet are found to be useful for time series classification (TSC). For several publicly available datasets from UCR TSC Archive and an industrial telematics sensor data from vehicles, we observe that a classifier learned over the TimeNet embeddings yields significantly better performance compared to (i) a classifier learned over the embeddings given by a domain-specific RNN, as well as (ii) a nearest neighbor classifier based on Dynamic Time Warping.

Motivation & Objective

• Motivate learning generic, unlabeled representations for time series as off-the-shelf features.
• Propose a sequence-to-sequence auto-encoder with a multilayered GRU encoder as TimeNet.
• Show that TimeNet embeddings improve classification over domain-specific encoders and DTW-based methods.
• Demonstrate robustness and well-separated embeddings via visualization (t-SNE).
• Evaluate performance across diverse UCR datasets and an industrial telematics dataset.

Proposed method

• Use a sequence-to-sequence auto-encoder with multilayered GRUs and dropout to encode time series.
• Train on diverse unlabeled time series from 18 training datasets (T ≤ 512) to learn a fixed-dimensional embedding from the encoder final state.
• Freeze the trained encoder to form TimeNet, a generic time series feature extractor.
• Compare embeddings from TimeNet (TN-C) with embeddings from domain-specific SAE encoders (SAE-C) and with DTW-based nearest neighbor (DTW-C) classifiers.
• Train non-linear SVMs with RBF kernels on embeddings.
• Analyze embedding quality with t-SNE visualizations.

Experimental results

Research questions

• RQ1 Can a multilayered RNN trained on diverse unlabeled time series serve as a generic time series feature extractor?
• RQ2 Do embeddings from TimeNet outperform domain-specific encoders and DTW-based classifiers on unseen datasets?
• RQ3 Are TimeNet embeddings robust when labeled data is scarce?
• RQ4 Do TimeNet embeddings form well-separated clusters across and within datasets?

Key findings

• TimeNet embeddings yield competitive or better classification performance than domain-specific SAE embeddings on 83% of test datasets and exceed DTW-C on 83% as well.
• TimeNet classifiers outperform DTW-C on 25/30 datasets and exceed SAE-C on 22/30 datasets; TimeNet also surpasses SAE-C on 18/30 datasets.
• Using two-thirds of the labeled data, TN-C still beats DTW-C on 20/30 datasets, indicating robustness with reduced supervision.
• Embeddings from TimeNet show well-separated clusters for time series from different classes and domains in t-SNE visualizations.
• Layer analysis indicates shorter time series may rely on a single Timenet layer, while longer sequences benefit from information across layers.