[Paper Review] Gated Transformer Networks for Multivariate Time Series Classification

Minghao Liu, Shengqi Ren+5 more|arXiv (Cornell University)|Mar 26, 2021

Time Series Analysis and Forecasting18 references100 citations

TL;DR

GTN extends Transformer with a gating mechanism to fuse step-wise and channel-wise relations for multivariate time series classification, achieving competitive results across 13 datasets and offering interpretability via attention maps.

ABSTRACT

Deep learning model (primarily convolutional networks and LSTM) for time series classification has been studied broadly by the community with the wide applications in different domains like healthcare, finance, industrial engineering and IoT. Meanwhile, Transformer Networks recently achieved frontier performance on various natural language processing and computer vision tasks. In this work, we explored a simple extension of the current Transformer Networks with gating, named Gated Transformer Networks (GTN) for the multivariate time series classification problem. With the gating that merges two towers of Transformer which model the channel-wise and step-wise correlations respectively, we show how GTN is naturally and effectively suitable for the multivariate time series classification task. We conduct comprehensive experiments on thirteen dataset with full ablation study. Our results show that GTN is able to achieve competing results with current state-of-the-art deep learning models. We also explored the attention map for the natural interpretability of GTN on time series modeling. Our preliminary results provide a strong baseline for the Transformer Networks on multivariate time series classification task and grounds the foundation for future research.

Motivation & Objective

• Motivate the need for effective multivariate time series classification using Transformer architectures.
• Propose Gated Transformer Networks (GTN) that merge step-wise and channel-wise representations via gating.
• Evaluate GTN on a diverse set of multivariate time series datasets with ablation studies.
• Analyze attention maps and embeddings for interpretability of learned features.

Proposed method

• Embed time series with a tanh-activated fully connected embedding layer and add time-aware positional encoding.
• Implement two Transformer towers: a step-wise (temporal) encoder and a channel-wise encoder to capture distinct correlations.
• Introduce a gating mechanism that weighs and combines tower outputs through learned gates g1 and g2.
• Use masking in the step-wise tower to preserve autoregressive-like attention and improve learning.
• Train GTN with Adagrad (lr=0.0001) and dropout (0.2), with learning-rate scheduling on plateau, evaluating on best-loss model.
• Provide ablation studies to show the impact of masking, tower components, and gating.

Experimental results

Research questions

• RQ1 Can a gated two-tower Transformer effectively model both step-wise (temporal) and channel-wise correlations in multivariate time series?
• RQ2 Does a gating mechanism outperform simple concatenation or single-tower designs in GTN?
• RQ3 How interpretable are the learned attention maps and embeddings for time series data?
• RQ4 What is GTN’s empirical standing against state-of-the-art deep learning baselines on a diverse benchmark?

Key findings

• GTN achieves competitive accuracy with state-of-the-art deep learning models across 13 datasets.
• Ablation studies show masking improves performance, and channel-wise or step-wise towers alone differ in effectiveness across datasets.
• Gating (learned weights) outperforms simple concatenation, enabling data-driven reliance on either tower per sample.
• GTN enables interpretable insights via attention maps and embedding visualizations, suggesting meaningful pattern discovery.
• Step-wise attention often dominates, but dataset-specific behavior shows both towers contribute to performance.
• GTN demonstrates strong baseline potential for Transformer-based time series classification.