[Paper Review] Focusing on What is Relevant: Time-Series Learning and Understanding using Attention

Phongtharin Vinayavekhin, Subhajit Chaudhury+5 more|arXiv (Cornell University)|Jun 22, 2018

Time Series Analysis and Forecasting22 references4 citations

TL;DR

This paper proposes a temporal attention layer that computes attention weights using the entire input sequence, enabling focused, interpretable attention for time-series tasks. The method achieves state-of-the-art accuracy in action classification, data completion, and key-frame detection while producing sharper, more intuitive attention visualizations than RNN-based attention mechanisms.

ABSTRACT

This paper is a contribution towards interpretability of the deep learning models in different applications of time-series. We propose a temporal attention layer that is capable of selecting the relevant information to perform various tasks, including data completion, key-frame detection and classification. The method uses the whole input sequence to calculate an attention value for each time step. This results in more focused attention values and more plausible visualisation than previous methods. We apply the proposed method to three different tasks. Experimental results show that the proposed network produces comparable results to a state of the art. In addition, the network provides better interpretability of the decision, that is, it generates more significant attention weight to related frames compared to similar techniques attempted in the past.

Motivation & Objective

• To improve interpretability of deep learning models in time-series applications by embedding interpretable attention mechanisms directly into the network architecture.
• To address the limitations of RNN-based attention mechanisms, which produce diffuse attention weights and obscure key temporal dependencies.
• To enable focused attention on relevant time steps, enhancing model explainability without sacrificing performance.
• To demonstrate the effectiveness of the proposed attention layer across diverse time-series tasks, including classification, completion, and key-frame detection.

Proposed method

• Introduces a temporal contextual layer that computes attention weights based on the full input sequence, not just a single latent vector.
• Uses a dense encoder to represent the entire sequence, enabling global context awareness for attention computation.
• Applies the attention mechanism to compute dynamic weights that emphasize relevant time steps for decision-making.
• Integrates the proposed layer into two architectures: an autoencoder for data completion and a classification model for action recognition.
• Employs self-attention mechanisms that allow the network to focus on salient frames while maintaining high accuracy.
• Uses a multi-head attention mechanism to capture diverse temporal dependencies across different representations.

Experimental results

Research questions

• RQ1 Can a self-attention mechanism based on the full sequence improve interpretability in time-series modeling compared to RNN-based attention?
• RQ2 Does the proposed attention layer produce more focused attention distributions than existing methods, especially in key-frame detection?
• RQ3 Can the model achieve competitive performance while providing clearer visualizations of decision-relevant frames?
• RQ4 How does the proposed method compare in accuracy and training efficiency to state-of-the-art models using BiLSTM or MLP architectures?

Key findings

• The proposed method achieved 85.9% ± 2.7 accuracy on the KIT- -action classification dataset, outperforming BiLSTM + Attention (85.4%) and matching the best-performing model in the comparison.
• The model with a dense encoder required only 56.5 seconds for training (52 epochs), significantly faster than the BiLSTM-based models (225.5–584.0 seconds).
• The number of parameters was drastically reduced—only 4,732 for the dense + proposed model, compared to 933,081 for a standard MLP.
• Attention visualization showed that the model focused on only a few key frames (e.g., frame 58 for 'bow'), which align with human intuition and are distinct from similar actions.
• In contrast, BiLSTM + feed-forward attention distributed attention more broadly, including non-relevant frames, reducing interpretability.
• The confusion matrix revealed perfect classification for 'bow', 'golf', and 'squat', with key frames clearly distinguishing these actions from others.