[Paper Review] Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks

Aya Abdelsalam Ismail, Mohamed K. Gunady+3 more|arXiv (Cornell University)|Oct 27, 2019

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology35 references19 citations

TL;DR

This paper proposes input-cell attention, a novel RNN cell architecture that mitigates the vanishing saliency problem in recurrent networks by enabling attention over input vectors from current and past time steps. By attending to relevant historical inputs through a learnable attention matrix, the method preserves saliency across time, enabling accurate detection of important features regardless of their temporal position—demonstrated on synthetic data and fMRI time series, where standard RNNs fail to detect early salient features.

ABSTRACT

Recent efforts to improve the interpretability of deep neural networks use saliency to characterize the importance of input features to predictions made by models. Work on interpretability using saliency-based methods on Recurrent Neural Networks (RNNs) has mostly targeted language tasks, and their applicability to time series data is less understood. In this work we analyze saliency-based methods for RNNs, both classical and gated cell architectures. We show that RNN saliency vanishes over time, biasing detection of salient features only to later time steps and are, therefore, incapable of reliably detecting important features at arbitrary time intervals. To address this vanishing saliency problem, we propose a novel RNN cell structure (input-cell attention), which can extend any RNN cell architecture. At each time step, instead of only looking at the current input vector, input-cell attention uses a fixed-size matrix embedding, each row of the matrix attending to different inputs from current or previous time steps. Using synthetic data, we show that the saliency map produced by the input-cell attention RNN is able to faithfully detect important features regardless of their occurrence in time. We also apply the input-cell attention RNN on a neuroscience task analyzing functional Magnetic Resonance Imaging (fMRI) data for human subjects performing a variety of tasks. In this case, we use saliency to characterize brain regions (input features) for which activity is important to distinguish between tasks. We show that standard RNN architectures are only capable of detecting important brain regions in the last few time steps of the fMRI data, while the input-cell attention model is able to detect important brain region activity across time without latter time step biases.

Motivation & Objective

• To investigate why saliency maps in RNNs, especially LSTMs, tend to vanish over time, biasing detection toward later time steps.
• To address the limitation of existing saliency-based interpretability methods in time series by proposing a mechanism that maintains feature importance across all time intervals.
• To evaluate the proposed input-cell attention mechanism on both synthetic time series and real-world fMRI data, demonstrating improved interpretability in neuroscience applications.
• To show that standard RNNs fail to detect salient features in early time steps, while the proposed method maintains consistent saliency detection across the entire sequence.

Proposed method

• Introduce a modified LSTM cell that incorporates an input-cell attention mechanism, where a learnable attention matrix attends to input vectors from current and previous time steps.
• At each time step, the input sequence is projected into a fixed-size matrix, and an attention mechanism computes weights over these inputs to form a weighted representation.
• The weighted input matrix is then used as the input to the standard LSTM cell, allowing the network to dynamically attend to salient features across time.
• The attention mechanism is differentiable and end-to-end trainable, enabling joint optimization of both the RNN and attention components.
• The method is architecture-agnostic and can be applied to any RNN or LSTM variant by replacing the input processing step with the attention mechanism.
• Saliency is computed via gradient-based methods (e.g., integrated gradients) to interpret feature importance across time.

Experimental results

Research questions

• RQ1 Does saliency in standard RNNs, particularly LSTMs, vanish over time, leading to biased detection of important features in later time steps?
• RQ2 Can an attention mechanism applied to input vectors across time steps mitigate the vanishing saliency problem in RNNs?
• RQ3 How does input-cell attention compare to self-attention and other attention mechanisms in preserving saliency across time in time series data?
• RQ4 Can input-cell attention improve interpretability in real-world neuroscience applications, such as fMRI-based task classification?

Key findings

• Standard LSTMs exhibit strong saliency decay over time, with feature importance scores significantly higher in later time steps, leading to biased detection of salient features.
• The proposed input-cell attention mechanism successfully preserves saliency across all time steps, enabling reliable detection of important features regardless of their temporal position in synthetic time series.
• On the MNIST-as-time-series task, input-cell attention recovers saliency values for features at all time steps, while vanilla LSTMs show consistent decay in saliency over time.
• In fMRI data from the Human Connectome Project, standard LSTMs only detect important brain regions in the final time steps, whereas input-cell attention identifies salient brain regions throughout the entire task sequence.
• When salient features are removed, the accuracy of standard LSTMs drops by only 2% when the top 10% salient features are dropped, indicating poor reliability; in contrast, input-cell attention drops by 9.5%, confirming more accurate saliency detection.
• More than 70% of the top 10% salient features identified by standard LSTMs are from off-task periods, highlighting their bias toward later, non-informative time steps.