[Paper Review] Recurrent Attention Unit

Guoqiang Zhong, Guohua Yue+1 more|arXiv (Cornell University)|Oct 30, 2018

Diet and metabolism studies29 references17 citations

TL;DR

This paper proposes the Recurrent Attention Unit (RAU), a novel RNN architecture that integrates an attention gate directly into the GRU cell to enable adaptive, dynamic focus on relevant sequence regions. RAU outperforms both GRU and LSTM across image classification, sentiment classification, and language modeling tasks, demonstrating superior long-term memory retention and reduced information redundancy.

ABSTRACT

Recurrent Neural Network (RNN) has been successfully applied in many sequence learning problems. Such as handwriting recognition, image description, natural language processing and video motion analysis. After years of development, researchers have improved the internal structure of the RNN and introduced many variants. Among others, Gated Recurrent Unit (GRU) is one of the most widely used RNN model. However, GRU lacks the capability of adaptively paying attention to certain regions or locations, so that it may cause information redundancy or loss during leaning. In this paper, we propose a RNN model, called Recurrent Attention Unit (RAU), which seamlessly integrates the attention mechanism into the interior of GRU by adding an attention gate. The attention gate can enhance GRU's ability to remember long-term memory and help memory cells quickly discard unimportant content. RAU is capable of extracting information from the sequential data by adaptively selecting a sequence of regions or locations and pay more attention to the selected regions during learning. Extensive experiments on image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other baseline methods.

Motivation & Objective

• To address the limitation of GRU in adaptively focusing on important sequence regions, which can lead to information redundancy or loss.
• To enhance long-term memory retention in RNNs by integrating attention mechanisms directly into the GRU architecture.
• To develop a simpler, more trainable model than existing attention-augmented RNNs by embedding the attention gate within the GRU cell rather than as an external module.
• To evaluate the effectiveness of RAU across diverse sequence modeling tasks, including image classification, language modeling, and sentiment classification.

Proposed method

• The RAU introduces an attention gate within the GRU’s memory cell, which dynamically modulates the hidden state based on input relevance.
• The attention gate computes a context-aware weighting over the input sequence, allowing the model to selectively focus on salient features at each time step.
• The attention mechanism is seamlessly integrated into the GRU’s update and reset gates, enabling end-to-end training without additional modules.
• The model uses a soft attention mechanism that assigns different weights to different input positions, emphasizing important content and suppressing irrelevant information.
• The architecture maintains the computational efficiency of GRU while enhancing its representational power through attention-driven feature selection.
• The method is applied to sequence modeling tasks using standard training protocols, including dropout and learning rate decay for regularization.

Experimental results

Research questions

• RQ1 Can integrating an attention mechanism directly into the GRU cell improve long-term memory and sequence modeling performance?
• RQ2 Does the proposed RAU model reduce information redundancy and improve focus on salient input features compared to standard GRU and LSTM?
• RQ3 How does RAU perform across diverse sequence tasks such as image classification, sentiment classification, and language modeling?
• RQ4 Is the attention gate integration into the GRU cell more effective and efficient than external attention modules in RNNs?

Key findings

• On the Penn Treebank (PTB) small dataset, RAU achieved a test perplexity of 113.89, outperforming both GRU and LSTM.
• In the PTB-medium configuration, RAU significantly reduced validation loss compared to GRU and LSTM, as shown in the training curves.
• On the PTB-large model, RAU maintained strong performance with a test perplexity competitive with state-of-the-art models, despite increased model complexity.
• In sentiment classification on the IMDB dataset, RAU achieved comparable accuracy to GRU and LSTM, with stable training curves using a learning rate of 10^-5.
• The RAU model showed consistent improvements in all evaluated tasks, demonstrating its generalization across different sequence learning problems.
• The integration of the attention gate within the GRU cell led to a simpler, more trainable architecture than external attention modules, without sacrificing performance.