[Paper Review] Survey on the attention based RNN model and its applications in computer vision

Feng Wang, David M. J. Tax|arXiv (Cornell University)|Jan 25, 2016

Advanced Image and Video Retrieval TechniquesComputer Science34 references100 citations

TL;DR

This survey reviews attention-based RNN models for sequence-to-sequence tasks, detailing four attention mechanisms and their applications in computer vision, with discussion of advantages and future directions.

ABSTRACT

The recurrent neural networks (RNN) can be used to solve the sequence to sequence problem, where both the input and the output have sequential structures. Usually there are some implicit relations between the structures. However, it is hard for the common RNN model to fully explore the relations between the sequences. In this survey, we introduce some attention based RNN models which can focus on different parts of the input for each output item, in order to explore and take advantage of the implicit relations between the input and the output items. The different attention mechanisms are described in detail. We then introduce some applications in computer vision which apply the attention based RNN models. The superiority of the attention based RNN model is shown by the experimental results. At last some future research directions are given.

Motivation & Objective

• Explain the concept of attention and its relevance to RNNs for sequence-to-sequence problems.
• Describe the encoder-decoder framework and why attention is beneficial for handling variable-length inputs/outputs.
• Detail four attention mechanisms (item-wise/ location-wise, soft/hard) and their integration into end-to-end models.
• Discuss applications of attention-based RNNs in computer vision and potential future research directions.

Proposed method

• Define and formalize the attention-based RNN model within the encoder-decoder paradigm.
• Present four attention mechanisms: item-wise soft attention, item-wise hard attention, location-wise hard attention, and location-wise soft attention.
• Explain how attention weights are computed (e.g., e_jt = f_att(c_t, h_{j-1}) and alpha_jt via softmax) and how the intermediate code is formed.
• Discuss training as end-to-end differentiable or extended with reinforcement-learning for hard attention.
• Illustrate how attention provides better intermediate representations and interpretability.

Experimental results

Research questions

• RQ1 What is the role of attention in improving encoder-decoder RNNs for sequence-to-sequence tasks?
• RQ2 How do the four attention mechanisms differ in terms of input type, computation, and differentiability?
• RQ3 What are the advantages of attention-based RNNs over traditional RNNs for vision-oriented sequence problems?
• RQ4 How can attention mechanisms be applied to various computer vision tasks and data modalities?
• RQ5 What are the suggested future directions and challenges for attention-based RNN models?

Key findings

• Attention-based RNNs assign differential weights to input parts, capturing implicit input-output relations.
• Four attention mechanisms are presented: item-wise soft, item-wise hard, location-wise soft, and location-wise hard attention.
• Hard attention can reduce computation by focusing on relevant parts, with learning facilitated by reinforcement learning techniques.
• Soft attention is differentiable, enabling end-to-end training of both the RNN and attention module.
• The survey reports experimental results showing the superiority of attention-based RNNs in the surveyed contexts.
• The model decomposes the sequence-to-sequence problem into encoder, attention, and decoder components, enabling flexible inputs (features maps or explicit item sequences).