[Paper Review] Query-based Attention CNN for Text Similarity Map

Tzu-Chien Liu, Yu-Hsueh Wu+1 more|arXiv (Cornell University)|Sep 15, 2017

Topic Modeling7 references7 citations

TL;DR

This paper proposes Query-based Attention CNN (QACNN), an end-to-end neural network for multiple-choice question answering that uses a two-staged CNN architecture with query-based attention to improve text similarity matching. By integrating word-level and sentence-level attention mechanisms, QACNN achieves 79.99% accuracy on the MovieQA dataset, setting a new state-of-the-art result.

ABSTRACT

In this paper, we introduce Query-based Attention CNN(QACNN) for Text Similarity Map, an end-to-end neural network for question answering. This network is composed of compare mechanism, two-staged CNN architecture with attention mechanism, and a prediction layer. First, the compare mechanism compares between the given passage, query, and multiple answer choices to build similarity maps. Then, the two-staged CNN architecture extracts features through word-level and sentence-level. At the same time, attention mechanism helps CNN focus more on the important part of the passage based on the query information. Finally, the prediction layer find out the most possible answer choice. We conduct this model on the MovieQA dataset using Plot Synopses only, and achieve 79.99% accuracy which is the state of the art on the dataset.

Motivation & Objective

• To address the limitations of existing compare-aggregate frameworks in matching passage and query by incorporating attention mechanisms.
• To improve feature extraction across multiple levels—word-level and sentence-level—using a two-staged CNN architecture.
• To enhance model performance by focusing on semantically relevant parts of the passage based on query context through query-based attention.
• To achieve state-of-the-art results on the MovieQA dataset for multiple-choice question answering.

Proposed method

• The model uses an embedding layer to convert passage, query, and answer choices into word vectors using pre-trained GloVe embeddings.
• A similarity mapping layer computes cosine similarity between passage sentences and the query, and between passage sentences and each answer choice, generating 2D and 3D similarity maps.
• The two-staged CNN architecture first performs word-level matching using convolutional layers on the passage-query and passage-choice similarity maps, then applies sentence-level matching on the pooled features.
• Query-based attention is applied at both word-level and sentence-level, using the passage-query similarity map to generate attention weights that highlight relevant words and sentences.
• The attention-weighted features from passage-choice matching are passed to a prediction layer that selects the most likely answer choice.
• The model is trained end-to-end on the MovieQA dataset using only plot synopses as passage input.

Experimental results

Research questions

• RQ1 Can a two-staged CNN architecture improve feature extraction in text matching for multiple-choice question answering?
• RQ2 Does query-based attention at both word and sentence levels enhance the model's ability to identify relevant information in the passage?
• RQ3 How does the integration of attention mechanisms affect performance compared to a passive aggregation approach?
• RQ4 Can the model achieve state-of-the-art performance on the MovieQA benchmark using only plot synopses?

Key findings

• QACNN achieved 79.99% accuracy on the MovieQA test set, setting a new state-of-the-art result for the dataset.
• The two-staged CNN architecture significantly outperformed a one-stage version, which achieved only 66.8% accuracy on the validation set.
• The ablation study showed that removing both attention mechanisms reduced performance to 69.6%, indicating that attention is crucial for model effectiveness.
• Sentence-level attention contributed more than word-level attention, with QACNN using only sentence-level attention achieving 75.1% accuracy, compared to 72.5% with only word-level attention.
• Visualization confirmed that the sentence-level attention map correctly identified the sentence containing the correct answer, and word-level attention focused on key words at the end of that sentence.
• The model also achieved strong performance on the MCTest dataset, with 68.1% accuracy on MC160 and 61.5% on MC500, outperforming baseline models.