[Paper Review] Multichannel CNN with Attention for Text Classification

Zhenyu Liu, Haiwei Huang+2 more|arXiv (Cornell University)|Jun 29, 2020

Topic ModelingComputer Science30 references18 citations

TL;DR

This paper proposes Attention-based Multichannel Convolutional Neural Network (AMCNN) for text classification, combining bidirectional LSTM encoding with scalar and vectorial attention to generate multichannel representations that capture both word-level and feature-level importance. The model outperforms state-of-the-art methods on benchmark datasets by effectively extracting n-gram features with enhanced semantic richness and robustness.

ABSTRACT

Recent years, the approaches based on neural networks have shown remarkable potential for sentence modeling. There are two main neural network structures: recurrent neural network (RNN) and convolution neural network (CNN). RNN can capture long term dependencies and store the semantics of the previous information in a fixed-sized vector. However, RNN is a biased model and its ability to extract global semantics is restricted by the fixed-sized vector. Alternatively, CNN is able to capture n-gram features of texts by utilizing convolutional filters. But the width of convolutional filters restricts its performance. In order to combine the strengths of the two kinds of networks and alleviate their shortcomings, this paper proposes Attention-based Multichannel Convolutional Neural Network (AMCNN) for text classification. AMCNN utilizes a bi-directional long short-term memory to encode the history and future information of words into high dimensional representations, so that the information of both the front and back of the sentence can be fully expressed. Then the scalar attention and vectorial attention are applied to obtain multichannel representations. The scalar attention can calculate the word-level importance and the vectorial attention can calculate the feature-level importance. In the classification task, AMCNN uses a CNN structure to cpture word relations on the representations generated by the scalar and vectorial attention mechanism instead of calculating the weighted sums. It can effectively extract the n-gram features of the text. The experimental results on the benchmark datasets demonstrate that AMCNN achieves better performance than state-of-the-art methods. In addition, the visualization results verify the semantic richness of multichannel representations.

Motivation & Objective

• To address the limitations of RNNs in capturing long-term dependencies and CNNs in restricting n-gram feature extraction due to fixed filter width.
• To overcome the semantic ambiguity in RNN-based models that treat word order as secondary by integrating attention mechanisms.
• To enhance feature representation diversity and robustness through multichannel modeling using scalar and vectorial attention.
• To improve text classification performance by combining the strengths of bidirectional LSTM, attention mechanisms, and convolutional feature extraction.

Proposed method

• Uses a bidirectional LSTM to encode both past and future context of each word into high-dimensional representations.
• Applies scalar attention to compute word-level importance weights, emphasizing semantically significant words.
• Employs vectorial attention to compute feature-level importance, assigning different learning rates to hidden state dimensions based on informativeness.
• Generates multichannel representations by combining scalar and vectorial attention outputs, enabling diverse perspectives on sentence semantics.
• Applies a CNN with variable filter sizes to extract n-gram features from the multichannel representations instead of using weighted sums.
• Uses max-pooling and fully connected layers for final classification, with hyperparameter tuning for hidden size, channel count, filter size, and number of feature maps.

Experimental results

Research questions

• RQ1 Can combining scalar and vectorial attention with multichannel representations improve text classification beyond single-attention or single-channel models?
• RQ2 How does the integration of bidirectional LSTM with attention and CNN enhance feature extraction and semantic representation?
• RQ3 To what extent does multichannel modeling improve robustness and generalization in text classification tasks?
• RQ4 What is the optimal configuration of hyperparameters such as hidden size, number of channels, filter size, and number of feature maps for maximum performance?

Key findings

• AMCNN achieves state-of-the-art performance on benchmark datasets, including MPQA, SST-2, MR, and Subj, outperforming existing methods.
• The model performs best with 3 channels on MPQA, SST-2, and MR datasets, and with 4 channels on the Subj dataset, indicating optimal channel count varies by dataset.
• Classification accuracy increases with hidden size up to 128, after which performance plateaus or declines, indicating a trade-off between underfitting and overfitting.
• Filter size of 4–8 yields optimal balance between performance and parameter efficiency, with minimal improvement observed at larger filter sizes.
• Increasing the number of feature maps boosts accuracy initially but leads to overfitting due to rising parameter count, with performance stabilizing after a point.
• The vectorial attention mechanism improves performance by enabling selective feature tuning and enhancing model robustness through hidden state perturbation.