[Paper Review] Comparative Study of CNN and RNN for Natural Language Processing

Wenpeng Yin, Katharina Kann+2 more|arXiv (Cornell University)|Feb 7, 2017

Topic Modeling20 references894 citations

TL;DR

The paper systematically compares CNN, GRU, and LSTM across various NLP tasks, showing RNNs often excel in sequence understanding while CNNs can outperform on certain local-keyphrase tasks; hyperparameters like hidden size and batch size greatly affect performance.

ABSTRACT

Deep neural networks (DNN) have revolutionized the field of natural language processing (NLP). Convolutional neural network (CNN) and recurrent neural network (RNN), the two main types of DNN architectures, are widely explored to handle various NLP tasks. CNN is supposed to be good at extracting position-invariant features and RNN at modeling units in sequence. The state of the art on many NLP tasks often switches due to the battle between CNNs and RNNs. This work is the first systematic comparison of CNN and RNN on a wide range of representative NLP tasks, aiming to give basic guidance for DNN selection.

Motivation & Objective

• Assess the relative strengths of CNN, GRU, and LSTM across a broad set of NLP tasks.
• Investigate whether CNNs or RNNs better capture local vs. global semantic information in text.
• Provide guidance for DNN selection in NLP based on task characteristics.

Proposed method

• Implement CNN, GRU, and LSTM with a basic, from-scratch setup (no pretrained embeddings).
• Tune hyperparameters per task and model on dev data to ensure fair comparisons.
• Evaluate on tasks including sentiment classification, relation classification, textual entailment, answer selection, question-relation matching, path query answering, and POS tagging.
• Use one-pass left-to-right encoding for GRU/LSTM, with bi-directional RNNs evaluated for POS tagging.
• For CNN, use a convolution layer over word embeddings with max-pooling to produce a fixed-size representation.

Experimental results

Research questions

• RQ1 Do CNNs and RNNs provide complementary information for text classification tasks?
• RQ2 How does the importance of semantically understanding the whole sequence affect CNN vs RNN performance?
• RQ3 How sensitive are CNN and RNN performances to hyperparameters like learning rate, hidden size, and batch size?
• RQ4 Which architectures best handle long-range dependencies vs. local key-phrase cues across different NLP tasks?

Key findings

• CNNs and RNNs provide complementary information for text classification tasks.
• Performance depends on whether global sequence understanding is crucial for the task.
• RNN performances are generally robust and competitive across tasks, except for certain keyphrase-dependent settings where CNNs may excel.
• Learning rate changes influence performance smoothly, while hidden size and batch size cause larger fluctuations.
• Bi-directional RNNs can improve POS tagging over one-directional variants.