[Paper Review] Deep Learning applied to NLP

Marc Moreno López, Jugal Kalita|arXiv (Cornell University)|Mar 9, 2017

Topic Modeling13 references128 citations

TL;DR

A survey explaining Convolutional Neural Networks (CNNs) and their variations, and how they have been applied to a wide range of NLP tasks from sentiment analysis to machine translation and question answering. It also covers related deep architectures (RNNs, LSTMs, DCNNs, DSCNN, etc.) and their use in NLP.

ABSTRACT

Convolutional Neural Network (CNNs) are typically associated with Computer Vision. CNNs are responsible for major breakthroughs in Image Classification and are the core of most Computer Vision systems today. More recently CNNs have been applied to problems in Natural Language Processing and gotten some interesting results. In this paper, we will try to explain the basics of CNNs, its different variations and how they have been applied to NLP.

Motivation & Objective

• Explain the basics and variations of Convolutional Neural Networks (CNNs) and their adaptations for NLP.
• Survey how CNNs and related deep architectures are applied to diverse NLP tasks.
• Discuss data representations (word embeddings, one-hot) and architectural components (filters, pooling, dynamic pooling) in NLP contexts.
• Highlight key findings on performance and interpretability of CNN-based models in NLP.

Proposed method

• Introduce CNN basics and contrast with traditional neural networks.
• Describe input representations for NLP (sentences as matrices of word embeddings or one-hot vectors).
• Explain convolution and pooling operations in NLP contexts, including max pooling and dynamic k-max pooling.
• Survey multiple CNN-based architectures tailored for NLP (DCNN, DSCNN, multi-column CNN, context-dependent CNN, RCNN variants, etc.).
• Present task categories and representative models across semantic understanding, information extraction, summarization, translation, and QA.
• Discuss model interpretation and efficiency considerations in CNN NLP models.

Experimental results

Research questions

• RQ1 What CNN architectures and variations have been most effective for NLP tasks?
• RQ2 How do pooling, filter sizes, and input representations influence NLP performance?
• RQ3 What are the strengths and limitations of CNN-based approaches across different NLP tasks (e.g., sentiment, event extraction, QA, translation)?
• RQ4 How do hybrid architectures combining CNNs with RNNs/LSTMs perform relative to pure CNN or pure RNN approaches?

Key findings

• CNNs can be trained effectively on relatively small datasets when carefully designed and regularized.
• Max-pooling and multi-scale/dynamic pooling enable fixed-size representations for variable-length text and capture salient features.
• A range of architectures (DCNN, DSCNN, RCNN, multi-column CNN, context-dependent CNN) achieve strong results across tasks like event extraction, relation classification, and QA.
• CNN-based models can outperform traditional feature-based or linear models in several NLP tasks, and can be integrated with other components (e.g., gating, ranking, alignment) for improved performance.
• Some works demonstrate state-of-the-art or competitive results on benchmark datasets at the time, illustrating the versatility of CNNs in NLP beyond traditional image applications.
• CNNs offer interpretability advantages by analyzing the influence of n-grams and textual inputs on predictions.