[Paper Review] Very Deep Convolutional Networks for Natural Language Processing.

Alexis Conneau, Holger Schwenk+2 more|arXiv (Cornell University)|Jun 6, 2016

Topic Modeling13 references222 citations

TL;DR

This paper introduces a very deep convolutional neural network (up to 29 layers) that processes text directly at the character level using only small convolutions and pooling operations. It achieves state-of-the-art performance on multiple text classification tasks, demonstrating that deeper architectures significantly improve NLP performance, marking the first successful application of very deep CNNs to natural language processing.

ABSTRACT

The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which are very successful in computer vision. We present a new architecture for text processing which operates directly on the character level and uses only small convolutions and pooling operations. We are able to show that the performance of this model increases with the depth: using up to 29 convolutional layers, we report significant improvements over the state-of-the-art on several public text classification tasks. To the best of our knowledge, this is the first time that very deep convolutional nets have been applied to NLP.

Motivation & Objective

• To explore whether very deep convolutional networks—successful in computer vision—can be effectively applied to natural language processing tasks.
• To address the limitation of shallow architectures in NLP, such as those based on RNNs and standard CNNs, by proposing a deeper model.
• To demonstrate that increasing network depth improves performance in text classification without relying on recurrent structures like LSTMs.
• To establish a new baseline for text classification using only convolutional and pooling operations at the character level.

Proposed method

• The model processes raw text at the character level, feeding sequences of characters directly into the network.
• It employs only small convolutional filters (e.g., kernel size 3) and max-pooling layers to extract hierarchical features.
• The architecture stacks up to 29 convolutional layers, enabling deep hierarchical representation learning.
• Each convolutional layer applies ReLU activation and uses weight sharing across the sequence to capture local patterns.
• Pooling layers reduce spatial dimensions and preserve the most salient features across receptive fields.
• The final representation is read out via global max-pooling or fully connected layers for classification.

Experimental results

Research questions

• RQ1 Can very deep convolutional networks, inspired by success in computer vision, achieve superior performance in NLP tasks?
• RQ2 Does increasing the depth of a CNN architecture lead to improved performance in text classification when operating at the character level?
• RQ3 Can a purely convolutional architecture outperform recurrent models like LSTMs in text classification without attention or recurrence?
• RQ4 Is the performance gain from depth consistent across multiple public text classification benchmarks?

Key findings

• The model achieves state-of-the-art performance on multiple public text classification datasets, outperforming prior methods.
• Performance improves significantly with increased depth, demonstrating a strong positive correlation between depth and accuracy.
• The architecture achieves these results using only small convolutions and pooling, without recurrence or attention mechanisms.
• The model is effective at the character level, eliminating the need for word-level tokenization or pre-trained embeddings.
• The best performance is achieved with a 29-layer architecture, confirming the benefits of very deep networks in NLP.
• This work establishes the first successful application of very deep CNNs to NLP, setting a new benchmark for pure convolutional models.