[Paper Review] Do Convolutional Networks need to be Deep for Text Classification ?

Hoa T. Le, Christophe Cerisara+1 more|arXiv (Cornell University)|Jul 13, 2017

Topic Modeling31 references70 citations

TL;DR

The paper compares shallow and deep CNNs for text classification using character and word inputs, finding shallow word-level CNNs often outperform deep Char models and even rival or surpass state-of-the-art on several datasets, with a DenseNet adaptation for text.

ABSTRACT

We study in this work the importance of depth in convolutional models for text classification, either when character or word inputs are considered. We show on 5 standard text classification and sentiment analysis tasks that deep models indeed give better performances than shallow networks when the text input is represented as a sequence of characters. However, a simple shallow-and-wide network outperforms deep models such as DenseNet with word inputs. Our shallow word model further establishes new state-of-the-art performances on two datasets: Yelp Binary (95.9\%) and Yelp Full (64.9\%).

Motivation & Objective

• Assess the impact of network depth on text classification performance for character- and word-based inputs.
• Compare shallow-and-wide CNNs against deep CNNs and DenseNet-style architectures in text tasks.
• Identify which input granularity (characters vs words) benefits most from depth in CNNs.
• Demonstrate state-of-the-art results on select datasets using word-level shallow models.

Proposed method

• Implement a shallow-and-wide CNN adapted from Kim (2014) with multiple filter widths and global max-pooling.
• Adapt DenseNet architecture for text at the word level and compare with character-level variants.
• Train models on five datasets using Adam optimization with standardized hyperparameters; report accuracy.
• Use embedding representations: pre-trained word2vec for word inputs and one-hot character encoding for character inputs.
• Compare against a wide range of baselines and prior state-of-the-art methods.

Experimental results

Research questions

• RQ1 Does depth provide a significant advantage for character-level text classification models?
• RQ2 Does depth benefit word-level text classification, or can shallow word-level models achieve or exceed state-of-the-art results?
• RQ3 How do character-level and word-level inputs compare across standard text classification datasets when depth is varied?
• RQ4 Can a DenseNet-like text model improve performance at either input level?

Key findings

• Deep character-level models show some gains over shallow models but are not consistently superior across datasets.
• A shallow-and-wide word-level CNN achieves strong performance and often matches or surpass deep models at the word level.
• The word-level shallow model attains state-of-the-art results on Yelp Binary (95.9%) and Yelp Full (64.9%).
• DenseNet-style word-level variants can approach but do not consistently outperform the shallow word-level CNN.
• Overall, deep models do not universally outperform shallow models for text classification; depth benefits are dataset- and input-dependent.
• Character-depth improvements are more pronounced than word-depth improvements, but word-level shallow models remain highly effective.